ResearchSetup
Hypothesis is written after seeing results
Typical symptoms
- The research note reads like the result was known in advance
- Rejection criteria are missing
- Several metrics are cited but no primary metric is named
Likely causes
- Exploration and confirmation were mixed
- No pre-analysis note existed
- The team optimized the story after seeing outcomes
Checks
- Find the timestamped hypothesis note
- Identify the primary metric
- Ask what result would have rejected the idea
Resolution path
- Create a method brief before the next test
- Separate exploratory observations from confirmatory tests
- Label retrofitted conclusions as exploratory only
Related terms: hypothesis, p-hacking, pre-analysis, confirmation bias
RiskSafety
Transaction costs and frictions are ignored
Typical symptoms
- High-turnover results are presented without friction
- Small edge disappears under conservative assumptions
- Liquidity constraints are absent
Likely causes
- Headline returns were prioritized over implementability
- Turnover was not measured
- Spread and market impact assumptions were omitted
Checks
- Calculate turnover
- Apply conservative cost scenarios
- Compare performance before and after friction
Resolution path
- Report base, conservative, and severe friction cases
- Reject fragile results that only work without costs
- Add liquidity and capacity notes
Related terms: slippage, turnover, market impact, capacity
DataData
Analysis uses information that would not have been known
Typical symptoms
- Historical results look unusually smooth
- Signals depend on revised or future data
- Performance falls apart when timestamps are shifted
Likely causes
- Inputs were aligned to the wrong timestamp
- Corporate actions or classifications were applied retroactively
- Final-period values were used inside the period being tested
Checks
- Audit each input's availability time
- Shift features forward and backward to test sensitivity
- Review whether labels were known at decision time
Resolution path
- Use point-in-time datasets where possible
- Lag inputs conservatively
- Document any unavoidable timing approximation
Related terms: lookahead, timestamp, point-in-time, feature lag
DataData
Failed or delisted observations are missing
Typical symptoms
- The sample only includes current winners
- Older periods show suspiciously strong quality
- Universe construction is not documented
Likely causes
- Current constituents were applied backward
- Unavailable or failed observations were removed
- Data coverage limitations were not disclosed
Checks
- Rebuild the universe as of each historical date
- Count missing observations by period
- Compare current-only and historical-universe results
Resolution path
- Use historical membership where available
- Disclose coverage gaps
- Treat current-only analysis as limited exploratory evidence
Related terms: survivorship, universe, delisting, coverage
RiskRuntime
Risk is summarized by one flattering metric
Typical symptoms
- Only one performance or risk statistic is shown
- Drawdown path is missing
- Bad scenarios are explained away without evidence
Likely causes
- The review selected the metric that made the result look best
- Path-dependent losses were not inspected
- Stress cases were skipped
Checks
- Inspect drawdown, volatility, tail loss, and time under water
- Compare against a benchmark
- Review worst-month and worst-period behavior
Resolution path
- Use a balanced scorecard
- Add drawdown and stress sections to every report
- Make the decision label reflect downside evidence
Related terms: drawdown, tail risk, time under water, stress test
Educational boundary
This lookup is for research troubleshooting and reproducibility. It is not financial, investment, trading, tax, or legal advice. Prefer conservative assumptions and independent review when evaluating any analytical result.
InQuantWeTrust publishes educational content for quantitative research and analytical methods. It does not provide personalized advice, trade recommendations, managed services, or guaranteed outcomes.
