Market Modeling For Practitioners

A backtest that “looks right” often hides a quiet failure: your input data drifts out of sync by a few minutes, your corporate actions get applied twice, or your bars get stitched together across trading halts without you noticing. Those problems don’t always crash your pipeline. They just make the simulated strategy slightly too good, then slightly wrong out of sample, and you end up chasing ghosts.

Talia, 34, a prop trader who runs backtests daily, learned this the hard way. One afternoon her equity curve jumped exactly when she expected it to fade. The change wasn’t in the model logic; it was in the data. The vendor had updated the corporate action file, and her script had pulled the new split factors but left the old dividend adjustments in place. The simulation started trading on adjusted prices that no longer matched the volume and timestamp conventions the rest of her code assumed.

This chapter shows you how to clean, align, and transform raw market data into simulation-ready inputs you can trust. You’ll end up with a repeatable process that produces a consistent time index, correct price/return series, and feature columns that stay aligned with the trading logic in your backtest or agent-based simulation. You’ll also learn how to catch the specific failure modes that create “too-good” backtests: duplicate adjustments, time-zone mismatches, and silent bar construction errors.

Input Sanity Checklist: cleaning, aligning, and transforming without guessing

The core move is to stop treating data prep as a one-off script. You need a checklist you run every time you build a dataset, and you need rules that translate raw vendor fields into simulation primitives: a clean timestamp, an orderable price series, and a return series (or state variables) that align with your execution model. Think in terms of invariants: if an invariant breaks, you fix the data before you run the model.

Use this Input Sanity Checklist for every instrument and every dataset version. It focuses on the three things that break simulations most often: time alignment, corporate actions, and bar construction.

1. Lock the time axis first (timestamp hygiene). Convert every feed you use - trades, quotes, OHLCV bars, corporate actions - into one canonical timezone (usually UTC) and one canonical bar boundary rule. Then verify that your final index is monotonic and has no unexpected gaps. If you simulate minute bars, you want exactly one row per minute per instrument between start and end, or you want explicit markers for missing minutes.

2. Apply corporate actions once, in the right order. Adjust prices for splits and dividends exactly once, using the vendor’s official action effective dates. For OHLCV bars, apply adjustments to all OHLC fields consistently, and decide what you do with volume (many pipelines scale volume by split factors but keep dividend effects out of volume). Your invariant: adjusted prices should reconcile with unadjusted prices via the cumulative adjustment factor.

3. Choose one price definition for execution and stick to it. Decide whether your strategy trades on the bar’s open, close, or a synthetic “next-bar open.” Convert your OHLC into the exact input your execution model expects. For example, if you run “enter at next bar open,” you must shift the features and labels so that the prediction at time t maps to the execution at time t+1. Mis-shifts create look-ahead bias without any obvious error.

4. Rebuild returns from the same adjusted price series. Compute returns only after you’ve finalized adjusted prices. If you compute returns earlier and then adjust prices later, you break the math. Your invariant: the return series should satisfy the identity

\[

r_t = \frac{P_t}{P_{t-1}} - 1

\]

(or log returns, if you use them) using your final adjusted price series.

Here’s a concrete example you can apply immediately. Suppose your vendor provides daily OHLCV and a separate corporate actions file. You ingest both into a dataframe. First you standardize timestamps: you make each daily bar timestamp represent market close in UTC (or market open - just be consistent with your execution rules). Next you compute an “adjustment factor” column from the corporate actions file and apply it cumulatively to the daily OHLC. Then you compute returns from the adjusted close. Finally you create shifted columns for labels: if your strategy predicts tomorrow’s return, you shift the return target by one day and you keep all features unshifted.

If you also use intraday data (quotes or trades) to build features like spread or midprice, you align those features to the bar boundaries after cleaning. For minute bars, you construct midprice from the best bid and ask at each minute boundary, then forward-fill only within the bar grid you already validated.

The practical trick: you don’t “clean until it runs.” You clean until the invariants pass....