Backtest Data Pipelines: Collecting, Cleaning, and Blocking Look-Ahead Bias OnTech Insight

TECH INSIGHT / EXPLAINED

A reproducible backtest needs the data as it could have been known at each decision time, plus a record of every cleaning choice. Keep raw observations, normalized observations and derived features separate so a surprising result can be traced back.

Three adult colleagues explain data pipeline — AI-generated teaching illustration, not an actual operating screen or performance record.

Pagination must make measurable progress

Document whether the API cursor is inclusive or exclusive and which order it returns. Deduplicate by venue, instrument, interval and timestamp—not timestamp alone across symbols. Track the oldest timestamp and newly added keys each page. If a page repeats without progress, stop with an error instead of looping forever. Respect rate and page limits; pagination fetches successive pages rather than bypassing a provider’s limits.

Do not erase inconvenient market events

Use the exchange calendar to distinguish missing data, no trades, holidays and trading halts. Preserve flags and raw records before filtering. Dropping a halt or filling it with tradable prices can hide the exact period in which an exit was impossible. Validate OHLC relations, volume and corporate-action adjustments. NaN and infinity pass ordinary float parsing, so add finite-value checks:

import math

def finite_price(value):
    try:
        number = float(value)
    except (TypeError, ValueError, OverflowError):
        return None
    return number if math.isfinite(number) and number > 0 else None

assert finite_price("NaN") is None
assert finite_price("inf") is None
assert finite_price("0") is None
assert finite_price("100.25") == 100.25

Event time is not availability time

A bar ending at 10:01 is not available at 10:00 merely because its label uses the start time. Track event_time, available_at and ingested_at; join features backward on availability with a staleness tolerance. A macro series revised later requires its historical vintage, and the investment universe should include delisted names if they were eligible then. Today’s constituent list can cause survivorship bias.

Precomputation can be causal

A trailing moving average can be calculated once over ordered data without look-ahead. Centered rolling windows, backward fill from the future, full-sample normalization, and a scaler fit on the holdout are the hazards. Slicing the final array alone cannot repair those transformations. Perturb future observations and assert earlier features and decisions stay unchanged; independently verify the bar-to-order timing contract.

Freeze a data manifest

Store source and API parameters, retrieval time, timezone, price adjustment convention, schema version, cleaning log and checksums. Save raw and normalized artifacts. Pin the data manifest and code revision to each run so a later corrected provider file does not silently rewrite the reported experiment.

Documentation and next steps

Content and examples reviewed: 2026-09-29

Backtest Data Pipelines: Collecting, Cleaning, and Blocking Look-Ahead Bias