Backtest Data Pipelines: Collecting, Cleaning, and Blocking Look-Ahead Bias

🌐 한국어

Half of a backtest's credibility is decided by the data pipeline, not the strategy. Feed in garbage and even an excellent strategy produces garbage results. These are field notes from preparing market data for backtesting.

⚠️ This article is for educational and informational purposes only. It is not investment advice, and any losses are your own responsibility.

1. Collection — Gathering Candles (OHLCV)

The basic unit of a backtest is the candle. Broker APIs generally serve candles by period and interval.

• Daily/weekly/monthly — range queries (e.g. inquire-daily-itemchartprice)
• Intraday minutes — mostly same-day, returned in small reverse-ordered batches

Watch out for the per-request count limit. Minute candles might come 30 at a time, so covering a long span means moving a reference timestamp backward across repeated calls and stitching the results together.

# Concept: paginate by walking the reference time backward

all_candles = []

cursor = end_time

while len(all_candles) < needed:

    batch = fetch_candles(symbol, base_time=cursor, count=30)

    if not batch:

        break

    all_candles.extend(batch)

    cursor = batch[-1].time  # move to the oldest candle's timestamp

2. Cleaning — Making the Data Trustworthy

Raw data is messy. Clean it before it goes anywhere near a backtest.

• Sort chronologically — APIs frequently return reverse order, so flip it to ascending.
• Deduplicate — the same candle can appear on both sides of a page boundary. Make timestamps unique.
• Handle gaps and outliers — filter out empty candles on trading-halt days and any zero or negative prices.
• Convert types — numbers arriving as strings become floats. Handle parse failures safely as None.

def to_float(v):

    try:

        return float(v)

    except (TypeError, ValueError):

        return None  # mark parse failures explicitly instead of swallowing them

3. Storage — Make It Reproducible

Save cleaned data to files (CSV, Parquet, and so on) so you reuse what you've already fetched. Hitting the API every time is slow and, more importantly, can change your results. When you save, record the symbol, interval, and date range in the filename or metadata, so you can always trace which data a given backtest ran on.

Precomputing indicators (moving averages, RSI, Bollinger Bands) and storing them alongside the candles makes repeated backtests much faster.

4. The Most Dangerous Trap: Look-Ahead Bias

This is the bug that turns an entire backtest into a lie: quietly referencing a value that isn't knowable yet at the point of calculation. Say you're supposed to decide on today's open but accidentally reference today's close — now you have a bot that knows the future and performance improves like magic. That information doesn't exist live, so it collapses the moment you deploy.

There's exactly one way to prevent it structurally: pass only the data the bot could actually know at that moment.

# Bad: pass the whole array and leave room to peek at the future

signal = strategy(all_candles, i)   # risk of referencing all_candles[i+1]

# Good: slice up to the current point before passing

window = all_candles[: i + 1]       # at step i, this is all that's knowable

signal = strategy(window)

The same applies to indicator computation. A moving average at step i must be computed from data up to i only. Code that "computes indicators once over the full dataset and then slices" leaks look-ahead bias easily at the boundaries, so be especially careful there.

Summary

• Collect candles with pagination to work around per-request limits
• Clean via sorting, deduplication, outlier handling, and type conversion
• Save to files for reproducibility, and precompute indicators
• Block look-ahead structurally by slicing data to the current point before passing it

When a backtest looks "too good," suspect the pipeline before you suspect the strategy. It's usually the data or a look-ahead leak.