Bot Development Milestones: Observe, Replay and Rehearse Recovery OnTech Insight

TECH INSIGHT / EXPLAINED

Incremental development works best when every stage ends with evidence, not merely another enabled feature. Keep the authority to submit orders separate from the ability to collect data or produce a signal.

M1, M2 and M3 are the development stages proposed in this article, not an industry standard or a validated trading strategy. They mean observe, replay without orders, and rehearse recovery. Deciding whether to enable live execution comes afterward as a separate decision.

Three engineers explain observing data in M1, replaying recorded inputs in M2, and reconciling ledgers after a connection failure in M3 — Record what was observed, replay the same input, and reconcile state after a disconnection. AI-generated explanatory illustration.

1. M1: make data quality visible

A price on a screen is not enough evidence that a strategy can use it. Keep a sample of the received message beside its parsed representation. Distinguish the instrument identifier, price units, market timestamp and local receive timestamp. When updates stop, a screen that retains the last price must also show that the data is old.

For a two-instrument calculation, check whether one quote is fresh while the other is several minutes old. Choose freshness limits from the strategy horizon and the provider's delivery conditions; an arbitrary number of seconds is not a universal standard. Keep the order adapter disconnected and begin with only the read permissions needed for collection.

Evidence to retain	How to check it
Field interpretation	Compare a raw sample and its parsed values with the official data specification.
Missing or duplicate inputs	Feed an empty instrument, a duplicate message and out-of-order timestamps; record the result.
Stale-data detection	Stop new messages and verify that both display and decision state change.
Session boundaries	Check how the market time zone and trading date select the reference price.

2. M2: replay decisions without submitting orders

Have a strategy function return a decision from explicit inputs; keep HTTP order submission elsewhere. The output of this stage is a record that a candidate would have been produced at a particular point. If you model fills, record fee, delay and partial-fill assumptions. Simulated results are not executed performance.

This small Python example illustrates reproducibility. It reads string prices as Decimal values and runs the recording twice. The threshold is a toy condition, not a trading strategy or crossover signal, and the example makes no exchange calls.

from decimal import Decimal

def replay(prices, threshold):
    limit = Decimal(threshold)
    if not limit.is_finite() or limit <= 0:
        raise ValueError("threshold must be positive and finite")
    decisions = []
    for sequence, raw in enumerate(prices):
        price = Decimal(raw)
        if not price.is_finite() or price <= 0:
            raise ValueError("price must be positive and finite")
        decisions.append((sequence, "candidate" if price >= limit else "observe"))
    return decisions

recording = ["100.00", "101.00", "99.00"]
first = replay(recording, "101.00")
assert first == replay(recording, "101.00")
print(first)

Verified with Python 3.12: [(0, 'observe'), (1, 'candidate'), (2, 'observe')]. The example also rejects zero, negative values, NaN and Infinity. A real collector needs a separate policy for conversion failures and retaining malformed source messages.

Replay becomes unreliable if a decision function silently reads the current clock or an uncontrolled random value. Make market time, configuration version and any necessary random seed explicit inputs. Identical outputs demonstrate reproducibility; they do not establish that the strategy is sound.

3. M3: inject failures and rehearse recovery

Prepare alerts, stopping behavior and recovery before a live rollout. In a test environment, introduce disconnections, stale data, process restarts and delayed responses, then inspect the resulting state transitions.

Failure scenario	Behavior to verify
Market-data connection stops	Block new decisions and show the last healthy receive time and reason.
Simulated missing order response	Query the existing order by identifier rather than assuming failure and resubmitting.
Process restart	Reconcile stored intent with external open orders, fills and balances before allowing new orders.
Stop button	Distinguish blocking new entries from policies for canceling orders and handling positions.
Alert delivery fails	Retain the failure in logs and status views and verify another response route.

Stopped and fully closed are different states. Canceling an order cannot undo a fill, and a close request can be accepted, partially filled, completed or rejected. Recovery exercises should expose those distinctions instead of combining them into a single green indicator.

4. Keep an evidence record before considering live execution

For each stage, retain the input, expected outcome, observed outcome, failure response and configuration version. Link M1 raw samples, M2 recordings and M3 fault-injection results so that later changes can be traced to the assumptions they affect. A small amount of money or a short observation period is not a substitute for passing these checks.

Live execution requires a separate transition decision, exposure controls, order reconciliation, account permissions and an operational response plan. Testing cannot eliminate all failures or losses. Record engineering validation separately from profitability evaluation.

Bot Development Milestones: Observe, Replay and Rehearse Recovery