Structured Bot Logs: Reconstruct Decisions Without Leaking Secrets

한국어로 읽기

TECH INSIGHT / EXPLAINED

Useful logs answer a concrete incident question: was the order never submitted, accepted but unconfirmed, partly filled, or already reconciled? Event names must distinguish those states.

AI-generated educational illustration. Three colleagues follow a decision through event records while keeping credentials outside logs.

Define events before selecting levels

Use DEBUG for expensive diagnostic detail, INFO for expected state transitions, WARN for degraded behavior and ERROR for failed operations needing attention. A routine cancellation is not necessarily a warning; an expected authentication refresh is not necessarily an error. Classify by operational meaning.

Give each decision a stable local ID and link its orders and unique fills. Log UTC event time, environment, application version, decision ID, local order reference, venue order ID when known, state, reason and attempt. Use decimal strings for prices and quantities.

package main
import ("log/slog"; "os")
func main() {
    log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
    decision := log.With("decision_id", "demo-001", "order_ref", "local-001")
    decision.Info("order_submit_started", "qty", "0.010", "price", "65000.00")
    decision.Warn("order_outcome_unknown", "reason", "read_timeout", "action", "reconcile")
}

A timeout is an unknown outcome

The sample records submission start followed by a read timeout. It deliberately does not claim order rejection or cancellation. A reconciliation worker must query the venue and attach the discovered result to the same decision. A single trace ID helps navigation, but durable order state remains the accounting source of truth.

For a hypothetical incident, find decision demo-001, compare send and receive times, locate the first venue acknowledgement, then compare the persisted fills. This is a reproducible investigation path; a vague “order done” message is not.

Keep secrets out, including partial secrets

Do not log API keys, bearer tokens, signed query strings, full account identifiers or raw authentication payloads. Masking the first and last characters of a key still reveals part of the secret. Use a non-secret configured credential label such as trading-reader instead.

Adopt an allowlist of safe fields and inspect exception handlers: dumping a request object may bypass normal redaction. Limit untrusted text lengths and use the JSON encoder rather than concatenating log lines.

Make observation survive an incident

Bound log retention by size and time, restrict access and monitor disk pressure. Group repeated alerts and give them a cooldown so one failing dependency does not drown out an order-state problem. Track data age, pending-order age, reconciliation backlog and error rate alongside logs; a process can remain alive while doing no useful work.

Keep occurrence time separate from collection time

An exchange event can arrive after network delay, making its source timestamp different from the time the application writes a log. The OpenTelemetry log data model distinguishes Timestamp from ObservedTimestamp. If source time is unavailable, represent that absence instead of relabeling receipt time as the original event time. Clock differences also mean the subtraction is not necessarily pure network latency.

Use a monotonic clock for elapsed intervals inside one process, and consider a run identifier plus local sequence numbers to distinguish restarts. When an older response is logged late, event identities and state versions help establish whether it overwrote newer knowledge or was merely observed later. Wall-clock timestamps alone cannot establish a global causal ordering.

A log schema is a deployed interface

If a dashboard aggregates the reason field and a new release moves the information into free-form message text, alerting can quietly break. Manage event names, data types, required fields, and schema versions as a small contract. If prices are decimal strings, test that no code path emits numbers instead. Distinguish an unknown value from a known numeric zero.

Add fields so older consumers can ignore them, but version changes to existing meanings deliberately. Replay sample records and confirm that both old and new queries still identify critical events during migration. Define required fields separately for submission, execution, and reconciliation events instead of putting every possible field into every record. This keeps the contract understandable without producing large objects full of meaningless empty values.

Put incident identities in logs, not metric labels

Decision and order identifiers are useful for individual investigation, but using them as metric labels continuously creates new time series. The Prometheus label guidance warns against unbounded cardinality. Aggregate error rates by service, environment, and a limited set of reason codes. Link investigators to logs or traces for the detailed identity.

Sampling verbose diagnostics can be reasonable, but randomly dropping transitions into unknown-order state or reconciliation completion destroys recovery evidence. Preserve required audit events under a separate policy with explicit retention and deletion rules. Keeping logs for longer does not substitute for durable order-state storage; each serves a different recovery requirement.

Investigate one synthetic failure end to end

Build an offline rehearsal around submission started, response timeout, unknown state, lookup started, acceptance discovered, and reconciliation completed. Shuffle records or duplicate them and check whether identities still allow the actual transitions to be reconstructed. If no execution was confirmed, the completion record must not claim a fill. Missing intermediate evidence should remain visible as a gap, not be silently invented by the investigation view.

Then inject a fake secret marker, an external error containing newlines, oversized fields, and a collector that simulates a full disk. Confirm that the marker never appears in final output and external strings cannot forge additional records. Do not report logging failures indefinitely through the same broken logger. Surface a bounded independent failure counter and explicit degraded state. These rehearsals use synthetic events and fake collectors; they do not claim that a live account incident occurred.

Sources and further reading

Go slog · OWASP logging guidance

Content and examples reviewed: 2026-10-01