Structured Logging and Observability for Trading Bots

🌐 한국어

Your bot placed a strange order at 3 AM. You go looking for the cause the next day and the only log is a single println("placed order")—that incident stays permanently unsolved. For an unattended bot, logs aren't a luxury; they're the black box.

⚠️ Live automated trading can incur losses from software and network failures. This article is for educational and informational purposes.

1. Log Levels — Separating Noise from Signal

Write everything in the same tone and the things that matter get buried. Split into at least four tiers.

• DEBUG: detail for development and reproduction (depth snapshots, intermediate calculations). Off by default.
• INFO: milestones of the normal flow (entries, exits, successful reconnects).
• WARN: odd but survivable (cancelled unfilled order, delayed data).
• ERROR: needs intervention (order failure, expired auth, recovered panic).

In production it's convenient to write INFO and above to file with DEBUG behind a switch. Levels are an index that lets you filter later.

2. Why Structured (JSON) Logs

Prose logs written for humans are hard for machines to search and aggregate. Emit JSON with fields and queries like "only today's ERRORs" or "only fills on symbol A" become immediate.

// String log (search hell)

log.Printf("symbol %s buy %d shares @ %.0f", code, qty, price)

// Structured log (queryable by field)

logEvent(map[string]any{

    "level":  "INFO",

    "event":  "order_filled",

    "code":   code,

    "side":   "buy",

    "qty":    qty,

    "price":  price,

    "trace":  traceID,

})

// → {"level":"INFO","event":"order_filled","code":"069500","qty":200,...}

In Go's standard library, log/slog gives you exactly this shape. No heavyweight framework required.

logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))

logger.Info("order_filled",

    "code", code, "qty", qty, "price", price, "trace", traceID)

3. Correlation Tracing — The Full Arc of One Decision

One bot cycle runs signal → validation → order → fill → state persistence. Attach the same trace ID across that flow and, when something goes wrong, you can follow that decision end to end in a single thread.

trace := newTraceID()               // one cycle = one ID

logger.Info("signal_fired",  "trace", trace, "gap", gap)

logger.Info("depth_check",   "trace", trace, "askQty", askQty)

logger.Info("order_sent",    "trace", trace, "qty", qty)

logger.Warn("order_timeout", "trace", trace)   // ← it went wrong here

logger.Info("order_cancel",  "trace", trace)

Filter by trace later and the full context of "why did that order get cancelled unfilled" reconstructs itself in chronological order. Your logs stop being scattered dots and become a connected line.

4. What to Keep and What to Throw Away

Keep everything and you blow out the disk while burying what matters. The test is "is this line needed to explain an incident?"

• Always keep: every state transition (entry/exit/halt), every external side effect (orders, cancels, API call results), every ERROR/WARN.
• Drop or demote to DEBUG: raw depth arriving dozens of times a second, intermediate values computed every loop. Turn it on only when needed.
• Never keep: secrets such as API keys, tokens, and account numbers. Leaking through logs is still leaking.

// Record sensitive values only in masked form

func maskSecret(s string) string {

    if len(s) <= 8 {

        return "****"

    }

    return s[:4] + "****" + s[len(s)-4:]

}

logger.Info("auth_ok", "key", maskSecret(appKey))

5. Operational Hygiene — Rotation and Alerts

• Log rotation: split files by date or size so no single file grows without bound. Delete old ones automatically.
• Push ERRORs outward: don't just pile them into a file—forward them immediately as messenger alerts. Nobody can watch logs all day.

Summary

• Separate noise from signal with levels — INFO and above in production, DEBUG behind a switch
• Use structured JSON logs so you can query and aggregate later (in Go, slog is enough)
• Reconstruct the full arc of a decision with a trace ID
• Keep state transitions, external side effects, and errors — and mask secrets
• Protect the disk with rotation, and push ERRORs out as alerts

Observability means "can I reconstruct exactly what the bot did, after the fact?" Long before a fancy dashboard, a single log line that can explain an incident is the safety net of unattended operation.