Multi-RPC Rotation and Failover for On-Chain Bots

🌐 한국어

Run an on-chain bot long enough and infrastructure trips you up well before strategy does. The worst offender is the RPC node. Public RPCs time out without warning, fall several blocks behind, and some days simply die. Tie your bot to a single node and the bot dies the moment that node does. The fix is to keep several endpoints and switch automatically to a healthy one.

Measuring a Node's Health

"Alive" is judged on two axes: does it respond (latency), and does it know the latest block. A node that connects fine but sits twenty blocks behind is every bit as dangerous as a dead one—quote on stale state and you're chasing an opportunity that already vanished.

type RPCNode struct {

    URL     string

    Block   uint64        // latest block height this node knows about

    Latency time.Duration // response latency

    Healthy bool

}

Health-Check Every Node in Parallel

Check endpoints one at a time and a slow node near the front blocks every check behind it. Probe them all concurrently and collect the results. Put a timeout on each probe so a dead node can't hang forever.

func (b *Bot) refreshRPCHealth(ctx context.Context) {

    var wg sync.WaitGroup

    results := make([]RPCNode, len(b.rpcURLs))

    for i, url := range b.rpcURLs {

        wg.Add(1)

        go func(idx int, u string) {

            defer wg.Done()

            node := RPCNode{URL: u, Healthy: false}

            start := time.Now()

            c, err := ethclient.Dial(u)

            if err != nil {

                results[idx] = node // dial failed → unhealthy

                return

            }

            defer c.Close()

            // Drop it if it can't return the latest block within 3 seconds

            checkCtx, cancel := context.WithTimeout(ctx, 3*time.Second)

            defer cancel()

            block, err := c.BlockNumber(checkCtx)

            if err != nil {

                results[idx] = node

                return

            }

            node.Block = block

            node.Latency = time.Since(start)

            node.Healthy = true

            results[idx] = node

        }(i, url)

    }

    wg.Wait()

    // ... ranking sort below

}

Ranking Rules — Healthy, Then Freshest, Then Fastest

Sort the results so the "best node" lands at the front. The priority is unambiguous: healthy nodes always first, then the node at the higher block, and finally the node with lower latency.

sort.Slice(results, func(i, j int) bool {

    // 1) healthy nodes to the front

    if !results[i].Healthy && results[j].Healthy {

        return false

    }

    if results[i].Healthy && !results[j].Healthy {

        return true

    }

    // 2) higher (fresher) block to the front

    if results[i].Block != results[j].Block {

        return results[i].Block > results[j].Block

    }

    // 3) lower latency to the front

    return results[i].Latency < results[j].Latency

})

Failover at the Point of Use

When pulling a client from the ranked list, walk from the front and take the first healthy node that actually connects. The ranking reflects the last health check, but a node can have died in the interval—so verify with a real connection.

func (b *Bot) getClient() *ethclient.Client {

    b.rpcMu.RLock()

    nodes := b.rpcNodes

    b.rpcMu.RUnlock()

    for _, n := range nodes {

        if !n.Healthy {

            continue

        }

        if c, err := ethclient.Dial(n.URL); err == nil {

            return c // best node that's actually alive

        }

    }

    // If everything fails, fall back to plain index rotation (last resort)

    url := b.rpcURLs[b.rpcIdx%len(b.rpcURLs)]

    b.rpcIdx++

    c, _ := ethclient.Dial(url)

    return c

}

Re-run the health check periodically (say, every few rounds) to refresh the ranking. A node that was down and came back rejoins the candidate pool, and one that has slowed down drifts to the back.

Field Notes

• Mix providers. Five nodes from the same company all go down when that company has an outage. Mix endpoints from different providers.
• Separate reads from writes. Send queries to any healthy node, but route transaction broadcasts to a more reliable—ideally paid—node.
• Set a block-lag threshold. Add a guard that excludes any node more than N blocks behind the highest, even if it reports healthy.

Not depending on a single node—this one simple principle raises your bot's uptime dramatically. Before any flashy strategy comes the plain requirement of continuing to run without dying.

This article is for educational and informational purposes and is not investment advice. On-chain automated trading carries risk of principal loss, and you alone are responsible for the outcomes.