Multi-RPC Rotation and Failover for On-Chain Bots

한국어로 읽기

TECH INSIGHT / EXPLAINED

RPC failover is useful only if the replacement node serves the right chain and compatible state. A fast connection or a larger reported block number alone is not proof of a trustworthy quote.

Three adult colleagues explain rpc failover
AI-generated teaching illustration, not an actual operating screen or performance record.

Local execution of the actual project

These captures show the existing project UI or functions executed locally with non-private sample inputs. Each caption states the scope of the execution.

The rpc_urls field in the actual ArbitrageBot settings screen, running locally. Both .invalid endpoints are non-connecting samples. This shows the configuration location, not a live failover test.
The rpc_urls field in the actual ArbitrageBot settings screen, running locally. Both .invalid endpoints are non-connecting samples. This shows the configuration location, not a live failover test.

Probe the service, not just the constructor

For HTTP transports, creating a client can succeed before any RPC reaches the server. Put a deadline around dialing and an actual chain/header request. Use bounded concurrency for endpoint probes and close unused clients. The integration excerpt below rejects a wrong chain; full health selection must also assess header time, synchronization, errors and state agreement.

// Integration excerpt: imports context, errors, math/big, time,
// and github.com/ethereum/go-ethereum/ethclient.
func checkedClient(parent context.Context, url string,
    wantChain *big.Int) (*ethclient.Client, error) {
    ctx, cancel := context.WithTimeout(parent, 3*time.Second)
    defer cancel()
    c, err := ethclient.DialContext(ctx, url)
    if err != nil { return nil, err }
    id, err := c.ChainID(ctx)
    if err != nil { c.Close(); return nil, err }
    if id.Cmp(wantChain) != 0 {
        c.Close(); return nil, errors.New("wrong chain")
    }
    if _, err = c.HeaderByNumber(ctx, nil); err != nil {
        c.Close(); return nil, err
    }
    return c, nil // caller owns Close; this is only a basic probe
}

Use one coherent state for a decision

A route quoted from several nodes at each node’s latest block can combine states that never existed together. Choose an agreed block/hash and request state at that reference where supported. Record the reference in the quote; detect reorgs. Safe/finalized semantics vary by chain and provider, so check support rather than reuse Ethereum assumptions blindly.

An explicit unavailable result is safer than a nil client

Copy endpoint snapshots under their lock before iterating. Protect rotation counters and handle an empty list without modulo by zero. If every candidate fails, return an error and pause state-dependent actions. Do not ignore Dial errors and hand callers a possibly nil client. Use cooldowns and a few recovery probes to avoid hammering a failed provider.

Broadcast failover must preserve transaction identity

An RPC timeout after submission leaves the outcome unknown. Persist signed transaction bytes, hash, chain and nonce before broadcast, then query receipt/pending state. Rebroadcasting the same signed transaction is different from creating a fresh transaction with a new nonce. Fee replacement is a deliberate same-nonce policy, not a generic retry. Separate read availability from transaction lifecycle reconciliation.

Specify what makes an endpoint replaceable

Maintain an endpoint capability inventory: expected chain ID, required methods, historical-state coverage, block-hash selection support, and request-size limits. A server that returns the current header may lack state for an older block, so it cannot replace every read provider. Run the relevant checks at registration and recovery. Matching chain IDs alone does not finish the trust assessment.

You can also compare against trusted network configuration and known checkpoints. Several URLs backed by one operator and the same infrastructure provide weaker independence than their count suggests. Assess provider, region, and backend dependencies. Replace credential-bearing URLs with non-secret aliases in diagnostic records.

Treat one quote as a coherent read bundle

For an illustrative two-pool quote, select the chain, block height, and block hash first. If the first pool is read from server A and processing moves to B, B must supply state for that same hash. EIP-1898 defines blockHash and requireCanonical selection for supporting state-query methods. Verify support for the actual method and provider rather than assuming universal implementation.

If B does not know the selected hash, do not finish the quote against B's latest state. Find a suitable endpoint or restart the entire calculation at a new reference. requireCanonical checks the serving node's canonical-chain view; it does not promise permanent finality. Comparing hashes around number-based reads detects some reorganizations but is not equivalent to hash-selected state. Requiring compatible providers is clearer for a path that needs the stronger consistency property.

Different failures need different circuit policies

FailureResponse
Wrong chainExclude the candidate and investigate configuration
Rate limitReduce load and wait at provider scope
Specific historical state unavailableExclude from that capability only
Valid contract revertClassify the call result separately from server failure

Rotating on every revert merely distributes a bad request across providers. A transient transport problem can instead trigger a cooldown followed by a few recovery probes. Do not move all traffic back after one fast response; observe sustained success for the required capability and restore load gradually. These thresholds are operational choices, not universal RPC-specification constants.

Budget requests before injecting failures

Ten strategy workers trying three providers twice can produce up to 60 calls for one decision bundle. Per-worker retry limits therefore do not control aggregate pressure. Share concurrency and elapsed-time budgets at the dependency boundary. Cancel or collect remaining attempts after a successful path completes so abandoned work does not consume hidden capacity.

A local fake RPC can inject a wrong chain, different hashes at the same height, an old but valid response, a rate limit, and missing historical state. Success means more than eventually obtaining a value: results from different references must never be combined, and both time and request-count budgets must hold. This rehearsal evaluates the read selector. It does not broadcast transactions, establish live-trading safety, or prove that a provider reports the truth.

Documentation and next steps

Content and examples reviewed: 2026-10-01

댓글