Using Willow behind a router

In an RPC router, Willow should be the first upstream for the methods it serves, with other providers handling all other methods and any request Willow refuses. Willow serves only requests for data it holds in full and returns a specific error for all others, so a router can determine when to fall back.

Methods to route to Willow

Route these 13 methods to Willow: eth_blockNumber, eth_chainId, net_version, eth_getLogs, eth_getBlockByNumber, eth_getBlockByHash, eth_getBlockReceipts, eth_getTransactionByHash, eth_getTransactionReceipt, eth_getTransactionByBlockNumberAndIndex, eth_getTransactionByBlockHashAndIndex, eth_getBlockTransactionCountByNumber, and eth_getBlockTransactionCountByHash.

If the key’s plan includes state methods, Willow can serve verified account state as well: also route eth_getBalance, eth_getTransactionCount, eth_getCode, eth_getStorageAt, and eth_getProof on every chain, and eth_call on Ethereum. eth_sendRawTransaction and the gas methods (eth_estimateGas, eth_gasPrice, eth_maxPriorityFeePerGas, eth_feeHistory, and eth_blobBaseFee) are forwarded by Willow to a provider, and can be routed to Willow or directly to another provider. On a plan that does not include state methods, they all return -32601. See State methods.

Every other method returns -32601, so routing it directly to another provider avoids a round trip.

The proof methods (willow_*) are specific to Willow and should be called on Willow directly.

Fallback rules

Fall back to the next provider on -32000, -32002, -32601, and -32603. Only -32603 indicates a failure at Willow: -32000 indicates that the data is not held, -32002 that the request exceeds what Willow serves in one call, and -32601 that the method is not supported.
Back off on HTTP 429 (-32005).
HTTP 402 indicates that the balance is exhausted: fall back, and top up on the Billing page.
-32602, -32600, and -32700 indicate errors in the request itself.
An eth_call that reverts (code 3) or halts (-32000 without “not found”, such as out of gas) is a result, not a refusal. Another provider returns the same result for the same block, so return it to the client.

The complete table is in Limits and errors.

venn

A whitelist filter prevents every other method from reaching Willow’s remote:

filters:
  - name: willow
    whitelist: true
    methods:
      eth_blockNumber: true
      eth_chainId: true
      net_version: true
      eth_getLogs: true
      eth_getBlockByNumber: true
      eth_getBlockByHash: true
      eth_getBlockReceipts: true
      eth_getTransactionByHash: true
      eth_getTransactionReceipt: true
      eth_getTransactionByBlockNumberAndIndex: true
      eth_getTransactionByBlockHashAndIndex: true
      eth_getBlockTransactionCountByNumber: true
      eth_getBlockTransactionCountByHash: true
chains:
  - name: ethereum
    id: 1
    block_time_seconds: 12
    remotes:
      - name: willow
        url: https://api.willow.tech/v1/rpc/1/{YOUR_KEY}
        priority: 0
        filters: [willow]
      - name: your-provider
        url: https://example.com/rpc
        priority: 1

venn falls back on Willow’s refusals without additional configuration. On -32000, it falls back only when the message contains “not found”, which every Willow refusal of these methods does. On -32002, it falls back without pausing Willow.

eRPC

List the same methods in the upstream’s allowMethods, with ignoreMethods: ["*"], and eRPC sends no other methods to Willow. eRPC treats every -32000 refusal of these methods as missing data and tries the next upstream. After a -32002, it splits an eth_getLogs range once no other upstream has served it.

eRPC (the gfx-labs fork at gfx-v0.0.5) was tested in front of Willow on all seven chains with an indexer-like workload. Four settings determined how much of the traffic Willow served and whether its refusals reached the other providers:

Hedging. Omit hedge from the network failsafe, or set maxCount: -1. eRPC converts maxCount: 0 to 1, so a value of 0 enables hedging. With hedging enabled, the upstream that responds is determined by a race, and a provider closer to the client’s servers wins most races. eRPC’s hedge rule can also return Willow’s -32000 refusal to the client and cancel the other request, which held the data.
Selection. eRPC orders upstreams by its own scores, not by their order in the configuration, so listing Willow first does not make it the first upstream. The policy below places Willow first and every other upstream after it.
Tags. Do not tag other providers tier:fallback while Willow is the only other tier. eRPC’s default policy then returns only Willow’s tier, so every method Willow does not serve, and every block it cannot serve, returns an error instead of reaching the other providers.
High block rates. Robinhood Chain produces about 10 blocks a second and Arbitrum One about 4. Poll each upstream’s head every 2 seconds (statePollerInterval: 2s) and allow about 2 seconds of head lag (headLagToleranceBlocks: 20 on Robinhood Chain, 8 on Arbitrum One, 2 elsewhere); otherwise, eRPC can mark Willow as lagging.

Configuration for Ethereum, with another provider listed after Willow:

projects:
  - id: main
    upstreams:
      - id: willow-1
        endpoint: https://api.willow.tech/v1/rpc/1/{YOUR_KEY}
        tags: [tier:cheap]
        ignoreMethods: ["*"]
        allowMethods: [eth_blockNumber, eth_chainId, net_version, eth_getLogs, eth_getBlockByNumber,
          eth_getBlockByHash, eth_getBlockReceipts, eth_getTransactionByHash, eth_getTransactionReceipt,
          eth_getTransactionByBlockNumberAndIndex, eth_getTransactionByBlockHashAndIndex,
          eth_getBlockTransactionCountByNumber, eth_getBlockTransactionCountByHash]
        evm:
          chainId: 1
          statePollerInterval: 2s
          headLagToleranceBlocks: 2
        failsafe:
          - matchMethod: "*"
            timeout: {duration: 10s}
            retry: {maxAttempts: 1}
      # your other providers here, untagged (not tier:fallback)
    networks:
      - architecture: evm
        evm:
          chainId: 1
        failsafe:
          - matchMethod: "*"
            timeout: {duration: 60s}
            retry: {maxAttempts: 3}
            # no hedge key (maxCount: 0 means 1; -1 turns hedging off)
        selectionPolicy:
          evalInterval: 2s
          evalFunc: |
            (upstreams, ctx) => {
              const healthy = upstreams
                .removeCordoned()
                .excludeIf(all(samplesAbove(10), errorRateAbove(0.7)))
                .whenEmpty(() => upstreams)
                .sortByScore(PREFER_FASTEST);
              return healthy.byTag('tier:cheap').union(healthy).union(upstreams);
            }

Chain head

If the router derives the chain head from Willow, the head follows Willow’s. Willow refuses latest once its newest block is older than the chain’s allowance, so the router fails over to another provider within seconds. If the router derives the head from another provider, a request for the newest block can reach Willow before Willow holds it; Willow refuses it with -32000, and the router passes it to the next upstream.

eth_getLogs ranges

Do not pre-split eth_getLogs ranges for Willow. Its index has no block-range cap, so a filtered query over a wide range is a single call. A query that is too large is refused with -32002 and a message that specifies a range that fits (Try with this block range [0x…, 0x…]), so a router that splits on errors splits only when necessary. The rules are described under Range limits.

eRPC. Leave getLogsAutoSplittingRangeThreshold at 0 on Willow’s upstream, and getLogsSplitOnError on (the default). The network’s getLogsMaxAllowedRange (30,000 blocks by default) refuses wider ranges before any upstream receives them, so raise it if clients request wider ranges.
venn sends -32002 to the next provider without pausing Willow.
Concurrency. An account may have two eth_getLogs calls reading blocks concurrently (one on Free). Beyond that, a call is refused with -32000 within 50 ms and is passed to the next provider. If the router can limit concurrent eth_getLogs calls per upstream, a limit of 2 holds those calls for Willow instead.