Health and status

Five public endpoints report service status. They require no key, and their responses are not cached.

Endpoint	Reports	503 when
`/v1/rpc/health`	Every chain: newest block, age, verification, and which chain is canonical	Any chain is unhealthy
`/v1/rpc/health/full`	Full blocks on every chain	Too many of a chain’s recent blocks are held without their full form
`/v1/rpc/health/proofs`	Proven ranges, proof jobs, and proof storage	Proving is unavailable, or proof storage is behind
`/v1/rpc/health/finality`	The `finalized` and `safe` tags on the six L2s	Either tag cannot be verified on a chain
`/v1/rpc/health/history`	Fetches of older blocks on every chain	Fetches are paused on a chain

GET /v1/rpc/health

Reports the status of every chain. A chain is healthy when its newest block is at most 180 seconds old, it is not stalled on a block that failed verification, and its canonical-chain verification is ok. The endpoint returns HTTP 200 when every chain is healthy, and 503 otherwise; consult the entry for the relevant chain.

curl -s https://api.willow.tech/v1/rpc/health

Response (Ethereum shown, truncated):

{
  "chains": {
    "1": {
      "ageSeconds": 3,
      "canonicalConfirmedBlock": 26262675,
      "canonicalFinalizedBlock": 26262589,
      "canonicalHeliosBlock": 26262675,
      "canonicalLagSeconds": 25,
      "canonicalMismatches": 0,
      "canonicalSource": "helios",
      "canonicalStatus": "ok",
      "from": 26133199,
      "fullBlocks": {
        "eventIndex": {"history": [[0, 26133198]], "historyRuns": 1, "indexed": [26183600, 26262676], "pending": 0},
        "from": 26183600,
        "healthy": true,
        "keep": "forever",
        "maxLogsBlocksPerRequest": 1000,
        "maxLogsRange": 100,
        "maxLogsWalkMs": 2000,
        "recentBlocks": 256,
        "recentSkipped": 0,
        "skipped": 0,
        …
      },
      "healthy": true,
      "latestMaxAgeSeconds": 180,
      "tip": 26262676,
      "upstreamHead": 26262676,
      "verifiedFrom": 26133199,
      "verifyFailures": 0
    },
    …
  },
  "healthy": true
}

healthy: the chain is current, not stalled, and its canonical-chain verification is ok.
tip: the newest block Willow holds for this chain.
ageSeconds: the age of that block, from its timestamp.
latestMaxAgeSeconds: the maximum age of that block at which eth_blockNumber and latest still return it.
verifiedFrom: the first block of the contiguous run Willow follows at the chain head. Every block from it to the tip was verified. Blocks below it are history: held, or fetched on request. from is the same block.
upstreamHead: the newest block known to be available from Willow’s providers. The difference between it and tip is Willow’s lag behind them.
verifyFailures: the number of times a provider’s copy of a block failed verification since the service started. A failed copy is never stored; the block is fetched again from another provider.
stalledAtBlock: present only while the chain is halted at a block that no provider served in a verifiable form.
lastVerifyError: present after a failure: the block, the provider (by position), the reason, and the time (block, endpoint, reason, at in Unix seconds).
fullBlocks: blocks held in full, with every transaction and receipt.
- from: the first block of the run at the tip that is held in full. keep is the retention period for blocks: forever.
- skipped counts blocks held without their full form since the service started (see Fields in full blocks), and recentSkipped those among the newest recentBlocks blocks stored.
- eventIndex: the blocks covered by the eth_getLogs index. indexed is the range at the tip, history the history ranges, historyRuns their count, and pending the blocks awaiting indexing.
- maxLogsRange, maxLogsBlocksPerRequest, maxLogsWalkMs: the eth_getLogs range limits.
- cold, keptReceiptProofs, blobUpdateFraction: Willow’s long-term storage of blocks, its stored receipt proofs, and, on Ethereum, the blob fee fraction used to compute blobGasPrice.
canonical… on Ethereum: the light-client verification. canonicalStatus is ok; lagging when the newest confirmed block is more than 120 seconds old; mismatch while Willow withholds blocks because the providers disagree with Helios; or unavailable when Helios has not responded for 120 seconds. canonicalConfirmedBlock is the newest block Helios confirmed, canonicalLagSeconds its age, canonicalFinalizedBlock the newest block that matched Helios’s finalized block, and canonicalMismatches the number of disagreements observed since the service started.
canonical… on the other chains: the sequencer-signature verification, with canonicalSource sequencer. canonicalStatus is ok; lagging when the newest confirmed block is more than 60 seconds old; mismatch while Willow withholds blocks because the providers disagree with the sequencer; or unavailable when no signed blocks were read for 60 seconds, or the signing keys were last read from Ethereum more than 30 minutes ago. canonicalSignedBlock is the newest block the sequencer signed, canonicalSigners the accepted signing keys, as read from Ethereum, and canonicalSignerAgeSeconds the time since they were read. canonicalSignedFrom is the source of the signed blocks: peer counts on the OP Stack chains, and the sequencer feed connection status on Arbitrum One and Robinhood Chain.
canonicalMismatch, canonicalError: present only during a disagreement, or after an error.
finalized…, safe…, derivation… on the six L2s: the verification underlying the finalized and safe tags. These fields never mark a chain unhealthy on this endpoint; they are reported by /v1/rpc/health/finality.

GET /v1/rpc/health/full

Reports full blocks on every chain. It returns HTTP 200 while every chain holds its newest blocks in full, and 503 once 16 or more of a chain’s newest 256 blocks were stored without their full form. This occurs when a field Willow computes no longer matches the providers’ value, for example after a fork changes how the field is computed. Logs and headers continue to be served in that case, and the full-block methods refuse the affected blocks with -32000, so a router sends those requests to another provider.

curl -s https://api.willow.tech/v1/rpc/health/full

Each chain’s fullBlocks is the same object as in /v1/rpc/health.

GET /v1/rpc/health/proofs

Reports proofs on every chain: which blocks are proven, how the chain is proven, whether proving on request is accepting requests, and the state of proof storage. It returns HTTP 200 when proving is available on every chain and proofs are stored on schedule, and 503 otherwise.

curl -s https://api.willow.tech/v1/rpc/health/proofs

Response (Ethereum shown, truncated):

{
  "chains": {
    "1": {
      "firstProvenBlock": 26226570,
      "healthy": true,
      "keptFrom": 26133199,
      "lastSegmentEnd": null,
      "leafBlocks": 1,
      "newestProvenBlock": 26226601,
      "newestProvenNearTip": null,
      "proofJobs": {
        "available": true,
        "queued": 0,
        "running": 0,
        "runner": {"ageSeconds": 13, "paidOnly": false, "paused": false, "pod": "none", "stale": false, …}
      },
      "provenRanges": [[26226570, 26226571], [26226600, 26226601]],
      "tip": 26262677,
      "windowBlocks": 64,
      …
    },
    …
  },
  "healthy": true
}

healthy: proving is available on this chain and its proofs are stored on schedule.
tip: the newest block Willow holds.
leafBlocks, windowBlocks: blocks per leaf and per final segment on this chain (see Proof construction).
firstProvenBlock, newestProvenBlock: the first and last proven block.
provenRanges: proven block ranges, [first, last], oldest first (up to the 32 newest).
lastSegmentEnd: the last block of the newest final segment, or null.
newestProvenNearTip: a proven block within 64 blocks of the tip, or null. Blocks are proven on request, so the value is usually null.
keptFrom: the first block for which proofs can be generated on request.
proofJobs: available indicates whether proof requests are accepted. queued and running count open jobs on this chain across all keys. runner describes the prover: pod is none, starting, or warm; paused; paidOnly, when it accepts only paid jobs; and stale is true when it has not reported for more than 300 seconds.
archive, disk, lastPrune: proof storage. archive is the copy of every proof in long-term storage, disk the disk space the proofs occupy on the server, and lastPrune the last time proofs already in long-term storage were removed from the server’s disk.

GET /v1/rpc/health/finality

Reports the finalized and safe tags on the six chains other than Ethereum, with the same finalized…, safe…, and derivation… fields as /v1/rpc/health and a healthy flag per chain. It returns HTTP 200 when every chain is healthy, and 503 otherwise.

A chain is unhealthy when the finality service has not reported ok for 10 minutes; immediately when the block it identifies as finalized is stored with a different hash, or a provider’s chain has a different block at or below a block identified as finalized or safe; when an identified block has waited 10 minutes for every provider’s safe block to reach it; or when the finalized block is more than two hours old. An unhealthy chain continues to serve all requests except those for the two tags.

curl -s https://api.willow.tech/v1/rpc/health/finality

GET /v1/rpc/health/history

Reports fetches of older blocks on every chain. It returns HTTP 200 while every chain accepts fetches, and 503 while fetches on a chain are paused or stopped, or Willow cannot read account balances.

curl -s https://api.willow.tech/v1/rpc/health/history

bytesToday, maxBytesPerDay, maxBytesPerChainPerDay: block data fetched today (UTC) on all chains, before compression, and the daily limits across all accounts.
chains: per chain, paused (fetches paused while disk usage exceeds maxDiskUsedPercent), bridgeUnpinned (the reason fetches await a server update), bytesToday, queue (fetches and blocks waiting), and, on Ethereum, backfill, Willow’s own fetch of Ethereum’s history.