Reading QA test results

The qa-* workflows run on Erigon's self-hosted QA runners (bare-metal Hetzner machines) and take hours. They are not ordinary unit tests: a red QA check almost always means "Erigon misbehaved on real chain data", not "a flaky CI runner".

Each workflow follows the same shape:

1. build Erigon from the branch under test;
2. run a Python driver from the erigon-qa repo ($ERIGON_QA_PATH = /home/qarunner/erigon-qa) that starts Erigon, tails its log, polls its JSON-RPC and Prometheus endpoints, and applies pass/fail thresholds;
3. write result-<chain>.json and upload it plus the Erigon debug log, metric plots and an FD-leak report as artifacts;
4. push the measurements to MongoDB → Grafana (https://monitoring.erigon.io/d/ddqiwbfvrgwlcd/erigonqa).

Triage order — cheapest signal first

Never start by dumping the whole run log. A tip-tracking step log is hours of Erigon debug output (hundreds of MB). Work down this list and stop as soon as you have the answer:

1. result-<chain>.json — the verdict in one small file.

gh run download <run-id> --repo erigontech/erigon -D /tmp/qa   # all artifacts
# or list first, artifact names vary per workflow:
gh api repos/erigontech/erigon/actions/runs/<run-id>/artifacts --jq '.artifacts[].name'
jq '{outcome, reason, measures: (if (.measures|type)=="object" then (.measures|keys) else .measures end)}' /tmp/qa/test-results*/result-*.json

{ "outcome": "FAILURE", "reason": "in sync less than 75% of the time",
  "exit_code": 1, "measures": { "total-sync-time_%": 63, ... } }

outcome is one of SUCCESS / FAILURE / Unexpected error (ERROR in the clean-exit and stage-exec drivers). See references/result-json.md for what each field and measure means, and how outcome maps to root cause.

2. The *** report lines in the test log. Every line the driver considers part of the final report is marked with three asterisks, so the whole verdict is one grep away:

gh run view <run-id> --repo erigontech/erigon --log 2>/dev/null | grep -F '***'
# or, on a downloaded artifact / local run:
grep -F '***' test_execution.log

3. The Erigon debug log (erigon-logs* artifact, or .../logs/erigon.log* inside test-results-*) — only once you know which phase failed and roughly when. Grep around the failure timestamp for [EROR], panic, SIGSEGV.

4. The stack dump at the bottom of the test log. On abort the driver sends SIGUSR1 to Erigon to dump every goroutine — the go-to evidence for a suspected deadlock or a stuck stage.

Reading the test log

The step log is an annotated Erigon log:

--> ... [INFO] *** ... is both: a driver line that belongs to the report. The clean-exit and stage-exec drivers use a simpler *** - <utc-ts> - <msg> form and prefix each Erigon line with OK-> / !!->.

The report block sits at the very end of a completed run. A canonical passing tip-tracking report:

--> [...] [INFO] *** Tip tracking completed
--> [...] [INFO] *** Total tracking time: 7200 secs
--> [...] [INFO] *** Total sync time: 7063 secs (98%)
--> [...] [INFO] Final value of metric 'exec_steps_in_db': 15.75
--> [...] [INFO] *** Tip tracking completed successfully
--> [...] [INFO] *** In-sync delay: 1979.9 secs
--> [...] [INFO] *** Snapshots download time: 6.1 mins
--> [...] [INFO] *** Data-dir size increment: 83.75 mb
--> [...] [INFO] *** Block height increment: 1276
--> [...] [INFO] *** Per-block size increment: 0.07 mb
--> [...] [INFO] *** Blocks per second: 0.18

Full field-by-field reference, including the metric block that follows it: references/sync-tests.md.

Which step holds the log. For the sync tests the report and the Erigon output you want are in the step Run Erigon, wait sync and check ability to maintain sync. Tip-tracking runs a preparatory step first, Run previous Erigon version and wait for sync (stabilization step), which brings the pre-built DB up to the tip using a previous Erigon release — its logs are normally irrelevant to the verdict, but if it fails (the reference version couldn't reach the tip) the branch under test never got to run, so check it before blaming the PR.

The three test categories

The documentation groups the qa-* workflows into three families, each with its own log shape:

• Sync tests — tip-tracking (pre-built DB) and sync-from-scratch (blank DB). Both drive the same tip-tracking Python driver and share the *** report and thresholds described below.
• RPC tests — integration (response diffs) and performance (latency at rising QPS). Different log format entirely: references/rpc-tests.md.
• Miscellaneous — snapshot download and clean exit. See Other test families.

The three ways a Sync test fails

These cover qa-tip-tracking*, qa-constrained-tip-tracking, qa-sync-from-scratch*, qa-sync-with-externalcl and qa-sync-test-bisection-tool — they all run the same tip-tracking/run_and_check_tip_tracking.py driver.

a) Sync-time threshold — Erigon reached the tip but could not hold it

*** Total sync time: 4536 secs (63%)
[ERROR] *** Test failed: total sync time below threshold (75%), final value: 63%

The node must be within 2 × block_time of wall-clock (24 s mainnet/sepolia/ hoodi, 10 s gnosis/chiado, 4 s bor-mainnet/amoy) for ≥ 75 % of the tracking window. Below that → FAILURE, reason: "in sync less than 75% of the time". It's a performance regression: something made block processing slower than the chain produces blocks. Cross-check exec_mgas_sec, the block_consumer_delay_hist_bucket percentages and the metric plots.

b) exec_steps_in_db threshold — state is not being collated fast enough

Final value of metric 'exec_steps_in_db': 378.656
[ERROR] *** Test failed: exec_steps_in_db exceeded threshold (300), final value: 378.656

Undigested execution steps piling up in the DB. Only enforced for Erigon ≥ 3.5; on 3.4.x the driver logs a warning and skips the check (the block-snapshot collation cap there produces benign spikes).

c) Deadline reached — the run timed out before reaching the tip

*** Deadline reached
[ERROR] *** Aborting test: Deadline reached

Read this carefully — it is the most misread outcome. It means Erigon never got to the chain tip within TOTAL_TIME_SECONDS, and the driver could not attribute it to any single check. It surfaces as outcome: "Unexpected error", reason: "Deadline reached".

This is not an infrastructure problem. It is an Erigon problem that needs investigation. Do not report it as "runner too slow" or "CI flake" without evidence.

Investigate: find where the log stops progressing, identify which stage was running (snapshot download / execution / commitment), and read the goroutine stack dump at the end for a stuck or deadlocked stage.

Aborts also happen for Erigon process terminated unexpectedly, Erigon in ERROR: ... (any [EROR] line while the test is active), SIGSEGV, and SyncSentinel error — all outcome: "Unexpected error" with the cause in reason. references/triage.md has the full decision tree, including which failures are genuinely environmental.

RPC tests

Not a ***-report log — a per-request pass/fail list (integration) or a per-QPS-stage latency list (performance), each with its own verdict location. Full log-reading guide: references/rpc-tests.md.

• integration — replays ~1000 requests and marks each OK or failed: diff mismatch (vs stored expected responses on historical data, vs a reference client — geth/nethermind — on tip data). Verdict is in results/test_report.json + output.log + a summary.md echoed into the job summary, not in result-<chain>.json. The suite retries up to 5× to shake out transient tip inconsistency; a per-test actual/expected/diff folder ships in the artifact, also browsable at http://rpctests.erigon.io/hive/main/index.html.
• performance — Vegeta at rising QPS; each stage logs success=…% lat=[max=…], then HDR percentiles per method. outcome=success in the log is not the verdict: the job fails on change-point detection, and a red run can reflect an open change-point issue from an older regression, not this run.

Other test families (Miscellaneous)

Each has its own driver, thresholds and report vocabulary:

• snap-download — counts snapshots, download rate, per-phase completion; fails with *** Snapshot download completed with failure: <why>.
• clean-exit — sends Ctrl-C and measures exit time; fails on panic, segmentation fault, or not exiting within the threshold.
• stage-exec — runs integration stage_exec and scans for [EROR], catch panic, wrong receipt, SIGSEGV, EXCEPTION.

Details and the per-workflow table (runner labels, driver script, timings, artifact names, --test_name used in MongoDB): references/workflows.md.

Rules of thumb

• The result JSON is the verdict; the log is the explanation. Quote reason before theorising.
• Measures are as important as pass/fail. A green run whose per-block-size-increment_mb doubled is a regression the thresholds missed — compare against Grafana history rather than judging one run in isolation.
• A test that never produced result-<chain>.json (no test-results artifact, test_executed != true) failed before the test ran — build, datadir restore, or runner problem. That, unlike Deadline reached, really is infrastructure.
• Warnings are not failures, but Node in false sync condition, batch processing with #blocks=N and head updated with age=N are the breadcrumbs that explain a sync-time failure.
• Never propose muting or skipping a QA test to get a check green; see the project's test-skip policy in CLAUDE.md.