Methodology

This page explains exactly what we check, what we publish, and how to contest a result. Checker version 0.2.0. Schedule: daily at 14:17 UTC.

What we publish

For each service, only factual results of our own checks, each with a timestamp (UTC) and a link to the raw log:

• Reachable: an HTTP response arrived within 20 seconds (any status code).
• HTTP status and latency: the status code and the time from sending the request to receiving the full response (up to 256 KB), measured once from a GitHub-hosted runner or our own machine.
• Valid x402 challenge: the unpaid request returned HTTP 402 with payment terms we could read, either from the PAYMENT-REQUIRED header (x402 v2) or from the JSON body (x402 v1).
• Advertised price: the price in the public listing where we found the service (the source is linked on each service page).
• Price in the 402 challenge: the amount the endpoint itself asks for, using the USDC "exact" option on Base. Matches listing compares the two.
• Charged price and delivered a valid response: only for paid checks. Charged price is the amount we signed and the endpoint accepted. A valid response means HTTP 2xx with a non-empty body that parses as JSON when it says it is JSON. When the latest check made no payment, these read "not tested (no payment made)".

We do not publish grades, scores, rankings, or labels such as "good" or "bad". A single failed check can be a temporary network problem on either side, so read results over time (each service page shows recent history).

The free check

One request per service, using the sample request from the public listing (method, query or body). No payment is made. We record the response and parse the payment challenge. Requests identify themselves with the user agent WithGrokBot-catalog-checker/0.2.0, and redirects are not followed.

The paid check

Paid checks are off by default and never run in the daily job. We start them by hand, on our own machine, from time to time. When run, the checker pays like any other customer: it signs a USDC transfer authorization (EIP-3009) for the price in the 402 challenge (only if it equals the listed price), sends it, and records the response and the settlement transaction if the endpoint returns one. It signs only for USDC on Base, only with the "exact" scheme, and never for other tokens or networks.

Hard spend caps, checked before anything is signed: at most $0.1 per call, $1 per UTC day, and $20 in total, ever. Every attempt counts toward the caps, even if the endpoint fails. The spend ledger is public at results/spend_ledger.json.

Known-answer tests

Every paid call is also checked against a known answer for that service, and the result is pass or fail. Examples: a token balance must equal balanceOf from a free public Ethereum RPC; a web search for "x402 payment protocol" must return at least one result on x402.org or github.com/coinbase/x402; a weather reading must be within 3 °C of a free public weather source. The full list, with inputs and pass rules, is in data/quality_tests.json; each service page shows its test. The test runs on the full response when it arrives, and the observed and expected values are kept in the raw log.

• A paid call that settles but fails its test counts as a failed call.
• If our own reference source cannot be reached (for example a public API is rate-limited), the call gets no pass/fail result rather than a fail.
• Services that are broken on the seller's side (for example an HTTP 500, or asking to be paid a second time) are marked "facts only": we publish what happened but give no pass/fail result. We lift this by hand after a clean paid call.
• Paid calls made before these tests existed (the first paid round on 2026-09-30) have no pass/fail result.

All paid calls are listed in receipts.json with time, amount charged, the settlement transaction, whether a valid response came back, and the known-answer result.

Money

We never hold, split or forward anyone else's funds. Buyers pay sellers directly. The only money we receive is the lookup fee for our own query endpoint ($0.02 USDC via x402 after 5 free lookups per client per UTC day); it buys query access only and never changes results, sort order or listings. The only money we spend is our own, on our own checks and self-tests, under the caps above.

Raw logs

Every check writes a raw log (JSON) with the request, the status, selected headers (the payment challenge decoded), a hash of the body, and the first 4 KB of the body. Response text from services is untrusted: we store it as data, never render it as a web page, and redact email addresses and a few local-path-like strings (the body hash covers the full, unredacted body).

Where the listings come from

Services are taken from public directories of x402 endpoints (currently the public CDP x402 Bazaar discovery API). We record the source and the date we read it. We did not invent any listing. Listing text (names, descriptions) is the seller's own, not verified by us.

Contest a result

If you run a listed service and think a result is wrong, or you want your service removed or added:

1. Open a GitHub issue on our public repository (all issues). Include the service id (from the URL of its page), the check time you disagree with, and what you expected.
2. We re-run the check, publish the new raw log, and reply on the issue with what we found.
3. If our check was wrong (for example, a bug in the checker or a bad sample request), we correct the record and say so on the issue. We do not remove accurate results, but you can always add context on the issue.

Limits

• One request per service per run, from one location. Latency depends on where we check from.
• "Reachable" does not mean the service works. A free check only shows that the endpoint answers and what it asks to be paid; only a paid check shows whether it delivers.
• The sample request comes from the listing. If it is out of date, the endpoint may answer with an error that a real customer would not see.