Documentation → Data freshness and jobs

Documentation

Data freshness and jobs

Freshness per data pool, coverage, throughput and control over the background runs. The /health and /status endpoints need no key.

What it is good for is on the product page: Data freshness and jobs

GET/coverageGET/healthGET/health/ingestGET/jobsGET/jobs/lanesPOST/jobs/reapGET/jobs/{job_id}POST/jobs/{job_id}/cancelGET/stats/throughputGET/statusGET/status/datasetsPOST/sync/all

GET /coverage

Coverage All

Inventory across all statement pools plus the backlog per lane.

Per pool: documents, companies and the period they cover. Per lane, additionally, how much is still outstanding. Meant as the one call that lets you spot a gap in the inventory before it turns up in an analysis.

No parameters.

Example

curl "$API/coverage" \
  -H "X-API-Key: $KEY"

GET /healthno key

Health

No parameters.

Example

curl "$API/health"

GET /health/ingestno key

Health Ingest

Freshness overview of the statement pipelines: the last run per source (from ingest_runs, written by refresh.py) plus a coarse staleness signal. For us it additionally reports the DERA quarterly lag — a missing dataset does not turn the run red, but it does leave reports missing. Needs no key (it sits under /health). Usable for monitoring and alerting.

No parameters.

Example

curl "$API/health/ingest"

{
  "lanes": {
    "us":  { "lastRun": "2026-09-01T02:14:00Z", "stale": false },
    "eu":  { "lastRun": "2026-09-02T03:40:00Z", "stale": false },
    "de":  { "lastRun": null, "stale": true, "reason": "never ran" }
  }
}

GET /jobs

Jobs List

Bulk jobs, newest first.

Name	Type	Meaning
`limit`	integer, in query optional · default `20`	—
`lane`	string, in query optional	Lane filtern, z. B. earnings.dates.repair
`status`	string, in query optional	Status filtern: pending, running, done, failed, cancelled, orphaned
`type`	string, in query optional	Job-Typ filtern

Example

curl "$API/jobs" \
  -H "X-API-Key: $KEY"

GET /jobs/lanes

Jobs Lanes

Per lane, the most recent job plus how many are currently running.

This is the polling endpoint for chunk-and-loop operation: it answers "is something running" and "how far along is it" in a single call.

No parameters.

Example

curl "$API/jobs/lanes" \
  -H "X-API-Key: $KEY"

{
  "lanes": [
    { "lane": "statements.esef", "running": 1, "backlog": 412 },
    { "lane": "earnings.dates.repair", "running": 0, "backlog": 0 }
  ]
}

POST /jobs/reap

Jobs Reap

Marks orphaned running jobs as orphaned.

A job counts as orphaned when no living process owns it any more — either it carries the identity of a different process, or its heartbeat is older than max_heartbeat_age_s.

This runs at startup and periodically thereafter on its own; the endpoint exists for when you do not want to wait for the next pass.

Name	Type	Meaning
`max_heartbeat_age_s`	integer, in query optional · default `180`	—

Example

curl -X POST "$API/jobs/reap" \
  -H "X-API-Key: $KEY"

GET /jobs/{job_id}

Jobs Get

Name	Type	Meaning
`job_id`	string, in path required	—

Example

curl "$API/jobs/job_7f3c" \
  -H "X-API-Key: $KEY"

POST /jobs/{job_id}/cancel

Jobs Cancel

Request cancellation.

The worker pool polls cancel_requested every 5 s; items already in flight are finished. The job then ends with status: "cancelled".

Name	Type	Meaning
`job_id`	string, in path required	—

Example

curl -X POST "$API/jobs/job_7f3c/cancel" \
  -H "X-API-Key: $KEY"

GET /stats/throughput

Stats Throughput

Throughput per data pool: total and daily average within the chosen window.

Answers "how much does the system get through per day?" — as opposed to /coverage (inventory) and /health/ingest (freshness). Three averages are given per pool: per calendar day, per business day (the meaningful figure, because nothing is published or filed at weekends) and per day with activity.

dateKind states what is being counted: business = events in the market (inflow), ingest = records processed by us (pipeline output).

Name	Type	Meaning
`days`	integer, in query optional · default `30`	Fenster in Tagen (rueckwaerts von heute)
`series`	boolean, in query optional · default `False`	Tageswerte mitliefern (fuer Charts)
`pots`	string, in query optional	Komma-getrennte Auswahl, z.B. directors_dealings,earnings_dates

Example

curl "$API/stats/throughput" \
  -H "X-API-Key: $KEY"

GET /statusno key

Status Overview

Data state per pool plus running background jobs — one call.

Per pool: documents, companies, the time of the last update (together with where that timestamp came from), a freshness signal, the backlog and the associated job. text is the readable short form, e.g. "29,363 documents from 29,363 companies — updated 11 Aug 2026 12:10 UTC (3 hours ago)".

The default is built for polling (metadata counts, indexed timestamps). ?deep=true returns exact figures and the expensive timestamps — that then takes seconds to minutes.

Name	Type	Meaning
`deep`	boolean, in query optional · default `False`	Exakte Zaehlungen und die teuren, direkt aus den Daten gelesenen Zeitstempel (Voll-Scans, Sekunden bis Minuten)
`group`	string, in query optional	Nur eine Gruppe: earnings, financials, statements, market, derived, reference
`dataset`	string, in query optional	Komma-separierte Keys, z. B. earnings.dates,prices
`recent_jobs`	integer, in query optional · default `8`	Wie viele beendete Jobs

Example

curl "$API/status"

GET /status/datasetsno key

Status Datasets

Which pools exist and where their figures come from — without touching the database.

Useful for building ?dataset= without paying for the full query.

No parameters.

Example

curl "$API/status/datasets"

POST /sync/all

Sync All

Trigger every lane — the single button.

Does not block and waits for no lane: each continues as its own background job. If a lane is already running, or nothing is due, it is skipped rather than failing (409 and 422 are not errors here but the normal answer).

Name	Type	Meaning
`lanes`	string, in query optional	Komma-separiert; leer = alle
`dry_run`	boolean, in query optional · default `False`	nur die Rueckstaende melden

Example

curl -X POST "$API/sync/all" \
  -H "X-API-Key: $KEY"