Skip to content

Keep long-running tasks from going stale - #58

Merged
snopoke merged 10 commits into
mainfrom
sk/keep-alive
Jul 30, 2026
Merged

Keep long-running tasks from going stale#58
snopoke merged 10 commits into
mainfrom
sk/keep-alive

Conversation

@snopoke

@snopoke snopoke commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Tasks with a stale_timeout go stale unless something updates them, so today a long-running Celery or Procrastinate task has to ping itself from the task body. heartbeat_interval makes the worker do it — settable on the system integration, on the task, or per call.

Design decisions the diff doesn't show:

  • One daemon thread per worker process pings every in-flight task, rather than a thread per task. It's started lazily at task_prerun, so publishers never get one, and a forked prefork child builds its own (an inherited thread reports is_alive() == False).
  • The interval is resolved wherever the task is created — publish handler, or _maybe_create_task for eager/canvas tasks — so it always matches the stale_timeout it's protecting, instead of a worker inventing an interval for a timeout that was never set.
  • stale_timeout defaults to 2 × interval, the same ratio taskbadger run already uses.

Worth a closer look:

  • exit_session ownership (a23ada4). f8ea1cf fixed a session leak for eager tasks but made them close a session they never opened — an eager task inside a caller's with taskbadger.Session(): popped the caller's frame, so the caller's __exit__ raised IndexError. The handlers now mark the request when they open the session. Those two commits are worth squashing if you'd rather the series bisect clean.
  • _taskbadger_system timing (4f4496c). It's now set before the idempotency return, so a @tracked task registered before ProcrastinateSystemIntegration(...) inherits the system's options. That also restores the record_task_args inheritance track()'s docstring already promises — a behaviour change to existing functionality, deliberate.

Both integration tests were verified to fail with Heartbeat.start stubbed to a no-op, so they aren't vacuous passes. They ran against the real API with local Redis and Postgres (7 passed).

Unrelated, noticed while testing: the Celery suite uses Redis DB 0, so messages from any other project on the same box can make the worker thread snapshot an unconfigured Badger before _bind_settings runs and fail a handful of pre-existing tests. Pinning celery_config to a dedicated DB would fix it; left alone here.

snopoke and others added 10 commits July 30, 2026 12:25
Eager tasks carry the task ID on the request, not in the message headers, so
`exit_session` bailed out and leaked the client.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A single daemon thread pings all in-flight tasks so that tasks with a
`stale_timeout` don't go stale while they are still working. Adds the
`heartbeat_interval` and `stale_timeout` options to the system integrations.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Starts on `task_prerun` and stops on `task_postrun`. The interval is resolved
where the task is created so that it matches the task's `stale_timeout`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covers sync and async tasks; `track` accepts `heartbeat_interval` and
`stale_timeout`.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Checks that a long running task's `updated` time advances while it runs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`run_worker` blocks the test, so the task samples its own `updated` time
either side of the sleep.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Only close the session that the signal handlers opened; an eager task inside a
caller's `Session()` was closing it out from under them. Also read per-call
heartbeat options back out of the headers, which is where they stay when
`before_task_publish` doesn't run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A `@track`ed task kept `_taskbadger_system = None` when the integration was
constructed after it was registered, so it never picked up the system's
options.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Keep the beat loop alive across unexpected errors, and round the derived
stale timeout up so a sub-second interval can't produce 0.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@snopoke
snopoke marked this pull request as ready for review July 30, 2026 14:47
@snopoke
snopoke merged commit 4de064c into main Jul 30, 2026
15 of 18 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant