Skip to main content
← All case studies
SaaSClapwork Platform Reliability

Closing the Silent-Failure Gap in Background Workers

A pre-launch audit found four background-task surfaces that silently swallowed failures. This study walks through how we restored Celery beat coverage, added per-entity dedup, and made email-preferences actually load-bearing.

35

New Tests

4

Tasks Restored

3

Incident Classes Closed

The trap

Celery tasks that catch broad Exception and log-and-continue feel safe — until the task is a notification dispatcher and 12% of your messages never leave the queue. Each catch hid one failure mode: bad enum values, missing model fields, expired task references in django_celery_beat.PeriodicTask rows that survived a settings-file removal.

The fixes (all merged to main)

1. Restored four Celery beat schedules the team had pruned from settings.py but forgot to drop the orphan PeriodicTask rows.

2. Per-entity dedup keys on notification dispatch — same (user_id, type, related_object_id) cannot fire twice in 60s.

3. *Wired `NotificationPreference.quiet_hours_`** into the service layer so the placebo setting actually silences email + WebSocket pushes.

4. Audit script that walks every tasks.py and flags any except Exception with no re-raise or escalation.

Outcomes

  • 24 new tests specifically around quiet-hours enforcement.
  • 11 priority-mapping tests covering the previously placebo ?priority= query parameter.
  • Zero regressions: full pytest suite still green (6994 passed, 11 skipped).

Curious how Clapwork could fit your team?