Scheduled Cron Job Skipped Silently With No Error Logged
A scheduled job never fired and nothing showed up in logs. Fix by going UTC-only, adding heartbeat metrics, and alerting on missed execution counts.
#Backend
Articles tagged with #Backend
Database Migration Stuck on ALTER TABLE in Production
An ALTER TABLE migration hangs forever on prod. Find the blocker in pg_stat_activity, terminate it, and re-run the migration with a lock_timeout.
Docker Container Randomly Restarts With Exit Code 137 (OOM Killed)
Containers restart with exit 137. The OOM killer hit your --memory limit. Find the leak, profile heap, set sensible limits, and stop the bleeding.
gRPC DEADLINE_EXCEEDED Errors Under Load
gRPC clients return DEADLINE_EXCEEDED when traffic rises. Propagate deadlines, set sensible per-RPC timeouts, and add a retry policy plus circuit breaker.
JWT 'Token Expired' Errors on Fresh Tokens (Clock Skew)
JWT verification fails intermittently with 'token expired' even on tokens issued seconds ago. Fix the server clock drift with NTP and add JWT leeway.
MongoDB Aggregation With $lookup and $group Runs for 30 Seconds
MongoDB pipelines with $lookup + $group crawl. Use explain('executionStats'), add compound indexes, push $match early, and split with $facet.
RabbitMQ Consumers Connected But Not Processing — Queue Depth Growing
RabbitMQ shows healthy consumer connections but the queue keeps growing. Fix prefetch, unacked messages, and dead-letter routing for stuck consumers.
S3 Presigned URL Returns 403 Mid-Upload on Large Files
AWS S3 presigned URL works for small files but 403s mid-upload on large ones. Fix with longer TTL, multipart upload, or the SDK upload manager.
Kafka Consumer Lag Keeps Growing Even After Scaling Consumers
You added more consumer pods. Lag is still going up. The bottleneck is almost never "not enough consumers" — it is partition count, poison messages, or commit-offset drift.
Postgres Autovacuum Stalled by a Long-Running Transaction
Table bloat grows, queries get slower, and pg_stat_progress_vacuum shows nothing running. A single forgotten transaction is holding back the entire vacuum horizon.
GraphQL Rate Limit Cascade Failure
One slow resolver triggers rate limiting that cascades to every query sharing the upstream. Fix by adding per-resolver complexity costs, DataLoader batching, and circuit breakers.
Message Queue Dead-Letter Queue Building Up
DLQ growing in SQS / RabbitMQ / Kafka without bound. Fix by classifying failures, fixing root-cause poison messages, and adding retry-with-backoff.
Postgres Connection Pool Exhausted Under Load
Postgres throws 'remaining connection slots reserved' under traffic. Fix by sizing the pool, adding PgBouncer, and killing long-idle connections.
Redis Cluster Failover Stuck Promoting Replica
Master node went down but no replica gets promoted; cluster stays in fail state. Fix by checking quorum, network partitions, and replica priority settings.
Auth Redirect Returns to Wrong URL
Sign-in flow redirects to localhost or the wrong domain. Configure allowed redirect URIs.
CORS Error Calling Your Own API
Browser blocks request with CORS error — server config or wrong origin.
Edge Function Timeout — Vercel / Cloudflare Fix
Edge function exceeds the time limit on Vercel / Cloudflare / Netlify.
Firebase Deploy "Permission Denied"
Firebase deploy fails with permission errors — IAM, login, or project mismatch.
Firebase Function Not Found (2026)
Calling a function returns "not found." Likely region, name, or deploy.
Firebase Hosting Rewrites Not Firing (2026)
You configured rewrites for SPA / functions but they don't trigger.
Works Locally, Fails in Production
Works locally, breaks on Vercel or Render. Diff env vars, runtime version, filesystem case-sensitivity, network, and build process — the systematic checklist.
Hit External API Rate Limit
Stripe, Twilio, SendGrid all return 429 once traffic ramps. Read Retry-After, add exponential back-off, cache GETs, and batch writes — in that order.
Object Storage Upload Denied
S3 / Firebase Storage / Supabase Storage upload 403s — IAM, signed URL, or bucket policy.
Supabase Env Vars Missing in Production
Supabase URL or anon key undefined in prod — host env config or prefix mismatch.
Supabase RLS Policy Blocks Data
Empty results despite data in the table. RLS policy too strict.
Webhook Not Firing — Provider 200, Endpoint Silent
Provider says webhook delivered, your endpoint never sees it.