Reconciling a Drifted Production Lambda Without Exposing Care Data

Context

ParkinSync is a small, real-world care-analytics system: it OCRs a paper caregiver log, enriches it with weather, and writes a normalized daily schema. The OCR step runs as an AWS Lambda in production, processing a family member's real care records daily.

The handoff notes carried a nagging risk: "production Lambda code may not match main." Until that was resolved, the repository's tests couldn't be trusted to describe production behavior — a dangerous gap for a system on a health-safety path.

Constraints

• Real production, real care data. Read-only only; no invocations, no writes, nothing that could touch or expose care records.
• Health-safety labeled. A wrong redeploy could regress a system people depend on.
• Public repository. No account IDs, environment values, secrets, or health details could appear in evidence.
• Solo, part-time. No team to review or pair; governance had to be built into the process itself.

What I did — and deliberately didn't do

I did not "just redeploy main," the tempting one-line fix. I first proved what production actually ran, then chose remediation from evidence.

The operational story (the core)

Using read-only AWS APIs (get-function / get-function-configuration), I exported the deployed function's code only — never its environment variables (which can hold secrets) — and hashed it. The result was decisive:

• The deployed handler was byte-identical (SHA-256) to an unrelated-history branch, not to main.
• main was a simpler, older implementation. Production had been quietly running the isolated branch's hardened code for months.

The divergence went both ways: the deployed branch had operational hardening main lacked (idempotent S3 processing, OCR-failure quarantine + notification, filename-based date recovery, broader date parsing), while main had a richer 25-column output schema and safer URL-decoding of S3 keys the deployed code had dropped. Neither side was simply "newer."

Decision & tradeoff

Redeploying main would have regressed production by removing idempotency and failure-quarantine from a health-safety path. Adopting the deployed code wholesale would have lost the canonical output schema. I recommended and executed a third option: port the hardened capabilities into main by hand, keeping main's better schema and decoding — reconciling the two rather than picking a loser.

Architecture & technical decisions

• A forensics-first approach: export → hash → structural diff of the handler contract → capability-by-capability comparison. Cheap, safe, repeatable.
• The ported hardening reflects real serverless-ops concerns: idempotency via S3 object tags, failure quarantine (copy the bad object to a review/ prefix + SNS notify, non-fatal), and fallback date parsing. On unexpected errors the handler quarantines and re-raises so Lambda can retry / route to a DLQ, with the idempotency tag preventing double-processing.
• A guardrail in deploy.sh refuses to deploy until main is reconciled, so the exact regression I'd diagnosed cannot recur by accident.

Delivery & governance

With no team, I made the process the reviewer: framed the work as GitHub issues #26/#27 with acceptance criteria and a release gate; shipped via reviewed PRs #30 and #31, each green on CI; wrote a sanitised reconciliation doc capturing hashes, the decision, a non-sensitive smoke test, and rollback evidence — with placeholders instead of real identifiers.

Outcomes

• Production behavior is now provably characterized (hash-pinned) and main reconciled to match/supersede it, verified by 29 unit tests (up from ~5).
• The rollback path is explicit: the exact production artifact is recoverable because it's byte-identical to a committed branch.
• The regression that caused all this is now structurally blocked by the deploy guardrail.

Honest limitations

• Production observability is thin — I could confirm what runs, but there's little telemetry on how well. Minimal operational metrics/alerting is a real next step.
• The smoke test was offline (syntax/compile) — I deliberately did not invoke the live function, so end-to-end verification still depends on a supervised manual run.
• Preventing future out-of-band deploys ultimately needs a proper CD pipeline, not just a script guard.

Lessons

• "main matches production" is a claim that needs evidence. A read-only export + hash turned a months-old hunch into a fact in an afternoon.
• On a health-safety path, reconcile, don't overwrite. The fastest fix was the wrong one; the evidence changed the decision.
• When you're solo, encode governance into artifacts so the process reviews you.

Evidence

Repo: github.com/larai-w/ParkinSync — issues #26, #27; PRs #30, #31. Docs: PRODUCTION_LAMBDA_RECONCILIATION.md, OCR_HARDENING_PORT.md, the deploy.sh guardrail, and tests/test_lambda_function.py (29 tests).