Every BES reporting team I have worked with has the same story. The EGM rejection lands on a Monday morning. Someone opens Splunk or ELK, greps for the participant ID, finds a stack trace from three days ago, and then spends the next fourteen hours reconstructing what the pipeline believed to be true at each stage. By the time the correction is submitted, the compliance officer has already been on three calls with the regulator and the operations lead has aged a year.
The uncomfortable truth is that this is not a tooling problem. Teams have Grafana, they have Prometheus, they have log aggregation, they have alerting on DAG failures. What they do not have is a legally defensible record of what the pipeline knew, and when it knew it, at every temporal boundary the regulator cares about.
Logging Is Not Observability, and Observability Is Not Auditability
When a GEV file is rejected because a participant's contribution period overlaps incorrectly with a prior employer's exit date, the question is never "did the pipeline fail?" It did not fail. It produced output. The output was wrong. The question is: at which stage did the state of that participant first diverge from what the regulator would consider correct, and what upstream input caused it?
Standard observability stacks answer three questions well:
- Did a job run?
- Did it throw an exception?
- How long did it take?
BES, HAYMER, FATCA, CRS, and state contribution flows require answers to a different set:
- What was the exact record state at the end of each pipeline stage?
- Which stage first introduced the divergence from the source of truth?
- Can I reproduce, six months later, the exact set of inputs and transformations that produced a specific field in a specific submission?
- Can I prove to an auditor that between the ingestion timestamp and the submission timestamp, no unlogged mutation occurred?
Most teams cannot answer any of these without a forensic reconstruction exercise.
The Temporal Boundary Problem
Pension and tax reporting pipelines are defined by temporal boundaries the business does not control. FATCA and CRS have annual cutoffs with specific balance-as-of dates. BES has monthly contribution windows. HAYMER has its own reconciliation cadence. State contribution flows run on payroll cycles that do not align with any of the above.
The pipeline crosses these boundaries continuously. A participant record ingested on the 3rd of the month, transformed on the 5th, enriched with employer data on the 10th, and included in a submission on the 15th has passed through at least four temporal contexts. If the submission is rejected, the question "what was this record's state on the 5th?" is almost never answerable from logs alone, because logs capture events, not state.
Event sourcing solves part of this. Bitemporal tables solve more of it. But most BES pipelines I have seen do neither. They mutate a working table, log the mutation, and move on. When the regulator asks why a field changed, the answer is a git blame on the transformation code and a hopeful reading of the log timestamps.
What Actually Needs to Be Captured
After running these flows in parallel across multiple insurers, the same minimum viable audit surface keeps appearing:
- Immutable stage snapshots: at the end of every pipeline stage, the full record state is written to append-only storage with the stage identifier, the input hash, the transformation version, and the wall-clock timestamp. Not a diff. The full state.
- Input lineage per field: for every field in the final submission, a resolvable chain back to the source system, the extraction timestamp, and the specific transformation that produced its final value.
- Divergence detection at boundaries: automated comparison of record state at each temporal boundary against the previous boundary, with structured alerts when a field changes without an authorized upstream event.
- Regulator-facing replay: the ability to regenerate any historical submission byte-for-byte from stored inputs and pinned transformation versions. If you cannot do this, you cannot defend the submission.
None of this is exotic. All of it is expensive to retrofit and cheap to design in from the start. Almost no one designs it in from the start.
Why Alerting Is the Wrong Primitive
Alerting assumes you know what a bad state looks like. In pension reporting, the bad states are defined by the regulator, and the definitions change. A field that was valid last quarter may be invalid this quarter because a communiqué changed the interpretation of a code. If your observability strategy is "alert on known-bad conditions," you will always be one regulatory update behind.
The defensible position is the opposite: capture everything, structure it so any question can be asked retroactively, and treat alerts as a convenience layer on top of an already-complete audit surface. When the compliance officer asks "how many participants had this field set incorrectly between January and March," the answer should be a query, not a project.
The Practical Path Forward
For teams already running production BES pipelines, the retrofit sequence that has worked:
- Add append-only stage snapshots first. Even without lineage, having full state at each stage boundary eliminates most forensic work.
- Version-pin every transformation. If you cannot answer "which version of the enrichment logic ran on this record," you cannot defend the output.
- Store raw inputs indefinitely, or at least beyond the longest regulatory statute of limitations. Compression is cheap. Regulatory disputes are not.
- Build the replay capability before you need it. The first time you need it will be under time pressure, and you will not build it well.
The teams that treat observability as a compliance surface rather than an operations dashboard are the ones who do not spend Monday mornings reconstructing what their pipeline believed on Friday. Everyone else is one rejection away from a very long week.