A pipeline job finishes without an error. The new table has 18% fewer rows.
Should you rerun the pipeline, stop the release, or trust the new output?
Before rerunning the job, preserve the two states you need to compare. A rerun
can create another state and make the original change harder to explain. The
first useful question is: which two states are we actually comparing?
Pin down the before and after
“Yesterday” and “latest” sound precise until a file is replaced or a table
advances to another snapshot.
Give both states durable identities. That might mean a file checksum, an object
version, a Delta table version, an Iceberg snapshot, a catalog reference, or a
pipeline run ID. Record the local time and timezone too. They help when logs and
object histories use different clocks.
Suppose a Delta table’s row-count drop occurred between versions 842 and 849.
Those two numbers turn a vague report into a reproducible question. Without
them, another engineer may inspect version 850 and reach a different conclusion.
Test the smallest useful explanation
An unexpected nullable column and a lower row count point to different first
checks.
If a column became nullable, compare the two schemas before scanning rows.
Check whether the field changed type, moved inside a nested structure, or was
renamed. A CSV inference change and a Parquet schema change may look similar in
a grid, but they are different events.
If the row count changed, group counts by the partition or date most likely to
explain it. A full profile of every column may cost more than the original query
while hiding the useful difference.
Delta and Iceberg add another question: which logical table state did the
reader resolve? The newest objects in storage do not necessarily belong to the
selected state. Check the version or snapshot, the metadata chain, and the data
or delete files involved before blaming an individual file.
The aim is a smaller search area. A larger report can wait until the evidence
calls for one.
State what the check can prove
The same finding means something different when it comes from metadata, a
sample, or every relevant row.
Metadata can establish recorded schema, partition information, and available
statistics. It cannot prove a claim about every row. A sample helps discover a
pattern, but a missing value in the sample does not prove that the value is
absent. A full scan supports stronger row-level claims, provided it covered
every relevant partition and delete.
Record truncation, skipped objects, unsupported semantics, cancellation, and
fallback behavior. If one of them prevents a conclusion, call the result
incomplete. That is more useful than a clean-looking result with a hidden gap.
Leave evidence another engineer can challenge
A short record is enough when another engineer can understand the decision and
repeat the check.
A fictional record might look like this:
Field Recorded value Question Why did the table lose 18% of its rows? Inputs Delta table versions 842 and 849. Checks Schema comparison and row counts grouped by day. Scope Metadata plus a full scan of the affected partition. Outcome Warning. The drop is isolated to July 25, 2026. Limit Delete semantics were not checked, so the cause remains unproven.Keep credentials, signed URLs, private paths, secret-bearing SQL, customer data,
and private screenshots out of the record.
Before running the checks, define the finish line:
- Pass means the stated rule held over the recorded scope.
- Warning means the result needs review.
- Failure means the evidence disproved the rule.
- Incomplete means the available evidence cannot support a conclusion.
Leave enough evidence for another engineer to reproduce the result, disagree
with it, and run the investigation again.
One question for data engineers
Which part of this work usually costs you the most time?
- Understanding an unexpected schema change
- Finding why row counts or profiles changed
- Understanding the real state of a Delta or Iceberg table
- Comparing pipeline output before and after a change
- Proving that a delivery or release is safe
- Something else
A one-number comment is enough. Please do not post confidential data, file
paths, SQL, schemas, credentials, customer information, or screenshots.
If you prefer email, the same one-question prompt is available on the
BigdataSight research page.
Disclosure: I build BigdataSight, a native Mac data workstation. We are
researching evidence-backed tools for investigating changes in files and
lakehouse tables. The product direction described here remains research and is
not a shipping capability.
답글 남기기