There is a persistent illusion in data engineering that the pipeline is infrastructure, a plumbing problem, separate from the business questions it serves.
This is wrong in a way that produces expensive consequences.
Every architectural decision in a data pipeline what to collect, at what granularity, how to transform it, what to discard, how to surface it, is a decision about which questions can be asked and which cannot. It is a decision about which patterns become visible and which remain hidden. It is, whether the engineers who made it intended this or not, a decision architecture.
The Supply Chain Case
When I was Head of IT at Lithe Group, we had a demand forecasting problem. Order delays were endemic a 15% reduction target that the business had been trying to hit for years.
The existing data pipeline told us what had shipped, when, and to whom. It did not tell us why orders were delayed. That information existed in emails, in production logs, in supplier communication records but it was not in the pipeline because it had not been designed to capture it.
The question the business needed to answer was causal: what conditions predict a delay before it happens? The pipeline had been designed to answer a different question: what happened?
This is not an unusual situation. Most enterprise data pipelines are designed to answer retrospective questions because those are the questions that were articulated at the time the pipeline was built. The prospective questions the ones that would let you intervene before a problem occurs require different data, at different granularities, with different timeliness requirements.
Redesigning the pipeline to capture delay signals changed the ML model's performance dramatically. Not because we found a better algorithm, but because we finally gave the algorithm the information it needed to reason about the problem correctly.
What Gets Discarded Shapes What Gets Decided
The most consequential decisions in pipeline design are often the ones about what to discard.
Storage is cheap but not free. Compute on high-volume streams is expensive. Every pipeline makes choices about what to aggregate, what to sample, what to discard entirely. These choices are usually made on technical grounds cardinality, storage cost, query performance without full consideration of what analytical questions they foreclose.
A common example: event timestamps are often normalized to the nearest second or minute to reduce cardinality. This is technically reasonable. It also makes it impossible to analyze sub-second behavioral patterns which, in some domains, are exactly the signal you want.
The discarded information is invisible by design. You don't know what questions you can't answer. You only know what questions you can.
Designing for Cognition
The alternative is to design pipelines not for storage optimization, but for cognitive reach for the broadest possible set of questions the data could support.
In practice, this means:
Preserve raw events as long as economically feasible. Aggregation is irreversible. You can always aggregate stored raw data later. You cannot reconstruct raw events from aggregates.
Make the transformation logic explicit and auditable. Every transformation that converts raw data into analytical data makes assumptions. Those assumptions should be documented, versioned, and visible to the analysts who use the downstream data. Hidden assumptions in transformation logic are a source of systematic analytical error that is extremely difficult to detect.
Design for the questions you don't have yet. The most valuable data is often data whose value you don't understand until years after you collected it. The discipline is to collect more than you currently need, at higher fidelity than current use cases require, and to design the infrastructure to make that data queryable when the question eventually arrives.
The MLOps Dimension
This becomes critical in MLOps contexts. A model is only as good as the features it was trained on, and features are only as good as the pipeline that produced them.
Feature drift changes in the statistical properties of pipeline outputs over time is one of the most common sources of model degradation in production. It often has nothing to do with the model. The pipeline changed. An upstream schema was modified. A data source was deprecated and replaced with a slightly different one. The transformation logic was updated to fix a bug, which also changed the distribution of values.
If your pipeline does not have explicit versioning, schema contracts, and observable data quality metrics, you will not know when your features have drifted until your model's performance has already degraded and you will spend weeks figuring out why.
The pipeline is not infrastructure. It is the cognitive substrate of every intelligent system built on top of it. Design it accordingly.