Evidence — Engineering blueprint
Decision-intelligence architecture
How to turn scattered data and models into decisions people actually trust and act on — with uncertainty, human authority, and the outcome feedback loop designed in from the start, not bolted on. Its tradeoffs, failure modes, and limits included.
Engineering blueprint
A reference architecture, generalized from decision-support patterns. Not a product and not client work.
Purpose
What this architecture is for.
This architecture turns scattered data and models into decisions people actually trust and act on, with uncertainty, human authority, and the outcome feedback loop made explicit parts of the design rather than afterthoughts.
When it is appropriate
- Repeated decisions of the same shape, where outcomes can eventually be observed and measured.
- Situations where a score alone is not enough — someone must be accountable for the consequential call.
- Operations that already collect the data a decision depends on, or can begin to.
Architecture
Components, relationships, and boundaries.
A decision runs as a governed loop, not a one-way score. Each component has a defined role, and the outcome returns to improve the next decision.
Decision loop
Text description of this diagram
The decision runs as a governed loop:
- Inputs — the defined features the decision depends on.
- Model + confidence — an answer with a calibrated confidence, not a bare guess.
- Human authority — a person owns consequential or low-confidence calls.
- Action — the decision is applied.
- Outcome — the result is recorded and fed back to improve the model.
Components
- Inputs and features
- The defined signals a decision depends on, with their source and freshness known.
- Model and rules
- Models produce estimates; explicit rules encode the constraints and policy that must always hold.
- Calibrated confidence
- Every estimate carries a confidence that has been checked against reality, so low-confidence cases can be treated differently.
- Decision policy
- The thresholds and bands that turn an estimate plus its confidence into an action or an escalation — owned by a human, versioned, and auditable.
- Human authority
- A person owns consequential and low-confidence decisions, can override any of them, and sets the thresholds.
- Outcome capture and feedback
- The action and its later outcome are recorded and fed back to monitor quality and improve the model.
Data and authority boundaries
Data and authority are scoped so the decision can be governed and its behavior cannot drift unseen.
- The model that estimates and the decision policy that acts are kept separate, so thresholds can change without retraining and be reviewed on their own.
- Only authorized roles can change thresholds or override decisions, and every such change is recorded.
- Inputs live inside a data boundary with least-privilege access; features used in a decision are logged for later scrutiny.
- Outcomes are captured without leaking personal data into places it should not be.
Human authority
Where a person stays in control.
The human is not a rubber stamp at the end. Authority and the ability to shape the system are designed in.
- A person owns consequential decisions and every low-confidence one.
- People set and revise the decision thresholds; the system does not silently move its own goalposts.
- Overrides are expected, recorded with a reason, and used as a signal that the model or thresholds may need attention.
- The interface surfaces uncertainty and the reasons behind an estimate, so a decision is informed rather than deferred to a number.
Evaluation
How it is measured, and how uncertainty is handled.
A decision system is judged by the quality of its decisions against real outcomes, not by a model metric in isolation.
What is measured
- Decision quality against observed outcomes, once they are known.
- Calibration — does an 80% confidence mean right about 80% of the time?
- Precision and recall at the chosen thresholds, where the operating point actually sits.
- Override rate and the outcomes of overridden decisions.
- The real operational or business measure the decision is meant to move.
Uncertainty
Confidence is calibrated and carried through to the decision policy. An estimate whose confidence falls in an abstention band is escalated to a human rather than actioned automatically. Uncertainty is shown to the decision-maker, not hidden behind a single number.
Observability
Inputs, the estimate and its confidence, the decision, and the eventual outcome are logged so drift and miscalibration are detectable. Monitoring compares live behavior against the evaluation baseline and raises a review when they diverge.
Failure modes
How it breaks, and what holds.
| Failure mode | Mitigation |
|---|---|
| Miscalibrated confidence | Measure calibration explicitly, recalibrate, and monitor it in operation; treat overconfidence as a defect, not a detail. |
| Feedback leakage or label contamination | Hold out data honestly, wait for real delayed outcomes, and keep training separate from the live decision path. |
| Distribution shift | Watch input and outcome distributions, alarm on drift, and re-evaluate on a schedule rather than assuming yesterday's model still holds. |
| Automation bias | Surface uncertainty and rationale, require a reason for overrides, and review decisions where humans always agree with the model. |
| Optimizing a proxy instead of the real outcome | Tie evaluation to the actual operational outcome, and revisit the metric when the proxy and the outcome diverge. |
Rejected alternatives
What was considered, and why it was not chosen.
-
Fully automate the consequential decision
Considered It removes the human bottleneck and is consistent.
Rejected No one is accountable for a wrong consequential call, and there is no authority to stop it. Automation is reserved for low-consequence, high-confidence cases.
-
A black-box score with no confidence
Considered It is simple to expose and integrate.
Rejected Without calibrated confidence, the decision policy cannot treat uncertain cases differently, and the system cannot be governed. Confidence is made first-class instead.
-
A dashboard where humans do everything
Considered It keeps humans fully in control.
Rejected It does not scale and produces inconsistent decisions across people. The model handles the routine and surfaces the uncertain, keeping humans for judgment.
-
A pure rules engine
Considered It is transparent and predictable.
Rejected It is brittle where patterns are complex and cannot learn from outcomes. Where rules genuinely suffice, though, they are the right answer (see below).
Deployment & cost
How it is deployed, and what drives its cost.
- Batch scoring
- Decisions computed on a schedule when the operation does not need an instant answer — simpler and cheaper to run.
- Real-time scoring
- Decisions served on demand when the operation needs an answer in the moment, at higher operational cost.
- Shadow mode first
- The system runs alongside the current process, its decisions recorded but not acted on, until evaluation shows it is safe to give it authority.
Operating cost
Cost is driven by model development and validation, the monitoring and feedback infrastructure, and continued human review — not by scoring alone. It is described qualitatively; a real estimate depends on decision volume, latency needs, and how much human review the consequence demands.
When not to use this architecture
The honest limits.
- When a simple rule or a clear report already produces the right decision — add no model.
- When outcomes can never be observed, so the decision can never be evaluated or improved.
- When the decision is rare or one-off; the investment in evaluation and monitoring will not pay back.
- When the available data is too sparse or too biased to support a trustworthy estimate — fix the data first.
What this does and does not prove
It shows
- That uncertainty and human authority can be made first-class parts of a decision architecture.
- That the outcome feedback loop belongs in the architecture, not in a later phase.
It does not show
- That any model achieves a given decision accuracy.
- That automation is warranted for a specific decision.
How to inspect it
Read the component and authority diagram, the uncertainty and feedback treatment, and the 'when a simpler answer is better' section.
Engagement
Begin with what must change.
If decisions are slow or uncertain because data and judgment live apart, a mandate begins by making the decision, its uncertainty, and its authority explicit.
Or submit an RFP, or request an NDA first.