Evidence — Filled artifact
Threat, evaluation, and acceptance specification
For the Governed Case Operations System: the assets and threats and how they are mitigated, the datasets and metrics it is measured against, and the criteria that must hold before it earns operating authority.
Filled artifact
A filled artifact from the reference system. Not a client document.Thresholds are illustrative for the synthetic scenario, and no measured evaluation results are reported.
Threat model
Assets, threats, and what mitigates them.
Built with a STRIDE-style analysis over the system's assets. Each threat names the mitigation already present in the architecture — the threat model and the design are one artifact, not two.
| Asset | Threat | Category | Mitigation |
|---|---|---|---|
| Case intake | Malformed or adversarial input crafted to break parsing or downstream logic | Tampering | Server-side schema validation, normalization, and length limits before a case enters the workflow. |
| Model gateway | Prompt injection hidden inside a retrieved document | Elevation of privilege | All content is treated as data, never instructions; tools run only from the orchestrator; the action registry is allow-listed. |
| Case data | Exfiltration of sensitive data through the model path | Information disclosure | Context minimization at the gateway, a record of exactly what was sent, and egress limited to allow-listed destinations. |
| Action path | Unauthorized or excessive action on a system of record | Elevation of privilege | Actions only via integration adapters with scoped credentials; consequential actions require approval at the gate. |
| Identity | Spoofed actor or broken access control | Spoofing | Authenticated identities, role-based permissions, and least privilege for every human and service. |
| Audit log | Tampering with the record to hide an action | Tampering / Repudiation | Append-only log with integrity protection, written before an action completes, retained independently. |
| Model gateway | Denial of service or runaway cost against the model | Denial of service | Rate limits, timeouts, and graceful degradation to a manual queue when the model is unavailable. |
Evaluation plan
How it is measured.
Datasets
- A labeled set of synthetic cases spanning the routine distribution.
- An adversarial document set (injection attempts, contradictory sources, unreadable files).
- An edge-case set (missing fields, ambiguous policy, out-of-scope requests).
Metrics and thresholds
| Metric | Method | Threshold |
|---|---|---|
| Extraction accuracy | Compare extracted fields against labels on the synthetic set. | Set per field before release |
| Groundedness | Every model claim must map to a retrieved citation; measured by automatic check plus human audit sample. | No unsupported claim may pass the gate |
| Policy-check precision and recall | Compare flags against labeled policy outcomes. | Recall prioritized on consequential rules |
| Escalation correctness | Confirm consequential and low-confidence cases reach a human. | No consequential case auto-actioned |
| Human-override rate | Track operator corrections in shadow mode and operation. | Monitored for drift, not a fixed pass/fail |
| Latency and cost per case | Measure end-to-end on the synthetic set at representative volume. | Set against the operational budget |
Method
Offline evaluation on held-out sets, then a shadow-mode run alongside the current process where the system's outputs are recorded but not actioned, with a sampled human review. Only after the acceptance criteria are met does the system receive operating authority.
Acceptance
What must be true before it ships.
Each criterion is marked by its basis. “By design” holds by construction in this reference implementation; “pending measurement” and “pending review” are the honest gaps — they are not claimed as met.
- By design Every consequential action passes a human approval gate before it leaves the boundary.
- By design Every model claim in an accepted case is supported by a retrieved citation.
- By design The audit log records every actor, action, and citation, append-only.
- By design Actions reach systems of record only through integration adapters under scoped credentials.
- Pending measurement Extraction accuracy and policy-check recall meet their per-metric thresholds on the held-out set.
- Pending measurement Groundedness holds under the adversarial document set, including injection attempts.
- Pending measurement Latency and cost per case are within the operational budget at representative volume.
- Pending review An independent senior technical review and a buyer-comprehension review are completed.
What this does and does not prove
It shows
- That threat modeling, evaluation, and acceptance are defined before a system is called done.
- That SageTensor states acceptance criteria explicitly rather than declaring success.
It does not show
- That the reference system has passed its own acceptance criteria under measurement.
How to inspect it
Read the threat table, the evaluation plan, and the acceptance checklist. Each ties back to the flagship control and evaluation views.
Engagement
Begin with what must change.
A mandate produces this specification for your operation — the threats, the measures, and the bar it must clear.
Or submit an RFP, or request an NDA first.