Provability
EU AI Act for in-house software: the duties that hit your architecture.
The risk classes are quick to read. It gets interesting where a duty turns into a requirement on how the system is built: automatic logging, traceable data lineage, explainable decisions. This page covers exactly that part.
Classification is paperwork. Logging duties are architecture.
Most companies we talk to have already classified their AI use cases. What follows is the actual work: a high-risk system must automatically log what it did over its lifetime, and those logs must be retained. Not as debug output, but as evidence.
Then there is the data side. If a model operates on your business data, it must be traceable which data in which state went in. Systems that only store current state cannot do this: they no longer know what a record looked like at the moment of the decision.
And finally oversight: a human control instance needs something to control. Without a recorded decision context, “human oversight” stays a line in a concept document.
From duty to architecture requirement.
An extract of the duties for high-risk systems and for anyone embedding AI in their own software.
| AI Act duty | What the system must do | What you produce |
|---|---|---|
| Automatic logging across the lifecycle (Art. 12) | Every model run produces an entry with input reference, model version, result and timestamp — immutable. | The full trail for a single case, retrievable months later without developer help. |
| Retention of logs (Art. 19) | Logs live separately from mutable application state and survive migrations. | Demonstrable retention over the legally required period. |
| Data governance (Art. 10) | Data lineage and data state are reconstructable — which record applied at decision time. | The state of the input data on that date, not today's row in the same table. |
| Technical documentation (Art. 11, Annex IV) | Model versions, prompts, parameters and data sources are versioned and tied to a release. | Which version made which decision — proven, not remembered. |
| Transparency towards affected people (Art. 13, Art. 50) | The factors driving a decision are stored together with the outcome. | An understandable rationale that matches the stored history. |
| Human oversight (Art. 14) | Interventions, approvals and overrides by humans are themselves events in the system. | Proof that oversight happened — not just that it was foreseen. |
Implementation view, not legal advice. Which duties apply depends on your role (provider, deployer) and risk class; application dates are staggered.
What AI-driven development adds.
Not just the model in the product. Also the way software is now built.
Context is a production artefact
Prompt, system context and model version shape the outcome as much as code. Without versioning them, a decision from last quarter cannot be reproduced.
Provenance of changes
When a large share of code is machine-generated, “who decided this” becomes “what triggered this change and who approved it”. That belongs in the chain of evidence.
Faster delivery without a chain of evidence
More releases per week are only safe if every change carries its provenance. Otherwise the reconstruction gap grows faster than productivity.
Fewer handovers, fewer side effects
Small teams without role handovers produce less incidental documentation. What used to appear as a by-product must now be produced by the system.
How we build it.
Provability is cheaper when it is part of the first design.
1 — Name the decision points
We identify where a model decides or pre-decides something that affects a person or a contract. Only those need the full chain of evidence.
2 — Decision as an event
Outcome, input reference, model version and rationale are recorded as one immutable fact — not as a field the next run overwrites.
3 — Make oversight visible
Human approval, correction and rejection get recorded. That turns a process rule into evidence.
4 — Readable for the business and for auditors
One view that answers what happened in a specific case without SQL. Without it, every request becomes a ticket.
When this fits — and when it does not.
Fits when
- AI decides or pre-decides in a process affecting people or contracts
- a customer or auditor asks for explainability and logs
- an AI pilot should go live and gets stuck on the evidence question
- a new system is being built with AI designed in from the start
Does not fit when
- you need a conformity assessment or certification
- you are looking for legal advice on risk classification
- the question is pure model quality — better specialists exist for that
- AI is only a keyword in the project proposal
Frequently asked
- Does the AI Act apply if we only call a third-party model via API?
- Possibly, as a deployer. The duties differ from a provider's, but logging and human oversight often still land with you — because your software supplies the context, not the model.
- Is storing prompts and responses enough?
- As a start, yes. It holds up once the data state at decision time and the model version are part of it. Otherwise the case cannot be reproduced later.
- Isn't this just good logging?
- Good logging gets you halfway. The difference is immutability, and recording business decisions as decisions — not as a by-product of a technical call.
- What if we have no AI in the product yet?
- Then this is the cheapest moment. These requirements are the same ones an audit asks for without AI. Retrofitting is the expensive path.
Sources
All article references point to the published regulation text.
Before the pilot goes live: can you prove it?
The provability check runs ten questions to see whether your system answers what comes next.