Explainability vs Interpretability in AI
Last reviewed: · By Victor Humenhuk (AIGP certified)
In the most common framing, interpretability is about the model, meaning how far a human can follow the internal mechanics that turn an input into an output, while explainability is about the output, meaning whether a specific decision can be described in terms the affected person can act on. Interpretability is usually intrinsic, achieved by choosing a simple model, whereas explainability is usually added afterwards using tools such as SHAP, LIME or counterfactuals. Be careful, because reputable sources swap the labels: the NIST AI RMF describes explainability as a representation of the mechanisms underlying an AI system's operation and interpretability as the meaning of its output in the context of its designed functional purpose, which is the reverse of common usage. In any assessment or audit, state which definition you are using before you use it.
Explainability vs interpretability: the short version
| Interpretability (common framing) | Explainability (common framing) | |
|---|---|---|
| Question answered | How does the model work? | Why this particular output? |
| Level | Global, the model as a whole | Usually local, one decision |
| How it is achieved | By design, by choosing a transparent model | After the fact, with explanation tooling |
| Typical methods | Linear and logistic regression, decision trees, rule lists, generalised additive models | SHAP, LIME, counterfactuals, saliency maps, surrogate models, feature importance |
| Primary audience | Developers, validators, model risk and audit | Affected individuals, deployers, regulators, complaint handlers |
| Cost | May cost predictive performance | Adds tooling, and explanations can be approximate or unfaithful |
| NIST AI RMF usage | The meaning of a system's output in the context of its designed functional purpose | A representation of the mechanisms underlying a system's operation |
Read the last row twice. NIST inverts the everyday framing, and it treats explainable and interpretable as a single trustworthiness characteristic rather than two.
Why do sources define these terms in opposite ways?
Neither word is defined in the GDPR or the EU AI Act, so there is no authoritative legal meaning to appeal to. Much of the machine learning literature settled on interpretability as a property of the model and explainability as a property of an account given about a decision. NIST, working from a socio-technical starting point, went the other way: explanations describe mechanism, and interpretations give output meaning to a person in context. This site's governance and oversight vocabulary follows the common framing, so interpretability is understanding how the model works internally and explainability is being able to account for its results.
Practical consequences:
- When answering a question or completing a questionnaire that cites a specific framework, use that framework's definition rather than your own.
- In a policy, define both terms once, in your own glossary, and use them consistently. Some vendors will use the other convention.
- In a technical debate, avoid both labels and say precisely what you want: a global account of behaviour, a local account of one decision, or documentation the affected person can act on.
What does the law actually require?
Neither term appears as an obligation. What the law requires is information and oversight:
- GDPR Articles 13(2)(f), 14(2)(g) and 15(1)(h): for solely automated decisions within Article 22, meaningful information about the logic involved, and the significance and envisaged consequences of the processing. Recital 71 refers to obtaining an explanation of the decision reached, but recitals are not binding.
- CJEU, Dun & Bradstreet Austria (C-203/22, judgment of 27 February 2025): meaningful information about the logic involved means explaining the procedure and principles actually applied, so that the person can understand which of their personal data was used and in what way, in a concise, intelligible and easily accessible form. It does not require disclosure of the algorithm itself.
- AI Act Article 13: high-risk systems must be sufficiently transparent for deployers to interpret the output and use it appropriately, supported by instructions for use.
- AI Act Article 14: human oversight measures must let the people overseeing the system properly understand its capacities and limitations, remain aware of automation bias, and correctly interpret the output.
- AI Act Article 86: from the general application date, an affected person subject to a decision taken by a deployer on the basis of the output of an Annex III high-risk system, other than those in the critical infrastructure area at point 2, which produces legal effects or similarly significantly affects them adversely, may obtain clear and meaningful explanations of the role of the system in the decision procedure and the main elements of the decision.
Note the audience shift: Article 13 explains to the deployer, Article 86 explains to the individual, and the GDPR explains to the data subject. Designing one artefact for all three rarely works.
How do you make a system explainable in practice?
There is no single control. What works is a layered approach:
- Choose interpretability where you can. For credit, employment and benefits decisions, a well-specified logistic regression or a gradient-boosted model with monotonic constraints is often close enough in performance and considerably easier to defend.
- Add post-hoc explanations where you cannot. SHAP and LIME give local attributions, and counterfactuals of the form had your income been X, the outcome would have been Y are usually the most actionable form for an affected person.
- Validate the explanations. Post-hoc methods approximate. Test stability, so ask whether near-identical inputs produce wildly different explanations, and record the limits.
- Document at the system level. A model card, a system card and the AI Act instructions for use carry the global account that no per-decision explanation can.
- Train the humans. An explanation nobody in the oversight role understands is not oversight. Article 14 is explicit about competence and automation bias.
Where do the trade-offs bite?
Three tensions recur in governance reviews. First, performance: the most accurate model on a tabular problem is often not the most transparent one, and the honest answer is to test whether the accuracy gap justifies the explanation gap for that use case rather than assuming it does. Second, trade secrets and security: Article 86 and the GDPR case law both stop short of requiring source code or model weights, and detailed explanations can help bad actors game a fraud or credit model. Third, explanation quality: a feature-importance chart is not an explanation to a rejected applicant. If the person cannot tell what they would need to change, the artefact has not done its job, however technically faithful it is.
Related study notes
- Trustworthy AI: the HAT test
- Governance, assurance and oversight
- Features and Feature Engineering
- NIST AI RMF: the full kit
Frequently asked questions
Is there a right to an explanation in the GDPR?
The binding text gives a right to meaningful information about the logic involved, plus the significance and envisaged consequences, for decisions within Article 22. A general right to an explanation of a specific decision appears only in Recital 71, which is not binding, although the CJEU has read the Article 15(1)(h) duty as requiring an account of the procedure and principles actually applied.
Does the EU AI Act create a right to an explanation?
Article 86 does, in a targeted way. It applies to affected persons subject to decisions taken by a deployer on the basis of output from most Annex III high-risk systems, where the decision produces legal effects or similarly significantly adversely affects their health, safety or fundamental rights, and it entitles them to clear and meaningful explanations of the role of the system and the main elements of the decision.
Is it illegal to use a black-box model?
No law bans complex models outright. But the more opaque the model, the harder it is to satisfy AI Act Articles 13, 14 and 86, the GDPR transparency duties, and sectoral rules such as adverse action notices in consumer credit. Opacity is a compliance cost rather than a prohibition.
Which definition should I use?
Match the source you are working from. If NIST is cited, explainability concerns underlying mechanisms and interpretability concerns the meaning of outputs. Otherwise use the common framing, where interpretability is a property of the model and explainability is an account of an output. Being able to state that the definitions conflict, and to say which one you are applying, is the safest position.
Test yourself
Try the free AIGP practice questions, or read the full AIGP study guide - free.