What is explainable AI (XAI)?

    Explainable AI makes model outputs understandable enough to check, act on or challenge. Main techniques, their limits, and where the law requires an explanation.

    Explainable AI (XAI) is the set of methods and practices that make an AI system's outputs understandable to a specific audience — why this output, for this input, with what confidence — accurately enough that the person can act on it, check it or challenge it.

    Why explainability matters for governance

    Four governance controls depend on explainability.

    • Human oversight. A reviewer shown only a verdict can do nothing but approve it. Oversight needs the inputs and factors behind the output. See human-in-the-loop.
    • Contestability. People cannot challenge a decision they do not understand, and several laws give them the right to try.
    • Debugging and bias detection. Explanations show when a model leans on a proxy for a protected characteristic, or on an artifact of the training data. See bias in AI systems.
    • Assurance. An auditor testing whether a system behaves as documented needs some view of what drives it. See AI audit.

    Explainable vs interpretable vs transparent

    • Interpretable models can be understood directly: a short decision tree, a points-based scorecard, a sparse linear model, a generalized additive model. The explanation is the model, so it is exact.
    • Explainable AI, in common use, means generating explanations after the fact for models that cannot be read directly, such as gradient-boosted ensembles and neural networks. These explanations are approximations of the model, not the model.
    • Transparency is disclosure about the system as a whole: that AI is used, its data, purpose and limits. It is covered in what is AI transparency?

    Cynthia Rudin's widely cited argument (Nature Machine Intelligence, 2019) is that for high-stakes decisions on structured data, organizations should use interpretable models rather than explain black boxes, because the accuracy cost is often negligible and the explanation is then faithful by construction. Interpretable models are not always feasible.

    NIST's four principles of explainable AI

    NIST IR 8312, Four Principles of Explainable Artificial Intelligence (September 2021), is a widely used test of whether an explanation is fit for purpose:

    • Explanation. The system supplies evidence or reasons for its outputs.
    • Meaningful. The explanation is understandable to its intended recipient. What is meaningful to a data scientist is noise to a loan applicant.
    • Explanation accuracy. The explanation correctly reflects the process the system actually used to produce the output. This is distinct from the accuracy of the output itself.
    • Knowledge limits. The system operates only under the conditions it was designed for, and signals when it reaches insufficient confidence or is outside them.

    The third principle is the one most often unmet in practice. A plausible explanation that does not reflect what the model did does not satisfy it, however convincing it reads.

    The main XAI techniques

    Technique Scope What it shows Main weakness
    Interpretable models (scorecards, short trees, GAMs) Global and local The exact logic May cost accuracy on unstructured data
    Feature importance, partial dependence Global Which inputs matter overall and how Hides interactions; says little about an individual case
    SHAP (Lundberg and Lee, 2017) Local, aggregable to global Each feature's contribution to one prediction, based on Shapley values from game theory Depends on the background data chosen; correlated features distort attributions; costly for large models
    LIME (Ribeiro et al., 2016) Local A simple surrogate model fitted around one prediction Unstable: small changes in sampling can change the explanation
    Counterfactual explanations (Wachter et al., 2017) Local The smallest change to the inputs that would have changed the outcome ("if your income were $4,000 higher…") Several valid counterfactuals may exist; some suggest changes a person cannot make
    Saliency maps, integrated gradients Local Which pixels or tokens most influenced the output Can look convincing while being insensitive to the model's actual parameters
    Mechanistic interpretability Internal Features and circuits inside a neural network Research-stage; partial coverage; not yet audit-grade

    Counterfactuals. For affected individuals they are often the most meaningful format, because they answer the question people actually ask: what would have had to be different? They also map naturally to legal duties to give reasons.

    Known failure modes

    • Explanations can be gamed. Slack et al. (2020) showed that a model can be built to behave in a discriminatory way while presenting innocuous LIME and SHAP explanations, because those methods probe the model with synthetic inputs it can detect.
    • Methods disagree. Krishna et al. (2022) documented the "disagreement problem": different explanation methods often give conflicting attributions for the same prediction, and practitioners tend to pick whichever one confirms their expectations.
    • Explanation accuracy is rarely tested. Most deployments check that an explanation is produced, not that it is faithful.

    Explaining large language models

    Large language models break most of the classical toolkit. The inputs are long, the features are not human-defined, and the model can generate its own fluent account of its reasoning.

    That self-account is not an explanation. Anthropic's 2025 research, Reasoning models don't always say what they think, found that reasoning models frequently failed to mention hints that had demonstrably changed their answers. A cross-lab position paper in July 2025 (Chain of Thought Monitorability) argued that visible reasoning is a useful but fragile safety signal that developers should work to preserve. Visible reasoning is therefore useful for monitoring but unreliable as a record of cause.

    Mechanistic interpretability is the more ambitious path. It attempts to identify the internal features and circuits a model uses. Work such as Scaling Monosemanticity (2024) and On the Biology of a Large Language Model (2025) has traced specific concepts and multi-step reasoning inside production-scale models. It is relevant to AI safety research. It is not yet a way to explain an individual credit or hiring decision to a regulator.

    Governance implication. Where an LLM contributes to a decision about a person, the model's own account is not relied on as the explanation. Good practice is to constrain what it does, log inputs and outputs, keep a human with authority in the decision, and make the human's reasons the explanation of record.

    Where the law requires an explanation (status as of October 2026)

    European Union

    • GDPR. Where a decision is based solely on automated processing and produces legal or similarly significant effects (Article 22), individuals are entitled to "meaningful information about the logic involved" (Articles 13–15) and to safeguards including human intervention and contesting the decision. Recital 71 refers to a right "to obtain an explanation of the decision reached."
    • CJEU, SCHUFA (C-634/21, 7 December 2023). A credit score can itself be an automated decision under Article 22 where a third party draws strongly on it to decide.
    • CJEU, Dun & Bradstreet Austria (C-203/22, 27 February 2025). An individual refused a mobile phone contract after an automated credit assessment is entitled to an explanation of "the procedure and principles actually applied", in a concise, transparent, intelligible and easily accessible form, so they can understand which of their data were used and how. Handing over a complex algorithm is not enough, and neither is a generic description. A trade secret claim does not end the matter: the information goes to the court or supervisory authority, which balances the interests. The court indicated that explaining how a variation in the person's data would have led to a different result could be sufficient, which is in effect a counterfactual explanation.
    • EU AI Act, Article 86. A person subject to a decision taken by a deployer on the basis of a high-risk system listed in Annex III (except critical infrastructure), which produces legal effects or similarly significantly and adversely affects their health, safety or fundamental rights, may obtain "clear and meaningful explanations of the role of the AI system in the decision-making procedure and the main elements of the decision taken." It applies only where other EU law does not already provide the right. Because it is tied to Annex III high-risk systems, it applies alongside those obligations from 2 December 2027, after the Digital Omnibus deferral.
    • EU AI Act, Articles 13 and 14. Providers must make high-risk systems interpretable enough for deployers to use them appropriately, and human overseers must be able to "correctly interpret" outputs (Article 14(4)(c)).

    United States

    • ECOA and Regulation B. A creditor taking adverse action must give a statement of the specific principal reasons (12 CFR 1002.9). There is no exemption for complex models.
    • CFPB circulars, withdrawn (May 2025) pending review. Circular 2022-03 said creditors cannot avoid the reasons requirement because their algorithm is too complex to explain. Circular 2023-03 said generic sample-form reasons are not enough when the real reasons are more specific. Both were withdrawn on 12 May 2025 among 67 guidance documents (CFPB withdrawn guidance). The CFPB stated that the withdrawal is not final and remains under review, so as of October 2026 their status is withdrawn pending review. The underlying regulation did not change, and private plaintiffs and state regulators still enforce it. A lender that cannot produce accurate, specific reasons from its model has a Regulation B problem with or without the circulars.
    • State law. Colorado's re-enacted automated decision-making law, SB 26-189, applies from 1 January 2027. After a consequential decision with an adverse outcome, deployers must give the consumer a plain-language description of the automated system's role within 30 days, and consumers may request correction of inaccurate personal data and meaningful human review.

    Canada

    • Quebec, Law 25. Section 12.1 of the private sector privacy act, in force since 22 September 2023, applies when a decision is based exclusively on automated processing. The organization must tell the person, and on request explain the personal information used, the reasons and principal factors and parameters that led to the decision, and the right to have that information corrected. The person must also be able to submit observations to a staff member who can review the decision.
    • Federal government. The Directive on Automated Decision-Making requires a meaningful explanation to affected individuals for decisions made or supported by automated systems. Its 2025 amendments added explanation criteria.
    • Federally regulated financial institutions. OSFI's Guideline E-23 (effective 1 May 2027) brings AI and machine learning models inside an enterprise model risk framework, where the explainability expected rises with the model's risk. See AI risk in financial services.

    How to tell whether an explanation is any good

    An auditor or second-line reviewer can apply five tests:

    • Fidelity. Does the explanation match the model's real behaviour? Perturb the top-ranked factors and confirm the output moves as the explanation predicts.
    • Stability. Do near-identical cases get near-identical explanations? Unstable explanations cannot support reasons given to customers.
    • Actionability. Could the recipient do anything with it? "Your score was low" is a verdict, not an explanation.
    • Consistency with what was disclosed. Do the reasons in adverse action notices, customer letters and complaint responses match what the model actually used?
    • Documented limits. Is it recorded where the explanation method is unreliable (correlated features, out-of-distribution inputs) and what happens then?

    If a system's explanations fail these tests, the options are to change the model, narrow its use, or put a human decision-maker in front of it.

    For disclosure about the system as a whole, see what is AI transparency?. For the oversight control explanations feed, see human-in-the-loop. For who answers for the decision, see AI accountability. For testing, see AI audit and what IT auditors should know about it. For the personal-data rights involved, see AI privacy.

    Sources and references

    Frequently asked questions

    What is explainable AI?

    Explainable AI (XAI) is the set of methods and practices that make an AI system's outputs understandable to a specific audience: why this output, for this input, with what confidence. A good explanation is accurate enough that the person can act on it, check it or challenge it.

    Why is explainable AI important?

    Because people cannot oversee, debug, audit or contest a decision they cannot understand. Explanations let a human reviewer catch errors, let developers find bias and data problems, let auditors test controls, and let affected people exercise legal rights to challenge automated decisions under laws such as the GDPR, the EU AI Act, US credit law and Quebec's Law 25.

    What is the difference between explainable AI and interpretable AI?

    An interpretable model is one whose reasoning a person can follow directly, such as a short decision tree, a scorecard or a sparse linear model. Explainable AI usually means producing explanations after the fact for a model that is not directly understandable, such as a gradient-boosted ensemble or a neural network. Interpretable models give exact explanations. Post-hoc explanations are approximations.

    What are examples of explainable AI techniques?

    Common techniques include SHAP, which attributes a prediction to input features using Shapley values; LIME, which fits a simple local model around one prediction; counterfactual explanations, which state the smallest change that would have changed the outcome; partial dependence and feature importance plots for global behaviour; saliency maps and integrated gradients for images and text; and mechanistic interpretability research for large language models.

    Is there a legal right to an explanation of AI decisions?

    In several places, yes. Under the GDPR, people subject to solely automated decisions are entitled to meaningful information about the logic involved, which the EU Court of Justice in February 2025 said means the procedure and principles actually applied. The EU AI Act's Article 86 adds a right to an explanation for decisions based on Annex III high-risk systems. US lenders must give specific principal reasons for adverse credit decisions, and Quebec's Law 25 requires reasons and principal factors on request.

    Can large language models be explained?

    Only partly. A model's own stated reasoning is not a reliable explanation: research in 2025 showed reasoning models often do not mention factors that actually influenced their answers. Mechanistic interpretability can trace some internal features and circuits, but it is a research tool, not yet an audit-grade explanation. For LLM-based decisions about people, governance should rely on constrained design, logging and human review rather than self-explanation.

    Do SHAP or LIME satisfy adverse action notice requirements?

    Not automatically. Regulation B requires the specific principal reasons for an adverse credit decision. Feature attributions can help identify them, but they must be accurate for the individual, stable, and translated into reasons a consumer can understand and act on. The CFPB withdrew its 2022 and 2023 circulars on complex algorithms in May 2025, pending review, but the regulation itself did not change.

    Share on LinkedIn