A model that’s 95% accurate but can’t explain any individual prediction is a hard sell in a lot of real-world contexts: a bank can’t tell a rejected loan applicant “the algorithm said no” without being able to say why, a hospital can’t act on a diagnostic model’s output without a clinician understanding what evidence it weighed, and regulators increasingly require an explanation as a condition of deployment, not just good aggregate performance. Interpretability and explainability are the subfield concerned with answering “why did the model produce this specific output,” and the terms get used almost interchangeably, though some researchers draw a distinction: interpretability refers to models that are understandable by design, while explainability refers to techniques that explain an otherwise opaque model after the fact.
Why This Is Hard
The accuracy gains of the last decade came largely from more complex models, deep neural networks with millions or billions of parameters, that are fundamentally harder to inspect than a linear regression or a shallow decision tree. A linear model’s explanation is trivial: each feature has a fixed weight, and you can read off exactly how much each input contributed. A neural network with a hundred layers of nonlinear transformations has no such simple decomposition; the influence of any single input feature on the output is distributed across the entire network in a way that isn’t directly readable from the weights.
This creates a real tension that shows up constantly in applied ML: the most accurate models are frequently the least interpretable, and the most interpretable models (simple linear models, small decision trees) are frequently not accurate enough for the task. Interpretability techniques exist largely to get some of the explanation benefit of a simple model without giving up the accuracy of a complex one.
SHAP Values
SHAP (SHapley Additive exPlanations) borrows a concept from cooperative game theory, the Shapley value, which fairly divides credit among players who contributed to a joint outcome. Introduced by Scott Lundberg and Su-In Lee in “A Unified Approach to Interpreting Model Predictions” at NeurIPS 2017, SHAP’s contribution was showing that several previously separate explanation methods are all special cases of one additive feature-attribution framework built on Shapley values (Lundberg & Lee, NeurIPS 2017, official proceedings). Applied to ML, each input feature is treated as a “player,” and SHAP computes how much each feature contributed to moving the prediction away from a baseline (typically the model’s average output across the training set), by averaging that feature’s marginal contribution across every possible combination of other features being present or absent.
The output is a set of per-feature contribution scores for an individual prediction that sum up to exactly the difference between that prediction and the baseline, which is a mathematically clean guarantee that most other explanation methods don’t offer. The cost is computational: exact Shapley values require evaluating an exponential number of feature subsets, so practical SHAP implementations use approximations (and model-specific fast paths, like TreeSHAP for tree-based models) to make it tractable.
LIME
LIME (Local Interpretable Model-agnostic Explanations), introduced by Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin in “‘Why Should I Trust You?’: Explaining the Predictions of Any Classifier” at KDD 2016, takes a different, more intuitive approach (Ribeiro, Singh & Guestrin, 2016, arXiv:1602.04938): to explain a single prediction, generate a bunch of slightly perturbed variations of that input, get the black-box model’s prediction on each variation, and then fit a simple, interpretable model (usually linear) to approximate the black-box model’s behavior in that local neighborhood only. The simple model won’t match the complex model’s behavior globally, but it doesn’t need to, it only needs to be a locally faithful approximation around the one prediction being explained.
This is why it’s called “local” interpretability: LIME makes no claim about the model’s overall logic, only about what’s driving this particular prediction for this particular input. It’s model-agnostic in the sense that it treats the underlying model purely as a black box it can query, so it works identically whether the underlying model is a random forest, a neural network, or an ensemble of both — the paper’s own framing is explicitly about solving the “trust a prediction” and “trust a model” problems separately.
Attention Visualization
For transformer-based models specifically, the attention weights computed internally (which tokens attend to which other tokens) offer a more direct, architecture-specific window into what the model is “looking at” when producing an output. Visualizing attention maps, showing which input tokens received the most weight when generating a particular output token, is a common diagnostic tool in NLP research, and is genuinely useful for spotting certain failure modes, like a model attending to an irrelevant token due to a spurious correlation in training data.
The important caveat, well established in interpretability research, is that attention weights are not a complete explanation on their own: high attention to a token doesn’t always mean that token was causally responsible for the output, and models can produce the same output through different attention patterns. Attention visualization is best treated as one diagnostic signal among several, not a full explanation by itself.
Probing Classifiers
A fourth family of technique, less well-known outside NLP/ML research circles but increasingly used alongside the three above, is probing: training a small, simple classifier on top of a frozen model’s internal activations to test whether a specific piece of information (e.g., part-of-speech, sentiment, factual knowledge) is linearly recoverable from that layer’s representations. Unlike SHAP, LIME, or attention visualization, probing doesn’t explain a single prediction, it answers a narrower question about what information the model’s internal representations actually encode at a given depth, which makes it more of a research diagnostic than a deployable, per-decision explanation tool.
Comparing the Four Approaches
| Method | Explains | Model-Agnostic? | Best For | Key Limitation |
|---|---|---|---|---|
| SHAP | Individual prediction, feature-by-feature | Yes (with fast paths for tree models) | Regulated, high-stakes decisions needing a mathematically consistent explanation | Computationally expensive for exact values; approximations trade off some rigor |
| LIME | Individual prediction, via a local surrogate model | Yes | Quick, intuitive local explanations across any model type | Only locally faithful; unstable across very similar inputs in some cases |
| Attention Visualization | What the model “looked at” internally | No (transformer-specific) | Diagnosing NLP/transformer failure modes | Attention ≠ causal responsibility for the output |
| Probing Classifiers | What information a layer’s representations encode | No (requires access to internals) | Research into what a model has learned, not per-decision explanations | Doesn’t explain individual predictions at all |
Why This Is Becoming Mandatory, Not Optional
Regulatory frameworks including the EU’s AI Act increasingly require that high-stakes automated decisions be explainable to the people affected by them. The Act’s Article 13 is specific about this: high-risk AI systems must be designed so their operation is “sufficiently transparent to enable deployers to interpret a system’s output and use it appropriately,” and must ship with instructions for use that are “concise, complete, correct and clear” about the system’s characteristics, capabilities, and limitations (Article 13: Transparency and Provision of Information to Deployers, EU AI Act official text). Various sector-specific rules in finance and healthcare add their own explainability requirements on top of this.
This means interpretability has shifted from a research nicety to a deployment requirement for anyone building in a regulated industry. Practically, the choice of which model to deploy is no longer just about which one scores highest on a held-out test set, it also has to account for whether that model’s decisions can be explained well enough to satisfy the applicable regulatory and business requirements — part of why simpler, inherently interpretable models still see heavy use in finance and healthcare even where a more complex model might squeeze out marginally better accuracy. In practice, tracking these explanations over time is itself an MLOps concern, not a one-off audit — a model’s explanations need monitoring for drift just like its raw accuracy does.
