Explainable AI (XAI): What it is, how it works and why it’s important

Explainable AI (XAI) makes it possible to see why an AI system reached a particular result. An ordinary system might only tell you that a loan application was declined. An explainable system also tells you which information drove that decision and what would have changed it.

What is explainable AI (XAI)?

Explainable AI is not a separate type of AI. It is a set of methods that show how an AI system reaches a result. XAI is short for eXplainable Artificial Intelligence. The goal is that people can understand an AI decision, check it and challenge it if needed.

A good doctor does not just tell you that you have an infection, but also why: the fever, the blood results and where it hurts. Explainable AI asks the same of a computer.

Black box versus explainable AI

The part of an AI system that makes the decision is called the model. Many powerful models are black boxes. Data goes in and an answer comes out, but you cannot see what happens in between. Even their builders often cannot say why one result came out as it did.

Take a bank that uses AI to assess loan applications. A black box only says: declined. An explainable system gives the reasons, for example that repayments are too high compared to income. Explainable AI is the layer on top that makes the decision checkable.

What is the difference between AI, generative AI and explainable AI?

Traditional AI predicts or classifies, generative AI creates new content, and explainable AI makes the workings of both understandable.

Traditional AI: predicting and classifying

Traditional AI learns patterns from past examples and uses them to predict or sort things. Think of a spam filter, a bank that estimates whether someone can repay a loan or a streaming service that recommends shows. With decisions about people and money, you want to know why.

Generative AI: creating new content

Generative AI creates new content, such as text, images or computer code. You give it an instruction in everyday language, a prompt, and it writes an answer or makes an image.

What are LLMs and AI chatbots?

The best-known form of generative AI is the large language model, or LLM. An LLM has read huge amounts of text and learned which words usually follow each other. It does not know what is true, so it sometimes says convincing things that are wrong, known as hallucinations.

An AI chatbot is the app you use to talk to such a language model. Think of the language model as the engine and the chatbot as the car around it. Well-known chatbots are ChatGPT from OpenAI, Claude from Anthropic and Grok from xAI, Elon Musk’s AI company.

Explainable AI: the explanation layer for both

Explainable AI is not a third type of AI. It is an approach you apply to both traditional and generative AI.

Why is explainable AI important?

Explainable AI is important because AI increasingly makes or shapes decisions about people and money. A decision nobody can explain cannot be checked, corrected or defended. The main benefits of explainable AI are:

• It catches errors you cannot see in the scores. In the study that introduced LIME, researchers showed a model that seemed to tell wolves from huskies (dogs that look like wolves). In reality, it was looking at snow in the background. The explanation exposed that mistake at a glance.
• It exposes bias. In the Dutch childcare benefits scandal, the Tax Administration used nationality as a signal in a system that automatically flagged applications as high risk. The Dutch Data Protection Authority fined it €2.75 million in 2021 over this discriminatory use of nationality. Explanations per input make that kind of dependency visible.
• It builds the right level of trust. People accept advice more readily when they can follow it. The same explanation also shows when the model is out of its depth.
• It helps you comply with the law. The GDPR, the EU privacy law, and the EU AI Act require organizations to explain automated decisions to the people they affect.
• It speeds up approval. Risk managers, lawyers and business owners can review the logic of a model themselves instead of taking it on faith.

How does explainable AI work?

Understandable by design or explained afterwards

There are two ways to make AI explainable. The first is to choose a simple model you can follow yourself. Think of a decision tree: a series of yes-or-no questions, such as “is the income above €3,000?”. This is sometimes called a glass box.

The second is to study a complex model after the fact: you keep changing a small part of the data and watch how the result moves. That works for any model, but you are looking from the outside.

The whole model or one decision

An explanation can cover the whole model: what does it pay attention to in general? That is what developers want to know. It can also cover one decision: why was this applicant declined? That is what the applicant wants to know.

What are explainable AI (XAI) methods?

There are dozens of XAI methods. The table lists the best-known ones.

Method	Covers	Works with	Useful for	Watch out for
SHAP	One decision and the whole model	Any model	The weight of each input in a decision	Depends on the comparison group
LIME	One decision	Any model, including text and images	Quick checks	Results can differ between runs
Permutation feature importance	The whole model	Any model	Seeing which inputs the model relies on	Misleading when inputs are very similar
Partial dependence and ICE plots	The whole model	Any model	The effect of one input on the outcome	Averages can hide differences
Counterfactual explanations	One decision	Any model	Telling a person what would change the outcome	Must be realistic and achievable
Saliency maps (Grad-CAM)	One decision	Image models	Seeing where the model looked in an image	Can look convincing without being right
Readable models (decision trees, scorecards, EBMs)	One decision and the whole model	Explain themselves	Important decisions about people	Sometimes slightly less accurate

SHAP: how much each input counted

SHAP starts from the average, such as a 73% approval chance. You then see how much each input pushes that chance up or down.

LIME: what mattered for this one decision

LIME keeps making small changes to one application and watches how the answer changes. That shows which information mattered most here.

Feature importance: what the model relies on most

Feature importance shows what the model pays most attention to in general. Partial dependence and ICE plots also show how the outcome changes when one input changes, such as income.

Counterfactual explanations: what would have changed the outcome

A counterfactual is a what-if explanation: the smallest change that would have flipped the decision. For example: with lower repayments, the loan would have been approved.

Saliency maps: where an image model looked

A saliency map is a heat map on top of an image, such as an X-ray. The coloured areas show where the model looked. A well-known method is Grad-CAM.

Examples of explainable AI

Well-known examples are a bank that explains why a loan application was declined, a hospital model that shows which blood values raised a risk score and a chatbot that cites its sources.

A loan decision, step by step

Imagine a bank uses AI to decide who gets a loan. We built such a model with 30,000 made-up applications. It looks at six things: the debt-to-income ratio (the share of income that goes to debt repayments), missed payments, years with the current employer, income, loan amount and postcode. With a chance of 50% or more, you get the loan.

Step 1: What does the model pay attention to?

The debt-to-income ratio counts by far the most, followed by missed payments. Postcode and loan amount hardly matter. If the postcode had come out on top, that would have been a warning sign.

Step 2: Why was this application declined?

Take someone who earns €3,900 net a month, has a debt-to-income ratio of 48%, missed one payment and wants to borrow €18,000. The model gives a 25% chance, so the application is declined. The table shows how each input changed the chance, starting from the average of 73%:

Input (applicant’s value)	Effect on the chance (percentage points)
Starting point: average applicant	73%
Debt-to-income ratio (48%)	-25
Missed payments (1)	-13
Postcode area	-7
Years with employer (1.5)	-4
Loan amount (€18,000)	-3
Net monthly income (€3,900)	+4
Chance for this applicant	25% (declined)

The debt ratio and the missed payment weigh the most. What stands out is the postcode, which costs 7 points on its own.

Step 3: What would have changed the decision?

With a debt-to-income ratio of 32% instead of 48%, the chance rises to 54%, and the loan is approved. The applicant can work on that by paying off debt or borrowing less.

Step 4: Is the explanation right?

An explanation can be wrong too. So check it with a second method, such as LIME. Also look critically at the postcode, which can say something about income or background.

Step 5: How do you tell the customer?

The applicant has no use for charts, but does need two or three reasons in plain language. For example: “Your application was declined mainly because your repayments are high relative to your income, and because a payment was missed. With a debt-to-income ratio of around 32% or lower, an application of this size would likely be approved.”

Explainable AI in healthcare

A model that estimates the risk of a serious infection (sepsis) can show which blood values raised that risk. A doctor can then judge whether the advice makes sense. Such a model must first prove itself in practice.

The role of XAI in business decision-making

For organizations, explanations are most useful before a model goes live: does it rely on sensible information or on something random? This differs per sector:

Sector	What it is about	Why explanations are needed
Banking and financial services	Credit scoring and fraud detection	Customers are entitled to a reason for a rejection
Insurance	Claims and premiums	A rejected claim must be backed up
Healthcare	Risk scores and reading scans	A doctor must be able to check the advice
Government and municipalities	Benefits, allowances and fraud signals	Citizens must be able to challenge a decision
HR and recruitment	Selecting job applicants	Preventing discrimination and meeting strict AI rules
Manufacturing	Predictive maintenance and quality control	Engineers trust an alert sooner when they see why

Explainable AI for generative AI, LLMs and chatbots

With chatbots, explanation is harder. The question and the answer are plain text, and the model behind them calculates with billions of numbers.

Is ChatGPT explainable AI?

No. ChatGPT is generative AI and largely a black box. The same goes for Claude and Grok. Even when a chatbot explains its answer, that is just more text the model writes, not a record of what happened inside.

How to make LLMs and chatbots more explainable

You can still make a chatbot easier to check, usually in three ways:

• Have the chatbot say where its information comes from with every answer, so you can check it yourself.
• Keep track of which questions were asked and which documents the chatbot looked up.
• Let the chatbot show its reasoning steps. That is no guarantee. In a 2025 Anthropic study, Claude 3.7 Sonnet was given a hint towards the answer. When it used the hint, it mentioned that in its reasoning in only 25% of cases.

ChatGPT, Claude and Grok compared

None of these chatbots is explainable AI by design, but they differ in what they show (as of October 2026):

Chatbot	Cites sources	Shows reasoning	Openness about how it works
ChatGPT (OpenAI)	Yes, when it searches the web or your files	A summary	Publishes how its models are meant to behave
Claude (Anthropic)	Yes, when it searches the web or your files	A summary	Publishes the standing instructions Claude gets in its apps
Grok (xAI)	Yes, when it searches the web or X	Intermediate steps in thinking mode	Has published Grok’s standing instructions since May 2025

The differences are small. How you set the chatbot up matters most.

The limits of explainable AI

Explanations help, but they have four pitfalls.

Explanations are not always reliable

Different methods sometimes give different explanations for the same decision. SHAP and LIME can also be fooled, so that an unfair model looks fair.

Correlation is not causation

Explanations show what the model pays attention to, not what causes something in the real world. If the model links a short time with an employer to payment problems, changing jobs less often will not make you more creditworthy.

Explanations can mislead people

In a 2021 study, people followed AI advice more often when an explanation was attached, even when the advice was wrong.

Explainable does not mean fair or accurate

A model can explain its decisions very well and still be unfair. Fairness and accuracy need their own checks.

What does the EU AI Act say about explainable AI?

The GDPR, the EU privacy law, already gives people the right to an explanation when a computer decides about them automatically. According to a 2025 ruling by the EU Court of Justice, that explanation must be understandable. Handing over the formula or code is not enough.

The EU AI Act sets extra rules for high-risk AI, such as AI that selects job applicants or assesses loan applications. Such systems must be understandable and subject to human oversight. Under Article 86, affected people have a right to an explanation.

Because of a 2026 change to the law, the Digital Omnibus, these rules apply from 2 December 2027, or from 2 August 2028 for AI in medical devices and other regulated products. Our article on the EU AI Act’s risk-based classification helps you check whether your system is in scope.

How to implement XAI: Best practices

These seven practices cover most of what goes wrong in real projects:

1. Decide per audience what an explanation must achieve before you choose a model.
2. For important decisions, for example about customers, test a readable model first. Switch to a black box only if it is clearly better.
3. Combine a method for the whole model, such as feature importance, with a method per decision, such as SHAP or a counterfactual.
4. Check your explanations: rerun them with different comparison data and compare methods.
5. Also measure whether the model treats groups fairly, and look critically at inputs that reveal things indirectly, such as postcode.
6. Log every decision with its input data, model version and explanation.
7. Test explanations with the people who will use them, and keep a person accountable for decisions with real consequences.

Conclusion

Explainable AI shows which information drove a decision, how much it counted and what would have changed the outcome. It is not a separate type of AI, but an approach for traditional AI and for chatbots such as ChatGPT, Claude and Grok. Keep in mind that an explained decision is not automatically a fair one.

DataNorth can help at each stage. An AI ethics and bias assessment checks whether an existing model contains hidden unfair patterns. An EU AI Act assessment shows which risk class your system falls into. Our explainable AI consulting helps you choose the right approach, and we also develop AI systems with explanations and human oversight built in from the start.

Get in touch to discuss your use case.

Frequently asked questions about explainable AI (FAQ)