Many AI projects that disappoint were already in trouble before anyone chose a model, because the data was not ready. Data preparation for AI is the work that prevents this. It means getting the data a use case needs into a state the AI system can rely on, and keeping it there. Skip that work, and even an excellent model gives unreliable answers.
We see this happen in projects of every size. A proof of concept runs on a clean sample and impresses everyone. Then the team connects the real CRM, the shared drives and years of email, and the answers start to drift. The model has not changed. The data has.
That is why data preparation belongs in your AI strategy, not in a cleanup phase afterward. This guide explains why, and shows how a team does the work: the workflow, the tools and a before-and-after dataset.
In short:
- AI output is only as reliable as the data behind it, whichever model you choose.
- Readiness is specific to a use case, so you only prepare the data that use case relies on.
- Preparation does not end at launch, because the data keeps changing.
Why does data preparation decide whether an AI project succeeds?
An AI system can only work with the data it receives, so gaps, duplicates and contradictions end up in its output. The model does not know which of two addresses is current. It picks one anyway and presents it with confidence.
Research points the same way. RAND interviewed 65 experienced data scientists and engineers about failed AI projects. One of the five root causes they found is an organization that “lacks the necessary data to adequately train an effective AI model”. In February 2025, Gartner predicted that through 2026, organizations would abandon 60 percent of AI projects that lack AI-ready data.
The damage often shows up late. Of the 53 AI practitioners in high-stakes domains that Google researchers interviewed, 92 percent had run into a “data cascade”, a data problem that compounds downstream. The researchers chose a blunt title for their paper: “Everyone wants to do the model work, not the data work”.
A more advanced model rarely fixes a data problem. A vendor can improve the model, but only your organization can improve its data.
What does data preparation for AI actually involve?
Data preparation for AI comes down to four jobs: finding the data, fixing it, shaping it for the AI system and governing it over time. Many organizations picture only the second one, cleaning, but the other three matter just as much.
Finding the data means knowing what exists, where it lives and which source to trust for each fact. In nearly every AI assessment we run, at least one department turns out to be paying for a subscription IT does not know about. Whatever data sits in that tool is then missing from IT’s overview as well.
Fixing the data starts with profiling: measuring how complete, consistent and current the data is before you change anything. The usual findings are missing values, duplicates, outdated records and the same thing written in different ways, such as “Ltd” next to “Limited”. Outliers need care, because some are errors and some are the rare cases the system must handle. That matches our own document processing work: the straightforward path is quick to build, and the last stretch of accuracy comes entirely from handling the awkward cases.
Shaping the data depends on the type of AI. Machine learning needs features, such as the number of orders in the last 90 days instead of a raw order table. Generative AI needs PDFs, emails and wiki pages turned into clean text with metadata such as owner, date, version and access rights. When a model has to learn to make a judgment, for instance which support tickets are urgent, someone has to label examples first.
The fourth job, governance, covers privacy, permissions, lineage (where each value came from) and quality checks. It also makes the other three repeatable.
Why is enterprise data so hard to get ready for AI?
Enterprise systems rarely share data, access rights tend to be too wide and data quality often has no owner. Each system was bought to run one process, with its own identifiers and definitions. Sales may count a customer as active after one order in two years, while finance counts only customers with an open contract. An assistant that reads both systems can give two different answers to the same question.
Permission problems often come to light only after an assistant goes live. According to Microsoft’s privacy documentation, Microsoft 365 Copilot shows only data a user already has permission to view. That is the right design. But it also means the assistant can surface anything in a folder that was shared too widely years ago.
Then there is ownership. In many organizations, nobody is responsible for the quality of customer data itself, only for the processes that create it. Experienced employees cover for that because they know which spreadsheet is current. An AI system lacks that knowledge, so the quality problems show up in its answers.
How does a data preparation workflow for AI work in practice?
In practice, data preparation for AI runs as a pipeline of eight stages, from the source systems to monitoring after launch. The table follows one use case, the customer assistant in the next section. The tools in parentheses are examples of a category, not requirements.
| Stage | What happens (and the output) | Common approaches | What can go wrong |
|---|---|---|---|
| 1. Source systems | List the systems the use case needs and the authoritative source per field (source inventory) | Spreadsheet or data catalog (Microsoft Purview, DataHub) | A department’s own tool is missed |
| 2. Extraction | Copy data unchanged into a raw layer, in batches or as changes occur (raw tables with load dates) | Connector pipelines (Azure Data Factory, Fabric Data Factory), exports, APIs | Rows dropped silently, or a manual export nobody repeats |
| 3. Profiling | Measure completeness, uniqueness, validity and freshness per field (issues and their size) | SQL; Python profiling libraries (fg-data-profiling); column profiling in Power Query | Profiling a sample, such as Power Query’s default of the first 1,000 rows |
| 4. Cleaning | Fix formats, map variants to one value and flag rule breaks (cleaned dataset and change log) | SQL or dbt models; Python (pandas) | Overwrites nobody can trace |
| 5. Transformation | Merge customer records under stable IDs and parse contracts into text passages with metadata (curated tables and passages) | dbt models; Python; entity-resolution libraries (Splink); document parsers (Azure Document Intelligence, Docling) | Two companies merged, or features built with information from the future |
| 6. Validation | Test each batch against agreed rules and thresholds (pass or fail, with failed batches held back) | dbt data tests and model contracts; data-quality tools (Great Expectations, Soda) | Checking structure but not values |
| 7. Serving or indexing | Publish the data where the AI system reads it (a table, an API or a search index with embeddings and permissions) | Views, feature tables and APIs; for RAG only, search indexes or vector databases (Azure AI Search, pgvector) | Documents indexed without their access rights |
| 8. Monitoring | Recheck freshness, volumes and quality, and rerun known-answer questions (alerts to the data owner) | Scheduled checks in the orchestrator; source freshness checks in dbt | Alerts to a shared inbox, or unnoticed drift |
Some terms need a definition. dbt runs SQL transformations as versioned code and tests the results, and a model contract fixes the columns and data types a dbt model must produce. Entity resolution decides which records describe the same real-world customer. An embedding is a list of numbers that captures a passage’s meaning, so an assistant can search by meaning. An orchestrator (Airflow, Dagster or Fabric Data Factory) schedules and reruns all eight stages. Many teams call the raw layer bronze, the cleaned layer silver and the curated layer gold.
How do you make data quality measurable?
You measure data quality per field with four metrics, each with a threshold you agree on with the data owner:
- Completeness: the share of records with a value.
- Uniqueness: the share of keys that occur once.
- Validity: the share of values that pass the format rule.
- Freshness: the time since the last update.
What happens when you connect an AI assistant to your customer data?
An AI assistant connected straight to scattered customer data gives inconsistent and sometimes wrong answers, as this composite scenario shows. It is based on situations we see regularly, not on one client. A company wants an assistant that gives account managers a complete view of each customer. The data sits in a CRM, an ERP system, support tickets, SharePoint, email, Excel files and a product database.
Pointing the assistant at all of it is quick, but it produces four kinds of errors. The first is identity. The same customer appears under its legal name in the CRM and as an abbreviation in the ERP, so part of the history goes missing. Two companies with similar names can also end up merged.
The second is conflicting information, such as two contract versions side by side. The third is access: a trainee sees a customer’s margin analysis in an answer. The fourth is missing content: key facts sit in a personal inbox or a scanned contract that the assistant cannot read.
In a 2024 paper, Barnett and colleagues list missing content as the first of seven failure points in retrieval-augmented generation (RAG). That is the technique that lets an assistant answer from your own documents.
Data preparation tackles all four problems in seven steps:
- Decide which system is authoritative for each type of information. Record it per field, for example the address from the ERP and contract status from SharePoint.
- Merge duplicate customers under one stable customer ID. Match on the company registration number first. Where it is missing, fuzzy matching looks for names and addresses that closely resemble each other, and a person reviews uncertain pairs.
- Correct outdated and wrong records at the source, so the fix survives the next sync. Rejected records go to the system owner as a task.
- Extract the key facts from PDFs, scans and emails. A document parser with optical character recognition (OCR) turns them into text, and each passage gets its customer ID, version, status and owner.
- Review who can see what in each source system, then carry those access rules into the assistant. Each passage keeps the groups allowed to read it, and every search is filtered on the user’s identity before the model sees any text. This is called security trimming.
- Test the result with real questions whose correct answers you already know. Include questions some users must not get answered, and rerun the set after every change.
- Automate these steps as a pipeline that alerts you when quality drops. The data owner gets an alert when an agreed threshold is crossed, for example for stale records or new duplicates.
The discount matching we built for Friesland Lease shows what works when the data is not perfect. The application matches roughly 2,900 cars in the fleet database to discount lists that differ from brand to brand. Every match gets a confidence score, and low-confidence matches go to a verification screen where the team checks them. That is the realistic target: data that is reliable enough for the decision, plus a clear route for the cases where it is not.
What does the customer data look like before and after preparation?
Before preparation, one customer is spread over four records that contradict each other. After preparation, it is one validated record linked to one current contract. The data is illustrative and comes from the composite scenario, not from a client:
| Source | CRM customer ID | Customer name | Country | Contract start | Linked contract |
|---|---|---|---|---|---|
| CRM | C-1042 | Acme Limited | NL | 03/02/2024 | Contract_Acme_final.pdf |
| ERP | (none) | ACME Ltd | Netherlands | 2024-02-03 | (none) |
| Spreadsheet | (none) | Acme | Nederland | 3 Feb 2024 | Contract_Acme_final_v2.pdf |
| CRM | C-2291 | ACME LTD. | NL | 01-06-2021 | Contract_Acme_2021.pdf |
The name has four spellings, the country three and the dates four formats. The same company has two CRM IDs, and two rows have none. Three contract files compete, and none says which is current. After preparation, the customer is one record:
| Customer ID | Legal name | Country | Contract start | Source of truth | Validation |
|---|---|---|---|---|---|
| CUST-000417 | Acme Limited | NL | 2024-02-03 | Name: CRM · Contract: SharePoint | Passed: unique ID, valid country code, valid date |
The four rows share one company registration number (not shown), which is why they were merged, and a cross-reference table links each source row to CUST-000417. Each contract file now carries that ID plus its version, status, owner and access rights. Only the current contract, Contract_Acme_final_v2.pdf, is indexed. So the assistant retrieves only that contract, links it to one customer and shows it only to the groups allowed to read it.
How does data preparation differ for generative AI, machine learning and AI agents?
Each type of AI has its own demands: generative AI needs current documents, machine learning consistent records, and agents data that is correct when they act.
| Type of AI | What the data has to get right | Typical failure |
|---|---|---|
| Generative AI assistants and RAG | Current, deduplicated documents with metadata and access rules | A confident answer from an outdated or restricted document |
| Machine learning | Complete, consistent and representative records, without information from the future | Strong test scores that collapse in production |
| Predictive analytics | Long history with definitions that stay stable over time | A forecast that mistakes a system migration for a trend |
| AI agents and automation | Correct, validated data at the moment of action | The right action on the wrong customer, order or amount |
Machine learning has a specific trap called leakage: training data that contains information the model will not have at prediction time. Kapoor and Narayanan compiled cases from the literature: 294 published papers across 17 scientific fields in which leakage had been found. In some cases it led to “wildly overoptimistic conclusions”. For data with a time dimension, the usual defense is to split data by date rather than at random, and to build features only from data available at the moment of prediction.
Agents raise the stakes because they act on data instead of describing it. A wrong address becomes a wrong delivery.
Even a system that only gives information can make your organization liable. When Air Canada’s chatbot contradicted the airline’s own bereavement fare policy, a Canadian tribunal held the airline responsible for all the information on its website, chatbot included.
What are the most common data preparation mistakes?
The most common mistakes happen before anyone touches the data, when the work is scoped. The first two come from the same error: taking on all the data at once.
Some organizations try to clean everything first. Without a use case, that work has no definition of good enough and no end date. Others connect every source they have, assuming more data is better. But two sources that contradict each other make an assistant less reliable than one source that is right.
A third mistake is treating preparation as a one-off project. Without automated checks, a system that was accurate at launch degrades quietly as the data changes.
The fourth is leaving permissions, privacy and ownership for later. Personal data falls under the GDPR from day one. For high-risk systems, Article 10 of the EU AI Act also requires governance of data preparation steps such as labeling, cleaning and enrichment. Those rules apply to the systems listed in Annex III from December 2, 2027. That is the timeline in force in September 2026, as our EU AI Act overview explains.
Is your data ready for AI?
Your data is ready for AI when you can answer ten questions about one specific use case with evidence rather than assumptions. According to Gartner’s February 2025 press release, AI-ready data must be representative of the use case, errors and outliers included.
| Question | A good answer looks like | Warning sign |
|---|---|---|
| 1. Do we know which sources matter for this use case? | A short list of systems and document sets, each with a reason | “Everything, just in case” |
| 2. Can we access the data we need? | Technical access, plus permission to use the data for this purpose | Data locked in a vendor system or available only as a manual export |
| 3. Do our systems agree on key definitions? | One written definition for terms such as “customer” and “active customer” | Each department uses its own |
| 4. Is the data complete and current enough? | Measured completeness and age for the fields that matter | Nobody knows how old the records are |
| 5. Is there an authoritative source for each fact? | A named system of record for each data element | Several systems that each claim to be the source |
| 6. Are documents structured and searchable? | Text-readable files with version, date, status and owner | Scans, duplicates and files named “final_v3_FINAL” |
| 7. Are permissions and personal data handled correctly? | Permissions reviewed, personal data mapped and minimized | Widely shared folders and unmapped personal data |
| 8. Can we measure data quality? | Automated checks with thresholds and alerts | Quality judged by feel |
| 9. Can we repeat the preparation for new data? | A scheduled, documented pipeline | A one-off script on someone’s laptop |
| 10. Who owns the data and its quality? | A named business owner for each data set | “IT”, or nobody |
The people who own the process should help answer the questions, not only IT. Every question you cannot answer with evidence becomes a work item. If the gaps include permissions, personal data or ownership, pause the build: those gaps can turn a disappointing pilot into an incident. You can download the questions as a one-page worksheet and fill it in for each use case.
Where should you start with data preparation for AI?
The best starting point is one clearly defined use case rather than your entire data landscape.
- Choose a use case with a measurable outcome and a business owner. Scoring your use cases on value and complexity helps you avoid starting where the data is weakest.
- List the fields, documents and history it needs, with the authoritative source for each.
- Profile that data only, and collect real test cases with known answers.
- Fix what materially affects the output, and add safeguards such as human review for the rest.
- Build the pipeline, data tests and monitoring before go-live, then bring the next use case onto the same foundation.
Your own history is often the best test set. On a purchase invoice automation project, we benchmarked the system on a full year of invoices, roughly 7,000 of them. It processed 88 percent straight through, without manual work. Invoices that carried no annotation were booked to the right ledger account against booking history in 93.5 percent of cases.
What should you settle before choosing an AI model or platform?
Before you choose a model or platform, you need to know whether your data can support the use case you have in mind. That question is cheaper to answer now than after a failed pilot.
In our experience, the AI projects that reach production usually have someone who knows which data to trust, and why.
We have run more than 40 AI assessments over the last 18 months, for municipalities, manufacturers, construction firms, healthcare organizations and investment houses. For your first use case, an assessment shows which data you need, where that data lives and what has to improve before you build.
Frequently asked questions about data preparation for AI (FAQ)
What is AI-ready data?
AI-ready data is complete, consistent, current, accessible and governed well enough for one specific AI use case. The bar is set by the decision the AI supports, so the same data can be ready for one use case and not for another. Its gaps are known, and each fact has one authoritative source.
Do you need perfect data to start an AI project?
No. Almost every organization has gaps in its data, and waiting for perfect data only delays the project. Measure whether the data is good enough for your use case, then add safeguards such as confidence scores and human review where it falls short.
What tools do you need for data preparation for AI?
No single tool is required. SQL and Python (pandas) cover profiling and cleaning. dbt makes transformations and data tests repeatable, and an orchestrator such as Airflow or Fabric Data Factory schedules them. You need a search index or vector database only for RAG. Start with the tools your team already knows.
How do you keep data ready after an AI system goes live?
You keep data ready by turning the preparation steps into an automated pipeline that runs whenever the data changes. Add data tests with thresholds, rerun a set of questions with known answers and review a sample regularly. Customers, products and documents keep changing, and each change can turn a correct answer into a wrong one.
