Data Preparation for AI Projects: Where to Begin

AI learns from the data it is shown, or answers by relying on it. An error in the data can become a lasting tendency in the system's behaviour. If a flooring manufacturer keeps its product details in three tables with three different units, an assistant working from them may quote different sizes for the same product.

Preparation is therefore not a cost paid at the start but a way of heading off the errors that would otherwise surface later. The shared cause of many failed projects is not the model but data that nobody trusts.

The first step is a list of what you hold: which information is kept where, who updates it, when it was last updated and in what format. The list is often surprising; the same customer detail may live separately in an accounts package, an email inbox and a spreadsheet.

For each source the inventory answers three questions: is it current, is it reliable, may an AI system be shown it. A source you cannot answer yes to on all three is left out of the project or fixed first.

• Source: where the data is kept, in which system or file
• Owner: who updates it and who answers for its accuracy
• Currency: when it was last reviewed
• Format: structured table, free text or scanned image
• Sensitivity: whether it holds personal, financial or commercially confidential material

Cleaning is three jobs: merging duplicates, resolving contradictions and filling or flagging gaps. A customer recorded under three spellings, a product listed at two prices, fields left blank: these are typical. Automated tools often get halfway, and someone who knows the business finishes the rest.

AI can help here too, for instance by grouping similar records and suggesting merges; but its suggestion may be wrong, and merging two customer records by mistake is hard to undo. Merge decisions are tested by eye on a small sample first and then widened.

Not all data has to be shown to AI. ID numbers, health details, bank details and private correspondence with customers are left out or masked unless the job truly needs them. The safest data is data that is never shown.

For the data that is shown, access is narrow: only the people on the project see it, every access is logged, and it is settled beforehand which service the data goes to, whether it is stored there and whether it is used to train the model. No firm ruling on legal obligations can be given here; before a project that touches personal data, take advice from a lawyer.

The most skipped part of preparation is the test set: a short list of real questions the system should answer correctly, with the right answers. Someone who knows the work writes it, independently of the system. Around a hundred questions is enough to begin with for most businesses.

Without a test set, whether the system works well is only an impression. Each time you update the data or change the tool, you rerun the same list and see whether it improved or slipped. AI sometimes answers wrongly; the test set shows how often and on which topics.

If you use AI only to fix spelling, summarise meetings or draft general text, you do not need your own data and detailed preparation is overkill. The real issue there is a simple rule about what must never be typed into the tool.

Preparation is also premature when the project's purpose is still vague. Cleaning data before you know what you want to solve risks fixing the wrong thing. Choose a small, clear question first, then prepare the data that serves it.

The most common mistake is trying to prepare all the data at once. A clean-up that runs for months wears people out before any result appears. Starting small, with the data for a single topic, gives a quick result and shows what works.

The second is orphaned data: a source with no clear person to update it goes stale again after a while. The third is sensitive data being forgotten during preparation and uploaded to a spreadsheet as it stands. The separation is done while the data is prepared, not left for later.