Data Governance for AI Models

It depends on what you do with the data, not on which model you use. Here is how we scope it, and where we disagree with vendors who sell one platform to every tier.

The opinion embedded in that table: buying software before you have an inventory produces a well-indexed list of unknowns.

A common gap we see in AI readiness assessments is simple. Nobody can say which datasets fed which model version. Engineering knows the pipeline, product knows the feature, legal knows the privacy notice, and nothing joins the three.

Fix that with one table, owned by one named person (usually the head of engineering or data), with a row per dataset and these columns: dataset name, system of origin, whether it holds personal information, the purpose stated to customers when it was collected, which models or indexes use it, who can read it, and its retention rule. For most companies under 200 people this is 10 to 40 rows. If it takes longer than a working session, that is itself a finding.

Include support tickets, call transcripts, analytics events and the shared drive your RAG index crawls. Also the prompts and outputs your own staff send to AI tools, which is where shadow AI exposure usually starts.

ISO/IEC 42001:2023, the AI management system standard published in December 2023, groups its data controls under Annex A.7 "Data for AI systems". In plain terms, A.7.2 to A.7.6 ask you to define and document data management for AI development, how data is acquired, quality requirements, provenance, and how data is prepared. The NIST AI Risk Management Framework (AI RMF 1.0, January 2023) covers the same ground in its MAP and MEASURE functions, and its Generative AI Profile, NIST AI 600-1 (July 2024), lists data privacy among 12 generative AI risks and recommends tracking data and content provenance.

You do not need to adopt either framework to borrow the evidence they expect.

Based on Gebru et al.'s "Datasheets for Datasets", keep two pages per dataset: why it was created, what it contains, how it was collected and consented, what was removed or de-identified, known gaps (missing customer segments, languages, date ranges), and who approved its use. Store it in the same repository as the pipeline so it versions with the code.

Model cards (Mitchell et al., 2019) record intended use, out-of-scope use, the datasets and versions used, evaluation results broken down by relevant groups, and known limitations. It answers the questionnaire item "how was this model trained?". Teams pursuing ISO 42001 certification will reuse both documents as audit evidence; our free ISO 42001 gap assessment shows which Annex A controls you already meet.

The fastest way to get into trouble is to reuse data collected for one purpose to train a model for another. Regulators on both sides of the border have said so directly.

PIPEDA's fair information principles require you to identify purposes at or before collection (Principle 4.2), obtain meaningful consent (4.3), and limit use and retention to those purposes (4.5). In December 2023, the federal, provincial and territorial privacy commissioners jointly issued principles for responsible, trustworthy and privacy-protective generative AI, which apply those same ideas (legal authority and consent, appropriate purposes, necessity and proportionality, openness, accountability, accuracy, safeguards) to developers and users of generative AI.

Quebec's Law 25 goes further. Since September 22, 2023, private-sector organizations must inform a person when a decision about them is based exclusively on automated processing, and must conduct a privacy impact assessment for any project to acquire or develop an information system involving personal information. Penal fines reach C$25 million or 4% of worldwide turnover for the preceding fiscal year, whichever is greater.

There is no federal privacy statute, so the pressure comes from the FTC and the states. The FTC has ordered companies to delete models built on data they should not have collected or used, including Everalbum (2021), and in December 2023 it banned Rite Aid from using facial recognition for surveillance for five years and required it to delete the images and any models derived from them. California's CPPA finalized regulations in 2025 that add risk assessments and automated decisionmaking technology (ADMT) obligations, with ADMT requirements phasing in from January 1, 2027. If you serve healthcare customers, PHI cannot go to a model vendor without a business associate agreement under HIPAA.

GDPR Article 35 requires a data protection impact assessment for high-risk processing, and Article 22 restricts decisions based solely on automated processing that produce legal or similarly significant effects.

The test for each inventory row: would a customer reading your privacy notice expect this data to train or ground a model? If the honest answer is no, you need a new notice, consent, de-identification or a different dataset.

Retention rules for raw data are familiar. The hard part with AI is that deleting a record from the database does not remove its influence from a fine-tuned model. Machine unlearning is still a research problem, not a feature you can switch on.

That changes the order of work. Decide retention and deletion before training, not after a customer asks. Three rules hold up in practice:

When the data was the problem, the FTC's remedy was deleting the model.

A retriever with a service account that can read the whole shared drive will happily surface HR files to a sales rep. OWASP's Top 10 for LLM Applications 2025 lists sensitive information disclosure, and vector and embedding weaknesses, as separate risks for exactly this reason. The fix is to enforce the requesting user's permissions at retrieval time, not to ask the model to keep secrets. Clean up overshared folders before you index them, and log which chunks were retrieved for each answer so you can investigate later. Our post on MLOps pipeline security covers the build-side controls, and data poisoning attacks covers what happens when someone writes to the data you trust.

If the model only drafts marketing copy, a spot check is proportionate. If it ranks job applicants, scores leads that decide who gets a sales call, or sets prices, test error and outcome rates by relevant group before release and on every retrain. In US employment, the EEOC's four-fifths guideline (a selection rate below 80% of the highest group's rate signals possible adverse impact) is a common first screen. It is a screen, not a safe harbour.

Many teams over-build here. If you send prompts to a commercial API that does not train on them, most of the work above does not apply. OpenAI, for example, states that API data has not been used to train its models since March 1, 2023 unless the customer opts in, and keeps abuse-monitoring logs for up to 30 days, with zero data retention available to eligible customers.

In that case your governance is short:

What this will not fix: decisions about people. If the output decides who gets approved, flagged or priced, the purpose, transparency and automated-decision rules above still apply, whoever hosts the model.

In our experience this sequence fits SaaS teams of 30 to 200 people with one or two AI features, and it mirrors how we structure AI governance and data security and privacy work.

The free AI Governance Playbook gives you templates for the policy and risk register. If you would rather have someone run the 60 days with you, our vCISO service starts from US$2,000 a month, or book a 30-minute call and we will tell you which tier of the first table you are actually in.