Data Security & Responsible AI

The numbers moved quickly in one year. IBM's Cost of a Data Breach Report 2026, which studied 602 breached organizations between March 2025 and February 2026, found more than 20% had a breach targeting AI models or applications. The most common entry points were compromised APIs, applications or plug-ins (27%) and cloud misconfigurations affecting AI workloads (27%). Only 37% of breached organizations said they encrypt sensitive data both at rest and in transit. The global average breach cost reached a record US$4.99 million.

A year earlier, IBM's 2025 edition put the share at 13%, and of those organizations, 97% reported not having AI access controls in place. One in five reported a breach due to shadow AI, and organizations with high levels of shadow AI saw breach costs about US$670,000 higher than those with little or none. In Canada, IBM's 2026 figures put the average breach at a record CA$7.11 million, with a 205 day lifecycle to identify and contain a breach.

Read those figures with care. IBM's sample is breached organizations, many of them larger than our typical client, so the percentages describe what goes wrong once something has gone wrong, not your odds. The pattern still matters: the failures are access control and configuration, not exotic model attacks.

If you never train or fine-tune a model, skip to the next section. If you do, the first control is boring and decisive: know exactly what went into each training run.

In practice that means three things. Every dataset gets a version and a hash, stored next to the model artifact it produced. Every source is listed with who supplied it and when. And nobody can write to the training bucket except the pipeline itself. Without those, you cannot answer the first question an incident responder or auditor will ask, and you cannot roll back a poisoned dataset because you do not know which version was clean.

A common pattern we see is a fine-tuning set assembled from support tickets, exported once by an engineer to a personal cloud folder, then reused for months. That file usually contains customer names, emails and sometimes payment details. It is also a personal information store under PIPEDA and Quebec's Law 25, and in the US it may fall under state breach notification laws or, for health data, HIPAA. Strip or tokenize identifiers before the data leaves the source system, not afterwards.

Prompts are the new copy and paste. Staff paste stack traces with API keys in them, customer emails with signatures, and spreadsheets with salary columns. Each paste lands in the vendor's logs, your own application logs, and sometimes a conversation history that other team members can search.

Three controls carry most of the weight here:

Canada's federal, provincial and territorial privacy commissioners said much the same in their December 2023 principles for generative AI: limit personal information to what is necessary and put safeguards in place proportionate to its sensitivity. Our post on shadow AI breaches covers the approved tools list in more detail.

In our experience, this is where the serious gaps hide. A retrieval augmented generation (RAG) system copies your documents into a vector store so the model can quote them. The copy usually loses the original permissions. The HR folder that only three people could open in SharePoint is now one embedding search away from anyone who can type into the assistant.

The OWASP Top 10 for LLM Applications 2025 lists this as LLM08, Vector and Embedding Weaknesses, and names the fix: permission-aware vector stores with strict logical partitioning, plus immutable logs of retrieval activity.

Ask your engineers one question: when user A asks the assistant something, are search results filtered by what user A can already access in the source system, before the results go to the model? If the answer is "the system prompt tells the model not to share confidential files," you have no access control. Models follow instructions most of the time. Access control has to work every time.

For multi-tenant SaaS products, add a second question: is each customer's data in a separate index or namespace, and is the tenant ID enforced in the query rather than in the prompt?

Outputs leak in two ways. The model can repeat personal or confidential data it retrieved or memorized, and your own logging can store every prompt and response in plain text for years.

For customer-facing features, filter outputs for the same identifier patterns you mask on input, and test it, including attempts to make the assistant reveal other users' data or its system prompt. If you already commission penetration testing, add the AI feature to the scope in writing. Our post on the risks of using LLMs lists common prompt injection patterns to try.

Logs need the opposite of the usual instinct. You want enough to investigate (user, time, documents retrieved, tool calls made), but full prompt and response bodies are a sensitive data store. Set a retention period, encrypt them, and restrict who can read them. If your AI feature can take actions, such as sending emails or updating records, log every tool call; securing AI agents goes further on that.

Every AI vendor contract answers four questions, sometimes by silence. Is our data used to train or improve your models? How long are prompts and files retained? Who at the vendor can see them, and in which country are they stored? Will you notify us of a breach, and how fast?

Get these in writing on the plan you are actually buying. Consumer and business tiers of the same product often have different answers. US regulators have taken this seriously: the FTC has said model-as-a-service companies that deceive users about how their data is used, including breaking promises not to use it to train models, may be violating the law, and it has required companies to delete models built on unlawfully obtained data. That protects you only if the vendor's promise is written down. For health data in the US, a vendor that creates, receives, maintains or transmits PHI on your behalf needs a business associate agreement under HIPAA. In Quebec, Law 25 requires a privacy impact assessment before personal information is communicated outside the province.

Vendors selling AI security products will disagree with the bottom half of this table. We rank by how much risk a 30 to 200 person company removes per hour of effort.

A policy says what you intend. It does not change what your system does. A common pattern we see is a well-written AI policy, often adapted from a template, sitting on top of a retrieval system that ignores document permissions.

Here is what a policy document will not do:

There is a fair counterpoint. In the 2025 study, 63% of breached organizations either had no AI governance policy or were still developing one, and a policy is what tells staff which tools are approved. Write it. Just write it after the first three rows of the table, and make each clause point to a control that enforces it. Our free AI Governance Playbook gives you a starting structure.

One gap we see in AI incident tabletops is not technical at all: nobody knows who can switch off the AI feature during an incident.

If you sell to larger customers, their security questionnaires now ask about AI data handling, and "we have a policy" no longer passes. Through our AI governance and data security and privacy services, or as part of a vCISO engagement for teams without a security lead, we can review your AI data map against the six control points above and help you sequence the fixes. vCISO tiers start from US$2,000 a month.

Book a 30 minute call and bring your AI data map, even a rough one. We will tell you which row of the table to fix first.