The Dark Side of AI

Three patterns cover almost everything we see in smaller companies.

Executive and vendor impersonation. The Arup case is the reference point. According to the South China Morning Post, a finance employee received a message in January 2024 that appeared to come from the UK-based CFO about a "secret transaction." The employee joined a video conference where the CFO and other attendees were digitally recreated, then made 15 transfers to five bank accounts. Arup confirmed it was the victim in May 2024. The sequence matters: a phishing message first, then a deepfake to remove doubt.

Voice clones at scale. In May 2025 the FBI warned that since April 2025 criminals had been impersonating senior US officials with AI-generated voice messages to build rapport and take over accounts. A month later the Canadian Centre for Cyber Security and the Canadian Anti-Fraud Centre issued a joint advisory about the same tactic aimed at business executives and senior officials in Canada. If officials are being cloned, your CEO's podcast is enough training data.

Synthetic identities in hiring. The FBI's January 2025 public service announcement says North Korean IT workers have used AI and face-swapping during video interviews, then stolen source code and extorted employers with it. In June 2025 the Justice Department announced searches of 29 suspected laptop farms across 16 states and said the schemes had placed workers at more than 100 US companies. One indictment alone cited at least $3 million in damages and more than 80 stolen US identities.

The FBI's 2025 Internet Crime Report is the cleanest primary data set. It recorded 1,008,597 complaints and about $20.9 billion in reported losses. Business email compromise alone accounted for 24,768 complaints and about $3.05 billion in adjusted losses. It also breaks out AI separately: 22,364 AI-related complaints with losses above $893 million, and more than $30 million in BEC losses where AI was involved.

That last number is small next to $3 billion. Good. It tells you something useful: most money is still lost to the plain version of the scam, and AI is an accelerant, not a new category. The controls that stop a badly written fake invoice also stop a perfectly written one.

Verizon's 2026 Data Breach Investigations Report found credential abuse in 39% of breaches and third parties involved in 48%. That is where impersonation lands: a stolen login or a trusted vendor. IBM's 2026 Cost of a Data Breach study puts the global average breach at $4.99 million.

This is where we disagree with much of what is being sold. Our view, from the controls we review across assessments:

The theme is simple. Stop trying to detect the fake and verify the request instead. A detector must win every time. A call-back rule just has to be followed.

Detection is not useless. Banks verifying identity at volume get value from liveness checks. But NIST's own report on synthetic content (NIST AI 100-4) describes detection as "a constant cat-and-mouse game." For a 40-person company, a written payment procedure beats a detection subscription every time.

Over-promising is how controls get ignored, so here are the limits.

And you probably do not need a dedicated deepfake program at all if your payments, vendor changes and hiring already run through written, verified procedures. Most companies under 200 people should spend the next dollar on the basics in the table, not on an "AI fraud" product.

The order of operations we use fits in a quarter and mostly costs time, not tools.

If you want a structured starting point, our Cybersecurity Baseline Assessment checks these controls, and our process, risk and controls work turns them into written procedures your auditors will accept.

Not IT alone. Payment fraud sits with finance, hiring fraud with HR, and someone must set policy across both. In most of our clients that person is a fractional security leader working with the CFO. A Virtual CISO engagement starts from $2,000 a month for 15 to 20 hours, compared with roughly $250,000 a year for a full-time CISO. If your team is also adopting AI tools internally, pair this with an AI governance policy so the same verification mindset covers agents and copilots.

Yes. The Arup employee joined a multi-person video conference and still made 15 transfers. A stressed employee on a short call with a senior "executive" is not hunting for artifacts.

Not first. Detection error rates are high on new generators, according to NIST. Put the money and effort into call-back and dual-approval rules, which work regardless of how good the fake is.

Look for reused phone numbers or email addresses across applicants, reluctance to appear on camera or in person, requests to ship equipment to a different address, and changes to payment details during onboarding. Verify identity at offer and again at onboarding.

It shapes them. In Quebec, using biometrics such as face matching to verify identity requires the person's express consent and prior disclosure to the Commission d'accès à l'information. A document check plus a live video call is simpler and avoids that. Our data security and privacy team can help you design checks that stay within PIPEDA and US state law.

Want a second set of eyes on your payment and hiring controls? Book a consultation with IRM Consulting & Advisory.