Your staff passed the AI course. Can they do the job?

Walk through the human-resources floor of almost any commercial bank in Nairobi or Lagos this quarter, and you will hear executives celebrating a training milestone: five hundred or a thousand employees through a certified prompt-engineering and AI-literacy course. Framed certificates sit in personnel files, LinkedIn glows with badges, and the learning-and-development budget gets ticked off as transformation delivered.

The pressure to show such numbers is now regulatory as well as reputational; the European Union's AI Act began phasing in an obligation on firms to ensure adequate AI literacy among staff in 2025, and supervisory expectations elsewhere are drifting the same way. Then Monday morning opens on the credit desk, and a newly certified analyst pastes a confidential borrower's unredacted tax return into a public browser tool, or rubber-stamps a fluency-polished credit summary that fabricated a debt-service coverage ratio out of thin air. A completion badge proves an employee watched a video. It provides no evidence that the employee can spot a machine error under deadline pressure.

The hollowing-out risk is measurable. In a 2025 study published in the Proceedings of the National Academy of Sciences, Wharton researcher Hamsa Bastani and colleagues ran a randomised field trial with nearly 1,000 high-school mathematics students in Turkey. One group practised with standard textbooks, a second used an unrestricted GPT-4 assistant, and a third used a safeguarded AI tutor designed to give hints rather than answers. During assisted practice, the unrestricted group scored 48 per cent higher than the control group. When the tool was removed for a closed-book exam, the same group scored 17 per cent worse than students who had never touched AI at all. They had used the machine as a cognitive crutch, hollowing out the underlying judgement it appeared to enhance. Equip bank staff with unguarded generative tools while testing them only on speed and attendance, and you manufacture the same skill atrophy across your risk and operations teams.

Separating tool familiarity from the decision on the desk.

When a bank buys training, it usually buys generic software instruction rather than role-specific operational judgement. Teaching a relationship manager, a call-centre clerk and a treasury reconciliation officer the same two-day module on how language models predict the next word misses the actual decision each employee is paid to make.

Different layers of automation demand different supervision skills. Supervising a deterministic rules engine requires an operations clerk to understand the exact threshold that triggered an automated account freeze and verify whether the source field was corrupted. Auditing a statistical credit or fraud model requires a credit officer to recognise confidence intervals, spot demographic or seasonal drift, and understand why a probability is not a certainty. Working alongside a generative assistant requires a compliance reviewer to treat fluent prose with suspicion, tracing every cited clause back to the primary contract. Overseeing an automated system allowed to initiate ledger entries demands real-time monitoring of velocity caps, idempotency locks, and kill-switch permissions, so a runaway loop can be severed in seconds. A generic course on writing clever prompts leaves three of those four supervisory jobs untaught.

Often, when frontline staff struggles, the bottleneck is not AI literacy at all. Before spending millions of shillings on external AI academies and recurring per-seat sandbox licences, an operations director should clean up the intranet so policy circulars are searchable in one place, publish a two-page decision tree for common exceptions and run weekly 30-minute peer reviews of real disputed files. Employees who cannot find the bank's current tariff sheet do not need a neural network. They need an organised filing cabinet.

Testing the pilot when the autopilot fails.

Aviation regulators do not license commercial pilots by checking whether they can switch the autopilot on. They put pilots in a simulator and fail an engine at night in a crosswind to see what happens when the automation disconnects. Recall the 2023 field experiment with 758 Boston Consulting Group consultants: on a problem designed to sit just outside the model's competence, consultants using AI were 19 percentage points less likely to produce the correct answer than colleagues working unaided, because the machine's persuasive explanation lulled trained professionals into switching off their critical faculties.

To verify whether training survives contact with the desk, banks need to replace multiple-choice quizzes with flight-simulator drills built on synthetic customer files. Hand a newly trained credit analyst five synthetic cases: three routine files where the assistant performs cleanly, one containing a subtle accounting contradiction between the M-PESA statement and the tax return, and one where the model has invented a non-existent Central Bank of Kenya grace period. Grade the employee on whether they caught the planted defect, verified the primary ledger before approving, refused to paste live personal data into an unapproved window in compliance with the Data Protection Act, 2019, and escalated the broken case through the proper risk channel.

Paying for demonstrated competence rather than classroom seats.

External training vendors have a strong commercial incentive to sell broad, entertaining awareness workshops where everyone passes, and human resources gets a glowing satisfaction score. When the same vendor designs the course, administers the test and sells the software licences used in the classroom, the assessment is a marketing brochure rather than an independent audit.

Before granting any employee production access to generative or automated banking tools in 2027, operations, human-resources and risk leads should require an independently graded synthetic-defect drill administered by internal audit, backed by 60-day post-deployment sampling of live work under adjusted KPIs that do not penalise staff for taking extra minutes to verify a suspicious output. At the second-quarter 2027 talent and risk review, strip training hours from the board pack and report the exact percentage of trained staff who caught a planted model hallucination during unannounced desk audits. Keep the attendance certificate in the HR folder, and reserve system permissions for people who prove they can fly the plane when the screen lies.