AI Governance: Choosing the Right Controls for Your Business

Start with the decision and a common data baseline

A supplier presents strong test results. Your business wants to use the AI system to prioritise customers, route service requests or recommend an action. Before approving it, you need to know whether those results support the use you have in mind.

That review starts with what could go wrong and who could be harmed. Would the proposed tests detect the problem? Who could intervene if it occurred in operation?

The answers depend on the decision, the people affected and the actions the system may take. The NIST AI Risk Management Framework connects evaluation to deployment context and calls for monitoring and intervention arrangements.[1] Every review also needs a common data baseline: provenance, permitted use, access, versioning and suitability. Dataset documentation should explain collection, composition and intended uses, as the Datasheets for Datasets proposal recommends.[2]

Then ask what the system learned from. Labels, discovered patterns and rewards introduce different assumptions to test. These learning paradigms are distinct from model architecture: deep neural networks can use different learning signals. Neither dimension determines how much authority the system should receive. That depends on the consequences of its proposed use and the evidence available.

Supervised learning: does the target represent the outcome you need?

In supervised learning, examples come with a target the model is trained to predict. Reviewing the accuracy of those labels is necessary, but the target itself also needs scrutiny.

A 2019 study by Obermeyer and colleagues examined a healthcare algorithm that predicted cost as a proxy for health need. Unequal access to care meant lower spending on Black patients, so predicting cost could understate their needs. Predictive performance against the chosen target did not settle whether that target served the intended decision.[3]

The business owner therefore needs to explain how the target relates to the outcome sought. That means examining how labels were created, whose outcomes are missing and whether historical decisions are being treated as evidence of what should happen next.

Useful evaluation starts with the errors that matter in the proposed use. A missed urgent case and an unnecessary escalation may have very different consequences. Review performance across relevant groups, with enough observations to interpret differences and uncertainty. Request this evidence from the developer and check it against the population you intend to serve.

Agree review thresholds before deployment and monitor errors as real outcomes become available, allowing for delays. A changed target definition or unexplained deterioration should trigger investigation and a review of permitted use. Diagnose the problem before deciding whether the remedy is better data, a different target, restricted use or retraining.

Unsupervised learning: what makes a pattern actionable?

Unsupervised methods can reveal structure without supplied outcome labels. In clustering, evaluation may compare groups with reference classes or assess properties such as internal coherence. Those measures have assumptions and limitations, as the scikit-learn documentation explains.[4]

The governance question begins when someone gives that structure business meaning. Imagine customer clusters that a manager names "low value", then proposes using to reduce service. The grouping and the decision to provide less service require separate justification.

We would ask a domain specialist to validate the interpretation against actual cases. Check whether the grouping changes materially with the data window, feature scaling or algorithm settings. Review who would be affected and the consequences of assigning someone to the wrong group. Stable clusters can still support an unsuitable decision.

The proposed use determines how far that validation needs to go. Exploring a segmentation is different from automatically excluding customers. Where evidence does not justify the consequential action, keep the output exploratory or limit its authority.

In operation, track changes in membership, reviewed errors and downstream effects. Record the number of cases reviewed alongside error and override rates; a low override rate says little if nobody meaningfully checks the decisions.

Reinforcement learning: can the reward rise while outcomes worsen?

Reinforcement learning trains an action policy using a reward signal. The control question is whether improving that reward could undermine the intended outcome. Research on AI safety identifies reward hacking, unwanted side effects, unsafe exploration and distributional shift as distinct concerns.[5]

Consider a hypothetical service-routing policy rewarded for shorter handling time. Transferring difficult cases repeatedly might improve the measured time while leaving customers without an answer. Test that possibility explicitly. Track resolution or repeat contacts separately from the reward, and define actions that remain unacceptable even when they improve the score.

Reward design is one part of the review. Another is when learning happens. Policies can be trained on offline data before deployment; reinforcement learning does not necessarily mean the system keeps learning in production. Research on offline deployment distinguishes using trained models from subsequently selecting or updating them with feedback.[6]

A fixed policy, a policy updated through controlled releases and one learning during operation need different change controls. Establish which applies to the system and how each version is identified.

Before allowing the system to act, test reward failure scenarios, action constraints and the ability to intervene. Use simulation where appropriate, then examine its limits against real operating conditions. If the system takes a prohibited action, suspend the relevant automation and use a tested fallback. Name who can do this, how quickly they must respond and what evidence is required before operation resumes.

LLMs: training history and explanations provide partial evidence

For large language models, ask which training stages actually apply. Self-supervised learning derives prediction targets from the data itself, such as predicting missing text.[7] Subsequent stages may use demonstrations or preferences. Direct Preference Optimization, for example, offers an alternative to the conventional pipeline of training a reward model and then optimising against it with reinforcement learning.[8] A supplier's description of the actual model is more useful than assuming a universal sequence.

A hybrid system may therefore need several kinds of assurance. Request the disclosed training stages, evaluations and known limits, then test the business task locally. Where supplier details are unavailable, record the gap and decide whether local evidence and operating restrictions are sufficient for the proposed use.

Explanations need scrutiny too. In May 2024, Anthropic reported identifying interpretable features in Claude 3 Sonnet and changing behaviour by manipulating them. Its own account cautioned that identifying representations did not explain how the model used them or establish that the findings could improve safety.[9]

Ask who needs an explanation, what decision it supports and how its accuracy will be assessed. NIST's explainability principles distinguish an understandable explanation from one that correctly reflects the system's process.[10] A convincing account alone cannot justify granting more authority.

For the deployed system, include retrieval, tools, permissions and escalation in the evaluation. A model's training history cannot establish that these components work together acceptably in your workflow. Record versions and repeat relevant tests after changes. Different answers following a prompt or retrieval change do not, by themselves, show that the model has learned in operation.

Record the evidence, owner and intervention before approval

The comparison below brings these checks together to help select questions for a particular system. It is CrmWare's practical synthesis, not a validated scoring framework or an exhaustive governance checklist.

For each proposed use, keep one decision record:

1. System and learning stages: the model version, relevant components and known gaps in the training history.
2. Failure and consequence: what could go wrong, who could be affected and which actions require restriction.
3. Evidence and acceptance: the test, population, acceptance criterion and explanation needed, where relevant. Turn lessons from verified, relevant incidents into validation scenarios, specifying which failure each scenario tests and what response is acceptable.
4. Owners: who supplies the evidence, operates the control and independently reviews its adequacy.
5. Monitoring: the measure, denominator, review interval and delay before real outcomes become observable.
6. Intervention: the escalation condition, authorised person, tested fallback and conditions for resuming operation.

The developer can supply training and validation evidence. The organisation deploying the system must decide whether that evidence supports its local use and implement the operating constraints. NIST explicitly addresses third-party monitoring and assigned responsibilities to disengage systems.[1] An independent reviewer assesses those arrangements; they do not become the operator.

If the supplier retains the only practical ability to investigate or intervene, examine which control rights remain yours. Keep the evidence of acceptable operation distinct from proof that the business gained an operational advantage.

The resulting decision may be approval within defined limits, a narrower use, further validation or a decision not to proceed. Where evidence is incomplete, make the consequence of that uncertainty explicit before assigning more responsibility to the system.

References

[1] NIST. Artificial Intelligence Risk Management Framework (AI RMF 1.0). 2023. https://nvlpubs.nist.gov/nistpubs/ai/NIST.AI.100-1.pdf

[2] Timnit Gebru and co-authors; Communications of the ACM. Datasheets for Datasets. 2021. https://arxiv.org/abs/1803.09010

[3] Ziad Obermeyer, Brian Powers, Christine Vogeli and Sendhil Mullainathan; Science. Dissecting racial bias in an algorithm used to manage the health of populations. 2019. https://pubmed.ncbi.nlm.nih.gov/31649194/

[4] scikit-learn developers. Clustering: clustering performance evaluation (documentation, version 1.9.1 inspected). n.d.. https://scikit-learn.org/stable/modules/clustering.html#clustering-performance-evaluation

[5] Dario Amodei and co-authors. Concrete Problems in AI Safety. 2016. https://arxiv.org/abs/1606.06565

[6] Ziniu Li and co-authors. Deploying Offline Reinforcement Learning with Human Feedback. 2023. https://arxiv.org/abs/2303.07046

[7] Facebook AI Research (now Meta). Self-supervised learning: The dark matter of intelligence. 2021. https://ai.meta.com/blog/self-supervised-learning-the-dark-matter-of-intelligence/

[8] Rafael Rafailov and co-authors. Direct Preference Optimization: Your Language Model is Secretly a Reward Model. 2023. https://arxiv.org/abs/2305.18290

[9] Anthropic. Mapping the mind of a large language model. 2024. https://www.anthropic.com/research/mapping-mind-language-model

[10] NIST. Four Principles of Explainable Artificial Intelligence. 2021. https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8312.pdf