Mitigating AI risks: Practical steps to AI governance

The rise of AI

In this article, we consider some of the key risks associated with developing and using AI and the practical steps organisations can take to mitigate these potential AI risks. 

Download your copy of our comprehensive Guide to AI Act Compliance.

Guide to AI Act Compliance

In the past 24 months, we have witnessed exponential growth in the development and use of artificial intelligence (“AI”). Rapidly evolving into arguably the most transformative technology of our generation, it is already demonstrating significant potential in many industries, for example, enabling organisations to enhance decision-making, optimise processes and identify new business models.

In the financial sector, organisations are using AI to detect fraud in real time by analysing transaction patterns and deploying it within “Know Your Customer” processes to identify potential issues before new customers are onboarded. We have also seen AI used in the healthcare sector to enhance the accuracy of diagnosis and within the retail industry to personalise customer journeys.

Alongside these advancements, we have seen legislators and rule makers adopt laws such as the EU’s AI Act and governments make statements of intent on how they will govern AI within their jurisdiction; last month, the EU Commission signed the first binding international instrument on AI to address the risks posed by AI to human rights, democracy, and the rule of law. The Biden administration issued an Executive Order on AI in the US outlining principles for responsible development and use of AI. Other countries such as Singapore, Canada and China are also introducing AI regulatory frameworks, which aim to allow innovation to flourish by underpinning it with ethical oversight and accountability. You can also learn more here about what assessments are required under the AU’s AI Act.

Whilst the benefits of AI are vast, its use has the potential to cause serious harm. It is therefore imperative that organisations identify any AI risks at the outset of development and before AI technologies are deployed. Adopting this approach will allow organisations to fully realise the benefits of AI; being better informed and consequently, able to ensure there are appropriate controls in place to mitigate identified potential issues.

Visit our AI Governance Service page to learn how we can support you in building a robust AI Governance framework. 

1. AI Bias and Fairness

AI systems are developed using vast amounts of data which can often contain biases that reflect historical inequalities or prejudices. These biases can often occur in two ways. Firstly, where there is bias in the training or in the input data that is supplied to the model once it is in operation, meaning AI systems can inadvertently perpetuate or amplify biases when they learn from biased datasets. Secondly, the selection factors and the weighting assigned during the design of the underlying system can cause bias to occur.

In a world where we already face many challenges in respect of equality and fairness, this is particularly concerning where AI is used in areas such as recruitment, policing and lending.  In 2018, for example, Thomson Reuters reported that Amazon abandoned an AI recruitment tool that it was developing as it exhibited bias towards men. The test data used to train the AI system largely related to men and therefore discriminated against women, preferring male candidates. This case highlights the importance of using diverse data but also ensuring the developers appreciate the importance of diversity in their development of such tools and bake it into their design processes.

Practical steps to mitigate “AI Bias”

1. Diverse Data Collection: in order to reduce the likelihood of bias occurring, organisations should train their AI systems with diverse and representative datasets. This should mitigate the risk of skewed outcomes that disproportionately affect specific demographics. As such, data should be gathered from reliable sources and be diverse in nature. For instance, data should not be collected from one source and where there are other options available, it should be collected from multiple sources.

Organisations should ensure that teams developing AI tools proactively consider bias and how to minimise its effects. The more diverse the team developing AI, the more likely it is for them to take such an approach.

2. Supplier due diligence: where organisations procure AI tools, they should assess the relevant third party’s approach to dealing with bias. Third parties should be willing to disclose to you how they have considered bias when developing their tools if this information is not readily available in their technical documentation.
3. Bias audits: biases can be identified and mitigated by carrying out regular audits of AI models. Systems should be frequently tested during the development life-cycle, for example, against specific group fairness metrics such as equal treatment across different demographic groups or causal metrics to identify whether changes in protected characteristics such as race/gender impact decision-making.
4. Human oversight: you must include human oversight in AI decision-making processes to mitigate the risks of bias. For instance, humans should be involved in any recruitment processes where AI is used to make decisions and similarly, where AI may be deployed in any law-enforcement context, humans should be involved to review any decisions made.

2. Accuracy and Reliability

We have highlighted above the risk posed by a lack of human oversight in AI decision-making. This risk is exacerbated when you couple it with the likelihood that many users of AI will rely on AI-based decisions without validating their accuracy; it was only last year that US lawyers cited fake cases generated by ChatGPT in a legal brief filed in a federal court, highlighting both the inaccuracies that can derive from the use of AI and the likelihood that users will not always verify outputs generated by AI. This could have seismic consequences in industries such as healthcare where misdiagnoses could have life-threatening consequences or in the financial sector, where those seeking loans could be affected by inaccurate credit checks. Against this backdrop, it is paramount that organisations implement controls to ensure that AI systems are reliable and accurate.

Practical steps to ensure “Accuracy and Reliability”

1. Data validation: organisations should ensure that the data used for training is accurate, up-to-date, and representative of the environments in which the AI system will be used before it is used to develop an AI system. Adopting this approach will mitigate the likelihood of inaccuracies occurring once the system is deployed.
2. Continuous monitoring: AI systems should be consistently monitored and updated to ensure they remain accurate and continue to work in the way expected. This is critical, as the way in which organisations use an AI model and the types of data being inputted may change over time. As a result, continuously monitoring and updating an AI system should ensure it remains aligned with the environment it was initially designed to operate in.
3. Fallback mechanisms: to ensure the robustness, safety and reliability of an AI system, they should be designed with fallback mechanisms. This is critical for high-risk applications, such as in healthcare, finance or autonomous driving. For example, if an autonomous vehicle’s perception system fails to recognise its environment and the AI system is developed without an appropriate fallback, the vehicle should slow down or come to a halt entirely. Again, human oversight is also important to ensure that when system errors occur, a human expert can be looped in to troubleshoot.
4. Board level buy-in: the costs of developing and using AI technologies responsibly can be eye-watering. Organisations may look to press on with the development and implementation of AI solutions without looking at wider implications such as compliance due to the costs involved. Consequently, organisations’ most senior members must be fully bought into using AI in a responsible manner as the controls outlined in this article will come at a cost. It is therefore essential to engage your board (and other key stakeholders) as early as possible in the AI lifecycle and ensure they appreciate the importance of responsible AI deployment. Providing AI literacy training to your board/C-suite will support this and enable you to highlight the benefits of AI from a strategic perspective whilst addressing the technical, practical and ethical aspects of AI.

3. Transparency and Explainability

Transparency and Explainability are key pillars of ethical and responsible AI. One of the significant challenges when developing AI systems is their “black box” nature. ‘Black-box’ AI refers to AI systems that are developed where their internal workings or decision-making processes are so opaque that they are not easily understandable to humans to the extent that even those developing such AI systems are not able to explain their decision-making processes. Effectively, when data is input into a black-box AI system and you receive an output, it is unknown how the AI system arrived at the conclusions or decisions that it presents.

This leads to a lack of transparency and could have significant adverse implications for individuals in areas such as recruitment, lending and criminal justice. For instance, if an individual has been denied a loan by an AI system, it could be challenging to understand the logic of that decision, which  makes it difficult to contest or rectify the outcome.

Practical steps to improve “Transparency and Explainability”

1. Use of explainable AI (“XAI”): there is a balance to be struck when developing AI models as the benefits of using black box AI (such as accuracy) may outweigh those of transparency for the circumstances an AI system is to be deployed in. As such, use case will be very relevant when determining the type of model that should be used for a particular AI model. Consequently, where possible, organisations should build more interpretable AI models, such as linear models or decision trees, rather than more complex deep neural networks which are less transparent. Organisations can also use XAI tools to assist them understanding and interpreting decisions made by AI models. Google, for example, has integrated such tools into a number of its products.
2. Robust documentation: detailed documentation should be maintained by organisations which set out how an AI system was developed through its entire lifecycle. Taking this approach will not only support in providing clarity to stakeholders but also assist with an organisation’s regulatory obligations.
3. Awareness training: it is not enough to only have AI governance policies and procedures in place, these policies and procedures must be operationalised to mitigate potential risks. As such, to enable more informed decision-making and foster trust, organizations must educate their employees on how AI systems work to ensure they understand their decision-making processes and limitations.
4. Ethical guidelines: organisations should adopt ethical guidelines which emphasise the importance of transparency, fairness and accountability when developing and using AI systems. This should encourage teams involved in AI development and deployment to consider the potential risks from the outset.

4. Data Protection and Information Security

AI systems are often trained on vast amounts of data, including personal data. As a result, data protection and information security compliance must be considered when developing and using AI tools. As disciplines, both data protection and information security can be extremely complex areas to navigate, this is only exacerbated when AI is added to the mix. Organisations must therefore underpin their development and use of AI with robust data protection and information security practices to mitigate all the potential risks associated with both disciplines. For example, robust security measures should be embedded in the design of AI systems to minimise their vulnerability to data breaches that could lead to unauthorised access to sensitive information and potential model takeover.

In addition, the lawfulness of processing and data subject rights are other areas which must be considered when developing and using AI systems. Earlier this year, the European Data Protection Board (“EDPB”) published an interim report setting out its efforts and primary findings with respect to the interplay between ChatGPT and its compliance with the General Data Protection Regulation (“GDPR”) GDPR. The report highlighted the challenges for exercising data subject rights and transparency issues that derive from web scraping. The report emphasises the importance of complying with data protection principles where personal data is processed, whilst highlighting that organisations developing AI tools need to ensure that they align their systems to legislation and guidance.

Practical steps to ensure “Data Protection and Information Security”

1. Compliance with data protection regulations: organisations should align their development and use of AI to applicable data protection laws. For example, if you are deploying AI systems in the EU and California, the GDPR and the California Consumer Privacy Act will be relevant so you will have to ensure your deployment meets the stringent requirements for elements such as user consent, transparency and data subject rights.

In March 2023, due to privacy concerns, we saw Garante issue an interim ban on OpenAI’s use of ChatGPT to process Italian users’ personal data, illustrating steps regulatory bodies will take when assessing whether AI systems comply with legislation. Similarly, the ICO issued a statement on September 2024 saying it has not provided Meta with its approval to use Facebook and Instagram user data to train generative AI, and therefore Meta must put safeguards in place to ensure this does not happen, as well as evidencing its compliance on an ongoing basis. Last month, following expression of dissatisfaction by its users, we saw LinkedIn’s ‘U-turn’ on its plans to use UK members’ data for generative AI training, illustrating the weight regulatory compliance can have on organisations’ strategic ambitions. As a result, developers and deployers of AI systems should ensure the features of their products align with regulatory requirements (in the jurisdictions in which they operate) insofar as designing functionality that will allow certain features to be turned on and off in certain jurisdictions.

Organisations should also consult guidance from the likes of The National Institute of Standards and Technology and Organisation for Economic Co-operation and Development as they are highly reputable sources of comprehensive guidance on AI risk management frameworks and can support identifying/mitigating risks associated with AI use and development.

2. Robust security practices: when developing/using AI systems organisations should deploy traditional security controls (e.g. access controls, encryption, logging etc) that will secure their underlying infrastructure but in addition, where appropriate, AI-specific controls such as adversarial testing should be utilised to deal with potential threats that are relevant to AI technologies (and not software applications generally). For example, threat actors may look to tamper with an AI system’s decision-making for malicious purposes (e.g. to change the outcome of financial loan decisions). Embedding AI security measures across an AI system’s lifecycle will guard against such attacks by making the technology more resistant/resilient against threats, with the aim of identifying issues before they arise and implementing controls to mitigate the risks posed by any potential vulnerabilities.
3. Anonymisation: organisations should anonymise data and use encryption where possible to protect personal data used by AI systems. Adopting this approach will mitigate the risks and challenges that come with personal data, including obtaining consent from data subject rights.
4. Data minimisation: the amount of personal data used by AI systems should be limited to only what is necessary to fulfil its purpose. Privacy enhancing techniques such as federated learning and synthetic data can be used to minimise personal data use and reduce the impact of potential security risks.

5. Contractual (& Legal) Risk

Due to the nature of AI and how it is used by organisations, the contractual/legal risks that can arise from it can be complex and novel (as generative AI is a relatively new technology).

Often, organisations procure third-party solutions to develop or deploy AI systems and numerous integrations can take place to develop one AI system, introducing a number of contractual and legal risks. It is therefore critical that organisations fully understand what is included in any AI provider contracts they enter into; for example, are there any disclaimers in respect of outputs generated by AI tools? Organisations should understand where liability lies; for instance, if the output generated by an AI system is not accurate and is subsequently relied upon by your organisation or another third-party you are providing services to – who is responsible for the inaccurate output? The answers to such questions should be clear before any agreements are entered into.

Where integrations are used, it will be necessary to review all the documentation that is relevant to all platforms involved as organisations that integrate with other organisations’ platforms will have varying approaches to data protection and security compliance.

Also, confidentiality issues may arise. For example, law firms are generally subject to stringent confidentiality restrictions when it comes to divulging client information and are often prohibited from sharing client data with third parties. Therefore, they must ensure (1) they do not input any client data into any third-party AI systems without consent and (2) data will not be used to train an AI model for any purposes outside of that law firm’s remit. As a result, organisations must understand what AI providers can do with their data,  as although, in most circumstances, where AI providers agree to not access data, there may be exceptions to this rule, for instance, for monitoring purposes which could lead to authorised access under a provider’s agreement but in turn, this access could breach an organisation’s confidentially obligations within client agreements. For example, earlier this year, a contractual loophole was identified in Microsoft’s Azure OpenAI Service allowing Microsoft employees to review certain prompts. Organisations should therefore carry out a detailed review of all the documentation available to them in relation to procured AI products, not only from the contract itself, but also in other documentation such as privacy notices.

Intellectual property is another area which has highlighted the complexities surrounding AI. As data is pulled from many sources to create generative AI models, questions have arisen around the ownership of content generated by generative AI systems. There are real concerns coming out of the creative industries such as film and art as to the potential adverse effects of AI.. There have already been a number of cases in respect of AI and intellectual property, the outcomes of which, for now, largely suggest it is very difficult to gain rights in materials generated by AI. In addition, ambiguity may arise as to responsibility if an AI system generates incorrect or harmful outputs. As such, organisations involved should know their position on intellectual property before they start using AI systems in order to manage expectations. Therefore, contracts should be clear on intellectual property rights for all use cases for any AI system.

Practical steps to address “Contractual Risks”

1. Comprehensive supplier due diligence: organisations need to ensure they fully appreciate what they are signing up to. AI is not like any other technology and brings its own unique risks and challenges. As such, organisations should thoroughly scrutinise AI provider contracts (and ancillary documents) to ensure they fully understand their relationship with that third party and, crucially, where liability lies. In addition, organisations must vet third-party service providers’ approaches to data protection and security to ensure they have robust measures in place. This due diligence should be on an ongoing basis to ensure nothing changes without your knowledge.
2. Detailed contractual terms: organisations developing AI tools should ensure their contracts are as detailed as possible to remove any ambiguity. Responsibilities regarding key elements such as data ownership, intellectual property, liability for AI-generated outputs, and confidentiality should be clearly set out in AI supplier agreements.
3. Review client agreements: if you are using AI in the provision of your services to other organisations, then you need to know what you are permitted to do under your client contracts. For example, are you allowed to share your client information with third parties? Do you need consent in order to appoint new third-party suppliers that will process your client’s data? Organisations must review their client contracts before inputting any client information into AI systems to ensure that they do not breach any of their contractual obligations.
4. Incident response: be prepared for issues that may arise. AI systems can experience issues such as hallucinations, forgetting and unauthorised access depending on the nature of the AI system. As a result, contracts with third parties should clearly set out what should happen in the event of an incident. Organisations should also have robust policies/procedures in place to respond to AI incidents in a timely manner; being able to respond quickly will mitigate the consequences of any incident.

Monitoring and Auditability

AI systems can experience new risks once they have been deployed or when new data is inserted into the tool’s algorithm. Consequently, whether you are using a third-party AI tool or have developed one in-house, it is imperative that you continue to monitor the output generated by your AI system to continually assess whether it is still working in the way expected/intended. Monitoring involves understanding documented purposes and risks that you have identified during the AI lifecycle.

Having robust audit controls in place, not only allows organisations to demonstrate compliance with legal and ethical standards, but also trace decisions back to specific inputs, models, or datasets. Without strong audibility, organisations will struggle to allocate appropriate resources to a system and find it difficult to identify issues when they arise, such as model drift. This lack of visibility could lead to significant operational, legal and reputational risks.

Practical steps to enhance “Monitoring and Auditability”

1. AI governance framework: Organisations should implement comprehensive compliance frameworks to govern AI throughout its entire lifecycle. These frameworks should include robust policies and procedures with strong oversight of AI systems, clear accountability structures must be embedded, and regular monitoring of AI-generated outputs should take place.
2. Maintain comprehensive logs: AI systems should be designed by organisations to maintain comprehensive logs of their decision-making processes. For example, snapshots of an AI system’s algorithm as well as its outputs should be retained so that if there is an issue, the organisation can return to a previous iteration.
3. Third-party reviews: Organisations should consider engaging third-party auditors to independently assess their AI systems for key elements such as transparency, fairness and accuracy. Taking this approach will provide an objective evaluation and likely, specialist knowledge that organisations lack internally, which will help enhance the performance of their system whilst mitigating the likelihood of inaccuracies and inefficiencies occurring.

Do you require help to create an AI governance framework?

As you can see, there are significant potential risks posed by developing and using AI. However, these risks are not unsurmountable provided appropriate steps are taken to mitigate them. Implementing strong AI governance practices will allow organisations to reap the reputational rewards and benefits that can stem from using AI. We therefore recommend that organisations start reviewing their use of AI, how their use of AI aligns to applicable legislation, the controls they currently have in place to meet such legislation and identifying where the gaps lie. Once this exercise has been carried out, ensure that you implement robust controls for your use case and keep these controls under regular review.

If you would like any help in understanding any of the above or would like to discuss how we can support your organisation from an AI governance perspective, please get in touch using the form below or visit our AI Governance Services page.