What is AI security?

AI security is the practice of protecting AI models, their training data and the pipelines around them from attack. It differs from conventional application security because the model itself is an attack surface: it can be poisoned during training, manipulated at inference, extracted through its API, or made to leak the data it was trained on. The main threat classes are set out in AI attack techniques, and the governing frameworks are compared in the AI security framework guide.

Since ChatGPT brought the world of generative artificial intelligence crashing into our lives in November 2022, vibrant discussions with my fellow security professionals have become a staple. One topic prompting debate surrounds the term "AI security" - it seems we each bring our own understanding of its meaning. I've identified three prevailing definitions:

The most common definition conceptualizes AI security as the controls protecting information technology systems embedded with artificial intelligence from attacks that could compromise functionality or sensitive organizational data. AI Security is therefore about "Securing AI-Powered Systems" or "security for AI systems"

A second definition centers on security solutions harnessing AI's machine language and deep learning capabilities to protect organizations from cyber attacks. Methods such as anomaly detection and predictive modelling thwart evolving threats in real time. These are "AI Security Solutions."

Some of my colleagues also added that there is a need for policy guidance governing employees responsible use of generative tools such as ChatGPT. I define these as "AI Governance."

AI Security refers to the measures taken to protect IT systems that use AI technology. It involves using cybersecurity solutions that leverage machine learning to analyze large amounts of data, providing organizations with enhanced protection. Additionally, AI security is about ensuring the responsible and safe use of AI within an organization through proper governance.

While nuances differ, our overarching aim aligns - to protect our organization's assets as we leverage AI technology. As security specialists, irrespective of taxonomy or wording, all perspectives around AI protection hold merit. AI security is about having a comprehensive approach to include all of the above in a solid AI security program.

Let's dive now deeper in the fascinating world where AI intersects with Security.

Securing ai-powered systems

The increasing prevalence of Artificial Intelligence (AI) systems across industries have brought about numerous benefits, such as accelerated content creation, data analysis and categorization, code review, and significant workflow improvement.

However, alongside these advancements, there are also concerns regarding potential AI risks that need to be addressed before deploying these AI systems.

Some are of these risks are related directly to security , whereas others are related to bias and transparency.

When implementing large language models and algorithms in business processes, such as automating customer service in a bank, it becomes crucial to ensure their proper security.

AI systems are susceptible to specific attacks, including input manipulation, data poisoning, and model inversion attacks.

In this context, AI security refers to the measures and controls employed to safeguard IT systems from potential attacks that could compromise functionality or sensitive data. In addition, privacy is particularly important when it comes to AI, as it involves protecting personal information from unauthorized access and ensuring its confidentiality and integrity.

To illustrate a potential threat, imagine a scenario where a threat actor manipulates a bank's automated customer service chatbot to gain access to personal information about other clients. Without proper controls, the attacker may even exploit the chatbot to initiate unauthorized cash transfers to their own accounts.

Data poisoning is another technique employed by attackers involving the manipulation of training data used for machine learning models. For example, an attacker may modify the data used to train a fraud detection model in a bank, making it more likely to approve fraudulent transactions. This attack can be successfully executed by combining other vulnerabilities in the AI ecosystem or by an insider who has access to the training data.

Another widespread attack is "prompt injection," where an attacker passes malicious instructions to the model to affect its behaviour and perform unauthorized actions. For instance, they might instruct the model to ignore previous instructions and provide a list of clients with high account balances.

Despite appearing simplistic, these attacks have been successfully demonstrated at DefCon 2023. Below is an example of a variation of a prompt injection attack where an evil model injects a compromised link during a casual interaction with the user. The Amazon voucher link is a fake one.

Source : Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection

AI SECURITY solutions

Let's now delve into the realm of AI Security Systems. In response to the growing concerns surrounding AI security risks, numerous cybersecurity firms have explicitly designed systems to address these challenges. Many of these solutions have been harnessing the power of large language models (LLM) and deep learning to improve security for several years now. 

Here are a few examples of how Artificial Intelligence can be leveraged in the cyber domain:

1. Advanced Threat Detection: AI algorithms analyze vast amounts of data to identify patterns and anomalies that may indicate potential threats, enabling proactive detection.
2. Real-time Monitoring: AI-powered security solutions continuously monitor networks and endpoints, swiftly detecting and responding to suspicious activities in real-time.
3. Automated Incident Response: AI automates incident response processes, enhancing the speed and efficiency of mitigating cyber threats.
4. User Behaviour Analysis: AI algorithms scrutinize user behaviour patterns, detecting any deviations or unusual activities that could indicate unauthorized access or malicious intent.
5. Malware Detection and Prevention: AI algorithms swiftly identify, alert and block malware, even unknown or zero-day threats, bolstering an organization's defence against cyber attacks.

AI therefore enables security analysts to automate their existing security processes and practices like threat intelligence, threat detection and vulnerability management. It allows them to focus on more complex strategic tasks like liaising with authorities when responding to a breach. The security team's use of deep learning techniques and other AI models enhance their response and reduce the impact of a breach. 

Today Cybersecurity professionals use AI in the security operations center (SOC) for automation for analyzing large volumes of data, triaging security alerts and blocking security threats. AI systems can provide relevant context from diverse sources for response and prioritization reducing false positives to security alerts.

Implementing these AI security solutions can significantly help organizations improve their risk management practices and proactively stay ahead of potential threats. However, it's crucial to exercise due diligence when considering these solutions.

While many vendors claim AI powers their offerings, it's essential to fully understand how AI is integrated, the underlying algorithms and models leveraged, and how they provide a competitive edge.

Moreover, as I consistently emphasize to my team, it's essential to recognize that merely implementing a technical solution is insufficient for comprehensive security.

Other aspects, such as establishing proper processes, providing security training, and implementing effective governance measures, must complement any security product or solution.

AI Governance

Integrating AI in businesses highlights the need for defining AI governance principles and developing a robust governance framework. Senior leaders and board members need to recognize the dangers of AI and take action to make it safe and secure. The organization must prioritize oversight, allocate budgets, and set policies to ensure AI is used properly.

As employees start using Generative AI tools like ChatGPT and Bing Chat, organizations need to develop a security framework including security measures, AI security standards and an AI Acceptable Use Policy They need to provide clear guidelines and define security practices on what employees can and are not allowed to do. For example, end users should be careful not to upload sensitive or personal information. Understanding AI will be a key success factor in leveraging this technology.

On the other hand, we have already started to see AI threat actors using the same tool for social engineering purposes, particularly through the generation of convincing phishing emails and the creation of voice and video deep fakes. Below is an example of a Business Compromise Email (BCE). I have seen and investigated many BCEs, but this one is just another level and could fool even experienced security professionals.

Source : Slashnext

In addition we have seen hackers are leveraging generative AI for malicious purposes and coming up with tools like WormGPT and FraudGPT.

Users must exercise heightened vigilance, as AI-generated content can be exceptionally persuasive due to its high quality. Your security awareness training program needs to be updated to reflect this new threat landscape powered by AI. We are also seeing the emergence of some AI governance tools in the industry that may be helpful.

AI regulations

As part of an overall AI strategy, your organizations should also closely monitor upcoming legislations like the EU AI act and other AI regulations in your specific industry and jurisdictions. 

Adherence to these legislations will be paramount and will ensure that AI security and privacy practices align with industry best practices and provide protection against potential legal and financial consequences.

Future of AI in cybersecurity

AI has the potential to increase organizations' security posture and their cyber resilience. From fast detection and automated response to risk prediction and increased productivity, machine learning and artificial intelligence will drastically change how cybersecurity teams work.

AI threat actors will also be leveraging this technology. We will see more fraudulent activities fueled by the use of AI like BCE and the emergence of more blackhats models like WormGPT. Many security firms and startups will also be coming up with innovative AI software systems to address security at different levels including endpoint security and at the security infrastructure level.

AI also offers an exceptional opportunity for organizations to transform their business and benefit significantly from this promising technology. Establishing a comprehensive governance framework and securing AI systems, including implementing AI cyber security solutions, can help organizations proactively address the security risks associated with AI yet benefit from this technological advancement. I urge you to stay abreast of this fascinating world of AI security. This is only the beginning of a new era for cyber security.