Do generative AI applications such as ChatGPT and DALL-E provide cybercriminals with potentially powerful tools?
It’s been the talk of the town.
No IT industry gathering in the past few months – formal or informal – gets away without someone raising the topic of generative AI.
ChatGPT and the likes have garnered a lot of attention lately due to generative AI’s potential for advancing various fields and allowing people with limited technical expertise to interact with technology in a simple and accessible way.
However, there are concerns about how dangerous it could be – including its ability to generate highly convincing fake text, perpetuate biases and discrimination, and be used for malicious purposes such as cyber-attacks or social engineering.
Are these fears unfounded? Is generative AI such a terrible tool in the hands of cybercriminals?
CybersecAsia finds out from Douglas McKee, Principal Engineer & Director of Vulnerability Research, Trellix:
The cybersecurity industry is rife with discussions about the potential of openly available generative AI in writing functional and convincing malware. But is it true that there are more limitations than potential? Are the fears unfounded?
McKee: While it is possible for generative AI to be used for malicious purposes, there are challenges in making it functional and convincing. In our experience, our researchers attempted to force ChatGPT, a language model, into writing a template that may be used for malware and this resulted in a basic server-client architecture. So, while generative AI can be used to create malware, it still requires human input and guidance to be effective.
Furthermore, many security measures are being developed to detect and counteract malicious software, such as advanced threat detection and machine learning algorithms. Today, language models like ChatGPT have severely limited if not outright blocked the ability for the system to create malware.
Additionally, AI-generated malware may not be able to adapt to changing environments or defenses as quickly as human-written malware. This means that it may be easier for security professionals to detect and mitigate AI-generated malware.
However, it is not entirely accurate to say that the fears surrounding generative AI and malware are unfounded. For instance, new techniques are implemented at a rapid pace by advanced threat actors and defenders race to understand them.
To illustrate, Microsoft’s GitHub Copilot uses OpenAI Codex to suggest functional code by ingesting all public code repositories on GitHub and going through a prediction model. The writer can add comments to the code, and Copilot will suggest functional code.
Unlike ChatGPT, which is designed for conversational text, Copilot is specifically designed for code generation and outshines ChatGPT in offensive tooling capability. Copilot can suggest functional code by providing the intended purpose, which can then be compiled into an executable.
The Copilot extension is available for multiple programming languages, making it versatile for targeted malware generation. However, a skilled author needs to ensure proper functionality. In this regard, it is important to remember that cybersecurity is an ongoing process, and advancements in technology and security measures will continue to evolve to address new threats.
What are the technical challenges that need to be overcome to weaponize generative AI for malicious purposes?
McKee: There are several technical challenges that need to be overcome to weaponize generative AI for malicious purposes. We must understand that generative AI is resource-heavy and requires significant computing resources to train and operate effectively. The presence of data is a key challenge. Generative AI models require large amounts of training data to learn from, and obtaining the necessary data for malicious purposes can be challenging.
Besides that, generative AI models are vulnerable to adversarial attacks, where attackers manipulate input data to cause the model to generate incorrect or malicious output. Developing defenses against adversarial attacks is an ongoing challenge in the field of AI security.
Unlike a cognitive AI model, generative AI models can be opaque, making it difficult to understand how they arrive at their outputs. This lack of transparency can make it difficult to detect and prevent malicious use.
How is GPT-4 set up to reduce the risk? What other possible obstacles may be put in the way of cybercriminals exploiting AI?
McKee: As mentioned previously, attempting to force the language model into writing a template that could be used for malware is currently blocked. But advanced threat actors have constantly demonstrated a level of expertise and finesse needed to complete their goals.
Open AI has cited that they have spent 6 months making GPT-4 safer and GPT-4 is 82% less likely to respond to requests for disallowed content and 40% more likely to produce factual responses.
However, there are additional ways to make it safer. For example, incorporating mechanisms for detecting attacks can be effective to prevent attempts to manipulate or subvert the behavior of AI systems like language models. This could involve incorporating techniques like robust training, and mechanisms like Extended Detection and Response (XDR) to identify threats effectively and proactively.
In several test cases, the malware that was created by ChatGPT was non-functional or immediately detected by Trellix security solutions, demonstrating its lack of uniqueness and creativity required in today’s evolving threat landscape.
The model offered a great blueprint to help understand different methods of implementing software solutions, but it is far from being a viable product for offensive operations. Overall, there are many potential strategies for reducing the risk of exploitation, but the key will be continued collaboration and innovation within the community to continue developing and implementing effective risk mitigation strategies.
What precautions should organizations adopt to prevent malware generated by AI from penetrating their systems?
McKee: The use of AI in creating malware is a growing concern for organisations, and there are many precautions organizations can take to prevent malware generated by AI from penetrating their systems. Implementing a robust security system to detect and block malware generated by AI is one of the most effective methods of preventing cyberattacks.
In addition to this, organizations should conduct regular security audits to identify vulnerabilities in your organization’s systems and processes, which can be exploited by AI-generated malware. Lastly, it is important to conduct training for employees to recognise and avoid phishing scams and suspicious emails.
