At Bugcrowd, we believe that the human ingenuity unleashed by crowdsourced security is the best tool available for meeting AI security goals in a scalable, impactful, and economically sensitive way.
Just a few weeks ago, we announced incremental updates to the Vulnerability Rating Taxonomy (VRT), an ongoing effort to define and prioritize vulnerabilities in a standard, community-driven way so that hackers and customers alike can participate in the process. Since 2016, the Bugcrowd Platform has incorporated the VRT alongside a CVSS conversion tool. This integration has enabled us to validate and triage hundreds of thousands of submissions from the crowd at scale. Our platform’s rigorous validation process ensures that these submissions are consistently recognized as valid vulnerabilities by program owners.
The VRT is designed to constantly evolve in order to mirror the current threat environment, and thus helps bug bounty program owners create economic incentive models that engage and motivate hackers to spend their valuable time looking for the right things with the expectation of a fair reward. Now, with the mainstreaming of generative AI and the appearance of government regulation including Executive Order 14410 and the EU Artificial Intelligence Act, it’s time for the VRT to take another evolutionary step to account for vulns in AI systems, particularly in Large Language Models (LLMs).
With these AI security-related updates to the VRT (and more to come) and our experience working with AI leaders like OpenAI, Anthropic, Google, the U.S. Department of Defense’s Chief Digital and Artificial Intelligence Office, and the Office of the National Cyber Director, the Bugcrowd Platform is positioned as the leading option for meeting that goal.
Bringing AI security into the ecosystem
Although AI systems can have well-known vulnerabilities that Bugcrowd sees in common web applications (such as IDOR and Broken Access Control vulns), AI technologies like LLMs also introduce unprecedented security challenges that our industry is only beginning to understand and document, just as we had to contend with new classes of vulnerabilities introduced by mobile technology, cloud computing, and APIs.
For that reason, our announcement today of VRT version 1.12 is a milestone in the crowdsourced cybersecurity industry: For the first time, customers and hackers will have a shared understanding of how the most likely emerging LLM-related vulnerabilities are defined and should be prioritized for reward and remediation. With this information, hackers can focus on hunting for specific vulns and creating targeted POCs, and program owners with LLM-related assets can design scope and rewards that produce the best outcomes.
In the interest of alignment with industry-standard definitions, the updates below overlap with the OWASP Top 10 for Large Language Model Applications. Special thanks to Ads Dawson, a senior security engineer for LLM platform provider Cohere and a core team member of the OWASP LLM Top 10 project, for inspiring these updates and his contributions to VRT v1.12!
What’s inside VRT v1.12?
Update to existing category:
- Varies: Application Level DoS > Excessive Resource Consumption – Injection (Prompt)
In the context of LLMs, application-level DoS attacks take the form of engineered prompts that can crash the client or otherwise make it unusable by others. When the LLM is integrated with other systems, the damage can also spread beyond the application.
New “AI Application Security” category and “Large Language Model (LLM) Security” subcategory:
- P1: AI Application Security > Large Language Model (LLM) Security > Prompt Injection
In “Prompt Injection”, an attacker manipulates the prompt in a way that causes the LLM to behave maliciously – such as by jailbreaking via “Do Anything Now” (DAN), developer mode, and roleplaying.
- P1: AI Application Security > Large Language Model (LLM) Security > LLM Output Handling
As LLMs become more common, there is a risk that LLM output will be accepted unconditionally by other applications. This can introduce vulnerabilities that invite Cross-Site Scripting (XSS) attacks, privilege escalation, and more.
- P1: AI Application Security > Large Language Model (LLM) Security > Training Data Poisoning
In Data (or Model) Poisoning attacks, a threat actor gets their input or prompts to influence the model for nefarious purposes. Model Skewing–in which the attacker attempts to pollute training data to confuse the model about what is “good” or “bad” – is among the most common Data Poisoning techniques.
- P2: AI Application Security > Large Language Model (LLM) Security > Excessive Agency/Permission Manipulation
Excessive Agency flaws are ones in which an LLM has more functionality, permissions, or autonomy than is intended, enabling an attacker to manipulate it to reveal sensitive data (including its own source code) or do other unexpected tasks. When the model is integrated with other systems, those flaws can have particularly dangerous consequences such as privilege escalation. (Note: Data leakage, which we expect to become a common LLM vulnerability in itself, is accounted for in the VRT’s existing “Disclosure of Secrets” category.)
According to Ads Dawson, “The main intention of this VRT update is to capture the correlation between LLM/AI-based vulnerabilities inline and application-based taxonomies – because one associated risk within each realm can trigger a downstream exploit, and vice versa. This not only opens up a new form of offensive security research and red teaming to program participants, but helps companies increase their scope to include these additional attack vectors inline with security researcher testing, receiving submissions, and introducing mitigations to secure their applications. I am looking forward to seeing how this VRT release will influence researchers and companies looking to fortify their defenses against these newly introduced attack concepts.”
Contributions needed!
This update represents our first step in recognizing these attack vectors within the VRT, but is far from the last–the VRT and these categories will evolve over time as hackers, Bugcrowd application security engineers, and customers actively participate in the process. If you would like to contribute to the VRT, Issues and Pull Requests are most welcome!
