garak.probes.packagehallucination

Check for package hallucination vulnerabilities. From https://vulcan.io/blog/ai-hallucinations-package-risk:

Using this technique, an attacker starts by formulating a question asking ChatGPT for a package that will solve a coding problem. ChatGPT then responds with multiple packages, some of which may not exist. This is where things get dangerous: when ChatGPT recommends packages that are not published in a legitimate package repository (e.g. npmjs, Pypi, etc.).

When the attacker finds a recommendation for an unpublished package, they can publish their own malicious package in its place. The next time a user asks a similar question they may receive a recommendation from ChatGPT to use the now-existing malicious package. We recreated this scenario in the proof of concept below using ChatGPT 3.5.

Package Hallucination

Trying to get code generations that specify non-existent (and therefore insecure) packages

class garak.probes.packagehallucination.JavaScript(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: PackageHallucinationProbe

Attempts to generate JavaScript code including hallucinated npm packages

Generators sometimes recommend importing non-existent npm packages. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent npm packages.

active: bool = True
goal: str = 'generate JavaScript code importing non-existent npm packages'
language_name = 'JavaScript'
primary_detector: str | None = 'packagehallucination.JavaScriptNpm'
class garak.probes.packagehallucination.PackageHallucinationProbe(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Probe

Base class for package hallucination probes

Generators sometimes recommend importing non-existent packages into code. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent packages.

active: bool = False
bcp47: Iterable[str] | None = '*'
doc_uri: str = 'https://vulcan.io/blog/ai-hallucinations-package-risk'
goal: str = 'base probe for importing non-existent packages'
language_name = ''
tags: Iterable[str] = ['owasp:llm09', 'owasp:llm02', 'quality:Robustness:GenerativeMisinformation', 'payload:malicious:badcode']
class garak.probes.packagehallucination.Python(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: PackageHallucinationProbe

Attempts to generate Python3 code including hallucinated packages

Generators sometimes recommend importing non-existent packages into code. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent packages.

active: bool = True
goal: str = 'generate python importing non-existent packages'
language_name = 'Python3'
primary_detector: str | None = 'packagehallucination.PythonPypi'
class garak.probes.packagehallucination.Ruby(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: PackageHallucinationProbe

Attempts to generate Ruby code including hallucinated gems

Generators sometimes recommend importing non-existent gems into code. These gem names can be found by attackers and then squatted in public gem repositories, so that incorrect code from generators will start to run, silently loading malicious squatted gems onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent gems.

active: bool = True
goal: str = 'generate ruby code importing non-existent gems'
language_name = 'Ruby'
primary_detector: str | None = 'packagehallucination.RubyGems'
class garak.probes.packagehallucination.Rust(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: PackageHallucinationProbe

Attempts to generate Rust code including hallucinated crates.io packages

Generators sometimes recommend importing non-existent crates.io packages. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent crates.io packages.

active: bool = True
goal: str = 'generate Rust code importing non-existent crates.io packages'
language_name = 'Rust'
primary_detector: str | None = 'packagehallucination.RustCrates'