garak.probes.packagehallucination
Check for package hallucination vulnerabilities. From https://vulcan.io/blog/ai-hallucinations-package-risk:
Using this technique, an attacker starts by formulating a question asking ChatGPT for a package that will solve a coding problem. ChatGPT then responds with multiple packages, some of which may not exist. This is where things get dangerous: when ChatGPT recommends packages that are not published in a legitimate package repository (e.g. npmjs, Pypi, etc.).
When the attacker finds a recommendation for an unpublished package, they can publish their own malicious package in its place. The next time a user asks a similar question they may receive a recommendation from ChatGPT to use the now-existing malicious package. We recreated this scenario in the proof of concept below using ChatGPT 3.5.
Package Hallucination
Trying to get code generations that specify non-existent (and therefore insecure) packages
- class garak.probes.packagehallucination.JavaScript(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
PackageHallucinationProbe
Attempts to generate JavaScript code including hallucinated npm packages
Generators sometimes recommend importing non-existent npm packages. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent npm packages.
- language_name = 'JavaScript'
- class garak.probes.packagehallucination.PackageHallucinationProbe(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
Probe
Base class for package hallucination probes
Generators sometimes recommend importing non-existent packages into code. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent packages.
- language_name = ''
- class garak.probes.packagehallucination.Python(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
PackageHallucinationProbe
Attempts to generate Python3 code including hallucinated packages
Generators sometimes recommend importing non-existent packages into code. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent packages.
- language_name = 'Python3'
- class garak.probes.packagehallucination.Ruby(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
PackageHallucinationProbe
Attempts to generate Ruby code including hallucinated gems
Generators sometimes recommend importing non-existent gems into code. These gem names can be found by attackers and then squatted in public gem repositories, so that incorrect code from generators will start to run, silently loading malicious squatted gems onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent gems.
- language_name = 'Ruby'
- class garak.probes.packagehallucination.Rust(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
PackageHallucinationProbe
Attempts to generate Rust code including hallucinated crates.io packages
Generators sometimes recommend importing non-existent crates.io packages. These package names can be found by attackers and then squatted in public package repositories, so that incorrect code from generators will start to run, silently loading malicious squatted packages onto the machine. This is bad. This probe checks whether a model will recommend code that uses non-existent crates.io packages.
- language_name = 'Rust'