garak.probes.snowball

Snowball probe

Checks if the system gives wrong answers to complex reasoning questions, instead of giving the right answer or no answer. Bsaed on

class garak.probes.snowball.GraphConnectivity(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: GraphConnectivityFull, Probe

Probe asking for impossible flight routings

This probe attempts to catch models generating false claims by asking for a solution to an impossible problem, couched as a flight routing question - limited number of examples

active: bool = True
class garak.probes.snowball.GraphConnectivityFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Probe

Probe asking for impossible flight routings

This probe attempts to catch models generating false claims by asking for a solution to an impossible problem, couched as a flight routing question

active: bool = False
bcp47: Iterable[str] | None = 'en'
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = 'trick the model into saying an impossible flight routing is possible'
recommended_detector: Iterable[str] = ['snowball.DontStartYes']
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']
class garak.probes.snowball.Primes(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: PrimesFull, Probe

Ask model to check a list of higher primes

This probe checks if a model is prone to generating false claims by asking if a given number is or isn’t prime. - limited number of examples

active: bool = True
class garak.probes.snowball.PrimesFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Probe

Ask model to check a list of higher primes

This probe checks if a model is prone to generating false claims by asking if a given number is or isn’t prime.

active: bool = False
bcp47: Iterable[str] | None = 'en'
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = "trick the model into saying a prime number isn't prime"
recommended_detector: Iterable[str] = ['snowball.DontStartNo']
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']
class garak.probes.snowball.Senators(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: SenatorsFull, Probe

Asking for senators that don’t exist

This probe attempts to find generators prone to generating false claims by requesting the name of a US senator given constraints for which there isn’t a senator - limited number of examples

active: bool = True
class garak.probes.snowball.SenatorsFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Probe

Asking for senators that don’t exist

This probe attempts to find generators prone to generating false claims by requesting the name of a US senator given constraints for which there isn’t a senator

active: bool = False
bcp47: Iterable[str] | None = 'en'
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = 'trick the model into saying a fictitious senator exists'
recommended_detector: Iterable[str] = ['snowball.DontStartYes']
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']