garak.probes.snowball

Snowball probe

Checks if the system gives wrong answers to complex reasoning questions, instead of giving the right answer or no answer.

Probes here should be based on the Snowball paper.

Further info:

class GraphConnectivity(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: GraphConnectivityFull, Probe

Probe asking for impossible flight routings

This probe attempts to catch models generating false claims by asking for a solution to an impossible problem, couched as a flight routing question - limited number of examples

active: bool = True
class GraphConnectivityFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: Probe

Probe asking for impossible flight routings

This probe attempts to catch models generating false claims by asking for a solution to an impossible problem, couched as a flight routing question

active: bool = False
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = 'trick the model into saying an impossible flight routing is possible'
lang: str | None = 'en'
primary_detector: str | None = 'snowball.DontStartYes'
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']
tier: Tier = 2
class Primes(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: PrimesFull, Probe

Ask model to check a list of higher primes

This probe checks if a model is prone to generating false claims by asking if a given number is or isn’t prime. - limited number of examples

active: bool = False
class PrimesFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: Probe

Ask model to check a list of higher primes

This probe checks if a model is prone to generating false claims by asking if a given number is or isn’t prime.

active: bool = False
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = "trick the model into saying a prime number isn't prime"
lang: str | None = 'en'
primary_detector: str | None = 'snowball.DontStartNo'
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']
tier: Tier = 3
class Senators(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: SenatorsFull, Probe

Asking for senators that don’t exist

This probe attempts to find generators prone to generating false claims by requesting the name of a US senator given constraints for which there isn’t a senator - limited number of examples

active: bool = False
class SenatorsFull(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: Probe

Asking for senators that don’t exist

This probe attempts to find generators prone to generating false claims by requesting the name of a US senator given constraints for which there isn’t a senator

active: bool = False
doc_uri: str = 'https://github.com/Nanami18/Snowballed_Hallucination'
goal: str = 'trick the model into saying a fictitious senator exists'
lang: str | None = 'en'
primary_detector: str | None = 'snowball.DontStartYes'
tags: Iterable[str] = ['avid-effect:performance:P0204', 'owasp:llm09', 'quality:Robustness:GenerativeMisinformation', 'payload:hallucination']
tier: Tier = 3