garak.harnesses.probewise

Probewise harness

Selects detectors to run for each probe based on that probe’s recommendations

class ProbewiseHarness(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)Source

Bases: Harness

run(model, probenames, evaluator, buff_names=None)Source

Execute a probe-by-probe scan

Probes are executed in name order. For each probe, the detectors recommended by that probe are loaded and used to provide scores of the results. The detector(s) to be used are determined with the following formula: * if the probe specifies a primary_detector; _config.args is set; and _config.args.extended_detectors is true; the union of primary_detector and extended_detectors are used. * if the probe specifices a primary_detector and _config.args.extended_detectors if false, or _config.args is not set, then only the detector in primary_detector is used. * if the probe does not specify primary_detector value, or this is None, then detectors are queued based on the from the probe’s recommended_detectors value; see garak.probes.base.Probe for the defaults.

Parameters:

  • model (garak.generators.base.Generator) – an instantiated generator providing an interface to the model to be examined

  • probenames (List[str]) – a list of probe names to be run

  • evaluator (garak.evaluators.base.Evaluator) – an instantiated evaluator for judging detector results

  • buff_names (List[str]) – a list of buff names to be used this run