garak.harnesses.probewise
Probewise harness
Selects detectors to run for each probe based on that probe’s recommendations
- class garak.harnesses.probewise.ProbewiseHarness(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
Harness
- run(model, probenames, evaluator, buff_names=None)
Execute a probe-by-probe scan
Probes are executed in name order. For each probe, the detectors recommended by that probe are loaded and used to provide scores of the results. The detector(s) to be used are determined with the following formula: * if the probe specifies a
primary_detector
;_config.args
is set; and_config.args.extended_detectors
is true; the union ofprimary_detector
andextended_detectors
are used. * if the probe specifices aprimary_detector
and_config.args.extended_detectors
if false, or_config.args
is not set, then only the detector inprimary_detector
is used. * if the probe does not specifyprimary_detector
value, or this isNone
, then detectors are queued based on the from the probe’srecommended_detectors
value; seegarak.probes.base.Probe
for the defaults.- Parameters:
model (garak.generators.base.Generator) – an instantiated generator providing an interface to the model to be examined
probenames (List[str]) – a list of probe names to be run
evaluator (garak.evaluators.base.Evaluator) – an instantiated evaluator for judging detector results
buff_names (List[str]) – a list of buff names to be used this run