garak.harnesses.base

Base harness

A harness coordinates running probes on a generator, running detectors on the outputs, and evaluating the results.

This module ncludes the class Harness, which all garak harnesses must inherit from.

class garak.harnesses.base.Harness(config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Configurable

Class to manage the whole process of probing, detecting and evaluating

DEFAULT_PARAMS = {'strict_modality_match': False}
active = True
run(model, probes, detectors, evaluator, announce_probe=True) None

Core harness method

Parameters:
  • model (garak.generators.Generator) – an instantiated generator providing an interface to the model to be examined

  • probes (List[garak.probes.base.Probe]) – a list of probe instances to be run

  • detectors (List[garak.detectors.base.Detector]) – a list of detectors to use on the results of the probes

  • evaluator (garak.evaluators.base.Evaluator) – an instantiated evaluator for judging detector results

  • announce_probe (bool, optional) – Should we print probe loading messages?