garak.evaluators.base

Base evaluators

These describe evaluators for assessing detector results.

class garak.evaluators.base.Evaluator

Bases: object

Class to be subclassed by evaluators.

Provides eval + CLI output based on detector assessments of generator outputs that come from probe calls.

evaluate(attempts: List[Attempt]) None

evaluate feedback from detectors expects a list of attempts that correspond to one probe outputs results once per detector

print_results_narrow(detector_name, passes, messages)

Print the evaluator’s summary

print_results_wide(detector_name, passes, messages)

Print the evaluator’s summary

test(test_value: float) bool

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0

class garak.evaluators.base.ThresholdEvaluator(threshold=0.5)

Bases: Evaluator

Evaluator class where a threshold for pass/fail is set

test(test_value)

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0

class garak.evaluators.base.ZeroToleranceEvaluator

Bases: Evaluator

Evaluator class that only lets items pass if the score is exactly 0.0

test(test_value)

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0