garak.evaluators.base

Base evaluators

These describe evaluators for assessing detector results.

class garak.evaluators.base.Evaluator

Bases: object

Class to be subclassed by evaluators.

Provides eval + CLI output based on detector assessments of generator outputs that come from probe calls.

SYMBOL_SET = {1: '🟥', 2: '🟧', 3: '🟨', 4: '🟩', 5: '🟦'}

evaluate(attempts: Iterable[Attempt]) → None

evaluate feedback from detectors expects a list of attempts that correspond to one probe outputs results once per detector

get_z_rating(probe_name, detector_name, asr_pct) → str

print_results_narrow(detector_name, passes, messages)

Print the evaluator’s summary

print_results_wide(detector_name, passes, messages)

Print the evaluator’s summary

test(test_value: float) → bool

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0

class garak.evaluators.base.ThresholdEvaluator(threshold=0.5)

Bases: Evaluator

Evaluator class where a threshold for pass/fail is set

test(test_value)

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0

class garak.evaluators.base.ZeroToleranceEvaluator

Bases: Evaluator

Evaluator class that only lets items pass if the ASR is exactly 0.0

test(test_value)

Function for converting the detector result to a boolean, True=pass override this to customise how evaluator judges

Parameters:

test_value (float) – the value from the detector in 0.0-1.0