run.spec selection grammar

garak/_spec.py implements the unified run.spec selection grammar: a single internal Spec (a list of Selector with explicit polarity) that both transports parse to.

CLI string (comma separated), via parse_spec_string
config file form (YAML/JSON include/exclude lists), via parse_spec_file

Selectors carry a category-prefixed plugin path (probes.<module>[.<Class>], buffs.<module>[.<Class>]), a probe filter (tag:<prefix>, tier:<N|name>), an intent typology code (intent:<code>, or intent:* / intent:all for every intent), or the explicit empty selection none / probes.none (distinct from an unspecified spec, which defaults to probes.*). A leading - excludes; tier:N is inclusive (“log level”: tiers 1..N).

intent: is a separate selection axis consumed by the intent service. When no intent: selector is given, the default scope S (the top-level Safety branch of the intent typology, garak._spec.DEFAULT_INTENT_SCOPE) is injected at resolve time. Typology expansion (child codes) and detectorless filtering happen in the intent service, governed by the run.* intent modifiers (run.serve_detectorless_intents).

garak/_spec.py covers the grammar: parsing and serialisation. Resolving a Spec to concrete plugin names (against active/tier/tag state) is done by garak._selection.resolve_spec; the single plugin-path resolution core is shared with the parse_plugin_spec adapter used for detectors.

See Configuring garak for the user-facing grammar and examples.

Code

garak._spec

Unified run.spec selection grammar.

A single internal Spec (a list of Selector with explicit polarity) is produced by two transports that parse to the same object:

CLI string (comma separated), via parse_spec_string()
config file form (YAML/JSON include/exclude lists), via parse_spec_file()

This module covers the grammar: parsing and serialisation. Resolving a Spec to concrete plugin names (against active/tier/tag state) lives in garak._selection.resolve_spec(); the parse_plugin_spec detector adapter shares the same resolution core.

class Resolution(selected: ~typing.Dict[str, ~typing.List[str]], rejected: ~typing.List[str], inactive: ~typing.List[str] = <factory>, empty_reason: str | None = None, intents: ~typing.List[str] = <factory>, blocked_intents: ~typing.List[str] = <factory>, intents_explicit: bool = False)Source 

Bases: object

Resolved selection.

selected maps each plugin category to its canonical category.module.Class names (e.g. {"probes": [...], "buffs": [...]}); a further category can be added without changing this type. rejected lists unknown selectors; inactive lists bare-module selectors that exist but whose plugins are all inactive (known-but-empty, distinct from unknown - see issue #830); empty_reason is set when the spec resolves to no probes. probes and buffs are convenience accessors over selected.

The intent axis is separate: intents/blocked_intents are raw typology codes (IntentService expands and detectorless-filters them); intents may be the injected DEFAULT_INTENT_SCOPE. intents_explicit flags a user-supplied intent: (vs the default) to warn when no IntentProbe is selected.

blocked_intents: List[str]

property buffs: List[str]Source 

empty_reason: str | None = None

inactive: List[str]

intents: List[str]

intents_explicit: bool = False

property probes: List[str]Source 

rejected: List[str]

selected: Dict[str, List[str]]

class Selector(kind: str, value: str, include: bool = True, category: str | None = None)Source 

Bases: object

A single selection clause.

kind is one of "plugin_path", "none", "tag", "tier" or "intent". For plugin_path and none selectors value carries the category-prefixed token (e.g. "probes.dan" / "probes.none") and category is set; for tag/tier/intent the selector applies to a non-plugin axis and category is None (tag/tier filter probes; intent carries a raw typology code for IntentService). A none selector is an explicit empty selection for its category, distinct from an unspecified spec (which defaults to all active probes at resolve time).

category: str | None = None

include: bool = True

kind: str

value: str

class Spec(include: List[Selector] = <factory>, exclude: List[Selector] = <factory>)Source 

Bases: object

Internal representation of a unified selection spec.

exclude: List[Selector]

include: List[Selector]

to_file_dict() → dictSource : Serialise to the config-file form (include/exclude lists).

legacy_selection_spec(probe_spec: str | None, buff_spec: str | None, probe_tags: str | None) → dict | NoneSource 

Build the run.spec file-form dict from the deprecated selection keys.

Returns None when none of probe_spec/buff_spec/probe_tags carry a meaningful value (absent / empty / auto). This is the single mapping shared by the config-load shim (garak._config._map_legacy_selection()), the CLI flag handling, and the run.spec fixer migration.

parse_spec_file(node: dict | None) → SpecSource 

Parse the config-file transport ({"include": [...], "exclude": [...]}).

List items are either plugin-path strings ("probes.dan") or single-key mappings for filters ({"tag": "owasp:llm01"}, {"tier": 1}).

parse_spec_string(spec: str) → SpecSource 

Parse the CLI string transport into a Spec.

Selectors are comma separated; a leading - excludes, + (or no prefix) includes. Inter-selector whitespace is rejected (outer whitespace is stripped, blank clauses tolerated) so the comma list needs no shell quoting; a * glob still does. Dedup is implicit at resolve time.

validate_intent_specifier(intent_specifier: str) → boolSource : Validate a single intent typology specifier (format only).