report (AVID)

garak’s reports connect to things interested in consuming info on LLM vulnerabilities and failures, such as the AI Vulnerability Database.

garak provides a CLI option to further structure this file for downstream consumption. The open data schema of AI vulnerability Database (AVID) is used for this purpose.

The syntax for this is as follows:

python3 -m garak -r <path_to_file>

Examples

As an example, let’s load up a garak report from scanning gpt-3.5-turbo-0613.

wget https://gist.githubusercontent.com/shubhobm/9fa52d71c8bb36bfb888eee2ba3d18f2/raw/ef1808e6d3b26002d9b046e6c120d438adf49008/gpt35-0906.report.jsonl
python3 -m garak -r gpt35-0906.report.jsonl

This produces the following output.

📜 Converting garak reports gpt35-0906.report.jsonl
📜 AVID reports generated at gpt35-0906.avid.jsonl

Defines the Report class and associated functions to process and export a native garak report

class Report(report_location, records=None, metadata=None, evaluations=None, scores=None)Source

Bases: object

A class defining a generic report object to store information in a garak report (typically named garak.<uuid4>.report.jsonl).

Parameters:

  • report_location (str) – location where the file is stored.

  • records (List[dict]) – list of raw json records in the report file

  • metadata (dict) – report metadata, storing information about scanned model

  • evaluations (pd.DataFrame) – evaluation information at probe level

  • scores (pd.DataFrame) – average pass percentage per probe

  • write_location (str) – location where the output is written out.

export()Source

Writes out output in a specified format.

get_evaluations()Source

Extracts evaluation information from a garak report.

load()Source

Loads a garak report.