garak.generators.rest

Flexible connector for REST-based APIs.

Uses the following options from _config.plugins.generators["rest.RestGenerator"]:

  • uri - (optional) the URI of the REST endpoint; this can also be passed in –model_name

  • name - a short name for this service; defaults to the uri

  • key_env_var - (optional) the name of the environment variable holding an API key, by default REST_API_KEY

  • req_template - a string where the text $KEY is replaced by env var REST_API_KEY’s value (or whatever’s specified in key_env_var) and $INPUT is replaced by the prompt. Default is to just send the input text.

  • req_template_json_object - (optional) the request template as a Python object, to be serialised as a JSON string before replacements

  • method - a string describing the HTTP method, to be passed to the requests module; default “post”.

  • headers - dict describing HTTP headers to be sent with the request

  • proxies - dict passed to requests method call. See required format.

  • response_json - Is the response in JSON format? (bool)

  • response_json_field - (optional) Which field of the response JSON should be used as the output string? Default text. Can also be a JSONPath value, and response_json_field is used as such if it starts with $.

  • request_timeout - How many seconds should we wait before timing out? Default 20

  • ratelimit_codes - Which endpoint HTTP response codes should be caught as indicative of rate limiting and retried? List[int], default [429]

  • skip_codes - Which endpoint HTTP response code should lead to the generation being treated as not possible and skipped for this query. Takes precedence over ratelimit_codes.

  • verify_ssl - (optional) Enforce ssl certificate validation? Default is True, a file path to a CA bundle can be provided. (bool|str)

Templates can be either a string or a JSON-serialisable Python object. Instance of $INPUT here are replaced with the prompt; instances of $KEY are replaced with the specified API key. If no key is needed, just don’t put $KEY in a template.

The $INPUT and $KEY placeholders can also be specified in header values.

If we want to call an endpoint where the API key is defined in the value of an X-Authorization header, sending and receiving JSON where the prompt and response value are both under the text key, we’d define the service using something like:

{
   "rest": {
      "RestGenerator": {
         "name": "example service",
         "uri": "https://example.ai/llm",
         "method": "post",
         "headers": {
            "X-Authorization": "$KEY"
         },
         "req_template_json_object": {
            "text": "$INPUT"
         },
         "response_json": true,
         "response_json_field": "text"
      }
   }
}

NB. response_json_field can also be a JSONPath, for JSON responses where the target text is not in a top level field. It is treated as a JSONPath when response_json_field starts with $.

To use this specification with garak, you can either pass the JSON as a strong option on the command line via --generator_options, or save the JSON definition into a file and pass the filename to --generator_option_file / -G. For example, if we save the above JSON into example_service.json, we can invoke garak as:

garak --model_type rest -G example_service.json

This will load up the default RestGenerator and use the details in the JSON file to connect to the LLM endpoint.

If you need something more flexible, add a new module or class and inherit from RestGenerator.

REST API generator interface

Generic Module for REST API connections

class garak.generators.rest.RestGenerator(uri=None, config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)

Bases: Generator

Generic API interface for REST models

See reference docs for details (https://reference.garak.ai/en/latest/garak.generators.rest.html)

DEFAULT_PARAMS = {'context_len': None, 'headers': {}, 'max_tokens': 150, 'method': 'post', 'proxies': None, 'ratelimit_codes': [429], 'req_template': '$INPUT', 'request_timeout': 20, 'response_json': False, 'response_json_field': None, 'skip_codes': [], 'skip_seq_end': None, 'skip_seq_start': None, 'temperature': None, 'top_k': None, 'verify_ssl': True}
ENV_VAR = 'REST_API_KEY'
generator_family_name = 'REST'