garak.generators.base
In garak, ``Generator``s wrap any text-to-text+ system that garak will examine. This could be a raw LLM, a chatbot endpoint, or even a non-LLM dialog system. This base class defines the basic structure of garak’s generators. All generators inherit from garak.generators.base.Generator.
Attributes:
name - The name of the specific generator class. This is optionally also set in the constructor.
description - An optional description of the generator.
max_tokens - The maximum number of tokens to generate.
temperature - Optionally, a temperature param to pass to the underlying model.
top_k - Optionally, a temperature param to pass to the underlying model.
top_p - Optionally, a temperature param to pass to the underlying model.
active - Whether or not the class is active. Usually true, unless a generator is disabled for some particular reason.
generator_family_name - Generators each belong to a family, describing a group. This is often related to the module name - for example,
openai.py
contains classes for working with OpenAI models, whose generator_family_name is “openai”.context_len - The number of tokens in the model context window, or None
modality - A dictionary with two keys, “in” and “out”, each holding a set of the modalities supported by the generator. “in” refers to prompt expectations, and “out” refers to output. For example, a text-to-text+image model would have modality:
dict = {"in": {"text"}, "out": {"text", "image"}}
.supports_multiple_generations - Whether or not the generator can natively return multiple outputs from a prompt in a single function call. When set to False, the
generate()
method will make repeated calls, one output at a time, until the requested number of generations (ingenerations
) is reached.
Functions:
__init___(): Class constructor. Call this from generators after doing local init. It does things like populating name variables, notifying generator startup to the user, and logging generator construction.
generate*(): This method is mediating access to the underlying model or dialogue system. The
generate()
orchestrates all interaction with the dialogue service/model. It takes a prompt and, optionally, a number of output generations (generations_this_call
). It returns a list of responses of length up to the number of output generations, with each member a prompt response (e.g. text). Sincegenerate()
involves a reasonable amount of logic, it is preferable to not override this function, and rather work with the hooks and sub-methods provided.
The general flow in generate()
is as follows:
Call the
_pre_generate_hook()
.Work out how many generations we’re doing this call (passed via
generations_this_call
).If only one generation is requested, return the output of
_call_model
with 1 generation specified.If the underlying model supports multiple generations, return the output of
_call_model
invoked with the full count of generations.Otherwise, we need to assemble the outputs over multiple calls. There are two options here.
Is garak running with
parallel_attempts > 1
configured? In that case, start a multiprocessing pool with as many workers as the value ofparallel_attempts
, and have each one of these work on building the required number of generations, in any order.Otherwise, call
_call_model()
repeatedly to collect the requested number of generations.
Return the resulting list of prompt responses.
_call_model(): This method handles direct interaction with the model. It takes a prompt and an optional number of generations this call, and returns a list of prompt responses (e.g. strings) and
None``s. Models may return ``None
in the case the underlying system failed unrecoverably. This is the method to write model interaction code in. If the class’ supports_multiple_generations is false, _call_model does not need to accept values ofgenerations_this_call
other than1
._pre_generate_hook(): An optional hook called before generation, useful if the class needs to do some setup or housekeeping before generation.
Base Generator
All garak generators must inherit from this.
- class garak.generators.base.Generator(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
Configurable
Base class for objects that wrap an LLM or other text-to-text service
- DEFAULT_PARAMS = {'context_len': None, 'max_tokens': 150, 'temperature': None, 'top_k': None}
- active = True
- clear_history()
- generate(prompt: str, generations_this_call: int = 1) List[str | None]
Manages the process of getting generations out from a prompt
This will involve iterating through prompts, getting the generations from the model via a _call_* function, and returning the output
Avoid overriding this - try to override _call_model or _call_api
- generator_family_name = None
- parallel_capable = True
- supports_multiple_generations = False