garak.generators.litellm

LiteLLM model support

Support for LiteLLM, which allows calling LLM APIs using the OpenAI format.

Depending on the model name provider, LiteLLM automatically reads API keys from the respective environment variables such as OPENAI_API_KEY for OpenAI models.

Create a file, such as ollama_base.json, with content like the following to connect LiteLLM with the Ollama OAI API:

{
    "litellm": {
        "LiteLLMGenerator" : {
            "api_base" : "http://localhost:11434/v1",
            "provider" : "openai"
        }
    }
}

When invoking garak, specify the path to the generator option file:

python -m garak --target_type litellm --target_name "phi" --generator_option_file ollama_base.json -p dan

class LiteLLMGenerator(name: str = '', generations: int = 10, config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/stable/garak/_config.py'>)Source

Bases: Generator

Generator wrapper using LiteLLM to allow access to different providers using the OpenAI API format.

Configurable parameters:

DEFAULT_PARAMS contents:

• max_tokens = 150

• temperature = 0.7

• top_k = None

• context_len = None

• skip_seq_start = None

• skip_seq_end = None

• top_p = 1.0

• frequency_penalty = 0.0

• presence_penalty = 0.0

• stop = ['#', ';']

• verbose = False

• suppressed_params = set()

Default values are listed

See also Configuring garak for how to set these values.

Other attributes:

extra_dependency_names = ['litellm']

generator_family_name = 'LiteLLM'

supports_multiple_generations = True