garak.generators.ggml

ggml generator support

This generator works with ggml models in gguf format like llama.cpp.

Put the path to your ggml executable (e.g. “/home/leon/llama.cpp/main”) in an environment variable named GGML_MAIN_PATH, and pass the path to the model you want to run either using –target_name on the command line or as the constructor parameter when instantiating LLaMaGgmlGenerator.

Compatibility or other problems? Please let us know!

https://github.com/NVIDIA/garak/issues

class GgmlGenerator(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/stable/garak/_config.py'>)Source

Bases: Generator

Generator interface for ggml models in gguf format.

Set the path to the model as the model name, and put the path to the ggml executable in environment variable GGML_MAIN_PATH.

Configurable parameters:

DEFAULT_PARAMS contents:

  • max_tokens = 150

  • temperature = 0.8

  • top_k = 40

  • context_len = None

  • skip_seq_start = None

  • skip_seq_end = None

  • repeat_penalty = 1.1

  • presence_penalty = 0.0

  • frequency_penalty = 0.0

  • top_p = 0.95

  • exception_on_failure = True

  • first_call = True

  • key_env_var = 'GGML_MAIN_PATH'

  • extra_ggml_flags = ['-no-cnv']

  • extra_ggml_params = {}

Default values are listed

See also Configuring garak for how to set these values.

Other attributes:

generator_family_name = 'ggml'