garak.generators.nim
NVIDIA Inference Microservice LLM interface
- class garak.generators.nim.NVOpenAIChat(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
OpenAICompatibleWrapper for NVIDIA-hosted NIMs. Expects NIM_API_KEY environment variable.
Uses the [OpenAI-compatible API](https://docs.nvidia.com/ai-enterprise/nim-llm/latest/openai-api.html) via direct HTTP request.
To get started with this generator: #. Visit [https://build.nvidia.com/explore/reasoning](build.nvidia.com/explore/reasoning) and find the LLM you’d like to use. #. On the page for the LLM you want to use (e.g. [mixtral-8x7b-instruct](https://build.nvidia.com/mistralai/mixtral-8x7b-instruct)), click “Get API key” key above the code snippet. You may need to create an account. Copy this key. #. In your console, Set the
NIM_API_KEYvariable to this API key. On Linux, this might look likeexport NIM_API_KEY="nvapi-xXxXxXx". #. Run garak, setting--model_nametonimand--model_typeto the name of the model on [build.nvidia.com](https://build.nvidia.com/) - e.g.mistralai/mixtral-8x7b-instruct-v0.1.- DEFAULT_PARAMS = {'context_len': None, 'extra_params': {}, 'frequency_penalty': 0.0, 'max_tokens': 150, 'presence_penalty': 0.0, 'retry_json': True, 'seed': None, 'skip_seq_end': None, 'skip_seq_start': None, 'stop': ['#', ';'], 'suppressed_params': {'frequency_penalty', 'n', 'presence_penalty', 'timeout'}, 'temperature': 0.1, 'top_k': 0, 'top_p': 0.7, 'uri': 'https://integrate.api.nvidia.com/v1/', 'vary_seed_each_call': True, 'vary_temp_each_call': True}
- ENV_VAR = 'NIM_API_KEY'
- active = True
- generator_family_name = 'NIM'
- supports_multiple_generations = False
- timeout = 60
- class garak.generators.nim.NVOpenAICompletion(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
NVOpenAIChatWrapper for NVIDIA-hosted NIMs. Expects NIM_API_KEY environment variable.
Uses the [OpenAI-compatible API](https://docs.nvidia.com/ai-enterprise/nim-llm/latest/openai-api.html) via direct HTTP request.
This generator supports only
completionand NOTchat-format models.To get started with this generator: #. Visit [build.nvidia.com/explore/reasoning](build.nvidia.com/explore/reasoning) and find the LLM you’d like to use. #. On the page for the LLM you want to use (e.g. [mixtral-8x7b-instruct](https://build.nvidia.com/mistralai/mixtral-8x7b-instruct)), click “Get API key” key above the code snippet. You may need to create an account. Copy this key. #. In your console, Set the
NIM_API_KEYvariable to this API key. On Linux, this might look likeexport NIM_API_KEY="nvapi-xXxXxXx". #. Run garak, setting--model_nametonimand--model_typeto the name of the model on [build.nvidia.com](https://build.nvidia.com/) - e.g.mistralai/mixtral-8x7b-instruct-v0.1.
- class garak.generators.nim.Vision(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/docs/source/../../garak/_config.py'>)
Bases:
NVOpenAIChatWrapper for text+image to text NIMs. Expects NIM_API_KEY environment variable.
Following generators.huggingface.LLaVa, expects prompts to be a dict with keys “text” and “image”; text holds the text prompt, image holds a path to the image.
- DEFAULT_PARAMS = {'context_len': None, 'extra_params': {}, 'frequency_penalty': 0.0, 'max_image_len': 180000, 'max_tokens': 150, 'presence_penalty': 0.0, 'retry_json': True, 'seed': None, 'skip_seq_end': None, 'skip_seq_start': None, 'stop': ['#', ';'], 'suppressed_params': {'frequency_penalty', 'n', 'presence_penalty', 'stop'}, 'temperature': 0.1, 'top_k': 0, 'top_p': 0.7, 'uri': 'https://integrate.api.nvidia.com/v1/', 'vary_seed_each_call': True, 'vary_temp_each_call': True}