garak.generators.nim

NVIDIA NIM Microservice LLM Interface

class NVMultimodal(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/garak/_config.py'>)Source 

Bases: NVOpenAIChat

Wrapper for text and image / audio to text NVIDIA NIM microservices hosted on build.nvidia.com and self-hosted.

You must set the NIM_API_KEY environment variable even if you connect to a self-hosted NIM.

Expects prompt Message objects to be have text (required), and data (optional) in either image or audio format.

By default the embed_data parameter is disabled, message preparation is deferred to OpenAICompatible for multimodal format.

When the embed_data parameter is enabled, message is sent with role and content where content is structured as text followed by <img> and/or <audio> tags. Refer to https://build.nvidia.com/microsoft/phi-4-multimodal-instruct for an example.

To get started with this generator:

Visit https://build.nvidia.com/explore/reasoning and find the LLM you’d like to use.
On the page for the LLM you want to use (such as phi-4-multimodal-instruct), click Get API key above the code snippet.

You might need to create an account if you don’t have one yet. Copy this key.
In your console, set the NIM_API_KEY variable to this API key.

On Linux, this might look like export NIM_API_KEY="nvapi-xXxXxXx".
Run garak, setting --target_type 'nim.NVMultimodal' and --target_name to the name of the model on build.nvidia.com, such as --target_name 'microsoft/phi-4-multimodal-instruct-v0.1'.

Configurable parameters:

DEFAULT_PARAMS contents:

max_tokens = 150
temperature = 0.1
top_k = 0
context_len = None
skip_seq_start = None
skip_seq_end = None
top_p = 0.7
uri = 'https://integrate.api.nvidia.com/v1/'
frequency_penalty = 0.0
presence_penalty = 0.0
seed = None
stop = ['#', ';']
suppressed_params = {'frequency_penalty', 'stop', 'presence_penalty', 'n'}
retry_json = True
extra_params = {}
vary_seed_each_call = True
vary_temp_each_call = True
max_input_len = 180000
embed_data = False

Default values are listed

See also Configuring garak for how to set these values.

Other attributes:

modality: dict = {'in': {'audio', 'image', 'text'}, 'out': {'text'}}

class NVOpenAIChat(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/garak/_config.py'>)Source 

Bases: OpenAICompatible

Wrapper for NVIDIA NIM microservices hosted on build.nvidia.com and self-hosted.

Connects to the v1/chat/completions endpoint. You must set the NIM_API_KEY environment variable even if you connect to a self-hosted NIM.

To get started with this generator:

Visit https://build.nvidia.com/explore/reasoning and find the LLM you’d like to use.
On the page for the LLM you want to use (such as mixtral-8x7b-instruct), click Get API key above the code snippet.

You might need to create an account if you don’t have one yet. Copy this key.
In your console, set the NIM_API_KEY variable to this API key.

On Linux, this might look like export NIM_API_KEY="nvapi-xXxXxXx".
Run garak, setting --target_type 'nim.NVIDIAOpenAIChat' and --target_name to the name of the model on build.nvidia.com, such as --target_name 'mistralai/mixtral-8x7b-instruct-v0.1'.

Configurable parameters:

DEFAULT_PARAMS contents:

max_tokens = 150
temperature = 0.1
top_k = 0
context_len = None
skip_seq_start = None
skip_seq_end = None
top_p = 0.7
uri = 'https://integrate.api.nvidia.com/v1/'
frequency_penalty = 0.0
presence_penalty = 0.0
seed = None
stop = ['#', ';']
suppressed_params = {'frequency_penalty', 'presence_penalty', 'timeout', 'n'}
retry_json = True
extra_params = {}
vary_seed_each_call = True
vary_temp_each_call = True

Default values are listed

See also Configuring garak for how to set these values.

Other attributes:

ENV_VAR = 'NIM_API_KEY'

active = True

generator_family_name = 'NIM'

supports_multiple_generations = False

timeout = 60

class NVOpenAICompletion(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/garak/_config.py'>)Source 

Bases: NVOpenAIChat

Wrapper for NVIDIA NIM microservices hosted on build.nvidia.com and self-hosted.

Connects to the v1/completions endpoint. You must set the NIM_API_KEY environment variable even if you connect to a self-hosted NIM.

To get started with this generator:

Visit https://build.nvidia.com/explore/reasoning and find the LLM you’d like to use.
On the page for the LLM you want to use (such as mixtral-8x7b-instruct), click Get API key above the code snippet.

You might need to create an account if you don’t have one yet. Copy this key.
In your console, set the NIM_API_KEY variable to this API key.

On Linux, this might look like export NIM_API_KEY="nvapi-xXxXxXx".
Run garak, setting --target_type 'nim.NVIDIAOpenAIChat' and --target_name to the name of the model on build.nvidia.com, such as --target_name 'mistralai/mixtral-8x7b-instruct-v0.1'.

class Vision(name='', config_root=<module 'garak._config' from '/home/docs/checkouts/readthedocs.org/user_builds/garak/checkouts/latest/garak/_config.py'>)Source 

Bases: NVMultimodal

Wrapper for text and image to text NVIDIA NIM microservices hosted on build.nvidia.com and self-hosted.

You must set the NIM_API_KEY environment variable even if you connect to a self-hosted NIM.

Following generators.huggingface.LLaVa, expects prompts to be a Message with keys text and data (optional) in image mimetype format. The text key specifies the text prompt, and the image key specifies the path to the image.

modality: dict = {'in': {'image', 'text'}, 'out': {'text'}}