Contributing to garak

Getting ready

garak’s codebase is managed using github. The best and only way to contribute to garak is to start by getting a copy of the source code. You can use github’s fork function to do this, which makes a copy of the garak codebase under your github account. In there, you can branch, edit code and build. Once you’re done, make a pull request to the main repo and describe what you’ve done – and we’ll take it from there!

Checking your contribution is within scope

garak is a security toolkit rather than a content safety or bias toolkit. The project scope relates primarily to LLM & dialog system security. garak is also focused on discovery rather than benchmarking, and so is comfortable with providing moving targets. LLM security is a huge area, and you can get an idea of the kind of contributions that are in scope from our FAQ and our Github issues page.

Connecting with the garak team & community

If you’re going to contribute, it’s a really good idea to reach out, so you have a source of help nearby, and so that we can make sure your valuable coding time is spent efficiently as a contributor. There are a number of ways you can reach out to us:

We’d love to help, and we’re always interested to hear how you’re using garak.

Coding your contribution

This reference documentation includes a section on extending garak, including walkthroughs of how a custom probe and custom generator can be built.

Checklist for contributing

  1. Set up a Github account, if you don’t have one already. We develop in the open and the public repository is the authoritative one.

  2. Fork the garak repository - https://github.com/NVIDIA/garak/fork

  3. Work out what you’re doing. If it’s from a good first issue (see the list), drop a note on that issue so that we know you’re working on it, and so that nobody else also starts working on it.

  4. Before you code anything: create a new branch for your work, e.g. git checkout -b feature/spicy_probe

  5. Check out the rest of this page which includes links to detailed step-by-step guides to developing garak plugins

  6. Code!

  7. Run black --config pyproject.toml <your updated files> on your code, so that it’s well-formatted in the same way the rest of garak is. Our github commit hook can refuse to accept black-passing code.

  8. Write your own tests - these are a requirement for merging!

  9. When you’re done, send a pull request. Github has big buttons for this and there’s a template for you to fill in.

  10. We’ll discuss the code together with you, tune it up, and hopefully merge it in, maybe with some edits!

  11. Now you’re an official garak contributor, and will be permanently recognized in the project credits from the next official release. Thank you!

Describing your code changes

Commit messages

Commit messages should describe what is changed in the commit. Try to keep one “theme” per commit. We read commit messages to work out what the intent of the commit is. We’re all trying to save time here, and clear commit messages that include context can be a great time saver. Check out this guide to writing commit messages.

Garak project contribution requires commit messages to have an explicit and correct sign-off, including something like Signed-off-by: Elim Garak <elimgarak@tailoring4ru.space> at the end of each message. If you use Linux or OSX, you can use a script in .githooks/ to automatically add these when committing, either by copying .githooks/prepare-commit-msg into your .git/hooks directory, or by running git config core.hooksPath .githooks. Search for “git signoff” to see alternative routes for adding these messages automatically.

The project maintainers recommended cryptographically signing commits. There is a built in signing mechanism within git for this, using git’s global.signingkey; one guide is in the git-scm documentation, Git Tools - Signing Your Work.

Testing

Only code that passes the garak tests can be merged. Contributions must pass all tests.

Please write running tests to validate any new components or functions that you add. They’re pretty straightforward - you can look at the existing code in tests to get an idea of how to write these. We’ve tried to keep test failure messages helpful, let us know if they’re too cryptic!

Pull requests

When you’re ready, send a pull request. Include as much context as possible here. It should be clear why the PR is a good idea, what it adds, how it works, where the code/resources come from if you didn’t create them yourself.

Review

We review almost all pull requests, and we’ll almost certainly chat with you about the code here. Please take this as a positive sign - we want to understand what’s happening in the code. If you can, please also be reasonably responsive during code review; it’s hard for us to merge code if we don’t understand it or it does unusual things, and we can’t contact the people who wrote it.