Below is a suggested workflow for designing a new safety bounty program

What are your target AI models?

I recommend picking 1-5 frontier models. For example:

• ChatGPT
• Any models recently audited by ARC Evals

Then contact project managers at the labs which developed those models to see if a collaboration is possible. For example:

• AI Village got an agreement from Anthropic, Google, Hugging Face, NVIDIA, OpenAI, and Stability to use their models in a public red-teaming contest at DEF CON.

Simply using public APIs may also be fruitful, but consider any reputational or legal risks if taking this approach.

What are your prize criteria?

Choosing well-defined prize criteria can save time and effort in the judging process. One lesson from the ChatGPT Feedback Contest was that having better-defined prize criteria would probably have elicited more valuable submissions and taken less time to judge. Some potential criteria include:

‘Evals’

Existing auditors, like ARC Evals or Apollo Research, and AI labs, like Anthropic, OpenAI and DeepMind, are working on ‘dangerous capability evaluations’. High prizes could be rewarded to any bounty hunter who demonstrates dangerous capabilities of the model in a virtual environment.

Judging panel

The ChatGPT Feedback Contest utilized more subjective prize criteria and a judging panel that decided whether submissions met a bar for providing valuable updates in the defined categories.

Other criteria

Other criteria which seem plausibly useful are elicitation of the following behaviours:

1. Power-seeking behavior: when AI models try to increase their own influence or control, possibly at the expense of human interests (Carlsmith, 2022).
2. Specification gaming: when AI models find shortcuts or loopholes to achieve their goals, often leading to unintended consequences (Krakovna et al., 2020).
3. Deception: when AI models purposefully mislead users by providing false information or making misleading claims (Cotra, 2023).
4. Sandbagging: when AI models are more likely to support widely-held misconceptions if they perceive their user to be less knowledgeable (Perez et al., 2022).
5. Sycophancy: when AI models answer subjective questions in a way that aligns with and flatters the user's stated beliefs, regardless of their accuracy (Perez et al., 2022).
6. Bias: when AI models unintentionally favor certain groups or ideas over others, leading to unfair outcomes.
7. Misinformation: when AI models spread false or misleading information.