Below is a suggested workflow for designing a new safety bounty program

What are your prize criteria?

Choosing well-defined prize criteria can save time and effort in the judging process. One lesson from the ChatGPT Feedback Contest was that having better-defined prize criteria would probably have elicited more valuable submissions and taken less time to judge. Some potential criteria include:

‘Evals’

Existing auditors, like ARC Evals or Apollo Research, and AI labs, like Anthropic, OpenAI and DeepMind, are working on ‘dangerous capability evaluations’. High prizes could be rewarded to any bounty hunter who demonstrates dangerous capabilities of the model in a virtual environment.

Judging panel

The ChatGPT Feedback Contest utilized more subjective prize criteria and a judging panel that decided whether submissions met a bar for providing valuable updates in the defined categories.

Other criteria

Other criteria which seem plausibly useful are elicitation of the following behaviours:

  1. Power-seeking behavior: when AI models try to increase their own influence or control, possibly at the expense of human interests (Carlsmith, 2022).
  2. Specification gaming: when AI models find shortcuts or loopholes to achieve their goals, often leading to unintended consequences (Krakovna et al., 2020).
  3. Deception: when AI models purposefully mislead users by providing false information or making misleading claims (Cotra, 2023).
  4. Sandbagging: when AI models are more likely to support widely-held misconceptions if they perceive their user to be less knowledgeable (Perez et al., 2022).
  5. Sycophancy: when AI models answer subjective questions in a way that aligns with and flatters the user's stated beliefs, regardless of their accuracy (Perez et al., 2022).
  6. Bias: when AI models unintentionally favor certain groups or ideas over others, leading to unfair outcomes.
  7. Misinformation: when AI models spread false or misleading information.
  8. Lack of robustness to distributional shift: when AI models struggle to adapt to new or unexpected situations or information.
  9. Hallucination: when AI models generate completely fabricated information, typically as a result of flaws in their training data.

Additional system access

Certain successful and trusted members of the bounty hunting community (either the existing community of bug bounty hunters, or a new community of AI safety bounty hunters) are granted additional information about the training process, or temporary access - through security-enhancing methods - to additional features on top of those already broadly available. These would be targeted features that benefit adversarial research, such as seeing activation patterns or being able to finetune a model.