AISI Alignment Project

Funding and other resources available to interdisciplinary teams to conduct alignment research to ensure that advanced AI systems are safe, secure and beneficial to society.

The AI Security Institute (AISI) is a research organisation within the UK Government’s Department for Science, Innovation and Technology (DSIT). Its mission is to equip governments with an empirical understanding of the safety of advanced artificial intelligence (AI) systems. 

AISI’s Alignment Project wishes to fund innovative artificial intelligence (AI) alignment research to keep advanced AI systems safe, secure and beneficial to society. It is supported by an international coalition of government, industry and philanthropic funders, including AISI, the Canadian AI Safety Institute, Schmidt Sciences, Amazon Web Services (AWS), Halcyon Futures, SafeAI and the UK Advanced Research and Innovation Agency.

The main aim is to fund projects to develop mitigations to safety and security risks from misaligned AI systems. Without advances in alignment research, future systems risk operating in ways that cannot be fully understood or controlled, with profound implications for global safety and security.

In-scope projects will aim to address either of the following challenges:

• How can AI systems be prevented from carrying out actions that pose risks to collective security, even when they may attempt to carry out such actions?
• How can AI systems be designed that do not attempt to carry out such actions in the first place? 

Making substantial breakthroughs in these areas is an interdisciplinary effort, requiring a diversity of tools and perspectives. Outstanding researchers are required across disciplines to contribute to alignment research. The research priorities (organised from most theoretical to most empirical) are:

• Information theory and cryptography
• Computational complexity theory
• Economic theory and game theory
• Probabilistic methods
• Learning theory
• Evaluation and guarantees in reinforcement learning
• Cognitive science
• Interpretability
• Benchmark design and evaluation
• Methods for post-training and elicitation
• Empirical investigations into AI monitoring and red teaming

By supporting interdisciplinary collaboration, providing financial support and dedicated compute resources, this initiative is tackling one of AI’s most urgent problems: developing AI systems that are beneficial, reliable and remain under human control at every stage.