Artificial intelligence for risk analysis and the risks of artificial intelligence: Part 1

Abstract

The revolution created by artificial intelligence and machine learning (AI-ML) is affecting our society in a profound way. The use of machine intelligence is entering an increasing number of human activities, with AI helping to automate tasks and augmenting human abilities. Although AI-ML has the potential to benefit society, helping us recognize risks more promptly and manage them better, the disruption caused by these new technologies also presents new challenges to society. In particular, unthinking or uninformed application of these technologies in decision-making can result in biased and unreliable decisions. In addition, concerns about data privacy and security pervade the use of AI-ML tools. The associated risks and the potential consequences are difficult to assess and predict. This special issue collects a series of papers that showcase recent advances in research on AI-ML technologies and their relationship to the practice of risk analysis and risk-informed decision-making. The papers clearly show that the transfer of knowledge and methods between risk analysis and AI-ML occurs in both directions. On the one hand, risk-analysis methods and theories contribute to framing and controlling the threats to society and human activities posed by AI-ML. On the other hand, AI-ML methods and tools are starting to be used in risk analysis. The use of AI-ML in risk analysis is the subject of a thorough investigation in Stødle et al. (2025). The authors frame this integration as an input−output process that helps to accomplish the three main tasks of consequence identification, uncertainty characterization, and knowledge management. Through this framework, they analyze current applications and discuss potential future uses of AI-ML in risk assessment. They conclude with several recommendations for risk researchers and practitioners, highlighting the opportunities and limitations of the use of AI-ML in risk analysis. Paté-Cornell (2025) identifies and investigates the concern that AI recommendations may not be consistent with the preferences and risk attitudes of the individuals involved in a given decision. Some of the questions that emerge are how to identify the risk attitude implied by an AI-ML tool, and how the results should be communicated and possibly modified before being integrated into the risk-analysis process. Baum (2025) proposes a risk assessment of AI takeover (i.e., a potential takeover of key decisions by generative AI), with potentially catastrophic consequences for humanity. The author develops a stylized model that compares the type of capabilities that generative AI would need to have for such a takeover to happen against the capabilities of current large language models (LLMs). Based on that assessment, the author provides recommendations on whether more aggressive governance of LLMs is needed. We leave the answer to the paper. Collier et al. (2025) investigate whether LLMs can play a role in product risk assessment. The authors began by using the popular ChatGPT to provide suggestions on tasks such as performing failure modes and effects analysis and recommending risk mitigations. The authors then presented the results to safety experts to evaluate ChatGPT's output. The same analysis was performed for additional LLMs. The expert examination identified significant limitations in product risk assessment, producing inconsistent, generic, and sometimes unrealistic guidance. However, researchers suggest these models may still be useful for initial ideation, with experts focusing on critically reviewing and refining AI-generated content to improve the overall risk assessment process. Faddi et al. (2025) fill in a gap in the literature by developing quantitative methods to assess the reliability and resilience of AI-ML models to be applied in safety-critical domains. The work highlights the dynamic nature of the problem, since reliability and performance change over time. The authors demonstrate the approach by app