In order to enable the safe handling of risks of AI systems that have an (in)direct influence on human users, it is essential to always have an overview of the actual effects of the software on all groups of people, not only during the development or purchase of software, but also during ongoing operation, and to check them using technical and ethical quality criteria.

To ensure that the control over AI systems can actually be implemented permanently, efficiently and effectively, an “AI cockpit” is needed as a central user interface that bundles all relevant process steps of AI systems on the software side. In this two-year research project, such an AI cockpit is to be developed, which can flexibly dock onto various AI systems in an operational context.

This software is intended to guide all participants step by step through the process of evaluating and reviewing AI systems and enable users to maintain an overview and control over AI systems not only in theory but also in practice. With this software-supported cockpit, we close the gap between the ethical requirements of a society and the decisions made automatically by AI systems. Only through the clear visualisation in the form of an AI cockpit is it possible to determine the actual effects of AI systems, to critically examine their results and also to operate the “switch-off button for AI” in an informed manner.

The project promotes the development of a framework for “human in command”, finds concrete software solutions in various application domains and addresses EU requirements for AI high-risk systems that should always be under human control (human in command). Using demonstrators in the areas of personnel management, transport and healthcare, the AI cockpit software will be developed with a human-centred approach in a participatory manner with companies and end users and then made available to the general public as an open source application.

In this project, nexus is coordinating the research and development in the application areas and the accompanying research as well as the project management in the consortium.

Duration: 07/2023 – 06/2025
Grant authority: Federal Ministry of Labour and Social Affairs
Team: Daniel Guagnin (Project management), Philipp Offermann (Project coordination), Matthias Bottel, Maria Jacob
Project partners: Hochschule Aalen, Universität Stuttgart, Institut für Arbeitswissenschaft und Technologiemanagement (IAT), Chemistree GmbH, Caritas Dortmund

Picture: Andrés Dallimonti – Unsplash