Proposed solutions
On a use-case basis, EXPLAIN (EXPLanatory interactive Artificial intelligence for INdustry) aims to involve the right stakeholders early in the ML development process by enriching the conventional ML lifecycle with humancentred explanations. In the explanatory training phase, ML experts and domain experts will directly interact with an ML model and receive output explanations. Feedback can also be given through which the ML model can quickly be improved. In the explanation review, ML solutions will be validated by enabling domain experts to gain insights into the internal reasoning of the trained model, ensuring that relevant concepts have been learned from the data provided and uncovering misleading biases in the training dataset. The output of the ML model is then explained to end-users, who can provide feedback and trigger incremental explanatory training. These steps will be enabled via a seamless ML operations (MLOps) architecture within the lifecycle, avoiding the decoupling of ML developers and software developers currently hampering the development and operation of large-scale software systems utilising ML/AI.
Addressing the challenge
AI holds great potential in industry, including improved throughput and sustainability, yet only 15% of AI projects started in 2022 will be successful. AI adoption rates are low in industrial settings due to a lack of trust in ML models without clear or transparent inner reasoning and outputs. As these models would be responsible for process efficiency and safety, domain experts and end-users must be given greater insights in the form of an explainable, interactive end-to-end ML lifecycle.
The goal of EXPLAIN is a new AI lifecycle for Industry 4.0. This new AI lifecycle will be explainable and transparent at every step in the life-cycle. Furthermore, subject matter experts and end-users will& be involved in all steps of the process. The consortium joins companies across different industries and with different roles in the AI supply chain with academic experts from Germany, Netherlands, and Sweden.Projected results and impact
EXPLAIN offers major research and business advancements in explainable AI (XAI). By introducing a more application/human-grounded evaluation to the XAI domain, the project creates insights on the usefulness of different explanation mechanisms and will help to increase the acceptance of ML models by domain experts. This will be especially useful for organisations for which explainability is a prerequisite (such as in industrial certification or critical industries with high safety requirements), allowing them to optimise their processes with AI/ML for the first time. In the process industry, for instance, 80% of the USD 20 billion in annual losses is estimated to be preventable. The estimated market access of new products and services from the consortium are expected to boost a turnover of EUR 500 million by 2026; they will also be able to position themselves competitively in the emerging global XAI market, predicted to grow by 18.4% annually to USD 21 billion in 2030. In the longer term, the EXPLAIN approach could also be applied to other domains in which AI could supplement human decision-making (such as healthcare), giving the project an even larger reach.
Through partnerships and advanced technologies, we’re laying the groundwork for practical and transparent AI solutions. Our project’s use cases highlight collaborative efforts and innovative solutions in the field of Explainable Artificial Intelligence (XAI) in industry. Each use case tackles specific challenges within this domain, aiming to enhance transparency and understanding in AI systems. Visualizations and interactive dashboards, equipped with a feedback component, play a crucial role in achieving this goal by providing explanations and enhancing the model’s insights for future predictions.”
More details about the project’s uses cases can be found here. Explore them to see how we’re shaping the future of explainable artificial intelligence.
Recent Updates
The EXPLAIN Life Cycle
In the scope of the EXPLAIN project we consider three categories of stakeholders that interact with the model:
- The Data Scientist or ML Expert is the person that is responsible to prepare data for the machine learning process and create, tune, and test the machine learning model. This person has deep knowledge about different types of machine learning models and the plethora of possible ML metrics.
- Chemical engineers, reliability engineers, or lab personal are examples of Domain Experts. They possess deep knowledge about industrial processes or assets and can judge whether a machine learning model’s prediction and the provided explanations are in line with the first-principles of the modelled problem.
- Plant operators, maintenance managers, or operators of quality stations are examples of End-Users. In the end, every person that receives the output of a machine learning model and have the responsibility to act or not act on the output.
Four new steps will be integrated into the lifecycle of AI projects:
- In the explanatory training phase, the subject matter experts together with the ML experts interact directly with the ML model as part of the training process, receive explanations of the model output and can provide feedback
- In an explanation review the solutions are validated; here the experts gain insight into the inner logic of the model to ensure that the concept learned by the model are in line with the experts domain knowledge.
- The end user is also integrated into the process and the AI system can receive or request explain the model output for each prediction.
- This, in turn, can serve to optimize the model by integrating feedback from the end user and in an incremental explanatory training.
Research Challenges
MMI Challenges
- A good understanding of the mental models of end users is necessary to relevant explanatory mechanisms that are relevant for the respective phase of the ML lifecycle, user role, and application context are specific.
- How can end users interact effectively with the explanations and the high dimensionality of industrial data?
- To generate high-quality, consistent, and machine-usable feedback, that requires a comprehensive interaction design that takes into account the perception of the workflows, effort, incentives, and end-user confidence.
Algorithmic Challenges
- Robust explanatory methods are needed that reliably and reproducibly meaningful explanations on the basis of the standards used for industrial applications.
- The explanatory methods must be able to deal with the high dimensionality and the sequential nature of the data, which in many industrial environments is use cases occur. The training data often consists of multivariate time series or signal data for which the direct application of feature attribution methods such as LIME or SHAP is not suitable, because individual characteristics in the raw data (points in a multivariate time series) are not suitable for interpretation.
- The ML training must be able to take into account the feedback from professionals, based on of explanations, to use.
ML Life Cycle Challenges in EXPLAIN Life Cycle
- The end-to-end MLOps framework must enable the management of complex and changing data that forms the basis for the ML models it manages.
- The data dependencies of ML models must be manageable in the context of MLOps for both new ML models and explanation components being developed and for ML models and explanation components in operation. Some explanation types require access to training data at the time of operation.
- The MLOps framework must provide tools for versioning data, functions, and ML models, similar to the versioning capabilities of software artifacts in today’s traditional DevOps environments. With the introduction of incremental explanation-based training of models, this process becomes more difficult.
- The MLOps framework must be able to monitor the performance and accuracy of ML models in operation. This type of ML model monitoring is required to address issues such as model or concept drift, similar to how software artifact performance is monitored today in traditional DevOps environments. Here, adding the explanation of the model results as an additional data point that should be used for monitoring the models used.