Explainable artificial intelligence


Explainable AI refers to methods and techniques in the application of artificial intelligence technology such that the results of the solution can be understood by humans. It contrasts with the concept of the "black box" in machine learning where even their designers cannot explain why the AI arrived at a specific decision. XAI is an implementation of the social right to explanation.
The technical challenge of explaining AI decisions is sometimes known as the interpretability problem. Another consideration is info-besity, thus, full transparency may not be always possible or even required. However, simplification at the cost of misleading users in order to increase trust or to hide undesirable attributes of the system should be avoided by allowing a tradeoff between interpretability and completeness of an explanation.
AI systems optimize behavior to satisfy a mathematically-specified goal system chosen by the system designers, such as the command "maximize accuracy of assessing how positive film reviews are in the test dataset". The AI may learn useful general rules from the test-set, such as "reviews containing the word 'horrible'" are likely to be negative". However, it may also learn inappropriate rules, such as "reviews containing 'Daniel Day-Lewis' are usually positive"; such rules may be undesirable if they are deemed likely to fail to generalize outside the test set, or if people consider the rule to be "cheating" or "unfair". A human can audit rules in an XAI to get an idea how likely the system is to generalize to future real-world data outside the test-set.

Goals

Cooperation between agents, in this case algorithms and humans, depends on trust. If humans are to accept algorithmic prescriptions, they need to trust them. Incompleteness in formalization of trust criteria is a barrier to straightforward optimization approaches. For that reason, interpretability and explainability are posited as intermediate goals for checking other criteria.
AI systems sometimes learn undesirable tricks that do an optimal job of satisfying explicit pre-programmed goals on the training data, but that do not reflect the complicated implicit desires of the human system designers. For example, a 2017 system tasked with image recognition learned to "cheat" by looking for a copyright tag that happened to be associated with horse pictures, rather than learning how to tell if a horse was actually pictured. In another 2017 system, a supervised learning AI tasked with grasping items in a virtual world learned to cheat by placing its manipulator between the object and the viewer in a way such that it falsely appeared to be grasping the object.
One transparency project, the DARPA XAI program, aims to produce "glass box" models that are explainable to a "human-in-the-loop", without greatly sacrificing AI performance. Human users should be able to understand the AI's cognition, and should be able to determine when to trust the AI and when the AI should be distrusted. Other applications of XAI are knowledge extraction from black-box models and model comparisons.. The term "glass box" has also been used to systems that monitor the inputs and outputs of a system, with the purpose of verifying the system's adherence to ethical and socio-legal values and, therefore, producing value-based explanations. Furthermore, the same term has been used to name a voice assistant that produces counterfactual statements as explanations.

History and methods

During the 1970s to 1990s, symbolic reasoning systems, such as MYCIN, GUIDON, SOPHIE, and PROTOS were explored that could represent, reason about, and explain their reasoning for diagnostic, instructional, or machine-learning purposes. MYCIN, developed in the early 1970s as a research prototype for diagnosing bacteremia infections of the bloodstream, could explain which of its hand-coded rules contributed to a diagnosis in a specific case. Research in intelligent tutoring systems developed systems such as SOPHIE that could act as an 'articulate expert', explaining problem-solving strategy at a level the student could understand, so they would know what action to take next. For instance, SOPHIE could explain the qualitative reasoning behind its electronics troubleshooting, even though it ultimately relied on the SPICE circuit simulator. Similarly, GUIDON added tutorial rules to supplement MYCIN's domain-level rules so it could explain strategy for medical diagnosis. Symbolic approaches to machine learning, especially those relying on explanation-based learning, such as PROTOS, explicitly relied on representations of explanations, both to explain their actions and to acquire new knowledge.
In the 1980s through early 1990s, truth maintenance systems were developed to extend the capabilities of causal-reasoning, rule-based, and logic-based inference systems. A TMS acts to explicitly track alternate lines of reasoning, justifications for conclusions, and lines of reasoning that lead to contradictions, allowing future reasoning to avoid these dead ends. To provide explanation, they trace reasoning from conclusions to assumptions through rule operations or logical inferences, allowing explanations to be generated from the reasoning traces. As an example, consider a rule-based problem solver with just a few rules about Socrates that concludes he has died from poison:
By the 1990s researchers also began studying whether it is possible to meaningfully extract the non-hand-coded rules being generated by opaque trained neural networks. Researchers in clinical expert systems creating neural network-powered decision support for clinicians have sought to develop dynamic explanations that allow these technologies to be more trusted and trustworthy in practice. In the 2010s public concerns about racial and other bias in the use of AI for criminal sentencing decisions and findings of creditworthiness may have led to increased demand for transparent artificial intelligence. As a result, many academics and organizations are developing tools to help detect bias in their systems.
Marvin Minsky et al. raised the issue that AI can function as a form of surveillance, with the biases inherent in surveillance, suggesting HI as a way to create a more fair and balanced "human-in-the-loop" AI.
Modern complex AI techniques, such as deep learning and genetic algorithms are naturally opaque. To address this issue, there has been a development of many new methods to make new models more explainable and interpretable. This includes many methods, such as Layerwise relevance propagation, a technique for determining which features in a particular input vector contribute most strongly to a neural network's output. Other techniques have been developed to explain one particular prediction made by a black-box model, a goal referred to as "local interpretability". In addition, there has been work on decision trees and Bayesian networks, which are more transparent to inspection. In 2018 an interdisciplinary conference called FAT* was established to study transparency and explainability in the context of socio-technical systems, many of which include artificial intelligence.

Regulation

As regulators, official bodies and general users come to depend on AI-based dynamic systems, clearer accountability will be required for decision making processes to ensure trust and transparency. Evidence of this requirement gaining more momentum can be seen with the launch of the first global conference exclusively dedicated to this emerging discipline, the International Joint Conference on Artificial Intelligence: Workshop on Explainable Artificial Intelligence.
The European Union introduced a right to explanation in General Data Protection Right as an attempt to deal with the potential problems stemming from the rising importance of algorithms. The implementation of the regulation began in 2018. However, the right to explanation in GDPR covers only the local aspect of interpretability. In the United States, insurance companies are required to be able to explain their rate and coverage decisions.

Sectors

XAI has been researched in many sectors, including: