Investigators: Jorge Ortiz, PhD
Summary
In order to build trust with machine-learning-based systems, systems must provide mechanisms that improve functional transparency and execution consistency. For example, models should explain how different features affect inference results. They should guide how decisions are made and allow users to reason about their output. In the literature, this is referred to model intelligibility, an emerging area machine learning. Low-levels of intelligibility leads to misuse or deactivation, as humans lose trust in the output of the model. While complex models in a highly interpretable domain, such as image understanding, have been studied extensively we propose to examine model intelligibility in the context of multi-modal sensor fusion-based control systems. These are systems that fuse multiple sensor streams to make control decisions. We will work in the context of smart buildings to understand the model explainability in the context of multi-sensor fusion. Adaptive buildings and sensing systems have limited or no user interfaces which leads to great uncertainty about how the data is being used in the built environment. Recent work on smart building controls has shown the benefits of fusion for optimizing energy consumption. However, such gains can be short-lived and it is not well understood how the inputs are affecting control decisions that lead to either gains or losses in efficiency. To study this problem, we propose to build a human-agent interactive control system, whereby the agent will explain the reasons for its control decisions and the human will provide feedback about both the decisions (i.e., is the control decision correct?) and the explanations (i.e., does the explanation make sense?).