Towards Explainable Navigation and Recounting

Explainability and interpretability of deep neural networks have become crucial in computer vision, alongside the need to understand increasingly complex models. This work proposes an explainable setting for visual navigation, in which an autonomous agent explores an unseen indoor environment while portraying and explaining interesting scenes with natural language descriptions. We combine advances in visual navigation, recounting, and explainability, applying explanation maps to images generated through agent-environment interaction. The resulting visual explanations highlight correlations between observed entities and generated words, helping focus on prominent objects encountered during navigation.

In International Conference on Image Analysis and Processing