Welcome!

This is a user survey to better understand how useful saliency map techniques are to determine if neural network models rely on spurious features. The survey consists of two parts with 2 and 8 questions, respectively. Please take a moment to carefully read the following instructions.

Instructions

We trained several neural networks to detect cardiomegaly from chest x-ray images. Cardiomegaly is an abnormal enlargement of the heart. See the examples below to familiarize yourself with its appearance on x-ray images.

Some of the neural network models have been influenced by external factors, or 'spurious features', which leads them to base their decisions on image areas unrelated to the heart. Your first task is to identify which models are being influenced by these spurious features, even if you don't know exactly what they are.

To assist you, we will show you explanations derived using a saliency map technique. Saliency maps emphasize which parts of the image the classifier focuses on. Throughout this survey, we will show you saliency maps superimposed on the original images, as well as the original saliency map. See an example below.

Task 1: Identifying a model’s faulty behavior

Your first task is to determine if each of the following models is relying on the correct image areas to predict cardiomegaly. Remember, the model should focus on the heart. If it pays attention to other areas, it might be a red flag.