Md Montaser Hamid
← All publications

"Loss in Value": What it Revealed about WHO an Explanation Serves Well and WHEN

Hamid, M. M., Dodge, J., Anderson, A., & Burnett, M.

CEUR Workshop Proceedings, Vol. 3957, Proceedings of the ACM IUI Workshops 2025, 9–15 · 2025

Measuring whether users can predict an AI's behavior is a common way to evaluate their mental models — but the usual binary right/wrong scoring can be misleading, because it ignores how wrong a prediction was. Applying Koujalgi et al.'s "Loss in Value" measure to our 69-participant XAI study revealed who our explanations served well and when — differences the binary measure could not show.

The problem

Prediction tasks — asking users to predict an AI's next move — are one of the most common ways to measure users' mental models of AI. But the usual binary framing (correct/incorrect) is very susceptible to floor and ceiling effects, and as an AI's output space grows, nearly 100% of predictions get classified as simply "incorrect". For adaptive XAI systems, whose reason for existence is delivering the right explanation to this user in this situation, aggregate-level binary measures can't answer the essential questions: which particular users does an explanation serve, and when?

What we did

We applied "Loss in Value" — one of four fine-grained prediction measurements recently proposed by Koujalgi et al. — to our 69-participant between-subject study of the Original vs. Post-GenderMag versions of a game-playing AI's explanations. For each of 17 prediction tasks across three games, we computed each participant's "PredError": the absolute difference between the score of the move the AI chose and the score of the move the participant predicted. "Who" information came from participants' problem-solving styles measured via the GenderMag facet survey; "when" information came from per-task PredErrors; and participants' free-text comments provided context.

Key results

  • "When": Loss in Value revealed 6 of the 17 prediction tasks with visibly noticeable differences between the treatments — the binary measure revealed only 1 of them.
  • Scrutinizing those 6 tasks exposed a double-edged explanation: the popular "Top 5 Moves" feature helped when the game stayed on trajectory, but caused overreliance when the trajectory changed — Post-GenderMag participants incorrectly picked (now-obsolete) top-5 moves significantly more often.
  • "Who": Loss in Value showed Abi-like participants' prediction errors differed between versions in all 6 tasks (binary obscured 3 of them), and Tim-like participants' in 4 (binary obscured 2).
  • Loss in Value also revealed an equity gap invisible to the binary measure: women had significantly (p < .05) fewer prediction errors than men in both versions.
  • Overall, Loss in Value proved a more accurate and more useful measure than binary scoring for the who/when/how questions XAI research needs to answer.

From the paper

From the paper's Figure 1: prediction errors for the 17 prediction tasks measured with Loss in Value (left) vs. binary scoring (right). Loss in Value revealed 6 tasks with noticeable treatment differences (bright); binary revealed only one (G3M13). Lower PredError is better.
From the paper's Figure 1: prediction errors for the 17 prediction tasks measured with Loss in Value (left) vs. binary scoring (right). Loss in Value revealed 6 tasks with noticeable treatment differences (bright); binary revealed only one (G3M13). Lower PredError is better.
Cite: Hamid, M. M., Dodge, J., Anderson, A., & Burnett, M. (2025). "Loss in Value": What it Revealed about WHO an Explanation Serves Well and WHEN. CEUR Workshop Proceedings, Vol. 3957, Proceedings of the ACM IUI Workshops 2025, 9–15.