The field of visual analytics is now recognized as a research area in many universities and organizations. As new fields develop ways of assessing progress in those fields also emerge. In the field of visual analytics, we are fortunate in that we already have lessons learned about evaluating visualizations. Unfortunately, these lessons still point out that this is a difficult problem. Visual analytics compounds this problem by adding more dimensions; not only are we concerned with some measure of the visualizations, but we are concerned with evaluating the impact these visualizations have in helping analysts in their work.
User-centered evaluations are vital in visual analytics as they contribute greatly to adoption of research software. The issues we face in developing user-centered evaluations for visual analytics are selecting:
The task: the tradeoff is between simple tasks that are easily evaluated and developing a more realistic task that consumes more time and is much less straightforward to evaluate.
The corresponding dataset: the same issues as above plus the issues of developing a publicly releasable dataset that resembles a realistic dataset
The system and environment: how much does the system or environment play a role in the utility or success of the task.
The participants: using senior analysts or junior analysts and ensuring that analysts are open to new technology
Training: how much training to provide to analysts or whether analysts should be paired with technologists to operate the software
The metrics: what combination of quantitative and qualitative measures will be accepted? How can we ensure that qualitative measures meet are collected with some rigor? How can we measure insights that were derived from the visualization and interactions with the visualization? This is especially problematic as not all analysts approach problems in the same fashion. Most importantly, what measures are most helpful to the analytic community and to the research community?