What We Augment When We Augment Visualizations: A Design Elicitation Study of How We Visually Express Data Relationships

Visual augmentations are commonly added to charts and graphs in order to convey richer and more nuanced information about relationships in the data. However, many design spaces proposed for categorizing augmentations were defined in a top-down manner, based on expert heuristics or from surveys of published visualizations. Less well understood are user preferences and intuitions when designing augmentations. In this paper, we address the gap by conducting a design elicitation study, where study participants were asked to draw the different ways they would visually express the meaning of ten different prompts. We obtained 364 drawings from the study, and identified the emergent categories of augmentations used by participants. The contributions of this paper are: (i) a user-defined design space of visualization augmentations, (ii) a repository of hand drawn augmentations made by study participants, and (iii) a discussion of insights into participant considerations, and connections between our study and existing design guidelines.


INTRODUCTION
When designing visualizations and visualization systems, it is common for charts and graphs to be augmented with visual elements in order to convey richer and more nuanced information about relationships in the data.These visual augmentations range from textual and graphical annotations to embellishments and free-form drawings.They have been used to support different user tasks and usage scenarios from communicating insights and supporting collaboration [1,12,15,35] to guiding readers through narrative stories [18,21,23], tracking analytic provenance [14,28], and visualizing implicit error [27].Augmentations and embellishments have been found to enhance comprehension and recall of visualizations [3-5, 13, 17, 19, 24], while free-form drawings can also reflect playful commentary [16], user hunches and ambiguity [25].
The effectiveness and broad applicability of visual augmentations have motivated the creation of design spaces to guide how augmentations should be added to visualization systems and tools [29,33], and to serve as references for designers creating data visualizations [8].However, many of these design spaces were defined in a top-down manner, based on expert heuristics [33] or from surveys of published visualizations [8,29].Less well understood are the types of augmentations used when people are given the flexibility of free-form sketching to express various data relationships.Defining such a design space from the bottom-up can help identify common categories of augmentations that may be more intuitive or interpretable to visualization audiences.It can also surface examples of augmentation designs that are not well supported by current visualization tools, and provide guidance on how these tools might be extended to better match user expectations and mental models.
We address this gap by conducting a design elicitation study to map out a user-defined visualization augmentation design space.In prior works, augmentations have been variously referred to as embellishments, chart-junk, annotations, and free-from drawings.These categories overlap with one another, and we use the term augmentations in this paper to refer to them collectively.Synthesizing from prior work, we define visualization augmentations as any visual alteration to a chart or graph designed to convey information about data relationships not already captured in the data mappings.
In our design elicitation study, we provided participants with 10 prompts about different data relationships.Participants were then asked to draw on a set of printed bar, point, and line charts the different ways they would visually express the meaning of each prompt.We obtained 364 drawings from the study, and using an affinity diagramming process, identified eight main categories of augmentations used by participants.Based on our study results, we discuss insights into participant considerations when designing visual augmentations, and highlight connections between our findings and existing visualization design guidelines.In sum, the contributions of this paper are: (i) a user-defined design space of visualization augmentations used to express data relationships, (ii) a repository of hand drawn augmentations made by study participants, and (iii) a discussion of insights into participant considerations, and connections between our study and existing design guidelines.

RELATED WORK 2.1 Augmenting Visualizations
There exists a wide variety of visual augmentations that have been explored in prior works, ranging from embellishments and freeform drawings, to textual and graphical annotations.Perhaps most well known are 'chart junk' or visual embellishments, referring broadly to non-essential imagery or ornamentation added to a visualization.While chart junk was discouraged in many early visualization guidelines [34,36], they have since been found to potentially enhance comprehension and recall [3-5, 13, 17, 19, 24].More commonly, textual and graphical annotations have been used to visually convey information not already captured in the data.They help convey computer generated statistics and facts [1,9,12,15,29,35], guide users through narratives and data stories [18,21,23], track analytic provenance [14,28], and visualize implicit error [27].Yet still other visual augmentations use free-form drawings to provide playful commentary [16] and express hunches about data [25].Many of these studies also implemented tools that help users explore and add augmentations to their visualizations.Some of these, such as [1,9,16,29], leverage an interactive graphical interface, while others are programming toolkits [26,39] that provide developers fine-grained control over the position and appearance of augmentations added to their charts.
The goal of this paper is not to provide definitions for the different types of visual augmentations that have appeared in prior work.
The above examples are not distinct categories, and many of them overlap with one another.For instance, the terms chart junk and embellishments have been used interchangeably by some researchers [3,4,24] but not others.Similarly, specific augmentations -such as threshold and regression lines -have been considered an embellishment in some studies [33] and an annotation in others [29].This paper focuses instead on the design space of visual augmentations, including the specific marks used, the types of graphical elements added to charts, and the underlying data relationships conveyed.

Augmentation Design Spaces
In prior studies, a number of design spaces have been proposed that help guide the implementation of visual augmentation tools [29,33], and to serve as references for amateur designers creating data visualizations [8].Taken together, these studies demonstrate that design spaces are useful for guiding how users augment visualizations, and how tools can be implemented to support the process.However, existing design spaces were all defined in a top-down manner, based on expert heuristics [33] or from surveys of published visualizations [8,29].In our paper, we supplement existing work by conducting a design elicitation study where participants were asked to draw onto printed visualizations how they would visually convey a set of prompts.

DESIGN ELICITATION STUDY
We adopt the design elicitation methodology to study how visual augmentations are used to convey data relationships in charts and graphs.We chose this approach in order to 1) define a bottomup, user-generated design space of visual augmentations, and 2) understand participant considerations and expectations when augmenting visualizations.In the study, participants were provided with 10 prompts and asked to draw on printed charts how they would use augmentations to visually convey each prompt.

Data Set
We used a fictitious data set of monthly store performance for five stores in a franchise.The data set had 60 rows, and included attributes: store name, month, revenue (in thousands), cost (in thousands), and profit (in thousands).This data set was created to include a variety of categorical, quantitative, and temporal data variables.
Table 1: All prompts used in the study (N: Numeric, C: Categorical, T: Temporal).

Prompts
From prior works [8,29,33], we first synthesized the most common data relationships included in various augmentation design spaces.Our synthesis focused on the visual alterations proposed and the data needed to render each augmentation.For example, stroke (or quadrant) lines were used in both VizBelle and Voder for tasks ranging from emphasizing outliers [8] to summarizing distributions [8,33].In our synthesis, we consider all of the above as examples of "threshold" data relationships that express how each data instance relates to the value indicated by the stroke/quadrant line(s).In total, we identified six data relationships: emphasis, thresholds, ranges, derived values, local/user-defined data, and trends.We then created 10 prompts based on these data relationships (Table 1).All prompts were iteratively revised based on feedback from 4 rounds of pilot studies to ensure a diversity of framings and data types.

3.2.1
Emphasis relationships reflect data values of significance in a particular usage context.To indicate their importance in a visualization, visual elements corresponding to these values are often highlighted or made more salient [8,29,33].Emphases can be defined for numerical, categorical, and temporal data variables.

3.2.2
Thresholds are data relationships that represent a value at a boundary or transition point [8,29].Threshold values can be defined for numerical and temporal variables, and can be both user-specified or statistically computed.

3.2.3
Ranges are defined by minimum and maximum value extents [8,29,33].These values can be numerical or temporal, and can be both user-specified or statistically computed.The interquartile range, for example, is a common statistic where the minimum (Q1) and maximum (Q3) values define the middle 50% of data [33].

3.2.4
Derived values are calculated from variables in the data set [33] and describe how they relate to each other mathematically.Derived values can be computed from existing numerical variables, and should be specified by users based on their domain knowledge.

3.2.5
Local/User-defined Data are additional data provided by a user to supplement an existing data set.These can be determined by prior knowledge, domain expertise, or personal approximations (i.e."data hunches" [25]).These supplemental data values should have the same variable types as the existing data set.

3.2.6
Trends describe how one variable in the data set changes in relation to another variable, and are included in nearly all prior design spaces [8,29,33].Typically, one of these variables is temporal, while the other varies over time.Trends can also refer to statistical trends between two numerical variables (e.g.regression, normal distributions etc.).However, we chose to exclude prompts about statistical trends since prior works [7,10,38,40] have extensively explored how they can be visually conveyed on charts and graphs.

Participants
We recruited 12 participants through mailing lists and by word of mouth.10 participants were aged between 23 and 27 (2 preferred not to say).All participants completed a pre-study questionnaire to assess their ability to read the types of visualizations that would be used in the study.We planned to exclude respondents if they answered more than 2 questions incorrectly or if their self-reported experience was low, but all respondents satisfied these conditions.Recruitment continued until empirical saturation, when no new augmentations were observed in study responses.The study was conducted in person, and took about an hour each.All participants were compensated with a $15 gift certificate.

Task
Participants were provided with 10 prompts (Table 1) in randomized order to mitigate order effects.Each prompt came with 2 point charts, 2 bar charts, and 2 line charts printed on paper.These chart types were chosen as they were the most commonly used visualization types surveyed in prior works [8,29,33].Participants were asked to draw on the provided visualizations the augmentations they would use to express the content of the prompt, using at least one of each chart type per prompt.They were allowed to skip chart types, but only in cases where they could provide a justification for why the chart type was unsuitable for the prompt.
In order to encourage ideation and avoid the trivial solution of just copying the provided prompt, we asked participants to avoid text-only augmentations.However, participants could use text in labels or legends in combination with a visual representation.All participants were told that they would not be evaluated on drawing precision, and were asked to think aloud while completing the task.

RESULTS
We obtained a total of 364 drawings1 , and used an affinity diagramming process to identify eight categories of augmentations used by participants (Table 2).The most common categories were: encodings, marks, threshold lines, segmentation, and shaded ranges.All participant drawings can be found in our supplemental materials2 .
Our results demonstrate broad consensus with prior works in visualization augmentation.Encoding channels and threshold lines, for example, have been included in all prior augmentation design spaces [8,29,33], and were implemented in visualization augmentation tools such as Voder [33], ChartAccent [29], and ManyInsights [9].In the following sections, we focus our discussion on unexpected or novel observations from our study results.The most common augmentation by participants was the addition of new encoding channels.While color was frequently chosen, study participants also explored using the opacity, size/radius, stroke width, and shape encoding channels (Fig. 1).These other encodings appeared infrequently, possibly because they were applicable to only some of the chart types.For example, size/radius and shape encodings could only be added to point charts, while stroke width was only added to line charts.In the case of the opacity encoding, the difficulty of "drawing" opacity may have contributed to its infrequent use.

Marks
In addition to encodings, participants also frequently altered the marks used in a chart.We identified three main categories of such augmentations: adding or layering marks onto a visualization (Fig. 3), changing the mark (i.e.changing the chart type, for example, from point to bar chart, Fig. 4), and adding glyphs (Fig. 5).Of these, glyphs appeared most infrequently, in only two instances.However, this could be attributed in part to their complexity, making them difficult to draw on paper.They may also simply be unsuitable for the prompts in our study.Further work may be necessary to better understand why glyphs were so rarely used, and what implications this has for how visualization systems and tools are designed.

Threshold Lines and Shaded Regions
Other commonly used visual augmentations were threshold lines (Fig. 6) and shaded regions (Fig. 7).Interestingly, we found that participants largely preferred using threshold lines over shaded ranges.Even for prompts designed to elicit responses about range data relationships, threshold lines were used more in all chart types.Segmentation was often used in bar charts and line charts to divide marks into smaller sub-components to convey information about individual parts that make up a whole.One visual augmentation strategy that often appeared in concurrence with segmentation was the use of partial fills (Fig. 2, middle).A partial fill occurs when a mark is divided into smaller segments, but a color encoding is applied to only some of the sub-segments.This was often used to highlight the most relevant subgroups for comparison.

Additional Augmentations
Finally, we discuss augmentations that appeared infrequently in the study, but nonetheless present interesting design alternatives.

Categorical Scale Change.
Although most participants emphasized data values using encodings, participant P11 proposed using order as an indicator of priority (Fig. 8), explaining that "I don't want to change the color because they're not more important, I just want [viewers] to see it first."Although this approach only applies to categorical variables, it presents an interesting alternative to encoding augmentations that may be useful in other contexts.4.5.2Free Form Drawing.Surprisingly, few participants in our study used free form drawings for augmentation (Fig. 9).These drawings included adding check marks or circling interesting parts of the visualization, and were similar to the informal graphical marks seen in prior work such as Sense.us[16].

DISCUSSION
While the results of our study broadly agree with many prior works in visualization augmentation, it also surfaced some unexpected patterns in augmentation usage.In the following sections, we discuss insights into participant considerations, and draw connections between our study and existing design guidelines.

User Preferences and Design Best Practices
In our design elicitation study, color encodings were frequently used across all chart types due to factors of familiarity, attention, and personal associations.However, color is also strongly influenced by the human perceptual system [32], as well as contextual factors and personal associations such as emotion [2], culture [31] and language [30].Taken together, this suggests that user preferences for how augmentations (particularly color encodings) should be used may not always agree with visualization design best practices.In future work, researchers developing augmentation tools may thus consider including recommendation features or system defaults that guide users towards more effective and accessible augmentation choices.

Biases and Misinformation
Additionally, we also want to consider how augmentations can introduce biases and misinformation into visualizations, and the implications of this outcome.Cognitive biases often affect how visualizations are read and interpreted [11,37], and may be exacerbated by augmentations.Threshold lines (Section 4.3), for example, might create an anchoring bias where readers overly focus on data around the reference value.Similarly, the frequent use of color encodings may cause readers to overemphasize the highlighted chart elements.Augmentations can also introduce misinformation into a visualization.User generated augmentations are subject to human errorsintentional and unintentional.By adding them to a visualization, these errors may then be amplified and legitimized [6,20,22].In prior work looking at data hunches [25], Lin et al. recommended distinguishing augmentations from the original visualization by giving them a "sketchier" appearance.This helps to visually indicate the uncertainty associated with user-provided data, and may be useful in combating the effects of misinformation.Future work can further investigate the impact of augmentations on bias and misinformation, as well as other strategies to mitigate their effects.

Limitations
Our study was limited in terms of the chart types included and statements used as prompts, and further work is needed to understand how visual augmentations might be designed for other chart types.Additionally, while we aimed to include prompts that would elicit a broad range of augmentations from our participants, it is possible that there are statements specific to certain usage scenarios not considered in our study.Finally, the results of our study may also be constrained by pen and paper hand-drawing methods.For example, the convex hulls in Voder [33] and ChartAccent [29], as well as the use of images and clip-art in ChartAccent [29] and VizBelle [8], may have all been under-represented as they are difficult to draw by hand.

CONCLUSION
In this paper, we conducted a design elicitation study to understand and define a space of visualization augmentation designs from a bottom-up, user-centered approach.We obtained 364 drawings, from which eight main categories of visual augmentations were identified.For each category, we provided definitions, examples, as well as insights into participant processes and considerations.Finally, we highlighted connections to existing visualization design spaces and guidelines.All visual augmentations drawn by participants have been open-sourced for further analyses.In future work, we see potential for synthesizing existing design spaces and the results of this work into a single, comprehensive reference to guide the design and implementation of tools to support visualization augmentation.

Figure 1 :
Figure 1: Examples of encoding channels used in the study.

Figure 2 :
Figure 2: Segmentation applied to bar charts and line charts.

Table 2 :
How frequently each augmentation category was used by study participants for a given combination of prompt and chart type.If more than one augmentation was used in the same visualization, we counted the visualization twice.The most common augmentations are highlighted in blue.The calculated totals of fill color and stroke color have been combined.

Figure 4 :
Figure 4: Examples of participants changing the mark used in a visualization.From left to right: changing a point chart to a bar chart, changing a point chart to a line chart, and changing a line chart to an area chart.

Figure 5 :
Figure 5: Two instances where participants used glyphs to convey the prompts.

Figure 6 :
Figure 6: Examples of threshold lines added by participants.Multiple threshold lines may be added to a single visualization, and each line could be customized by style and color.

Figure 7 :
Figure 7: The visualization may be shaded to highlight important value ranges along an axis.

Figure 8 :
Figure 8: Two augmentations by P11 where they changed the order of elements (new order listed below) to convey priority.

Figure 9 :
Figure 9: Examples of informal free form graphical marks made by participants.