Designing Grocery Shopping Experiences for Virtual Reality

Online grocery shopping offers time-saving efficiency and convenience, yet many people still prefer physical shopping for trust in food freshness and other sensory experiences. While online stores are evolving to offer new user experiences, such as supporting eco-friendly or ethical shopping, the desktop and mobile platforms limit the engagement of such experiences. Virtual Reality (VR) presents an opportunity to create immersive and rich grocery shopping experiences, closing the gap between the convenience of online shopping and the sensory experience of physical shopping. Yet, designing VR grocery stores remains relatively unexplored. In this paper, we investigate the long-term potential of VR grocery stores, focusing on meeting individual needs. Through a co-design workshop, participants brainstormed the design of product displays, in-shop navigation, shopping carts, social shopping, among others. Based on our findings, we provide design recommendations for future VR grocery shopping to develop meaningful alternatives to existing shopping experiences for groceries.


INTRODUCTION
The human-computer interaction (HCI) community has extensively explored ways to enhance the online grocery shopping experience to cater to a broad spectrum of user needs [40].Features such as supporting meal planning [38], facilitating healthier food choices [9], enabling eco-friendly product purchases [20,33], and assisting people with visual impairments [18] have been proven to be desirable by individuals.While online shopping is often considered convenient, it lacks the sensory-rich experiences of physical shopping, limited by device-centric, two-dimensional interfaces.This highlights the need to shift towards an experience-focused, heads-up shopping paradigm.The metaverse, which incorporates numerous technologies [19], including virtual reality (VR), opens up new possibilities for shopping.For example, by utilising VR, we can provide an immersive shopping experience where users can actively engage with three-dimensional products with the sense of touch, for instance, simulating food weight to increase realism and immersion [37].Moreover, virtual reality's unique affordance of shared co-presence [10] allows users to shop alongside their friends to make joint decisions without being physically co-located.Furthermore, personalised systems can support and simplify users' decision-making process, filtering out or navigating users through many products to find the perfect match.To explore how to design and interact in a VR grocery store, we conducted a multi-stage co-design workshop with six participants to explore how a VR grocery store can be designed.Our results suggest that VR grocery shopping could provide unique and engaging experiences that are difficult to achieve with traditional physical shopping or through two-dimensional screens.

RELATED WORK
Shopping technologies, including smart devices, robots, software services, can enhance user experience through features like Kahl et al. 's smart cart [16,17], humanoid social robots that boost shopper engagement [7] and multimodal shopping lists [14].Additional innovations [34], such as transparent voice-shopping systems [28] and Fornelos et al. 's conversational agents for virtual try-ons [11], offer on-demand support and improved shopping transparency.
Building on the benefits of existing shopping technologies, VR technologies further enhance customer experiences.They do this by promoting sustainable behaviour, healthier consumption, ethical decision-making, etc., through 3D effects and immersive storytelling.For instance, research demonstrates that 3D visualisations in VR have a more potent effect than two-dimensional pictures or texts, significantly enhancing VR's ability to influence user perception and behaviour towards sustainability [6].Besides, VR can reduce cognitive load by using cognitive adaptive VR [5] or utilising VR's unique spatial affordances [27].In VR, an intelligent shopping companion can guide users through various products by explaining the classification or styles, reducing the shopping workload.For instance, almost 25% of consumers are overwhelmed and spend more than 10 minutes in front of a wine rack not knowing which product to buy [30].Furthermore, users can get personalised recommendations based on their budget and preference of origin or even immerse themselves into a vineyard or winery if they are interested in the grape-growing or winemaking process, which can be further strengthened using olfactory feedback [3].
Nonetheless, it remains unclear how a VR grocery store can be designed.Initial work by Speicher et al. [36] explored the suitability of a virtual apartment environment for shopping.They discovered that customer satisfaction can be enhanced by allowing users to position products in situ within the virtual apartment.In another work by Speicher et al. [35], the authors explored the design and evaluation of a smartphone-based VR grocery shopping environment and recommended using speech input for mobile VR shopping.Others, such as Wang et al. [39], introduced UI design guidelines for mobile-based VR shopping, providing helpful references for the design and development of panoramic VR.While prior works provide promising research directions for VR-based shopping experiences, there is still a lack of user-centric design and an understanding of how customers envision shopping in VR.

CO-DESIGN WORKSHOP
Our co-design workshop aimed at a) having an initial understanding of the motivations and challenges when shopping in the real world and online, b) brainstorming ideas that better fulfil individual needs in VR, and c) deriving design concepts and mock-ups using generative AI for VR shopping experiences.
We recruited 6 participants (2 female, 4 male) aged 21 to 28 ( = 24,  = 2.37) for the intensive, hands-on co-design workshop.Participants were all students pursuing a Bachelor, Master or PhD degree in computer science or consumer science.All of them have shopped online before, but none of them do grocery shopping online.All participants have experiences with VR or AR in various settings, including university contexts, gaming, and museums.
The workshop lasted 5 hours, including a 1-hour break in the middle.Participants signed a consent form for data collection upon arrival in the lab.We audio-recorded the workshop and used a Miro board as a co-working space.First, we introduced the status quo of physical grocery shopping, online shopping and shopping in VR by showing pictures.Second, we conducted semi-structured group interviews to understand participants' positive and negative experiences with grocery shopping online and in physical environments.Third, we organised an in-depth brainstorming session using the How-Might-We method [31] to stimulate ideas about nine key topics.These topics, which emerged from internal discussions with two researchers, included 1) product display, 2) product search and in-store navigation, 3) product inspection and customer review, 4) shopping carts, 5) product comparison, 6) social aspect, 7) checkout procedure, 8) customer support, and 9) personal needs.During the brainstorming, participants wrote ideas on sticky notes for each topic and shared them with the focus group.Then, participants presented their ideas and grouped similar ideas.This helped identify emerging themes and provided a clearer overview of all ideas, simplifying the subsequent concept development phase.Participants then voted on the ideas based on their preferences.After this, two design directions emerged from previously generated ideas.We then assigned participants into two groups based on their interests of the design directions.The task was to develop more detailed concepts further and use Midjourney [24], a generative AI tool capable of creating images from natural language, to prototype mock-ups of possible VR store concepts (see Supplemental Material).
To analyse the interview data, all audio recordings were transcribed verbatim using WhisperX [21], and imported into Miro.We then conducted an affinity mapping to derive themes from the interview data, as described by Blandford et al. [2].Furthermore, for the analysis of the brainstorming idea data, two researchers iteratively analysed the data generated by the participants in Miro.To derive themes, we performed affinity diagramming [2] on participants' initial clustered data.Through this process, different sticky notes initially grouped by participants were rearranged and sorted into overarching themes derived from the affinity mapping.

RESULTS
In the group interview, participants preferred physical grocery shopping over online shopping.This preference can be attributed to several factors: 1) Participants prefer the tangible experience of in-person shopping, which includes physically inspecting items, assessing their quality, and finding new discovery while browsing the store's aisles.2) They favour the sensory elements, such as the olfactory ambiance of the store and the tactile nature of interaction with products.3) They are reluctant to pay delivery fees; especially for those living near the physical store, in-person shopping is more economical for small or frequent purchases.Conversely, participants opt for online shopping for its cost-effectiveness and broader product selection.The convenience of doorstep delivery, especially for bulky or heavy items, is a key factor in their preference.However, despite their preference for in-person shopping, participants report dissatisfaction with the physical grocery experience during peak hours, marked by crowds and limited product availability.They prepare shopping lists and strategically plan their visits to mitigate these issues.Yet, they remain open to exploration, often drawn to unexpected discounts and novel stores, particularly stores in foreign countries.In the following, we present the results for the nine topics addressed during the in-depth brainstorming session on designing VR shopping experiences.

Product Display
Most participants preferred 3D models over traditional 2D images for product visualisation in online shopping.Additionally, there is an emerging need for 3D representations of food items inside the packaging, or an interactive unpacking feature.This allows customers to virtually explore the contents inside before purchasing, fulfilling their curiosity or providing more information, such as checking the cookie filling.For products where freshness is crucial, using digital twin technology to create accurate replicas was highlighted.This paradigm aligns with the overarching design direction of Replicate the Physical Store.An alternative perspective is captured within the Explore New Possibilities in VR.Some participants desired VR stores not merely to mimic traditional physical stores, but to utilise VR's unique capabilities.This approach meets specific needs, such as offering package-free, eco-friendly options by providing customisable product sizes beyond standard packaging dimensions.

Product Search and In-store Navigation
Participants favoured streamlined navigation in the VR shopping environment, enabling direct product access and boosting efficiency.They appreciated the idea of automatically reading shopping lists and instantly teleporting users to each product's location, eliminating the frustration of navigating through numerous aisles and searching for hard-to-find products.Another preferred feature was a video game-inspired map interface, positioned within the user's field of view, enabling easy selection of areas of interest and movement across different store sections, thus promoting easier in-store movement.For those who wish to bypass the hassle of aisle navigation yet still enjoy relaxed product exploration, a sushi conveyor belt-inspired interface presents a novel solution.This approach displays products from the user's wishlist on a conveyor, allowing for a gradual and leisurely examination of items of interest.Alternatively, users could be positioned on a conveyor belt, allowing for a combination of exploration and transportation at a deliberate Additionally, participants expressed a preference for personalisation of the physical store, a design concept that organises products into themed rooms, such as 'party' or 'barbecue' rooms, tailored to specific occasions, trends, or seasons.This setup provides access to all relevant items in one place and offers inspiration through curated selections.Furthermore, users could customise the store by filtering out products based on individual health conditions, dietary restrictions, or fitness goals.Although this reorganisation is well received, it is not intended to completely replace the traditional layout.Instead, it aims to give users more control over their experiences by allowing adjustments to the store's structure.

Product Inspection and Customer Review
In multi-modal product info and text enhancement, participants advocated for a floating interface by each product for detailed information, with features like text magnification for better accessibility.They also favoured incorporating haptic feedback to convey sensory product details like weight and texture, which was perceived as instrumental in facilitating a nuanced evaluation of product quality.Moreover, incorporating videos or animations into the virtual store was well-received among participants, valued for demonstrating various aspects, such as food preparation techniques, culinary applications, and ethical supply chain practices.In augment reviews, participants suggested visual enhancements for product ratings, such as marking high-rated products with a golden hue for prominence and lower-rated ones in grey.

Shopping Carts
Design ideas for shopping carts were divided into two directions: simulating reality, with a preference for classic shopping baskets where customers engage in the hands-on process of grabbing items and placing them into the basket.However, these virtual baskets should be capable of holding an infinite number of products, shrinking, and smoothly transitioning to the corner of the viewport to mimic the location of the shopping cart icon in web-based stores.For exploring alternatives, a novel metaphor was introduced -a grid inventory akin to those found in video games.Beyond traditional baskets, items could also orbit around the shopper.

Product Comparison
In 3D comparison, the VR store could mimic natural comparison behaviours, where users use hand gestures to pick up and examine two products side by side, with enhancements such as highlighting differences in price and nutrition information.Furthermore, the recommender system could suggest similar items by making them float within the user's view or marking them on virtual shelves.Concurrently, products deemed less relevant could be visually deemphasised on the shelves.

Social Aspect
Enable social shopping highlights the demand for co-shopping experiences, including features like discussion rooms and the unique option to blend in or out other shoppers.In gamification, the store can integrate gamified elements into co-shopping by, for example, providing missions for customers to win prizes or incorporating mini-games, thus enhancing exploration and enjoyment.

Checkout Procedure
Simple and quick checkout reveals a preference for a straightforward, one-click checkout in VR, similar to traditional e-commerce, without having a virtual cashier.However, participants also highlighted the need for customisable checkout options, including interacting with virtual cashiers for those who prefer tangible checkouts with a human touch.

Customer Support
Participants emphasised the desire for an on-demand virtual AI assistant, to enhance customer support.Additionally, for a more personalised touch, the VR store could offer real assistants, through holographic humans or animated avatars, entering the user's virtual space for direct interaction, especially when AI support fails to meet expectations.

Personal Needs
Nutritional needs are voiced and can be fulfilled by tailored meal suggestions according to the items in one's shopping cart or refrigerator.The store can suggest additional purchases to meet nutritional goals.This idea can be enriched with a virtual cooking test station, letting users experiment with selected products.Additionally, animated cooking tutorials can be incorporated to help users follow recipes more effectively.Sustainability needs: The store could highlight products that are locally produced to support the local community and reduce logistical costs, contributing to sustainability.An impact visualisation could be displayed through a map-like interface, marking the origins of ingredients or production machinery, along with information on their CO2 impact or footprint.Moreover, prominently showcasing alternatives to meat and fish could address ethical concerns by providing choices that minimise dependence on animal products.

DISCUSSION
In the following, we discuss the tensions between the ideas of realworld physical replicas and the experiences beyond real-world, and the challenges that these designs could introduce.Traditional Supermarket Layout vs. Personalised Shopping Room: Following a traditional supermarket layout brings consistency, which is typically considered a fundamental usability metric [25].This is evident not only in physical stores, which share consistent layouts to reduce cognitive load and friction, but also in e-commerce websites that offer a consistent customer journey.Yet, personalisation could offer benefits such as enabling goal-nutrition-based shopping, such as creating shorter routes to all vegan shelves, which poses the challenge of maintaining consistency.Thus, deciding which aspect to prioritise and determining when and how to do so, poses interesting future research topics.Besides, Ploydanai et al. [29] found that a store's layout can impact customers' shopping behaviour.As VR store layouts could be more flexible and individualised, it is interesting for future research to explore how different layouts of VR stores influence customers' shopping attitudes and behaviours.The numerous layout possibilities could also bring the risk of dark patterns, such as misdirection or obstruction, by e.g.changing the layout to prolong the routes and adding more distance to the checkout point.Diegetic 3D Model vs. 2D User Interface: Most Participants perceived three-dimensional product visualisations with direct spatial hand interaction as beneficial, which aligns with the research from Ogier et al. [26].They compared a non-diegetic (list-based UI) to a diegetic in-game store (stock-on-shelf), finding that the simulation of a real-world store can improve the immersiveness of players.However, this conflicts with other literature and stores that apply card-like, non-diegetic UIs for displaying products.For example, Wang et al. [39] proposed a demo displaying the product in a two-dimensional user interface.Physical Cashier vs. One-Click Checkout: While a physical cashier, in the form of a VR avatar, could be retained to uphold some social elements in an entirely virtual shopping experience, the downside is that this could also include the cumbersome process of placing all shopping items from the baskets onto the conveyor belt.Alternatively, this could be redesigned to allow a shortcut action while still preserving the social interaction with the cashier.In contrast, the one-click checkout, which is common in e-commerce, can save significant time.Future work is encouraged to investigate the trade-off between convenience in VR shopping and social presence, and how this can be best balanced.Natural Locomotion vs. Teleportation: various locomotion techniques allow users to navigate in virtual environments [8,32].Our participants discussed two core directions for navigation in VR shopping experiences: exploratory locomotions and time-efficient locomotions.This is an exciting approach, as shopping has unique affordances that might result in different locomotion techniques.For example, many people make use "shopping lists" to support their shopping in the real world, yet they often also explore relevant discounts.Finding the balance between navigating in a VR shopping experience for "exploration" or "efficiency" remains an open question and builds an interesting future research direction.For example, while redirected walking in VR can contribute towards immersion and support exploratory experiences, walking using teleportation might be more time efficient.
Realistic Shopping Cart vs. Surreal or Creative Shopping Cart: While a realistic presentation of a shopping cart is preferred, participants also emphasised the importance of not being restricted by the functionalities of a physical cart.Moreover, the appeal of more surreal or creative shopping carts remains strong.Interestingly, the benefits of abstract shopping carts are highlighted by Speicher et al. [36], showing that an abstract shopping cart outperformed a realistic basket in terms of error rate, user experience, and workload.Blend In/Out Bystanders (Public Space vs. Personal Realm): The need to avoid crowds in the real world opens up interesting possibilities for future VR shopping, where users can blend in/out bystanders at any time depending on their preferences.For instance, some scenarios may benefit from the presence of virtual bystanders to facilitate casual conversations or small talk.While in other scenarios, users might simply prefer a quick and focused shopping experience.This leads us to discuss the factors that influence customer preferences between public spaces and personal realms, such as the feeling of loneliness, the need for realism, or fatigue.Besides, controlling the visibility of bystanders brings more benefits, especially when they feel being harassed or when the co-presence of additional customers introduce significant cognitive load.Likewise, an example from Chiossi et al. [4] adjusted the amount of NPCs according to users' cognitive loads in VR.Nonetheless, privacy issues may emerge if the settings for blending bystanders in or out are not mutual, allowing one user to observe another without their knowledge.Challenges of the VR Shopping: The brainstormed ideas did not consider technical and financial feasibility, presenting potential implementation challenges.The hurdles include creating digital product replicas, simulating physical properties, and providing multisensory experiences, which demand technical expertise and financial resources, even though recreating objects in virtual environments and developing high-fidelity digital twins is feasible [1,12,23].Besides, despite advancements in sensory technology [15], such as haptic gloves [13], accurately replicating food characteristics such as temperature, aroma, or flavour remains a challenge to address in future research.In addition, societal and privacy concerns can arise by employing VR shopping.Before deploying such a system in the wild, social risks tied to virtuality, such as diminished face-to-face interactions and a lack of connections to reality, need to be addressed.This aligns with [22], who uncovered the concerning phenomenon of individuals spending a lot of time in virtual world and becoming detached from reality.Furthermore, it's crucial to tackle privacy risks, including accessing shopping lists, personal dietary preferences, movement trajectories in the VR store, shopping schedules or frequency, and purchase preferences.

CONCLUSION
In this work, we contribute design concepts for VR grocery stores through a co-design workshop, covering topics of product display, in-store navigation, among others.We summarise the benefits of VR shopping, from time-saving efficiency, to cater to elevated needs, encompassing nutritional, environmental, ethical, and social aspects.We further discuss two main design directions for VR grocery shopping: (1) simulating existing physical stores and (2) creating new virtual stores based on VR's unique affordances.Our work thus provides design inspirations for future grocery shopping experiences.