Experiencing the Architectural Evolution of a Heritage Museum in Extended Reality Application

Although the use of Mixed Reality technology in knowledge, leisure, and cultural domains is ubiquitous, it has remained limited in heritage contexts. Museums are the DNA of each culture and contain exquisite heritage collections worth premier exhibit to be perceived, absorbed, and appreciated. When the heritage of the past is located within an architectural heritage, the connection and attachment to the roots of a culture are magnified. Hence, the heritage building of museums necessitates documentation and demonstration to visitors as much as the heritage artifacts. This paper uses a case study from DIKE (Digitization of Indigenous Knowledge for Extended Reality (XR) and Culture), German Summer School, 2023 to present a novel approach to utilizing Augmented Reality (AR) to develop a mobile application that facilitates the experience of the architectural evolution of Sarawak Museum. The APP »Museum Time Machine« showcases the architectural changes of Sarawak's heritage museum from 1891 to 2022.


INTRODUCTION
Museums are one of the most critical building typologies, which often create iconic landmarks in urban spaces and serve as a symbol of the art and culture of the society they represent.They are the repositories of culture, art, science, nature, documents, and traditional crafts.Museums contribute to preserving and conserving natural and cultural heritage through documentation and educating the community about the original collections [1].They safeguard and showcase the tangible and intangible heritage and keep the cultures alive [2], [3] and [4].In addition, these non-profit institutions oversee conducting research, development, and interpretation of the collections of original objects for the benefit of the public."They function publicly as places where people learn from and find inspiration and enjoyment through the display and research of original objects [5]." Museums often function as repositories of objects and specimens representing an objective record or collective memory [6].However, they can have a far more intellectual position related to anthropological memory and education that bridges the past and future.Museums indefinitely accumulate time, constituting a place of all times that is itself outside of time [7].Heritage museums have more layers of meaning than those immediately meeting the eyes.Although heritage museums are an essential part of human history, they are often more concerned with cultural and historical artifacts and fail to provide an authentic experience of architecture.Moreover, these cultural collections in the form of architectural features and spaces are too large to be exhibited and archived in museums [8].
Heritage within a heritage: Heritage museums reflect a society's cultural values and traditional aspirations, especially museums within heritage buildings.They are not just cultural buildings but historical monuments that hold architectural significance and must be preserved and learned.Many museums worldwide have heritage buildings, but only some are designed initially to be museums.Many were originally buildings with functions other than museums, such as Historical Palaces, e.g., the Uffizi Gallery in Florence, Italy, the National Museum of Qatar, and the Louvre Museum in Paris, France.On the other hand, some heritage museums were intentionally and designed initially to be museums, such as the National Museum of Natural History (Washington, D.C., USA), 1910, the National Museum of Korea, 1945, and The National Museum of Anthropology, Mexico, 1964 and Sarawak Museum, Malaysia, 1891.Preserving and learning about such museums fosters a sense of identity and connection to the past.The preservation of the museum provides a sense of identity [9].and belonging [1] to the community.Moreover, it teaches people the value of cultural heritage and local history [10].What is undoubtedly precious is creating a sense of collective memory through the education and conservation of the significant heritage of the past [11].
Heritage Conservation: Typically, precious museum collections are subjected to deterioration primarily when located within heritage as historic buildings are susceptible to risk due to factors such as aging [12], natural disasters, and fire [13], and environmental factors [14], urban development, or lack of maintenance.Hence, conserving historical museums is vital to mitigate the risk of deterioration.Consequently, Historic buildings usually go through various architectural modifications throughout their existence.
Architectural Evolution of Heritage: Museum modifications are commonly exerted to enhance the preservation of artifacts, visitors' experience, and the overall functionality of spaces, viz., expansion, accessibility improvements, renovation and restoration, and environmental upgrades.The alterations in the museums with heritage buildings are mostly conservation-related.The conservation aims to protect heritage buildings without compromising their architectural integrity.Visitors expect to observe the architectural integrity of the heritage buildings and deserve to know to what extent the historic building retains its original components, style, and structure.Many studies, such as [15], seek the public's perception (layperson and professionals) towards preserving historic buildings' architectural style and form despite the conservation efforts to maintain the architectural integrity of heritage museums.The primary responsibility is to keep the historical buildings, which demands specific alteration of their physical forms.Hence, the architectural evolution of museums throughout time was almost inevitable.
The application of Augmented Reality (AR) makes the learning environment more engaging [16] and concurrently develops a more positive emotional attachment [17].Both the former and the latter are essential in connecting the current and future generations with the heritage and values of their past.AR technology is known for the positive triggering experience in museum settings [18].However, most AR applications focus on supporting the learning process of history, science, art [16], and interior exhibits; they miss out on the fact that some museum buildings are the cultural heritage of the past that need to be treated as artifacts to be explored and learned.Studies have shown that Youngsters find it more interesting when museums are more engaging or immersive with the latest technologies, such as virtual reality.Despite the rapid development of XR, the application of Augmented reality (AR) in heritage is limited [19].Hence, this paper seeks to facilitate the experience and education of the architectural changes and conservation of the Sarawak Museum by leveraging XR technology.

STUDY CONTEXT
Sarawak Museum is the oldest museum in Southeast Asia, located in Sarawak (The eastern part of Malaysia), the western part and one-fifth of the Island of Borneo right across the equator.The museum was built in 1886-1888 during Sir Charles Brooke's era, who ordered this project under the influence of his friend Alfred Russell Wallace, the co-founder of the evolutionary theory (natural selection paper's author) [20].The museum incorporated Neo-Palladian architecture style, with prominent features such as symmetrical classical architectural elements and grand appearance, reflecting the colonial state identity of the city.At the time, the Sarawak Museum played a unique role in preserving social and racial traditions, the revival of local and national identity, and the revival of the pre-European past (ibid.p. 240).
Due to the rich wildlife and severality of racial groups with ideological cultures in Borneo Island, the heritage museum was designed to serve as a repository of natural and cultural artifacts.The collection of local fauna and artifacts inspired the creation of this museum, and it eventually displayed extensive ethnographical (clothing, tools, weapons, musical instruments) and natural (taxidermy of the specimens found in Sarawak) history artifacts that aimed to teach people about the rich history, biodiversity, and culture of indigenous people.Some of the artifacts contained information and materials related to orang-outang, flying squirrel, tarsier, hornbill, king cobra, giant toad, centipede, fish traps, mats, beadwork, ritual objects, agricultural tools, value beads, tattoo designs, hair-dresses, canoes, and many more natural and cultural assortments.
This iconic building is located in a unique context of Kuching yet within a garden that acts as an urban node that attracts local and international tourists to experience the culture, heritage values, and past stories.The architectural aesthetic and cultural significance of the Sarawak Museum has made it one of the most cherished landmarks in the city.Many architectural changes have been made to the museum during the past centuries.While most of the books and published scholarly works focus on museums in West Malaysia, e.g., [21], it is believed that the cultural and architectural values of heritage museums in East Malaysia need more devotion and attention.
The building of Sarawak Museum, often referred to as the gem of the East, has one of the oldest structures and has undergone many architectural changes due to conservation and other various reasons that are worth preserving and presenting to the public as the knowledge of the museum.Despite people being interested in their pre-colonial culture during the colonial era, currently, there is interest in knowing about the colonial buildings and the Charles Brook era.Besides, the museum has been closed to the public for over five years and has yet to open at its total capacity.Therefore, the present study sought to provide an alternative experience of the architectural evolution of the museum to the public through the application of augmented reality.

Accessibility Building Material used to construct the Museum
The entire museum envelope, including the new wing, was made of bricks, with both old and new floors constructed of Belian or hardwood (ironwood) which is found in the Borneo rainforest Higher floor beams are made of steel beams (L-shaped) with timber floorboards on top of them.The roof structure incorporated natural materials such as Belian timber or iron wood as supporting arches and beams.However, due to the scarcity of Belian timber, the additional structures were made of steel beams encased in timber to match the old structure.The main rafter of the roof has bearing blocks made of hand-carved ironwood.Concrete has been used in making parapet walls and decorations.The original roof was made of traditional and natural Bornean materials such as palm leaves, and thatch, which were suitable to keep the museum thermally cooler, but they were wrecked and eventually replaced by Belian wood shingles.Figure 1 shows the bird's-eye view of the museum as it is today and the image of the last conservation of the museum in 2022.

Environmental design
Certain architectural features in the Sarawak Museum are designed to respond to the climatic conditions of Borneo Island, mainly heat reduction and protection against heavy tropical rain.To combat the heavy rainfall in Borneo, a 45-degree steeped-pitched roof was

RESEARCH METHODOLOGY
Data Collection and Design Phase: The first phase of the APP Development was to find the architectural changes imposed on the building to land on accurate periods and trace back the evolution of the museum.We conducted an on-site observation by taking a tour with the entire team, Professor Jürgen Sieck, and the conservationist architect Ar.Haris Fadzilah around and through the museum, enabling us to take several reference photographs of the exterior facades and interior and observe the historical photos in the museum's archives to help us better understand the structure of the building.The conservationist architect explained the architectural features of the museum, the materials used, and construction and conservation techniques.In a semi-structural interview, he thoroughly explained the rationales behind every alteration to the buildings and the periods.We also took a photo of the large Tree of Life painting on a wall, which was later included in the introduction screen of the app.Subsequently, we collected the architectural 2D drawing from the conservationist, both hand-drawn and AutoCAD drawings, handwritten notes by the architect, and photographs of the conservation process and historic photographs from the architect's collection.In addition, the team collected visual documentation by Architect JFM and John Ting architects, created in 2016, illustrating the architectural changes in the front elevation of the Sarawak Museum from 1891 to the 1990s.Figure 1 shows the key architectural elements that were emphasized by the conservationists and were repeatedly mentioned in various primary and secondary data.
The 3D model of the museum, as it is today, was created using SketchUp software, based on the collected photographs during the visit, the 2D AutoCAD drawings, and accurate blueprints provided by the conservationist Architect Haris Fadzilah.In addition, the 3D models of the historical elements and the changes were created by referring to old photographs obtained from online resources and museum archives i.e., giant staircases, balconies, and dormer windows (Figure 2).
The 3D model of the museum as it is today was created using SketchUp software.In addition, the 3D models of the historical elements and the changes were created by referring to old photographs obtained from online resources and museum archives, i.e., giant staircases, balconies, and dormer windows.The Museum Time Machine app was developed using Unity and its AR Foundation package.The development team first obtained requirements and completed the design and planning process to know what needed to be done for the app.We determined we needed to show the changes made to the museum building over the years, highlighting its rich history.The app was to be developed using augmented reality, showing the building on top of a flyer.
The Museum Time Machine app was developed using Unity and its AR Foundation package.The development team first obtained requirements and completed the design and planning process to know what needed to be done for the app.We determined that we needed to show the changes that have been made to the museum building over the years, highlighting its rich history.The app was to be developed using augmented reality, showing the building on top of a flyer.The 3D model of the museum was separated into parts that could be individually animated, moving in and out of position around the virtual model as different time periods were selected.For instance, a roof appears from above and lands on the building at one stage but is removed later on corresponding with more recent changes made.Besides, 2D icons used in the app's menus were created mostly by the team using Inkscape, with some downloaded as free resources from Flaticon.Other features of the APP are Plane Tracking which allows the detection of flat surfaces using a device's camera and is used to anchor virtual objects to them in real space.Feature Points are specific points on the 3D model.They allow for extra contextual information to be displayed  Narrations-Audio: The content explaining each scene belonging to a specific era with the architectural change was narrated by one of the team members, recorded in an acoustic proof room, and synced to each particular frame.The development team developed a few lines of narration to be included in the app to enhance the experience.These lines play once every time the user selects the next stage of development of the building, orally explaining the changes that occurred.We also included sound effects obtained from Zapsplat, such as construction sounds, as a means of feedback to the user when the user selects a different period.
Design of Flyer-Augmented Reality Marker: The creation of the app's companion flyer using Canva occurred in parallel with the development of the app.We went through several iterations of the flyer, varying its information.The flyer intended to provide more information not shown in the app, such as the historical photos showing the original grand staircase.The visual of the building on the flyer was rendered using Lumion in a stylized sketch-like manner to draw attention.It was also crucial for the flyer to be distinctive, as it would act as the AR marker for the museum to appear on when the app is used in image-tracking mode.

RESEARCH FINDINGS 4.1 Content Analysis
The content analysis of the collected data from onsite observation and interviews with the conservationist architect led the team to seven significant architectural changes in Sarawak Museum in the past 135 years and their underlying reasons.The key findings are as follows: •

Augmented Reality Application
According to the findings of this study, the architectural changes in Sarawak museums were due to safety, environmental, and space constraints.Previous research, such as Barbieri et al. [22], indicated that space constraints in museums are inevitable and museums are constantly being found.Augmented Reality Markers are visual signals that trigger the exhibition of virtual information processed by an AR application or through a camera and are frequently applied in the museum context (Figure 4).Augmented Reality Markers are visual signals that trigger the exhibition of virtual information processed by an AR application or through a camera and are frequently applied in the museum context.The augmented Reality Marker of the Museum Time Machine APP has been designed as a brochure, which includes the instructions to use the Museum's APP in four stages, translated into English and Malay.In addition, the marker displays a timeline ranging from 1891 to 2022 that shows seven fundamental architectural changes made to the museum.Instruction number four requires visitors to scan the marker; as soon as the user's phone camera captures the marker image, the 3D presentation of the museum will appear on top of the marker sheet.The marker makes using the APP easier even with low Wi-Fi/internet connectivity and possible GPS signal loss.However, two options are given, image tracking or plane tracking, to make it possible for users without the booklet to use the app (Figure 5).
Furthermore, the Museum Time Machine APP includes a toggleable feature point scanning system, letting users find exciting points of the museum on their own.Toggleable feature points would facilitate further exploration of each architectural element of the museum, especially for visitors who have a background in architecture or are interested in engaging in such a learning process (Figure 6).

FUTURE IMPROVEMENTS AND DEPLOYMENT
We intended for the app to be released on the Google Play Store and the Apple Store after completion.The user could download and use the app in plane tracking mode with basic features.We planned for the flyer itself to only be obtainable at the museum.This way, we could restrict certain features to being accessible while using the app in image tracking mode to encourage visitors to visit the museum rather than check it out virtually in the app itself.It is possible to introduce features that would only be accessible with the flyer on hand to encourage users to visit the museum and obtain a flyer themselves.Gamification is also possible, in particular, a treasure hunt-like mini-game where users can explore in and around the virtual museum building to look for artifacts and collect points and collectibles.It is also possible to add the interior of the museum.gamification increases the level of interaction of the visitors with the APP as well as the museum and it makes the learning of the educational content and facts about the museums more enjoyable.The museum contains several old taxidermy exhibits that can be reproduced virtually using 3D scanning technology.Further enhancements could be made by adding relevant animal sounds to the exhibits looking at them.The architectural changes presented in the augmented reality App are not exhaustive; more modifications might be exerted on the Sarawak Museum.However, this study has tried to address the major ones and those which are architecturally significant externally.Once the museum is more accessible to the general public for a visit and completely reopens, additional changes to the interior spaces can be added to the existing APP.Expanding The application of XR to include the museum's interior could provide a more comprehensive experience but was not possible due to the closure of the museum-only a one-time visit was given to our team for a very short time to understand the museum's history and contents.The main limitation of this study was the need for more access and time to the interior space and artifacts of the museum, which was unfortunately impossible due to its ongoing renovation; otherwise, the team could have made the XR experience of the interior and exhibits, too.We recommend future studies to expand the XR experience to the interior space of the Sarawak Museum.This mobile application educates people through the virtual experience of architectural heritage.It enables the community to travel back in time and perceive the architectural heritage of the museum in its original form which helps in preserving the memory of the museum and simultaneously encourages the engagement of the local community in preserving the museum for future generations.The app is available to both visitors and those who are unable to visit the museum on-site.

Figure 1 :
Figure 1: Sarawak Museum in the present time, Right) Last Conservation of the museum in 2022

Figure 2 :
Figure 2: Left) Old photographs of the architectural elements of the museum from various sources: staircase, balcony, arches, and dormer windows (top, left to right); Right) Post conservation of 2022: Kayan's Tree of Life mural located on the first floor at the ethnological galleries.

Figure 3 :
Figure 3: Sarawak Museum in the present time, Right) Last Conservation of the museum in 2022 Technical development of Augmented Reality App for Sarawak Museum 1) 1891: The old Sarawak Museum had only a west wing and a grand staircase.• (2) 1911: The museum was extended by adding an east wing to make room for more exhibits -to increase interior spaces • (3) 1912: The grand staircase was removed -To create a clean and symmetrical façade.A new staircase was added to the south veranda to access the upper floor.• (4) 1940: The verandas on the ground floor and the firstfloor balconies were covered, and the front porch was added -expansion to increase interior spaces (by walling up of verandas and balconies) in the 1940s.A front porch was added to the Central Wing as a new main entrance.The wall behind the porch was opened to accommodate double timber doors.An internal grand timber staircase was introduced to the Central Wing during this period to improve circulation control of the building with only one entrance.• (5) 1958: The roof dormer windows were removed, another roof was added on top, and the faux bricks were painted white -The building seemed to suffer from a severe roof leakage problem.It was not due to the inappropriate materials, techniques, and detailing of the flat roof construction over the balconies to discharge rainwater.A new roof over the entire existing roof was constructed to solve the roof leakage problem by constructing a new roof over the entire existing roof.This new addition altered the roof's features, such as removing dormer windows and creating roof overhangs.The faux brick might have been covered with plaster and painted with lime wash during the 1940s renovation to suit the new wall.

Figure 4 :
Figure 4: A scene from the final augmented reality android APP of Sarawak Museum

Figure 5 :Figure 6 :
Figure 5: Augmented reality market and brochure of the Sarawak Museum's APP