Challenges in Cyber-Physical Attack Detection for Building Automation Systems

In recent years, the vulnerability of smart buildings to cyber-physical attacks has been drastically increasing. Due to enhanced device connectivity and increasing complexity of these systems, attacks are also becoming significantly more severe and can cause immense harm. This paper focuses on the challenges of implementing a physics-based attack detection scheme for HVAC systems in Building Automation Systems. Using a real-world HVAC system as a testbed, we apply a heat balance equation based model and non-parametric CUSUM statistic to detect false data injection attacks. The results highlight HVAC-specific challenges, pitfalls, and potential solutions, contributing valuable insights for future practical implementations in smart building security.


INTRODUCTION
The rise in cyber-attacks on smart buildings has become a pressing concern in recent years, with commercial buildings, research facilities, and critical infrastructure being particularly attractive targets for attackers.Building Automation Systems (BAS), especially heating, ventilation, and air conditioning (HVAC) systems, are at high risk due to their inherent vulnerabilities, increased connectivity, and potential to cause immense harm [5].These systems frequently operate on outdated protocols, lack advanced authentication or encryption methods, and have extended device/system life-cycles, making them susceptible to attacks [12].Therefore, it is crucial to prioritize the implementation of attack detection and mitigation strategies for these systems.
Attacks on HVAC systems can cause significant damage to the building itself, its inventory, and occupants.Additionally, these systems can be exploited as gateways for attacks on other critical infrastructure.This paper specifically addresses cyber-physical attacks, which cause physical changes in the attacked system.Cyberphysical attack detection on HVAC systems can generally be performed via rule-based attack detection schemes or via data-based attack detection schemes, which are often Machine Learning based [12].However, these two main categories are not mutually exclusive.Depending on the specifically used procedure, approaches have to be assigned differently.In this paper, we focus on physicalformula-based attack detection, which is considered a rule-based attack detection scheme.We select this scheme due to its high accuracy while still providing generalizability, as well as its explainability and ability to provide validation (see, e.g., [3]).While this approach has seen success in various Cyber-Physical Systems domains, like autonomous vehicles [6] or water distribution networks [4], physics-based attack detection has not been widely used for real-world HVAC systems yet.
The primary contribution of this paper is to highlight common challenges and pitfalls encountered when implementing physicsbased attack detection for real-world HVAC systems, covering the entire process from data collection to final attack detection.Furthermore, practical guidelines to circumvent these obstacles are presented.
This paper is structured as follows.Section 2 presents related work.Section 3 then describes the used experimental framework and results for cyber-physical attack detection on a real-world HVAC system, ranging from the execution of a real attack, to developing a physics-based model and the application of an attack detection strategy.The identified challenges and pitfalls that occurred when employing attack detection on a real system are presented in Section 4. Finally, Section 5 provides the conclusion.

RELATED WORK
In the HVAC domain, limited research exists on attacks and attack detection, especially concerning the detection of cyber-physical attacks.Few studies, such as [8,9,13,14], have simulated cyberphysical attacks on HVAC systems.Of these, only one [14] conducted an attack on a real-world system, but with no attempt to stay stealthy, and none have considered attack detection.A majority of the small number of existing related work focusing on cyber-physical attack detection on HVAC systems applies black-box Machine Learning (ML) based schemes, such as in [7,11,18].All of these also consider false data injection attacks but only present concepts [18] or use simulation-based data for modeling and attack detection [7,11].In some cases, a large amount of labeled attack data is necessary, which is difficult to obtain for real-world HVAC systems.These black-box models often face challenges in generalization and explainability.The most relevant related work is presented in [15,16] and uses physics-based attack detection for radiator-based heating systems.No real HVAC systems, which have different heating and additional cooling behavior, are considered.Furthermore, an abstract linear state space model is used with subspace identification for parameter fitting instead of a detailed physical-formula-based model.The used model is, thus, specifically adapted to the scenario via learned parameters and highly relies on a diverse data set.The model lacks generalizability and explicit interpretability inherent to more detailed physics-based models.Furthermore, only simulation-based attack data is considered.
Unlike previous studies, we highlight the challenges encountered while performing actual, stealthy attacks on an HVAC system and while applying a physics-based attack detection approach to a realworld HVAC system.

EXPERIMENTAL FRAMEWORK 3.1 HVAC System and Executed Attacks
Here, we focus on applying attack detection to individual thermal zones served by VAV-based HVAC systems, the most common system configuration for commercial buildings in the United States [1].On these systems, typical available sensor measurements are zone temperature, adjacent zones' temperatures, discharge temperature from the VAV box, and airflow within the VAV box.
The attacks considered here can be classified as false data injection attacks with the aim of overheating a zone.This type of attack is especially relevant in the considered context as it is a cyber-physical attack, can cause great harm, and can be designed in a stealthy way.The practically implemented overheating attacks were performed as sensor attacks on zone temperature sensors.For these attacks, the zone temperature sensors were re-calibrated.Re-calibration was performed in a way that the then incorrectly measured and shown false zone temperature   was gradually shifted to temperatures below the heating setpoint.As a consequence, heating was triggered.The actual zone temperature   remained hidden, masking the malicious manipulation.
The designed attacks were executed for three chosen zones/rooms on a real HVAC system via the web-based BAS platform.For the attacks, a negative offset was gradually added to   until the target offset was reached.Once the final target temperature   had been reached, the offset was maintained for a predefined period of time before the negative offset was gradually removed again.To start the attack in a stealthy way and to eventually trigger heating, a typical cooling behavior of the zone was imitated.To end the attack, a heating behavior was shown.Both had been learned from stored data and can be recognized in Figure 1.
One such attack experiment for a considered zone is highlighted in red in Figure 1, with   = 80 •  .In the initial phase of the attack, the used offset step-size was −0.3 •  /5, which corresponds to a typical rate for temperature decrease of the considered zone.After heating had started, the offset was manipulated less frequently In order to end the attack, a respective step-size was used to imitate a typical rate for temperature increase in the zone.Achievable options for   and the respective zone's heating/cooling behavior at higher temperatures were determined via preliminary experiments.In Figure 1, the disguised actual zone temperature   is also displayed for comparison but would only be available for the attacker.

Applied Physics-Based Model
Physics-based attack detection schemes leverage the fact that physical systems, such as HVAC systems, follow certain physical laws and compare actually measured data with model prediction for attack detection.An overview of physics-based models for HVAC systems can be found in [2,3].Models range from pure physicsbased white-box models, which often only focus on a certain aspect of the system and are difficult to align with real-world systems, to more accurate, here focused, grey-box models.These are based on physical laws while learning missing parameters from data [2].Contrary to many black-box models, grey-box models also offer the advantage of explainability and validation, providing a clear understanding of the system dynamics and facilitate the verification of model predictions [3].Typical physics-based models focusing on the thermal zone are thermal models based on heat balance or, for example, weighting factor methods, which however assume heat transfer is linear and system properties are constant.A third common category are models based on electric circuit theory, which can be derived from a heat balance model [2,17].Thus, a heat balance model is utilized here due to its direct applicability and simplicity.The used model is based on the zone air heat balance equation, see [2,19], adapted to the available information.Assuming temperature is uniform throughout the zone, zone heat capacity is constant, and the zone is located in the core of the building, the following model is applied: Figure 2 shows the resulting model prediction in comparison to the actual measured zone temperature by the thermostat on training data.A respective prediction on unknown data, even considering a rather extreme preliminary experiment and an attack experiment, can be seen in Figure 1.Especially when considering overheating attacks with temperatures of about 80 •  , the achieved accuracy of the model was considered sufficient, with a root mean squared error (RMSE) for the prediction of about  = 0.42 on test data.Thus, the resulting model was used as a basis for attack detection in Section 3.3.

Employed Attack Detection Scheme
For attack detection, the non-parametric CUmulative SUM (CUSUM) statistic, a form of statistical threshold method, is applied here to identify inconsistencies in physical features of sensor measurements using model prediction from Section 3.2 as expected behavior.The CUSUM statistic   at time step  is calculated recursively via: with  0 = 0, the residual   , i.e., the difference between model prediction and measured zone temperature, and a small constant  > 0 with  [|  | − ] < 0 for attack-free training data, where  [•] denotes the expected value, see for example [10].An attack is detected if   > ℎ for the chosen threshold ℎ.This type of attack detection is used as even small deviations of longer duration can be detected.Moreover, no alarm strikes in case of only single deviations due to single measurement errors, which can often occur in real-world HVAC systems.Each residual |  | >  increases   .Therefore,  defines the deviation from the expected value of the training residuals which can be detected in general.Here,  is chosen as the standard deviation of the residuals seen in training data.By contrast to , the threshold ℎ considers history of errors and determines which amount of significant error, |  | > , has to be accumulated until an alarm strikes.
The here presented attack detection scheme can effectively detect the executed false data injection attack on a real-world HVAC system, as shown in Figure 1.No false alarm was triggered during the depicted preliminary experiment, in which only setpoints were increased.For the two sets of executed attacks, the attack detection rate was 100%, as attacks heated zones to about 80 •  , which causes large residuals.No false alarms were triggered for any available data as a high value for ℎ was used, which was sufficient to detect the performed extreme attacks.Future work will include less drastic attacks for a more detailed algorithm performance evaluation.

IDENTIFIED CHALLENGES, PITFALLS, AND PROPOSED GUIDELINES
When practically employing cyber-physical attack detection for a real-world HVAC system, the following challenges were identified: 1) Trade-off between accuracy and spatial generalization: Learning specifics about a zone from data via parameter fitting enhances the model's accuracy by including more detail than provided by physical formulas and improves attack detection for that zone.However, this reduces the model's applicability for different zones, requiring time-consuming parameter fitting for each zone.Factors such as heat transfer to and from adjacent zones significantly influence zone temperature changes.Therefore, the temperature difference to adjacent zones is included in the used model in Section 3.2, weighted by parameters, such as the corresponding thermal resistance   .These can be estimated via building knowledge or, for more accuracy, be learned from data.Parameters learned from data improve accuracy compared to estimations but limit the model's generalizability.In planning attack detection for a large building, a balance must be struck between accuracy/sensitivity for attack detection and effort for individual parameter fitting.
2) Trade-off between accuracy and temporal generalization: Seasonal variations significantly impact the thermal dynamics of a zone, affecting the accuracy of models trained on data from a specific season.For instance, the hot water supply temperature for the reheating coil in winter is higher than in summer for our considered HVAC system, altering the rate of temperature rise.This necessitates repeated parameter fitting to maintain model accuracy across different seasons, especially during the first year of operation.
3) Trade-off between accuracy and over-fitting: It is a known phenomenon that model complexity, i.e., the number of learned parameters, has a crucial impact on the risk of over-fitting.Due to a large number of unknown influences in temperature prediction when using HVAC systems in occupied buildings or missing history for relevant sensors, parameters have to be included into the model and fitted to create an accurate model for attack detection.While adding/fitting parameters can improve model accuracy for attack detection, it also increases the risk of over-fitting.Therefore, when data is scarce, a simpler, less accurate model can be more suitable due to its better generalization.As seen during experiments, data tracking is often only started on request.Therefore, early data collection is crucial when planning to implement attack detection.Contrary to black-box models, the here applied grey-box models already partially address the stated challenge by utilizing valid physical formulas as a basis and only learn missing information from data.This mitigates, but does not eliminate, the accuracy vs. over-fitting trade-off, as well as the challenge stated next.
4) 'Unknown unknowns' and corner cases: Real-world scenarios can present unpredictable situations that are challenging to incorporate into models, leading to potential false alarms.Furthermore, there are certain corner cases which have been identified but occur only rarely and are, thus, underrepresented in training data.Two examples encountered during our practical experiments are: 1) People keeping doors open and, thus, preventing the room from heating up as predicted.This could be addressed by incorporating door sensor data into the model.2) An adjacent zone with a low cooling setpoint being indirectly heated by the HVAC system of the considered zone during a heating experiment, triggering cooling in the second zone and, thus, significantly reducing the temperature of the considered zone.This not only risks false alarms but also wastes energy and highlights the importance of careful setpoint selection.These unpredictable factors underscore the complexity of modeling real-world systems and the potential for unexpected challenges during both regular operation and executed experiments.
5) Dealing with poor data quality: Data quality, influenced by factors such as faulty sensors, noisy measurements, and significant unknown variables, is crucial for effective model prediction and attack detection.Therefore, data should be analyzed for prominent anomalies and plausibility during pre-processing.As recognized when choosing a suitable zone for experiments and identifying a faulty sensor, existing fault detection mechanisms in HVAC systems are currently mainly used for real-time maintenance.However, data detected as faulty could be immediately labeled correspondingly to ensure its appropriate handling in future applications.
6) Integrating the developed attack-detection strategy into an existing BAS: When working with different BAS on campus, extremely high alarm rates were recognized.The sheer volume of alerts coupled with limited personnel resources, however, makes each alert's detailed investigation difficult.Therefore, a low false alarm rate and mitigating the risk of alarm fatigue is crucial.However, maintaining a high detection rate is equally important to ensure significant anomalies are not missed.Balancing these conflicting objectives depends on individual risks and conditions.Furthermore, raising awareness of potential HVAC cyber-attacks is crucial to promote the integration of attack detection strategies in existing alarm systems.

CONCLUSION
The rising threat of attacks on BAS necessitates the integration of attack detection strategies, such as cyber-physical attack detection, in systems like HVAC systems.However, implementing these strategies presents unique challenges, from data acquisition and model generation to integrating a developed attack detection strategy into an existing BAS.This paper investigates cyber-physical attack detection based on a real-world attack and outlines identified challenges and pitfalls.Future work will include addressing these challenges, in particular spatial and temporal generalization, for physics-based attack detection for HVAC systems.

Figure 1 :
Figure 1: Successfully Detected Attack on Unknown Data •  •   with time , zone temperature  (), air flow rate   (), specific heat capacity of air   , discharge temperature  ℎ (), internal (), surface area   of the wall to adjacent zone , adjacent zone's temperature   , (), thermal resistance   of the wall between the zone and its adjacent zone , volume  of air in the zone, density of air , and thermal capacitance   of the zone.Here,   () =  •  2 () with unknown parameter  and measured  2 -level  2 ().The resulting differential equation determines the rate of temperature change in the considered zone and is used to predict a next zone temperature.Fitting of unknown parameters, such as ,   , and a factor within   , is performed by non-linear least squares optimization via the Levenberg-Marquardt algorithm.