Adversities in Abstract Interpretation - Accommodating Robustness by Abstract Interpretation

Robustness is a key and desirable property of any classifying system, in particular, to avoid the ever-rising threat of adversarial attacks. Informally, a classification system is robust when the result is not affected by the perturbation of the input. This notion has been extensively studied, but little attention has been dedicated to how the perturbation affects the classification. The interference between perturbation and classification can manifest in many different ways, and its understanding is the main contribution of the present article. Starting from a rigorous definition of a standard notion of robustness, we build a formal method for accommodating the required degree of robustness—depending on the amount of error the analyst may accept on the classification result. Our idea is to precisely model this error as an abstraction. This leads us to define weakened forms of robustness also in the context of programming languages, particularly in language-based security, e.g., information-flow policies, and in program verification. The latter is possible by moving from a quantitative (standard) model of perturbation to a novel qualitative model, given by means of the notion of abstraction. As in language-based security, we show that it is possible to confine adversities, which means to characterize the degree of perturbation (and/or the degree of class generalization) for which the classifier may be deemed adequately robust. We conclude with an experimental evaluation of our ideas, showing how weakened forms of robustness apply to state-of-the-art image classifiers.


INTRODUCTION
Robustness of classifiers [Albarghouthi 2021;Fawzi et al. 2017;Gehr et al. 2018;Madry et al. 2019] refers to the ability of classifiers to perform consistently and accurately across various conditions, including noisy or incomplete data, adversarial attacks [Szegedy et al. 2014], or distribution shifts.
5:2 R. Giacobazzi et al.A robust classifier should maintain its performance and make reliable predictions even when faced with challenging or unexpected scenarios.The underlying assumption is that tiny perturbations of the input will not affect the capability of (correctly) classifying that input; hence, a robust classifier should be resilient to common sources of errors, such as variations in input data, outliers, or attempts to deceive the prediction.From an epistemological point of view, robustness is essential to control and assess correct generalization from specific instances in inductive reasoning.Learning general rules, patterns, or principles that apply beyond the specific examples initially observed assumes some form of uniformity [Mill 1882] that, when applied to classifiers, requires robustness.The goal of a classifier is to capture the underlying regularities or trends in the data and apply them to unseen or new situations despite perturbations on the input.The strength of generalization, and therefore of (machine) learning, depends upon robustness, ensuring that the model predictions remain reliable and trustworthy in real-world applications.
Developing robust classifiers is quite complex, and it involves techniques like data augmentation, regularization, ensemble learning, and adversarial training.By enhancing the model's capability to generalize and handle uncertain or perturbed inputs, robust classifiers can mitigate the impact of noise or adversarial attacks, improving their overall performance and reliability.Given the intricate nature of ensuring and verifying robustness, it becomes imperative to have a comprehensive understanding of the fundamental constituents that contribute to this concept.Moreover, the adaptability of the notion of robustness to different contexts is crucial.Classifiers find applications in various fields, each with unique features and challenges.Therefore, it is essential to tailor the understanding of robustness to suit these specific contexts.This involves considering domainspecific factors, adjusting the model behavior to align with the application's requirements, and identifying relevant performance metrics for evaluating robustness.
By delving into the main constituents of robustness, we introduce weaker notions that can better model various situations, both in the case of classifiers and, more generally, in robustness program analysis.In particular, we model incomplete or noisy data through the notion of abstraction as introduced by Cousot and Cousot [1977].Abstraction serves two essential purposes: First, it offers a set of rigorous mathematical definitions for representing uncertainty in input data, allowing for a precise mathematical model of data perturbation.Second, it establishes a boundary determining whether perturbed data remains consistent with a given property.
Starting from the formalization of a standard definition of robustness, we introduce two main weakenings.The first one, called abstract robustness, corresponds to allow a form of classification error.In this case, robustness means that, by perturbing the input, the classifier returns the exact class approximation, i.e., the same error.Note that, when dealing with class abstraction, we assume that the classes exhibit a hierarchical structure, modeled as a poset.This naturally models class generalization.The second weakening, called hierarchy-aware abstract robustness, aims at exploiting such hierarchical structure by requiring that by perturbing the input, the classifier returns at most the exact class approximation as the original input. 1 These weakenings strongly depend on the choice of the input within a designated perturbation area.We introduce the notion of coherence to weaken this dependency.Coherence allows us to specify robustness by requiring a common property among the classifications provided for various perturbed objects, losing the connection with the chosen input so far.In particular, it simply requires providing any class, even more general (approximated) than the right one.We show that these weakenings can be effectively verified for neural networks provided by Singh et al. [2019] on the MNIST [LeCun et al. 2010] and CIFAR10 [Krizhevsky 2009] datasets, by adapting the ETH Robustness Analyzer for Deep Networks (ERAN) with abstractions modeling potential real approximations.

5:3
Abstract interpretation offers a significant advantage by providing a framework for adapting various domain refinement/simplification methods, known in program analysis and verification, to classifiers.Our findings demonstrate the feasibility of constraining adversities to specific abstractions derived from initial perturbations and an acceptable class approximation, thereby ensuring adequate forms of robustness.It is intriguing to note that these novel robustness concepts enable the discovery of a meaningful connection between information-flow policies in language-based security, specifically abstract non-interference [Giacobazzi andMastroeni 2010, 2018], and the robustness of classifiers.This connection can be understood in light of the degree of precision of abstractions, known in program analysis and abstract interpretation by the notion of completeness [Bruni et al. 2021[Bruni et al. , 2023;;Giacobazzi et al. 2000].Our ambition is to lay the foundation for extending methods and tools used to verify malicious information flows or to improve the precision of abstract interpreters, to the realm of classification and adversarial attacks in machine learning.
Paper road map.In Section 2, we describe the related works concerning robustness.Section 3 provides the necessary background for the mathematical notation and abstract interpretation.In Section 4, we provide a formal definition of robustness aiming to be general enough to capture several existing approaches but at the same time sufficiently expressive to capture the essence of the robustness notion, requiring insensitivity to input perturbation.In Section 5, we provide a framework for defining weaker notions of robustness by allowing some sort of error in the output classification.This error is modeled as an abstraction (in the abstract interpretation framework) of the class set.Just as it happens for robustness, the abstract robustness notion proposed here is strictly local as well, i.e., dependent on a fixed point: the center of perturbation.In Section 6, we make robustness a property of the perturbation area and not of the perturbation center.We show the relations between all of these notions, and in Section 7, we show how the so far defined notions have a meaning in the programming languages field, i.e., in language-based security and program verification.In Section 8, we exploit the formal framework proposed for confining adversities, namely for characterizing the degree of robustness in terms of the perturbation and/or the class abstraction, making the classifier acceptably robust.Once we have the framework for formally tailoring the robustness property we desire on the application context, we adapt an existing technique for checking robustness (ERAN, see Section 9 for details) to check whether a given classifier satisfies a weakened form of robustness.Hence, we can provide an experimental evaluation (Section 9) showing that checking abstract notions of robustness is possible (in a sound and complete way) and may provide interesting outcomes.In particular, we show how to check classifiers against robustness, avoiding potential false alarms due to a too-strong robustness formalization.Finally, we conclude with a discussion of future possible research directions (Section 10).

RELATED WORKS
In general, robustness concerns systems whose inputs can have small amounts of error and uncertainty.Depending on the application, the effect of the input on the output can be more or less tolerated for robustness to be validated.For instance, it is normal for software to produce different outputs given different inputs, but we still want it to not crash even on uncertain inputs [Chaudhuri et al. 2012].This software property has been rigorously defined by using continuity, and Lipschitz continuity in particular.That is, perturbations to the program inputs lead to at most proportional changes to its outputs [Chaudhuri et al. 2012].When dealing with Machine Learning, particularly classifiers, the term "robustness" usually does not admit an error in the result, and predictable output means the same classification.Classifier robustness has been characterized in several, more or less different, ways throughout the literature [Albarghouthi 2021;Gehr et al. 2018;Leino et al. 2021;Mirman et al. 2018;Ruan et al. 2018;Singh et al. 2018Singh et al. , 2019;;Zhang et al. 2020], often enormously depending on the context of application, e.g., computer vision, and sometimes defined through the technique used for enforcing it.Some approaches define robustness as a binary property that either holds or not [Albarghouthi 2021;Gehr et al. 2018;Hein and Andriushchenko 2017;Leino et al. 2021;Liu et al. 2021;Mirman et al. 2018;Singh et al. 2018Singh et al. , 2019;;Zhang et al. 2020].In contrast, others model it as a metric providing a numerical indication of just "how robust" a classifier is Fawzi et al. [2017], Giraudon et al. [2021], Madry et al. [2019], and Ruan et al. [2018].Among these works, modeling robustness as a metric, robustness is treated as some statistical proxy of the loss function [Madry et al. 2019] or as the radius of some ball in input space [Fawzi et al. 2017;Giraudon et al. 2021;Ruan et al. 2018].
Regarding the choice of inputs on which to check robustness, again there is a wide range of possibilities: The set of points within a maximum distance from a chosen center point (using different distance norms) [Giraudon et al. 2021;Hein and Andriushchenko 2017;Leino et al. 2021;Liu et al. 2021;Ruan et al. 2018], a pre-determined set of inputs [Gehr et al. 2018;Singh et al. 2018Singh et al. , 2019]], the collection of inputs satisfying a fixed set of conditions [Albarghouthi 2021], the inputs selected by adding a selection of perturbation vectors to some initial input point [Fawzi et al. 2017;Madry et al. 2019;Tsipras et al. 2019], the image of a function [Mirman et al. 2018;Zhang et al. 2020], and so on.Moreover, classifier robustness is often enforced for ensuring safety against adversarial attacks [Szegedy et al. 2014], which represents the agent able to control input perturbation.
Hence, the overall idea concerning classifier robustness is that a robust classifier always provides the same classification independently from errors that may be introduced, on purpose or not, in the input.Our conclusion is that, particularly regarding robustness for classifiers, it is not easy to find a uniform way to define different needs in the literature depending on the application field.This makes it hard to understand the limits and improvement possibilities of existing techniques precisely and to formally compare the different approaches to robustness.For this reason, our first aim is to provide a framework where it is possible to define different forms of robustness that can be accommodated, depending on the analysis constraints determined by the specific application context.Our idea is to exploit the rigorous approach proposed for software, based on Lipschitz continuity [Chaudhuri et al. 2010], admitting an error in the output, and therefore allowing the analyst to tune robustness, but based on a qualitative and not quantitative characterization of such error.This is possible by using the notion of abstraction developed in abstract interpretation [Cousot and Cousot 1977] as a means to establish the boundaries for imprecision: The output can change within a given abstraction.When applied to the output of a classifier, this allows us to model a further dimension in the generalization process: The possibility that the lack of robustness concerning some property means being robust for a more abstract one in the sense of abstract interpretation.

FUNDAMENTAL MATHEMATICAL BACKGROUND
Sets, order, and functions.Given two sets S and T, we denote with ℘(S) the powerset of S, with S T the set-difference between S and T, with S T strict inclusion, with S ⊆ T inclusion, with |S| the cardinality of S. A set S is finite if |S| < ω.A set L equipped with a partial order relation ≤ L is a poset, and it may be denoted as L, ≤ L .A poset L, ≤ L is a lattice, denoted L, ≤ L , ∨ L , ∧ L , L , ⊥ L , if for all x , y ∈ L we have that the least upper bound (lub) x ∨ L y, then the greatest lower bound (glb) x ∧ L y, the greatest element (top) L , and the least element (bottom) ⊥ L belong to L. It is complete when for every X ⊆ L we have ∨ L X , ∧ L X ∈ L. We may omit subscripts, like in ⊥ L , or ∨ L , used to disambiguate the underlying lattice L, when it is clear from the context.
Given f : S → T and g : T → Q, we denote with g • f : S → Q their composition, i.e., (g • f )x = g(f (x )).Function composition g • f (x ) is sometimes denoted gf (x ).id S : S → S is the identity function over S, and we omit the subscript when S is clear from the context.f : , it preserves least upper 5:5 bounds of arbitrary non-empty sets (co-additivity is dually defined, i.e., by preserving greatest lower bounds).If f : L → D, then we use the notation f ↑ : ℘(L) → D, D ∈ {D, ℘(D)}, for the additive lift of f to sets of inputs (i.e., for any S ∈ ℘(L) defined as f ↑ (S ) Abstract interpretation and Abstract Domain Completeness.In the standard framework of abstract interpretation [Cousot and Cousot 1977], abstract domains can be equivalently formulated either in terms of Galois connections or closure operators [Cousot and Cousot 1979].A pair of functions α : L −→ A and γ : A −→ L on posets forms an adjunction or a Galois connection if for any x ∈ L and y ∈ A we have α(x ) ≤ A y ⇔ x ≤ L γ (y).In this case, α (respectively, γ ) is the left-(respectively, right-) adjoint of γ (respectively, α), and it is additive (respectively, co-additive).An upper closure operator ρ : L → L on a poset L is monotone, idempotent, and extensive, i.e., ∀x ∈ L. x ≤ L ρ(x ).Closure operators on a complete lattice L are uniquely determined by the set of their fix-points fp(ρ The set of all upper closure operators on L is denoted by uco(L) and if L is a complete lattice then it forms a complete lattice uco(L), , , , λx .L , λx .x , where ρ η iff fp(η) ⊆ fp(ρ).
Any η ∈ uco(℘(L)) induces a partition on L where [x ] η def = y η(x ) = η(y) are the equivalence classes.The most concrete closure that induces the same partition of values as η is the additive lift of the function Hunt and Mastroeni 2005].A closure η is called partitioning if η = P[η] ↑ [Ranzato and Tapparo 2004].Abstract interpretation is generally used for abstracting computation functions.Given a function f : L −→ D, an input abstraction η ∈ uco(L) and an output abstraction ρ ∈ uco(D), we call best correct approximation (bca for short) of f in η, ρ the function ρ • f • η [Cousot and Cousot 1979].The bca computation is always correct, i.e., ∀x ∈ L. ρ , meaning that we may lose information when computing abstract inputs.The bca is complete when ∀x ∈ L. ρ [Cousot and Cousot 1979;Giacobazzi et al. 2000], i.e., when there is no loss of precision by computing with objects in the abstract domain.We call it local completeness when it is verified on a fixed input x ∈ L, i.e., ρ [Bruni et al. 2021].

DEFINING (CLASSIFIER) ROBUSTNESS
In this section, we introduce the notion of robustness as a safety property that is usually verified on a specific kind of programs known as classifiers.In the following, we formalize the notion of classifier for fixing the notation and the application context in which we move.

Classifiers
Intuitively, a classifier is any function taking an input sample of any form or shape and returning the class that it determines to be the most correct for the given sample in a discrete set of classes.For instance, a classifier might try to guess which type of animal is in a photograph or determine which song is a given audio waveform.Usually, a classifier might be tested by feeding it with some input samples for which the correct classes are known to see whether the predictions match the expectations.
Formally, let us denote the set of inputs by I and the set of classes by C (not necessarily discrete and/or finite).A classifier, is any function f : I → C, and its prediction for Classifiers may be implemented using different techniques, depending on the kind of implementation and/or on the field of application, potentially imposing restrictions on I and/or C. For instance, as it often happens in practical applications, we can assume that classifiers deal with a finite set of discrete classes.
Example 4. 1 (Support Vector Machines).A Support Vector Machine (SVM) is a binary classifier type known for its simplicity, elegance, and performance [Boser et al. 1992].The domain C is only composed of two classes: a positive class and a negative class (hence the name, "binary classifier"), i.e., C = {−1, +1}.A specific SVM is defined by its vector of weights w ∈ R n and its bias b ∈ R. Given a vector of features x ∈ I = R n , the corresponding prediction is sign(w T x + b).
Thus, when w T x + b ≥ 0, then x is predicted as belonging to the positive class; otherwise, it goes to the negative class.
Example 4.2 (Neural Networks).Neural networks are an extremely popular tool for solving machine learning problems, particularly for implementing classifiers [McCulloch and Pitts 1943].Intuitively, you can think of them as flexible templates that morph and adapt to mimic the behavior of any continuous function.Concretely, a neural network is composed of a sequence of layers, each containing many units called neurons.Together, the neurons take the (real) numerical values from the previous layer, apply an affine transformation followed by a non-linear activation function, and thus produce a new set of values.This process is called forward-propagation: The input values are fed into the first layer, and by reading the values in the last layer, you get a score for each class; the result of the prediction is finally obtained by taking the class with the largest score.
Let n be the number of input values the neural network expects and m be the number of output values that match the number of classes.Then, denote with g : R n → R m the function performing the forward-propagation step and with h : R m → [1, m] the function that takes the values in the final layer and predicts a class (usually h is simply argmax).
To cast neural networks into our framework, we simply let and assemble the classifier as f = h • g.The prediction for x is simply f (x ) = h(g(x )): The n-vector x is forward propagated through all layers by using g, and the winning class is decided by h.

Defining Robustness
Let us consider a generic subset of inputs Ω ⊆ I (Ω ), on which we are interested in assessing robustness. 2 Clearly, Ω can be just a single point (in which case we might talk about local robustness), the training, testing, or validation set for a machine learning model, or the whole I set.The larger Ω is, the stronger the robustness property becomes, especially when the points are close to the classifier decision boundary.
We have to formally characterize the set of input perturbations on which we require the exact prediction to guarantee robustness.We define the notion of perturbation as any function strictly extending a datum to a set consisting of what we model as its neighborhood.Formally, it is an extensive function on data different from the identity on the whole I. 3 In other words, a perturbation function can be anything at all as long as the perturbed datum is always included in the perturbation region together with at least another element.Without losing generality, we may suppose that a perturbation function is total on I by supposing it is the identity only on inputs that cannot be perturbed if they exist.It is worth noting that when robustness is defined as safety against adversarial attacks [Szegedy et al. 2014], then the perturbation function I precisely models the capability of the attacker to inject errors in the inputs.In the following, we will simply call I perturbation, and we specify the center when dealing with the perturbation of a specific point x .
In other words, I draws a region around each input in Ω. Intuitively, Robustness (Definition 4.4) consists precisely in verifying that all the points in a region, identified by I(x ), are classified with the same label/class.Clearly, the larger the region, the stronger the robustness property is.

Robustness Flexibility: Some Examples
The formal definition of robustness given above is flexible enough to cover most of the, more or less formally given, definitions we recalled before.In this section, we show the relation of Robustness in Definition 4.4 with some approaches existing in the literature.
AI 2 -style approach.Let us consider the AI 2 -style definition of robustness [Gehr et al. 2018;Singh et al. 2018]: "The image f (X ) of a set X is entirely contained in a set of admissible outputs Y ." Let x ∈ X be an element such that f (x ) ∈ Y .This point may not always exist, but we can simply take any point x ∈ X at all and, if f (x ) Y , then the property is proved to be false; otherwise, x is taken as reference point x = x .Let us define the function g as This classifier is depicted in Figure 1.The decision boundary, depicted as a blue line, is the set of points x = (x 1 , x 2 ) for which w T x + b = 0. Points lying above the line are classified as belonging to the positive class (+1 ∈ C).Let us consider the perturbation I r defined as the spheres around points, i.e., I r In Figure 1, the two circles on the left represent the perturbation region for the point (2, 6) with radii 1 (in green) and 3 (in red).On the right, there is the perturbation region of radius 1 centered in (6, 2).Then we can observe that f is robust, according to Definition 4.4, in (2, 6) w.r.tI 1 but not w.r.tI 3 .It also loses robustness when moving to a point x closer to the decision boundary, such as (6, 2).
Neural network classifiers.Neural networks are very pervasive and several robustness verification techniques already exist.For this reason, we are now going to show how the verification technique in Albarghouthi [2021]4 (verified on the abstract execution of the network) does imply Robustness (Definition 4.4) for the concrete network.
Let  The first step is lifting g to sets, When dealing with abstract interpretation of neural networks, we have to consider a family of abstractions on ℘(R j ), i.e., γ j • α j ∈ uco(℘(R j )), which are used for soundly approximating the lift of the forward propagation g ↑ of the neural network, let g ↑ Δ be such approximation, such that (*) . The final step of the verification process is asserting that h always gives the same class for every element of γ m g ↑ Δ α n (X ) (which by virtue of relation (*) implies that h gives the same class also for all the elements of g ↑ (X )).The issue is that g ↑ Δ α n (X ) might symbolically represent an infinite number of vectors.Therefore, it is defined a binary function p Δ : Δ m → {0, 1}, depending on the chosen output abstract domain Δ m , such that By assuming X to be the perturbation of x ∈ R n defined by I, then Equation ( 1) implies that f assigns the same class c ∈ C to each point in I(x ), namely it implies Robustness (Definition 4.4).

WEAKENING ROBUSTNESS BY ABSTRACT INTERPRETATION
Definition 4.4 is general enough to formally capture what we believe is the essence of the robustness notion for classifiers, namely the requirement to be insensitive to input perturbations.This is interesting, but such a neat formalization allows us to move further in understanding robustness.
In particular, we wonder whether this requirement could be considered too strong in contexts where some degree of (controlled) sensitivity to input perturbation could be acceptable, rejecting, so far, as not robust classifiers that may be deemed robust under particular circumstances.Hence, we exploit the given general formalization of robustness to investigate how we can formally model such a sensitivity to input perturbation to accommodate the enforced notion of robustness in the specific application context.This observation leads us to weaken robustness by abstracting the observation of the returned classes, namely by allowing some degree of mistake among the classes collected starting from sets of perturbed inputs.

Abstracting the Classification
The main idea behind the process of weakening Robustness (Definition 4.4) revolves around the idea that some classes might be so similar, or at least more similar than others, that we would consider mistaking among them as an acceptable error, as far as robustness is concerned.In other words, our idea is to model the situation where although the classifier f predicts at least two distinct classes c, c ∈ C for the points x ∈ I and x ∈ I(x ), respectively, the analyst might be interested in accepting the difference between those two classes and not fail robustness straight away.Hence, we introduce a class abstraction whose role is to strip a class of the details that we wish to ignore.Formally, if C is the collection of all the classes that the classifier may return, then the idea consists of taking as abstraction an upper closure operator (uco) on ℘(C), i.e., C ∈ uco(℘(C)) with only one exception.In fact, λc.C is a limit abstraction, abstracting any class in C, hence not distinguishing any class at all.It should be clear that the use of such an abstraction for robustness is nonsense, since it corresponds to accepting mistakes among all classes in output, and, therefore, not checking any form of robustness.Hence, the abstraction we use for robustness is It is worth noting that C itself may be equipped with a partial order ≤ C , namely C may be a poset.This means that the class domains C exhibit some sort of hierarchical structure, containing classes that are not entirely mutually exclusive but such that some are generalizations of others, as determined by ≤ C .This implies that a point can belong to a class and to other larger, more general, classes at the same time [Li et al. 2022;Lin et al. 2014;Zhou et al. 2016].This is especially typical of computer vision problems such as object detection and image segmentation (see Example 6.2 for a trivial example).In this case, the analyst may choose to abstract directly C, keeping the same hierarchical structure but deciding to accept some degree of approximation for the resulting classes.Formally, in this case, C ∈ uco (C). 5n the following, let us denote by C (called class domain) the partial order set, defined in terms of the class set C, on which we define the class abstraction.In particular, if C is a poset, then the analyst may choose C ∈ {C, ℘(C)} depending on how he/she aims at abstracting classes, by generalizing further (C) or by fixing the acceptable mistake degree (℘(C)); otherwise, C = ℘(C).For the sake of readability, when C = ℘(C) we abuse notation so that C(c) actually stands for C({c}) (c ∈ C).
The next example shows that class abstraction becomes necessary when the classifier itself is approximated by returning more than one class.
Example 5.1.If the classifier f is the output of a neural network, then what usually happens is that there are a few dense layers in the tail and the winning class is picked by taking the output neuron with the highest value.Sometimes, the margin between the winning and the second class is very low; meaning that a classifier labeled as not robust could be so only by a scratch.Imagine that there is a perturbation region in which the classifier always predicts two different classes but with a very similar score.Is that, deep down, truly not a robust network?The idea is therefore to consider an approximated classifier predicting the set of all the classes with a similar score w.r.t. a given threshold.
Formally, let C be the powerset of the actual set of classes C and consider the classifier approximation f : I → ℘(C) returning not only the winning class but all classes that are highly likely, meaning their activations in the output layer is higher than a chosen threshold.A proper C ∈ uco(℘(C)) can be built by choosing as fixpoints all sets of classes that f is allowed to produce throughout a single region on the track set.

Abstract Robustness
In the first weakening of Definition 4.4, instead of requiring the same class for all the points in the perturbation, we require the same error among the returned classes.Namely, all the We call this property abstract robustness, because the verification is performed by using classes that are approximations of what was originally given by the classifier.In the following, we call "concrete classes" the elements of C and "abstract classes" the fixpoints of C.
To show the meaning of abstracting classes, let us consider some simple examples.
Example 5.3.Let us consider a handwritten number classifier.Let f : I → C be a classifier that takes images of handwritten numbers in the [0, 99] interval and tries to guess which number it is.Suppose that for any input image x ∈ I, the perturbation consists of some minor mutations to it (e.g., a shift in brightness or the addition of some noise) and is supposed to allow some error, as long as the classifier predicts a number whose unit (right) digit may be different, while the tens (left) digit must remain stable.For instance, we tolerate 72 becoming 73 but not 82.
Let I be the function that takes a point in the input space and collects a set of perturbed points in its neighborhood by applying some mutations.Let C Example 5.4.Let f : I → C be a classifier for an object detection system: It predicts which entity (class) is depicted in an input image, e.g., among {car, lion, horse, oak, etc . . .}.The set of classes C can be a very vast domain but, for the sake of simplicity, we can simply consider a restricted one, e.g., C def = {Lion, Squirrel, Horse, Dandelion, Rose, Oak}.In this case, the analysts might accept f predicting two different species in x ∈ I and in x ∈ I(x ) as long as they are both animal species or both plant species.After all, confusing a lion for a horse is not as big of a mistake as confusing a squirrel for a dandelion.Note that abstract robustness is a strict weakening of robustness, since it is abstract robustness w.r.t. the identity class abstraction.
Theorem 5.5.Let I be a set of data, C a class set, C the chosen class domain, I ∈ Pert(I), and Ω ∈ ℘(I); then (1) These results, tell us that (1) robustness is an instance of abstract robustness and it is stronger than any abstract robustness instance but also that (2) abstract robustness is the robustness of the abstract observation of the classifier result w.r.t. the set of abstract classes.Hence, abstract robustness surely allows a change of perspective, but it does not really provide new insights into robustness.

Hierarchy-aware Abstract Robustness
As observed before, when introducing class abstraction we impose a sort of hierarchical structure on the way we look at classes in the set C. In other words, we allow some sort of mistake in classification by moving toward the power set of classes (C = ℘(C)), it is by generality when C itself is a poset (C = C) [Li et al. 2022;Lin et al. 2014;Zhou et al. 2016], but in any case, in such a situation, the equality required between the output (abstract) classes in Definition 5.2 may become too strong a requirement for defining a notion of robustness deeply exploiting the meaning of abstracting classes.
Imagine to have two points x , x ∈ I such that x ∈ I(x ), f (x ) = c and f (x ) = c with C(c ) C C(c). Abstract robustness (Definition 5.2) would dismiss f as not robust (being C(c ) C(c)), but is that truly not a robust classifier?Indeed, x , technically, has also the class property C(c) (being C(c ) ≤ C C(c)), but the classifier has provided a class with a more precise observation in C than that of c.
For this reason, we introduce a new weakening of robustness, allowing us to check robustness not by checking the equality of class properties but their (abstraction) relation.
Example 5.7.Imagine a classifier built into a system for autonomous driving that tries to recognize speed limit road signs.If the speed limit is underestimated, i.e., 40 is detected but the speed is actually 50, then this may be considered an acceptable mistake, not really affecting the classifier's robustness.This situation can be modeled by using Definition 5.6.
Let C def = Z.It is a poset with the standard numerical order.We can consider C as the identity function.Let Ω be a set of images of speed limit road signs, then Definition 5.6 requires that the perturbations of the images result in a predicted speed limit that is equal, or at least lower, than the one predicted for the untouched image.Beware that this does not mean that the predictions are correct, we are only stating that they can be considered acceptable, making the classifier sufficiently robust to adversarial attacks.
Example 5.8.Let f : I → C be the classifier in Example 5.4.Let us consider a restricted class domain C {Lion, Squirrel, Horse} (and again C = ℘(C), being C a set without an order).Imagine that we are interested in checking that, starting from an image x of a lion, f is not distracted by small perturbations applied to x and keeps predicting a lion for all x ∈ I(x ).Lions are dangerous animals and we do not want the classifier to mistake it for a squirrel just because we have perturbed, e.g., either brightened or darkened, the image.However, a squirrel suddenly becoming a lion may not be a big issue.
Let We may wonder whether this new definition can be seen as a particular case of abstract robustness, and indeed we can prove that in some conditions this is possible, but not for any class domain, and in particular, not when the class domain is a not flat poset.In particular, whenever the class abstraction C is a flat domain (i.e., the abstract classes are mutually exclusive with a top and a bottom), then this notion collapses to abstract robustness, since the only possibility for an abstract class to be contained in another one is to be equal to it.In all the other cases, abstract robustness, w.r.t. the same parameters, is strictly stronger than hierarchy-aware abstract robustness.These results tell us that hierarchy-aware abstract robustness is more general than abstract robustness, since the latter can be modeled as a special case of hierarchy-aware abstract robustness simply by forcing C (by embedding a flat order relation) to be flat and taking C = C.In this way, any of its abstractions is flat and the two notions collapse.
Another important aspect to observe of this notion is that the role of the direction in the relation between classes for checking hierarchy-aware abstract robustness still makes it strongly dependent on the choice of the region's center to check.This means that to regain such independence, we must further weaken the robustness definition.
Let us consider the scenario depicted in Figure 3 where on the left we have the complete lattice C, and, on the right, the perturbation regions of the input points x 1 and x 2 .Suppose f (x 1 ) = c u , f (x 2 ) = c d , while all other points in both the regions have class In this case, hierarchy-aware abstract robustness is satisfied in

COHERENCE: MAKING ABSTRACT ROBUSTNESS A REGIONAL PROPERTY
In the previous section, while weakening robustness, we made it strongly dependent on the chosen center of perturbation.This dependency may become particularly annoying, also when we do not abstract the classes, namely in standard (concrete) robustness (Definition 4.4).This may happen, in particular, when C is a poset, namely when the class domain shows a hierarchical structure itself.Let us explain this issue by means of an example.Suppose we have a classifier, and suppose we want to define a notion of hierarchy-aware abstract robustness dealing with regions of points instead of with single points.In this case, the idea is to deem a classifier as robust when the classification of the elements in the perturbed region all have some information in common; it is not important what kind of information, as long as it is meaningful.We consider "meaningful" any element in C (chosen/designed by the analyst) that is different from the top, since the top is the element corresponding to having no information at all about classes. 7Hence, if all the classes returned for the elements in a perturbation region share meaningful information, then we can say that the classifier provides a coherent answer, by not assigning completely unrelated classes to points inside the given region.Coherence needs abstractions defined on lattices, since we need a least upper bound defined on any pair of observed properties of classes and a top element (join-complete semi-lattice).Note that ℘(C) is always a complete lattice and therefore a join semi-lattice, and hence the constraint is only when we decide to keep the potential order structure embedded in C. C has to be a join complete semi-lattice, since we need the top element ( C ) and the least upper bound (∨ C ) for any (finite or infinite) set of elements for defining coherence.C represents a complete loss of information, something that C cannot represent meaningfully, while C( C C ) represents the smallest (abstract) class generalizing all the elements c ∈ C .Since C is the element that stands for a complete loss of information, by requiring C( C C ) to be less than C we require all the classes in C to have something meaningful in common, with respect to the designed class abstraction C. Example 6.2.Let f : I → C be a classifier for an object detection system as in Example 5.4.In this case, suppose that C is a complete lattice.This means that some classes are more generic than others.The classifier can predict a very specialized class if it can do so with confidence, but it can also predict a broader class when it is unable to nail down a more specific class.For instance, if it cannot decide whether something is a car or a van, then it can just say it is a vehicle.We make this class domain a complete lattice by adding the top ( ) element representing a special class that generalizes all the others.C can clearly be a very vast domain but consider, for the sake of simplicity, the one depicted in Figure 4 (where ≤ C and C = are those depicted in Figure 4).Let us consider C = C and the identity class abstraction on C, i.e., C = id.Let x ∈ I be such that f (x ) = Horse.To satisfy coherence, f may either predict all x ∈ I(x ) as animals, or all as vehicles.If the prediction mixes the two abstract classes, then the least upper bound goes to the top and coherence is no more satisfied.For instance, if ∀x ∈ I(x ) we have f (x ) ∈ {Lion, Horse}, then we have the following: But if ∀x ∈ I(x ), then we have f (x ) ∈ {Car, Lion}, and then coherence fails, Indeed, a horse is related to a lion as they are both animals, but a horse is also related to a car as they can both be used as vehicles.Conversely, a lion and a car are nothing alike, and mistaking them becomes unacceptable.
Finally, let us observe that, at first sight, it might seem that coherence is a further weakening of hierarchy-aware abstract robustness.Surely, hierarchy-aware abstract robustness allows us to accept points within a perturbation region to have a class that is less or equal to the region center class, while coherence accepts, in addition, a greater class.Unfortunately, limit cases make the two notions incomparable.It is worth noting that the abstract robustness properties defined so far (Definition 5.2 and Definition 5.6) are tightly bonded with the perturbation region center x ∈ Ω.If it goes to , then abstract robustness may still be satisfied.Coherence instead does not accept this exception, and is not allowed at all, being too imprecise.

5:15
Remark on flat class abstractions.Both Theorem 5.9 and Lemma 6.4 tell us that, when dealing with flat abstractions of classes, everything may collapse (under some additional conditions for coherence) to abstract robustness, which is indeed robustness w.r.t.abstract classes (Theorem 5.5-2).This observation is important, because it tells us that when the abstraction of classes C is flat, then the entire framework may collapse just to robustness w.r.t. a new domain of (incomparable) abstract classes, the ones identified by C. But, at the same time, this also gives value to the idea of using abstract interpretation, since it means that the proposed framework becomes novel and meaningful precisely when we exploit the power of abstract interpretation by considering strictly hierarchical (by generalization or by approximation) abstractions of classes, taking, so far, in consideration class relations that escape from the standard approaches to robustness.

BRIDGING THE GAP: ROBUSTNESS IN PROGRAMMING LANGUAGES
In the previous section, we introduce several weakenings of robustness for classifiers driven by the idea of accommodating such notions depending on the field of application, in particular allowing to mistake some classes or making robustness a regional notion, i.e., not depending on the center of perturbation.The approach used for such weakenings follows the standard framework of abstract interpretation [Cousot andCousot 1977, 1979] that has been introduced and developed for program analysis and, therefore, on program semantic functions.We strongly believe that the framework we develop here provides a bridge between classifier analysis (e.g., in neural networks) and program analysis.
In this section, we first show, using examples, how the proposed weakenings of robustness also have a meaningful interpretation in the context of (language-based) security, providing new insights into the abstract interpretation-based framework for modeling information-flow policies [Giacobazzi and Mastroeni 2018;Mastroeni 2005Mastroeni , 2013]].Afterward, we prove how the two abstract notions of robustness correspond (in some cases under specific conditions) to well-known and widely studied properties in program verification.Finally, we show that, instead, coherence provides a novel form of precision that can be required when dealing with program analysis [Giacobazzi et al. 2023].

Weakened Robustness in (Language-based) Security
In this section, we show, through examples, the role that the proposed notions of robustness may have in the context of language-based security.
7.1.1Abstract Robustness: Secure API Attack.In this section, we show an example of how abstract robustness corresponds indeed to a simplified and weakened form of non-interference, a widely used confidentiality policy in language-based security, defined in terms of generic perturbations on input (instead of ucos [Balliu and Mastroeni 2010;Giacobazzi and Mastroeni 2018]).Consider using a secure API to authenticate and authorize users to access an ATM cash machine.The user inserts the credit card and the PIN code into the machine.The PIN code gets encrypted and travels along the network until it reaches the issuing bank.At this point, a verifying API is executed to check whether the actual user PIN corresponds to the trial PIN inserted at the cash machine.The verifying API, called PIN V, is the one exploited by the attacker to disclose the actual PIN.A simplified code is given in Listing 1, where len is the length of the real PIN.The value to compare with PIN is obtained by the encryption of the validation data vdata (a kind of user profile) with the PIN derivation key pdk (x1), taking the len hexadecimal digits (x2), decimalizing through dectab (x5), and digit-wise summing modulo ten the offset (x4).EPB (Encrypted PIN Block) is the ciphertext containing the trial password encrypted with the key k.The trial PIN is recovered by decrypting EPB with key k (x5) and then compared with the computed encryption of inserted PIN code (x4).For the above snippet of code, there exists a brilliant attack used to disclose the exact PIN code just by modifying low integrity variables offset and dectab and observing the output response [Centenaro et al. 2009].8Let us consider, for the sake of simplicity, only the part of the attack involving dectab, allowing to determine the digits in the PIN (but not the right order), suppose Suppose in output we consider fp(C) def = { , {"PIN wrong }, ⊥}.Then, we avoid the attack if we cannot change the response for all the points in the perturbation.In particular, the semantics f of this code (executed with the right PIN encrypted in EPB) is not abstract robust when we use dectab n , with n in the PIN, instead of dectab.Indeed, C • f (dectab) = (being the trial PIN inserted right), while C • f (dectab n ) = "PIN wrong .Note that, in this case, we need abstract robustness requiring equality among all the elements in the perturbation, since it is precisely the change in the answer releasing information that the attacker can collect for disclosing the PIN.In contrast, the relation among answers is irrelevant.
7.1.2Hierarchy-aware Robustness: Buffer Overflow Bug.This section shows an example of hierarchy-aware robustness for detecting buffer overflow vulnerabilities.Consider the code in Listing 2 where the C++ (simplified) code snippet prompts the user to enter a password (line 5).It then compares this input against the stored password (line 7) it previously loaded (line 4).If they match, then the user is granted access.A mechanism like this could unlock the sharewareor trial-version of an application or give the user access to information and features inside the application by entering an administrator password.When we execute the program, as expected, entering the correct password (e.g., ripwd) grants us access.Conversely, entering an incorrect password denies access.But anyway, the program is vulnerable to buffer overflow, since the variable password address follows the variable input, meaning that if we enter an input that is eight chars long repeated two times, then we overwrite the content of password, forcing the test to be true.In this case, we would like to guarantee that the perturbation, repeating the string, of a wrong password does not make the test true, while vice versa is acceptable; we need hierarchy-aware robustness to model the situation.Let us define I(s) = {s n | n ∈ N} (where s n is the self concatenation of s n times).Suppose in output we consider fp(C) def = { , {"Access denied }, ⊥}.Then on the right password the program semantics f is hierarchy-aware robust, since ∀s ∈ I(ripwd) 7. 1.3 Coherence: Detecting the Heartbleed Bug in OpenSSL.This section shows how coherence could detect a publicly disclosed bug, such as the well-known Heartbleed bug in the Heartbeat process in OpenSSL.
The Heartbeat function tests TLS/DTLS secure communication links by allowing a computer at one end of a connection to send a Heartbeat Request message, typically consisting of a text string, along with a declaration of its length as a 16-bit integer.The receiving computer then must send the same text back to the sender (the main C instructions causing the Heartbleed bug of the openssl-1.0.1/ssl/t1-lib.ctls1_process_heartbeat function is provided in Listing 3, where [...] stands for omitted code).The Heartbleed bug consists of a data leakage caused by the unchecked correspondence between the declared amount of information to copy from memory (payload) and the size of the information to copy from the source (pl).This situation may be exploited in the presence of a man-in-the-middle attack by controlling the amount of information to copy value.Let us suppose that, as it happens in the OpenSSL functions, the length of information to copy is set to 16 (in the caller function); we should expect that also payload should set to 18 and therefore bp (the returned observable message) should be at most 37 bytes.Hence, any execution where the bp contains less than 37 bytes (and at least one) is safe, since, in this case, a subset of information to copy is indeed copied.With this code, if an attacker can crack the vulnerable website application by changing the payload value, then the code copies in bp not only the information requested (i.e., pl) but also other information stored in the memory.Hence, when a website or application is vulnerable to Heartbleed bug, then personal information, including passwords and credit cards, could be stolen by attackers.
Let us show how coherence may capture this vulnerability.Let us check the coherence w.r.t. the payload input (for the sake of simplicity, we consider it as a separate input), and let us consider the perturbation changing of one unit the value (plus or minus): . Consider, as output observation, the subset of interval domain on the length of bp (the message sent back, and therefore observable), where we can observe precisely all the bp lengths (the identity) but also the safe interval [20,37] (due to payload between 1 and 18), i.e., fp( [20,37]}.Then the code semantics f is coherent for instance on payload = 15, since the 20,37] .However it is not coherent on payload = 18 (or higher), since Note that the semantics f , in this case, does not satisfy hierarchy-aware robustness, since for instance C • f (14) = [33, 33] is not comparable with the result of the center of perturbation 34,34], meaning that coherence better captures the vulnerability when C minimally models what is supposed to be safe.

Weakened Robustness in Program Verification
To move in the program analysis field, we have to constrain perturbations to be specific extensive abstractions, namely upper closure operators.Then, we can show how the proposed weakenings of robustness may collapse to well-known properties in program verification.These connections are interesting, since they allow us to relate well-studied techniques developed for program verification by abstract interpretation to the context of classifier robustness and, therefore, to all the properties associated with robustness, such as safety.

Abstraction-based
Perturbations.Some perturbation functions are particularly interesting, since they can be modeled by collecting all the points sharing a common property, modeled by an abstract domain η ∈ uco(℘(I)). 9ormally, starting from η ∈ uco(℘(I)) we can define the abstraction-based perturbation as the partition induced by η (Section 3), i.e., I η x)} (see Section 3 for definitions).When this happens, we say that the perturbation is induced by η and its lift to ℘(I) becomes the partitioning closure of η, namely I ↑ η ∈ uco(℘(I)).Note that if η is partitioning, then I ↑ η = η.Hence, all ucos may induce an input perturbation function, but not all input perturbation functions are induced by an uco unless they are themselves ucos when lifted on ℘(I).For instance, the sphere perturbation is not abstraction based.However, any perturbation inducing a partition of input is induced by an uco and can be lifted to an uco (by additive lift).

Abstract Robustness: (Global) Domain Completeness.
The following result tells us that (global) abstract robustness, when dealing with ucos and monotone functions, is precisely (global) completeness of f w.r.t. the input observation η and the output observation C, which is indeed Abstract Non-interference [Giacobazzi and Mastroeni 2018;Mastroeni and Pasqua 2023].In the following, we have to use the additive lift of f : Hence, abstract robustness on all inputs w.r.t an abstract,ion-based perturbation and an additive class abstraction correspond to (global) completeness of these abstract domains for the additive lift of the classifier [Giacobazzi et al. 2000].To prove the equivalence, we need to consider the whole I (globality); otherwise, for a generic x , we could not guarantee, under the completeness hypothesis, that all the elements in the perturbation satisfy completeness.

Hierarchy-aware Robustness: Local Domain Completeness.
When dealing with hierarchyaware abstract robustness, in general, the binding with completeness is possible only with a subset of inputs.In particular, the relation holds locally for any element in Ω ⊆ I. Therefore, when it is a singleton, it corresponds precisely to local completeness [Bruni et al. 2021], namely completeness on a single abstract input.Theorem 7.2.Let I be a set of inputs, C a class set, C a chosen class domain, C ∈ uco(C) additive and η ∈ uco(℘(I)), then It is worth noting that, in general, without the additivity condition on C, the completeness-based characterization is always stronger than the corresponding robustness notion.

Coherence
It is worth noting how hierarchy-aware abstract robustness and coherence both compute Cf ↑ I η (x ) (7.2 and 7.3), except that the former enforces a particular target (requiring the equality with Cf (x )).At the same time, the latter is looser and accepts any value at all as long it can be meaningfully represented in the abstract class domain (different from C ).However, we can observe that Cf (x ) appears in both abstract robustness notions, while it is ignored in coherence.This is related to coherence's weak binding with the perturbation region center.

CONFINING ADVERSITIES BY ABSTRACT INTERPRETATION
In this section, inspired by the strong connection with abstract non-interference and domain completeness, we explore the possibility of characterizing the robustness property satisfied by a given classifier on a fixed input (locally).We consider only hierarchy-aware robustness and coherence, since abstract robustness (Definition 5.2), as previously observed, may be modeled as a particular case of hierarchy-aware abstract robustness.Still, it also closely corresponds to a form of Abstract Non-Interference/Completeness, which both have well-known characterizations [Giacobazzi and Mastroeni 2018;Giacobazzi et al. 2000;Mastroeni and Pasqua 2023].
In the following sections, we investigate the meaning and the source of abstract robustness violation, depending on the violated notion.Understanding where the violation comes from allows us to determine whether the classifier is indeed unsafe or whether the formalized robustness property to enforce is still too strong.

Confining Adversities to Hierarchy-aware Robustness
As far as hierarchy-aware abstract robustness is concerned, the violation may happen both for a too-coarse perturbation, which can be seen as the model of an adversarial attacker able or not to induce the classifier failure, or for the choice of the class abstraction, that can be still too concrete.Hence, we have two possible directions for confining the sources of violation: (1) to refine the partition, characterizing which is the smallest set of perturbations that have the same (abstract) class of the center, and (2) to further abstract the class domain, finding the classes that are indeed mixed up by perturbing the input.Direction ( 1) is the confinement of the perturbation power of an adversarial attack, while direction (2) is the confinement of the effects of the attack on the classification.
To refine the perturbation, we have to restrict the perturbation of x to all and only the perturbations whose abstract classification is approximated by the abstract class of x , formally we define When interpreting perturbations as adversarial attacks, this is the most abstract adversarial attacker w.r.t.x (i.e., able to generate the maximal amount of perturbations of x ) unable to induce the classifier in error, namely not violating the classifier (hierarchy-aware abstract) robustness.
The property may also fail, because the classification is still too restrictive.In this case, we could compute precisely the classes mixed up by perturbing x , to be able to decide whether the classifier is indeed not robust, or the notion we enforced was too strong.In this case, the problem is that the abstract class of x is not comparable with the abstract class of some of its perturbations.The only possibility is to further abstract the class associated with x to cover all the classes of perturbations.Let us define the transformer The resulting set is still a Moore family for the greatest lower bound properties and the new abstract class for x precisely characterizes the degree of class approximation necessary to guarantee robustness.It is the analyst that must decide whether this is acceptable.It is worth noting that, in the worst case, the minimal abstract class is C , meaning that the classifier cannot be robust w.r.t. the given adversarial attack, i.e., perturbation.

Confining Adversities to Coherence
As far as coherence is concerned, the situation is quite different, since, for the nature of the notion, the only possibility, if it exists, is to refine both the perturbation and the class abstraction.Let us identify the potential sources of violation.Let us consider first the perturbation.When dealing with classical perturbations (determined in terms of a kind of distance from the x ), it is pretty unrealistic to have f ↑ • I(x ) = C = C (for the chosen class domain C).
When, instead, we consider η ∈ uco(℘(I)) and we take as perturbation I η , then a wrong choice of η, determining the perturbation in terms of a shared property, may indeed cause a limit situation where f ↑ • I η (x ) = C , and in this case we would have first to adjust the perturbation, before checking coherence w.r.t. a class abstraction.Let us define the following refinement reducing the perturbation of x to a set not leading the classifier to the top: where the domain operation [Bruni et al. 2022] is defined as follows.Let ρ ∈ uco(L) and r ∈ L It is worth noting that, in the worst case, we add precisely x to I ↑ η , meaning that the perturbation I ↑ η cannot be adjusted for making the classifier coherent in x .The condition forced above is a necessary condition for coherence to hold.It is, instead, not sufficient, since still, C may lead to the top.Let us characterize the class Adversities in Abstract Interpretation 5:21 abstraction, making the classifier coherent w.r.t. a given perturbation.Let us define the following refinement, adding the most coarser abstract class capturing the class of all the inputs in a set X , In the worst case, we add precisely f ↑ (X ) ∈ C, identifying the classifier mistake among classes w.r.t. the perturbation.
Note that if we define coherence in terms of I ∈ Pert(I), hence such that f ↑ • I(x ) C C , then Proposition 8.2 alone enforces coherence.Otherwise, we need to combine the two transformations: Hence, as far as coherence is concerned, differently from hierarchy-aware abstract robustness and the other cited properties (abstract non-interference and completeness), it may be necessary to refine, both the perturbation and the class abstraction to characterize the coherence property locally satisfied by a classifier.

EVALUATION
In this section, we show the practical applicability of our formal framework by testing a few real neural network classifiers.Rather than implementing an analyzer from scratch, we have leveraged an existing toolkit: the ERAN.12

ERAN: A Neural Network Analyzer for Robustness
ERAN is a state-of-the-art neural network analyzer developed at ETH.Several different verification techniques are implemented in it, but we focused on their latest algorithm based on abstract interpretation: DeepPoly [Singh et al. 2019].Normally, the DeepPoly implementation works like this: (1) Select the first 100 samples from the testing set of a chosen dataset.
(2) Test the classifier as usual and filter out the samples incorrectly classified.
(3) For each sample that was correctly classified singularly, over-approximate the ∞ ball of radius ϵ ∈ R around it and abstractly interpret the network to obtain an upper bound and a lower bound for the score of each class.(4) Examine the resulting bounds to assess whether there is a possibility of some other class overtaking the original.If so, then the verification fails; otherwise, robustness around the point is proven.Once all the samples have been examined, DeepPoly prints the percentage of inputs that could be verified among those correctly classified.

Relaxed-ERAN: A Multi-modal Neural Network Analyzer
To achieve our goal, we have forked 13 the original code and changed it, and we named our tool Relaxed-ERAN.The main improvement is that Relaxed-ERAN can operate in three modes: robustness (Definition 4.4), hierarchy-aware abstract robustness (Definition 5.6), and coherence (Definition 6.1).The original ERAN corresponds effectively to the first mode only.At the same time, we ignore abstract robustness (Definition 5.2), since, as previously observed, it can be modeled as a particular case of Definition 5.6 (see Theorem 5.9).9.2.1 Relaxed-ERAN Inner Workings.With respect to the description of ERAN above, we modified point (4) so that the robustness of the neural network in a given input sample is determined by a subroutine whose behavior depends on the chosen operating mode. 14The arguments passed to the subroutine are as follows.
Center class.f (x ): The class assigned to the input sample x under examination.This is obtained simply by concretely executing the network on the single input x .
Perturbation region classes.{f (x ) | x ∈ I(x )}: The set of classes assigned to the points inside the perturbation region I(x ) -an ∞ ball centered in x .It is collected by reasoning on the computed lower and upper bounds.A class i can never be predicted if another class j exists that always has a larger score, i.e., its lower bound l j is larger than the upper bound of the former u i .When that does not happen, the class i can win somewhere in the sphere.Formally, class i is collected if j .u i < l j .
Pools.P ⊆ ℘(C): a collection of sets of classes through which the analyst intends to observe the outputs.We name each set of classes, i.e., each p ∈ P, a "pool." The pools are only used when operating in "hierarchy-aware abstract robustness" mode or in "coherence" mode-to reason about properties of predictions rather than raw classes.Note that we can generate pools in two ways.We can manually identify the set of generalized classes that we accept to observe in the output classification and then formally generate the abstract domain induced by the pool (by Moore closure of the pool, see Section 3 for details), or we can define the abstraction function of classes C, as usual in abstract interpretation and then generate the pool by taking all the abstract elements in C (except the top).
We denote by Relaxed-ERAN(f , x , ϵ, m, P) the Relaxed-ERAN-analysis of f : I → C, on the input x ∈ I, with perturbation function I(x and mode m ∈ {Rob, HawRob, Coher}.Furthermore, P might be omitted when m = Rob, since it is not used during its analysis and thus does not affect the outcome.We emphasize here that our modifications have virtually no net impact on the performance of ERAN, because step 4 is now O(|C 2 |)-which is constant time.Therefore, the time measurements reported in Singh et al. [2019] shall also be a rather faithful indication of the performance of Relaxed-ERAN.9.2.2 Verified Properties.We formalize here the properties verified by Relaxed-ERAN, depending on the mode m.For the sake of readability, let us denote f Proposition 9.1.Relaxed-ERAN(f , x , ϵ, m, P) verifies the following properties, depending on the mode m: Adversities in Abstract Interpretation 5:23

On the Relaxed-ERAN Soundness and Completeness
In this paragraph, we discuss the soundness and completeness of Relaxed-ERAN w.r.t. the notions of Robustness that we aim to enforce.In particular, the following theorem shows that fixing the pools of classes that the analyst aims at observing, then Relaxed-ERAN enforces a corresponding coherence property in a sound and complete way.
Then the property verified by Relaxed-ERAN(f , x , ϵ, HawRob, P) corresponds to hierarchy-aware robustness, i.e., On the contrary, the following result shows which set of pools we have to use in Relaxed-ERAN to verify a given hierarchy-aware abstract robustness property (where the perturbation is fixed to the sphere, and the poset of classes is its power set).
Then the property verified by Relaxed-ERAN(f , x , ϵ, Coher, P) corresponds to coherence, i.e., On the contrary, the following result shows which set of pools we have to use in Relaxed-ERAN to verify a given coherence property (where the perturbation is fixed to the sphere, and the poset of classes is its power set).

Relaxed-ERAN Experimental Evaluation
In this paragraph, we walk through a few experiments in which we tested -using Relaxed-ERAN -a few of the neural networks provided by Singh et al. [2019] on the MNIST [LeCun et al. 2010] and CIFAR10 [Krizhevsky 2009] datasets.
Concretely, we pitch robustness (Definition 4.4) against hierarchy-aware abstract robustness (Definition 5.6) on both datasets and then do the same against coherence (Definition 6.1); using different lists of pools P each time to model a realistic analysis scenario.
We used a CPU-optimized virtual machine hosted by DigitalOcean with 8 gigabytes of working memory and four virtual CPUs.
Experiment 1: Can mistake a digit for a lower one.We analyze a handwritten digit recognizer trained on the MNIST dataset.Suppose we accept that perturbing an image can only result in the classifier estimating an equal or lower number, but never higher.This can be useful, e.g., in the context of a self-driving car that should never go faster than the speed limit (see also Example 5.7).We compute Relaxed-ERAN(f , x , 0.3, HawRob, P 1 ), where and x varies among the database images.In Figure 5, we consider as x 1 the image 58 (in MNIST) on the left, and as x 2 the image 72 on the right.Then, f (x 1 ) = 9 and ∀x ∈ I(x 1 ).f (x ) ∈ {4, 9}.Since 4 is smaller than 9, robustness is verified.However, f (x 2 ) = 2, but unfortunately ∃x ∈ I(x 2 ).f (x ) = 3, meaning that the classifier can predict a larger number, raising a violation.
Experiment 2: Wild animal detector.Let us imagine that an animal association system uses a classifier f for the CIFAR10 dataset to detect the presence of wild animals in the meadows that surround a dangerous highway so that they can rescue them.Realistically, we allow the image classifier to lean on the conservative side and to mark a perturbed image to contain an animal, even if it is a vehicle.Concretely, the property that we test is Relaxed-ERAN(f , x , 0.008, HawRob, P 2 ), where P 2 = {{c | c is an animal}}, f def = ConvBig, and x varies among the database images.In Figure 6 we consider as x 1 the image 77 (in CIFAR10) on the left, and as x 2 the image 84 on the right.Then f (x 1 ) = Cat and ∀x ∈ I(x 1 ).f (x ) ∈ {Cat, Dog}.Since all perturbations are classified as animals, no violation is raised.However, f (x 2 ) = Bird but ∃x ∈ I(x 2 ).f (x ) = Airplane.Hence, an animal is no longer recognized once the image is perturbed, and therefore, a violation is raised.
Experiment 3: Closely packed numbers.Assuming again that we are working with a handwritten digit recognizer for the MNIST dataset, we want the classes assigned to the points inside the perturbation region to span a short interval.Testing Relaxed-ERAN(f , x , 0.3, Coher, P 3 ), with and x varies among the database images.Hence, we aim to check that any x in the database, even when perturbed, gets assigned to numbers within ±1 of each other.In Figure 7, we consider as x 1 the image 8 (in MNIST) on the left, and as x 2 the image 11 on the right.Then, f (x 1 ) = 5 and f I (x 1 ) = {5, 6}.These numbers are close enough that no violation is raised.However, f (x 2 ) = 6 but f I (x 2 ) = {0, 6} where the numbers are too far apart, and therefore, a violation is raised.
Experiment 4: Animals and means of transport.Given a network trained to classify images from the CIFAR10 dataset, we allow an image, plus its perturbations, to be classified as animals, or means of transport, but not as a mix of the two.Concretely, we execute Relaxed-ERAN(f , x , 0.008, Coher, P 4 ), with set of pools distinguishing animals from means of transport, i.     among the database images.An interesting spin, is that we consider the class Horse to be an animal and a means of transport at the same time.In Figure 8, we consider as x 1 the image 54 (in CIFAR10) on the left, and as x 2 the image 64 on the right.Then f (x 1 ) = Ship and f I (x 1 ) = {Ship, Airplane}.Both are vehicles; no violation is raised.However, f (x 2 ) = Frog, but f I (x 2 ) = {Cat, Dog, Frog, Truck}, meaning that there is a violation, because the classifier predicts a mix of animals and vehicles.
Results.We report the outcomes of all four experiments in Tables 1 and 2, tested on the correctly classified images, among the first 100 images of the datasets.In particular, in the MINST dataset, we tested abstract robustness verification on 95 images for the ConvBig classifiers (5 images were not correctly classified) and on 99 images for the classifier ConvSuper (1 image was not correctly classified).In the Cifar10 dataset, the amount of tests is smaller, since only 60 images were correctly classified.The resulting percentage scores refer to the ratio of correctly classified images that passed the verification using a specific mode m.Note that, when m = Rob, the results match (as expected) what is obtained by executing the unmodified ERAN analyzer. 16We can observe that both hierarchy-aware abstract robustness and coherence have higher scores than strict robustness, empirically proving that we can avoid a relevant amount of false alarms whenever the degree of abstraction considered for testing abstract robustness is deemed as acceptable.

DISCUSSION: WHAT IS NEXT
In this article, we tackle the problem of formalizing and accommodating the notion of classifier robustness to the specific field of application.In particular, we use abstract interpretation to model the degree of error that we aim to accept in the class prediction performed by a classifier, and we observe how the introduction of abstraction splits robustness in several potential notions with different peculiarities.In Figure 9, we synthesize all the implication relations proved among the robustness weakenings proposed, and we recall the main conditions making the implications hold.These weakenings are motivated in the context of classifiers, but we show how these notions can meaningfully model information flow policies in language-based security, noting that, while Abstract Robustness can be considered a form of Abstract Non-interference [Giacobazzi andMastroeni 2004, 2018], the other two policies are novel and provide new insights to the problem.Moreover, we also show how, under certain conditions, the robustness notions proposed here correspond to approximation precision (completeness) forms in program verification [Bruni et al. 2021;Cousot and Cousot 1979;Giacobazzi et al. 2000].We strongly believe this work is just a first step, since there are still many other directions to investigate, both from theoretical and practical perspectives.First, it would be interesting to investigate further the relation between the approaches proposed for analyzing and characterizing program robustness [Chaudhuri et al. 2012], also in the presence of active attackers modifying the code [Balliu and Mastroeni 2010], and the framework proposed here.Moreover, the proven connections between robustness, abstract non-interference and completeness, together with the definition of program robustness as continuity [Chaudhuri et al. 2010[Chaudhuri et al. , 2012] ] create a bridge also between continuity and completeness [Cousot and Cousot 1979;Giacobazzi et al. 2000], in particular with partial completeness [Campion et al. 2022], where the distance is not measured between values, but between their abstractions.Moreover, the recently proposed notion of adequacy in program analysis [Giacobazzi et al. 2023] is strongly related to coherence, and therefore, it could be interesting to understand the meaning of defining a parametric notion of coherence (substituting the top bound with a generic, potentially smaller, threshold) for classifiers.
In the context of classifiers, the first aspect that could be interesting to investigate is the connection with formal concept analysis, which is a method for data representation and analysis [Ganter et al. 2005;Škopljanac Mačina and Blašković 2014].From given tabular input data this approach aims to find all formal concepts shared by data and computes a concept lattice in which all formal concepts are hierarchically ordered.We think, that this could be an empirical way for automatically deriving, from the application context and from the effective collection of classified data, a suitable abstraction of classes, minimally abstracting classes.the proof of Theorem 5.9 for flat abstractions such that ∀x ∈ I. C(f (x )) C , this implies that ∃c ∈ fp(C) { C , ⊥ C } such that {Cf (x ) | x ∈ I(x )} = {c}, hence by transitivity of equality ∀x ∈ I(x ) we have Cf (x ) = Cf (x ), implying Cf (x ) ≤ C Cf (x ) (by Theorem 5.9 we already know that for flat abstractions abstract robustness and hierarchy-aware abstract robustness collapse).
However, we can show that if C is not flat, then we can find a witness f in Coher Definition 4.3 (Perturbation (Function)).Let I be a set of data.A perturbation function for I is a strictly extensive function I : I → ℘(I), i.e., such that ∀x ∈ I. {x } I(x ).The set of all perturbations on I is Pert(I).
Definition 4.4 (Robustness).A classifier f : I → C is robust w.r.t. the input set Ω ∈ ℘(I) and the perturbation I ∈ Pert(I), written f Robustness (Definition 4.4) of g, with Ω def = {x } and I(x ) def = X , is true if and only if f (X ) ⊆ Y .Support Vector Machine.Let us consider Support Vector Machine classifiers.Let I = R n , C = {−1, +1}, and f (x ) = sign(w T x + b), and the classifier f : I → C be the composition of a function g : R n → R m performing the forward propagation, and a function h : R m →[1, m]  selecting the winning class.

Fig. 2 .
Fig. 2. Hasse diagram of the class domain C = C of Example 5.3.classes returned should share a fixed property, chosen by the analyst and formalized as an abstraction.Definition 5.2 (Abstract Robustness).A classifier f : I → C is abstract robust w.r.t. the input set Ω ∈ ℘(I), the chosen class domain C, the perturbation I ∈ Pert(I), and the class abstraction C ∈ uco (C), written f ∈ AbsRob Ω, C (I, C), if ∀x ∈ Ω. ∀x ∈ I(x ).Cf (x ) = Cf (x ).
def= {c | c ∈ [0, 99]} be the domain of integers between 0 and 99 (Figure2), and suppose it is equipped with the partial order ≤ C defined as ∀a, b ∈ C. a ≤ C b iff a/10 = b, where / is the integer division.With C being a poset, we can consider C = C, and therefore we can define C ∈ uco(C) = uco(C) to be the class abstraction that takes a number and sets the unit digit to zero (depicted with blue arrows in Figure2).Formally, ∀a ∈ C the uco is defined as C(a) def = 10 * a/10 , namely its fixpoints are{0, 10, 20, 30, 40, 50, 60, 70, 80, 90}.If f is such that f (x ) = c and f (x ) = c c for x ∈ I(x ), then abstract robustness (Definition 5.2) still holds as long as c and c have the same left digit.For instance, if c = 82 and c = 88, then abstract robustness holds, since C(82) = 80 = C(88) (while robustness in Definition 4.4 does not).
Formally, the abstraction of a species c ∈ C should be the entire collection of animals (if c is an animal) or of plants (if c is a plant).Hence, we consider C = ℘(C) and therefore C ∈ uco(C) = uco(℘(C)).Therefore, we can define C on single classes as follows and then additively lift it to sets of classes C ∈ ℘(C) (hence abstracting to = C any set C containing both animals and plants): C(c) = {c | c belongs to animal species} = {Lion, Squirrel, Horse} if c is an animal {c | c belongs to plant species} = {Dandelion, Rose, Oak} if c is a plant Now, let us consider x ∈ I such that f (x ) = Lion and suppose that ∀x ∈ I(x ) in its neighborhood we have f (x ) ∈ {Squirrel, Lion}.Abstract robustness holds in this case, since C(Lion) = {Lion, Squirrel, Horse} = C(Squirrel).While, if ∃x ∈ I(x ).f (x ) = Dandelion, then robustness fails, since C(Lion) {Dandelion, Rose, Oak} = C(Dandelion).
Definition 5.6 (Hierarchy-aware Abstract Robustness).A classifier f : I → C is hierarchy-aware abstract robust w.r.t. the input set Ω ∈ ℘(I), the chosen class domain C, the perturbation I ∈ Pert(I), and the class abstraction us consider the class abstraction C ∈ uco(C) defined as: ∀c ∈ C C(c) c if c ∈ {∅, {Lion}, {Lion, Horse}} C otherwise At this point, let x ∈ I such that f (x ) = Squirrel, then even if, for instance, ∃x ∈ I(x ) such that f (x ) = Lion, then hierarchy-aware abstract robustness (Definition 5.6) holds, since we have that Cf (x ) = {Lion} ≤ C C = Cf (x ).Nevertheless, Definition 5.6 strongly depends on the center choice.For instance, if f (x ) = Lion and ∃x ∈ I(x ) such that f (x ) = Squirrel, then Definition 5.6 does not hold on x , because Cf (x ) = C C {Lion} = Cf (x ).

Definition 6. 1 (
Coherence).Let f : I → C be a classifier, C the join complete semi-lattice chosen class domain.Then f is coherent w.r.t. the input set Ω ∈ ℘(I), the perturbation I ∈ Pert(I) and the class abstraction C ∈ uco (C), written f ∈ Coher Ω, C (I, C), if ∀x ∈ Ω. C( C {f (x ) | x ∈ I(x )}) C

Fig. 4 .
Fig. 4. A toy complete lattice C for object detection.
Lemma 6.3.Let I be an input set, C be a class set, C the lattice chosen as class domain and let I ∈ Pert(I), Ω ∈ ℘(I) and C ∈ uco (C).Then HawRob Ω, C (I, C) ⊆ Coher Ω, C (I, C) if and only if ∀x ∈ Ω. Cf (x ) C .Lemma 6.4.Let I be an input set, C be a class set, C the lattice chosen as class domain and let I ∈ Pert(I), Ω ∈ ℘(I) and C ∈ uco (C).If C is flat and ∀x ∈ I.C(f (x )) C , then Coher Ω, C (I, C) ⊆ HawRob Ω, C (I, C).If C is not flat then Coher Ω, C (I, C) HawRob Ω, C (I, C).Therefore, in general, neither property necessarily implies the other.Theorem 6.5.Let I be an input set, C be a class set, C the chosen class domain and let I ∈ Pert(I), Ω ∈ ℘(I).Then we have that ∃C ∈ uco (C).HawRob Ω, C (I, C) Coher Ω, C (I, C) and ∃C ∈ uco (C).Coher Ω, C (I, C) HawRob Ω, C (I, C).

}
.] /*bp is the destination location where the copied data will be kept; 6 pl is the source address; 7 Payload signifies the length of the data to be replicated;*/ 8 memcpy(bp, pl, payload); 9 [...]/*Sends back bp to the user*/ 10 Listing 3. Core of the Heartbleed bug in OpenSSL.
: A New Model for Characterizing Domain Precision.In this section, we show that coherence is quite an interesting and almost unexplored property in the field of abstract interpretation.Let us first provide a different characterization of the property, showing that it corresponds to a recently proposed notion of precision for abstract domains, namely Adequacy [Giacobazzi et al. 2023] with threshold C , determining how much the abstract domain is sufficiently precise w.r.t. the program code to analyze.Theorem 7.3.Let I be a set of inputs, C a class set, C a chosen class domain, C ∈ uco(C) and η Theorem 8.3.Let f : I −→ C be a classifier, C the chosen class domain, C ∈ uco(C), η ∈ uco(℘(I)) and x ∈ I such that f (x ) C C .Then Cl X C • f ↑ • I(x ) C C , where I def = Pe x η and X def = I(x ).

Table 1 .
Scores for Experiments 1 and 3 on the MNIST Dataset Classifier f ϵ m = Rob m = HawRob, P = P 1 m = Coher, P = P

Table 2 .
Scores for Experiments 2 and 4 on the CIFAR10 Dataset Classifier f ϵ m = Rob m = HawRob, P = P 2 m = Coher, P = P 4 Fig. 9. Schema of new notions and proved relations.