skip to main content
research-article
Public Access

FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments

Published:21 July 2022Publication History

Skip Abstract Section

Abstract

As advances in Deep Neural Networks (DNNs) demonstrate unprecedented levels of performance in many critical applications, their vulnerability to attacks is still an open question. We consider evasion attacks at testing time against Deep Learning in constrained environments, in which dependencies between features need to be satisfied. These situations may arise naturally in tabular data or may be the result of feature engineering in specific application domains, such as threat detection in cyber security. We propose a general iterative gradient-based framework called FENCE for crafting evasion attacks that take into consideration the specifics of constrained domains and application requirements. We apply it against Feed-Forward Neural Networks trained for two cyber security applications: network traffic botnet classification and malicious domain classification, to generate feasible adversarial examples. We extensively evaluate the success rate and performance of our attacks, compare their improvement over several baselines, and analyze factors that impact the attack success rate, including the optimization objective and the data imbalance. We show that with minimal effort (e.g., generating 12 additional network connections), an attacker can change the model’s prediction from the Malicious class to Benign and evade the classifier. We show that models trained on datasets with higher imbalance are more vulnerable to our FENCE attacks. Finally, we demonstrate the potential of performing adversarial training in constrained domains to increase the model resilience against these evasion attacks.

Skip 1INTRODUCTION Section

1 INTRODUCTION

Deep learning has reached high performance in machine learning (ML) tasks in a variety of application domains, including image classification, speech recognition, and natural language processing (NLP). Many applications already benefit from the deployment of ML for automating decisions and we expect a proliferation of ML in even more critical settings in the near future. Still, research in adversarial machine learning showed that deep neural networks (DNNs) are not robust in face of adversarial attacks. The first adversarial attack against DNNs was an evasion attack, in which an adversary creates adversarial examples that minimally perturb testing samples and change the classifier’s prediction [68]. Since the discovery of adversarial examples in computer vision, a lot of work on evasion attacks against ML classifiers at deployment time has been performed. Most of these attacks have been demonstrated in continuous domains (i.e., image classification), in which features or image pixels can be modified arbitrarily to create the perturbations [6, 8, 11, 27, 39, 46, 53, 68].

ML has a lot of potential in other application domains, including cyber security, finance, and healthcare, in which the raw data is not directly suitable for learning and engineered features are defined by domain experts to train DNN models. Additionally, in certain application domains such as network traffic classification used in cyber security, the raw data itself might exhibit domain-specific constraints in the original input space. Therefore, techniques used for mounting evasion attacks in continuous domains will not respect the feature-space dependencies in these applications and new adversarial attacks need to be designed for such constrained domains.

In this paper we introduce a novel, general framework for mounting evasion attacks against deep learning models in constrained application domains. Our framework is named FENCE (Feasible Evasion Attacks on Neural Networks in Constrained Environments). FENCE generates feasible adversarial examples in constrained domains that rely either on feature engineering or naturally have domain-specific dependencies in the input space. FENCE supports a range of linear and non-linear dependencies in feature space and can be applied to any higher-level classification task whose data respects these constraints. At the core of FENCE is an iterative optimization method that determines the feature of the maximum gradient of the attacker’s objective at each iteration, identifies the family of features dependent on that feature, and modifies consistently all those features, while preserving an upper bound on the maximum distance from the original sample. At any time during the iterative procedure, the input data point is modified within the feasibility region, resulting in feasible adversarial examples. Existing evasion attacks in constrained environments, such as PDF malware detection [65, 69], malware classification [30, 66], and network traffic classification [1, 2, 28, 31, 59] do not support the entire range of complex mathematical and domain-specific dependencies as our FENCE framework. Moreover, some of these attacks result in worse performance [2] or operate only in a specific domain [59] or feature space [2, 28].

We demonstrate that FENCE can successfully evade the DNNs trained for two cyber security applications: a malicious network traffic classifier using the CTU-13 botnet dataset [25], and a malicious domain classifier using the MADE system [50]. In both settings, FENCE generates feasible adversarial examples with small modification of the original testing sample. For instance, by adding 12 network connections and preserving the original malicious behavior, an attacker can change the classification prediction of a testing sample from \( \mathsf {Malicious} \) to \( \mathsf {Benign} \) in the network traffic classifier. We perform detailed evaluation to demonstrate that our attacks perform better than several baselines and existing attacks. We show that the state-of-the-art Carlini-Wagner attack [11] designed for continuous domains does not respect the feature-space dependencies of our security applications. We compare two optimization objectives in our FENCE framework, the Projected Gradient Descent (PGD) [46] and the Penalty method [11], and show the advantages of the PGD optimization. We also study the impact of data imbalance on the classifier robustness and show that models trained on datasets with higher imbalance, as is common in security applications, are more vulnerable.

We also consider attack models with minimum knowledge about the ML system, in which the attacker does not have information about the exact model architecture and hyperparameters. We test several approaches for performing the attacks through transferability from a surrogate model to the original one, using the FENCE framework. We observe that the evasion attacks generated for different DNN architectures transfer to the target DNN model with slightly lower success than attacking directly the target model with FENCE. Finally, we test the resilience of adversarial training using our attacks as a defensive mechanism for DNNs trained in constrained environments.

To summarize, our contributions are:

(1)

We introduce a general evasion attack framework FENCE for constrained application domains that supports a range of mathematical dependencies in feature space and two optimization approaches.

(2)

We apply FENCE to two cyber security applications using different datasets and feature representations: a malicious network connection classifier, and a malicious domain detector, to generate feasible adversarial examples in these domains.

(3)

We extensively evaluate FENCE for these applications, compare our attacks with several baselines, and quantify the adversarial success at different perturbations. We also study the impact of data imbalance on the classifiers’ robustness.

(4)

We evaluate the transferability of the proposed evasion attacks between different ML models and architectures, and show that adversarially-trained models provide higher robustness.

Skip 2BACKGROUND Section

2 BACKGROUND

2.1 Deep Neural Networks for Classification

A feed-forward neural network (FFNN) for binary classification is a function \( y = F(x) \) from input \( x \in R^d \) (of dimension \( d \)) to output \( y \in \lbrace 0,1\rbrace \). The parameter vector of the function is learned during the training phase using back propagation over the network layers. Each layer includes a matrix multiplication and non-linear activation (e.g., ReLU). The last layer’s activation is sigmoid \( \sigma \) for binary classification: \( y=F(x) = \sigma (Z(x)) \), where \( Z(x) \) are the logits, i.e., the output of the penultimate layer. We denote by \( C(x) \) the predicted class for \( x \). For multi-class classification, the last layer uses a softmax activation function. There are other DNN architectures for classification, such as convolutional neural networks, but in this paper we only consider FFNN architectures.

2.2 Threat Model

Adversarial attacks against ML algorithms can be developed in the training or testing phase. In this work, we consider testing-time attacks, called evasion attacks. There exist several evasion attacks against DNNs in continuous domains: the projected gradient descent (PGD) attack [46] and the penalty-based attack of Carlini and Wagner [11].

Projected gradient attacks. This is a class of attacks based on gradient descent for objective minimization, that project the adversarial points to the feasible domain at each iteration. For instance, Biggio et al. [8] use an objective that maximizes the confidence of adversarial examples, within a ball of fixed radius in \( L_1 \) norm. Madry et al. [46] use the loss function directly as the optimization objective and use the \( L_2 \) and \( L_{\infty } \) distances for projection.

C&W attack. Carlini and Wagner [11] solve the following optimization problem to create adversarial examples against CNNs used for multi-class prediction: \( \begin{equation*} \delta = \operatorname{arg\,min}||\delta ||_2+ c \cdot h(x + \delta)\end{equation*} \) \( \begin{equation*}h(x+\delta) = \max (0, \max (Z_k(x+\delta):k \ne t) - Z_t(x + \delta))\end{equation*} \) \( \begin{equation*} \text{where} Z() \text{are the logits of the FFNN}\end{equation*} \)

This is called the penalty method, and the optimization objective has two terms: the norm of the perturbation \( \delta \), and a function \( h(x + \delta) \) that is minimized when the adversarial example \( x+\delta \) is classified as the target class \( t \). The attack works for \( L_0 \), \( L_2 \), and \( L_{\infty } \) norms.

Under the assumption that the DNN model is trained correctly, the attacker’s goal is to create adversarial examples at testing time. In security settings, typically the attacker starts with \( \mathsf {Malicious} \) points that he aims to minimally modify into adversarial examples classified as \( \mathsf {Benign} \).

Initially, we consider a white-box attack model, in which the attacker has full knowledge of the ML system. White-box attacks have been considered extensively in previous work, e.g., [8, 11, 27, 46] to evaluate the robustness of existing ML classification algorithms. We also consider a more realistic attack model, in which the attacker has information about the feature representation of the underlying classifier, but no exact details on the ML algorithm and training data.

We address application domains with various constraints in feature space. These could manifest directly in the raw data features or could be an artifact of the feature engineering process. The attacker has the ability to insert records in the raw data, for instance by inserting network connections in the threat detection applications. We ensure that the data points modified or added by the attacker are feasible in the constrained domain.

Skip 3METHODOLOGY Section

3 METHODOLOGY

In this section, we start by describing the classification setting in constrained domains with dependencies in feature space and the challenges of evasion attacks in this setting. Then we devote the majority of the section to present our new attack framework FENCE which takes into consideration the relationships between features that occur naturally in the problem space or are the result of feature engineering.

3.1 Machine Learning Classification in Constrained Domains

Let the raw data input space be denoted as \( \mathcal {R} \). This is the original space in which raw data is collected for an application. In healthcare, \( \mathcal {R} \) could be the space of all data collected for a particular patient. In network security, \( \mathcal {R} \) could be the raw network traffic (for example, pcap files or Zeek network logs) collected in a monitored network in order to detect cyber attacks.

Consider a fixed raw data set \( R = \lbrace r_1,\dots , r_M\rbrace \in \mathcal {R} \). The raw data is typically processed into a feature representation, denoted by \( \mathcal {F} \), over which the machine learning model is trained. In standard computer vision tasks such as image classification, the raw data (image pixels) is used directly as input for neural networks. Thus, the training examples \( x_i \) are the same as the raw data: \( x_i = r_i, i \in [1,M] \). In this case the feature space \( \mathcal {F} \) is the same as the input space \( \mathcal {R} \).

In contrast, in other domains, such as threat detection or health care, the feature representation is not always exactly the raw data. See Figure 1 for a visual representation of this process. In most application domains, there might exist dependencies and constraints in the feature space introduced either by the application itself or by the feature engineering process:

Fig. 1.

Fig. 1. Neural network training for images (left) and for constrained domains with feature space dependencies (right). In the vision domain the raw data space \( \mathcal {R} \) is the same as the feature representation \( \mathcal {F} \) , while in constrained domains they might be different. Additional dependencies in feature space might arise from natural dependencies in the raw data and the feature engineering process.

  • Dependencies among different features could manifest naturally in the considered application. For instance, the results of two blood tests are correlated and they result in correlated feature values for a patient data. In network security, the packet size and number of packets are correlated with the total number of bytes sent in a TCP connection. We denote by \( \mathsf {Feasible\_Set}(R) \) the set of all feasible points in the raw data space. \( \mathsf {Feasible\_Set}(R) \) is a subset of the raw data that encompasses the feasible values for the particular application. For instance, a network TCP packet size is upper bounded by 1500 bytes and the ratio between the number of bytes and the number of packets in a TCP connection needs to be lower than the maximum packet size. The feasible set \( \mathsf {Feasible\_Set}(R) \) will only include network connections in which the upper bound on TCP packets is enforced and the ratio constraint between the number of bytes and number of packets is satisfied.

  • Constraints in feature representations might also result from the feature engineering process performed in many settings. In this case, features in \( \mathcal {F} \) are obtained by the application of an operator \( \mathsf {Op}_j \) on the raw data \( R \in \mathcal {R} \): \( x_{ij} = \mathsf {Op}_j (R) \). Examples of operators are statistical functions such as \( \mathsf {Max} \), \( \mathsf {Min} \), \( \mathsf {Avg} \), and \( \mathsf {Total} \), as well as linear combinations of raw data values, and other mathematical functions such as product or ratio of two values. The set of all supported operators applied to the raw data is denoted by \( \mathcal {O} \). This process creates \( N \) training examples \( x_1,\dots ,x_N \) in the feature space \( \mathcal {F} \), each being \( d \)-dimensional, with \( d \) the size of the feature space. The feature engineering process creates additional dependencies in feature space. For instance, if we consider the \( \mathsf {Max} \), \( \mathsf {Min} \), and \( \mathsf {Avg} \) number of connections for a particular port in a given time window, the average value needs to be between the minimum and the maximum values.

A data point \( z = (z_1,\dots ,z_d) \) in feature space \( \mathcal {F} \) is feasible if there exists some raw data \( R \in \mathcal {R} \) such as for all \( i \), there exists an operator \( \mathsf {Op}_j \in \mathcal {O} \) with \( z_i = \mathsf {Op}_j (R) \). The set of all feasible points in feature space for raw data \( R \) and operators \( \mathcal {O} \) is called \( \mathsf {Feasible\_Set}(R,\mathcal {O}) \). This space includes the set of feasible points \( \mathsf {Feasible\_Set}(R) \) (obtained for \( \mathcal {O} = \emptyset \)). Examples of feasible and infeasible points in feature space are illustrated in Table 1. The constraints in this example are that the sum of feature values must sum up to one. This may arise in situations when the subset of features represents ratio values, for example, the ratio of connections that have a particular result code.

Table 1.
FeatureFeasible pointInfeasible point
\( F_1 \)0.20.5
\( F_2 \)0.130.13
\( F_3 \)0.330.33
\( F_4 \)0.340.4
  • We show an example of a feasible point and an infeasible one in feature space \( \mathcal {F} \).

Table 1. Example Feature Values for Four Ratio Features, Whose Sum Needs to be 1 in the Feasibility Region

  • We show an example of a feasible point and an infeasible one in feature space \( \mathcal {F} \).

3.2 Challenges

Existing evasion attacks are mostly designed for continuous domains, such as image classification, where adversarial examples have pixel values in a fixed range (e.g., [0,1]) and can be modified independently [6, 11, 46]. However, many applications in cyber security use tabular data, resulting in feature dependencies and physical-world constraints that need to be respected.

Several previous works address evasion attacks in domains with tabular data. The evasion attack for malware detection by Grosse et al. [29], which directly leverages JSMA [53], modifies binary features corresponding to system calls. Kolosnjaji et al. [36] use the attack of Biggio et al. [8] to append selected bytes at the end of the malware file. Suciu et al. [66] also append bytes in selected regions of malicious files. Kulynych et al. [38] introduce a graphical framework in which an adversary constructs all feasible transformations of an input, and then uses graph search to determine the path of the minimum cost to generate an adversarial example.

Neither of these approaches is applicable to our general setting because the attacks do not satisfy the required dependencies in the resulting adversarial vector. Crafting adversarial examples that are feasible, and respecting all the application constraints and dependencies poses a significant challenge. Once application constraints are specified, the resulting optimization problem for creating adversarial examples includes a number of non-linear constraints and cannot be solved directly using out-of-the-box optimization methods.

In order to measure the feasibility of adversarial examples, we run the existing Carlini and Wagner (C&W) attack [11] on a malicious domain classification. The details of the attack adaptation for this application are given in Section 4.2. We considered the balanced case, in which the number of \( \mathsf {Malicious} \) and \( \mathsf {Benign} \) examples is equal in training. While the attack reaches 98% success at a distance of 20, the resulting adversarial examples are outside the feasibility region. An example is included in Table 2, and the description of the features is given in Table 6. In this case, the average number of connections is not equal to the total number of connections divided by the number of IPs contacting the domain. Additionally, the average ratio of received bytes over sent bytes should be equal to the maximum and minimum values (as the number of IPs contacting the domain is 1).

Table 2.
FeatureDescriptionInputAdversarial ExampleCorrect Value
Num_IPNumber of IPs111
Num_ConnNumber of connections15233.56233.56
Avg_ConnsAvg. connections by IP1559.94233.56
Avg_Ratio_BytesAvg. ratio bytes8.27204.01204.01
Max_Ratio_BytesMax. ratio bytes8.27240.02204.01
Min_Ratio_BytesMin. ratio of bytes8.27119.12204.01
  • The inconsistent feature values are in red, while the correct values are in green. We consider all the IPs that connect to the domain and all the connections made to the domain. The ratio of bytes features are computed as ratio of received bytes over sent bytes per IP. When there is a single IP connecting to the domain, the number of connections should be equal to the average number of connections by IP, and the three ratio bytes features should all be equal.

Table 2. Infeasible C& W Adversarial Example for the Malicious Domain Classifier

  • The inconsistent feature values are in red, while the correct values are in green. We consider all the IPs that connect to the domain and all the connections made to the domain. The ratio of bytes features are computed as ratio of received bytes over sent bytes per IP. When there is a single IP connecting to the domain, the number of connections should be equal to the average number of connections by IP, and the three ratio bytes features should all be equal.

3.3 The FENCE Framework

To address these issues, we introduce the FENCE framework for evasion attacks that preserves a range of feature dependencies in constrained domains. FENCE guarantees by design that the produced adversarial examples are within the feasible region of the application input space.

The starting point for the attack framework are gradient-based optimization algorithms, including projected [8, 46] and penalty-based [11] optimization methods. Of course, we cannot apply these attacks directly since they will not preserve the feature dependencies. To overcome this, we use the values of the objective gradient at each iteration to select features of maximum gradient values. We create feature-update algorithms for each family of dependencies that use a combination of gradient-based method and mathematical constraints to always maintain a feasible point that satisfies the constraints. We also use various projection operators to project the updated adversarial examples to feasible regions of the feature space.

Algorithms 1, 2, and Figure 2 describe the general FENCE framework. We consider binary classifiers designed using FFNN architectures. However, the framework can be extended to multi-class scenarios by modifying the optimization objective. For measuring the amount of perturbation added by the original example, we use the \( L_2 \) norm.

Fig. 2.

Fig. 2. Flow of the FENCE evasion attack framework.

The input to the FENCE framework consists of: an input sample \( \boldsymbol {x} \) with label \( y \) (typically \( \mathsf {Malicious} \) in security applications); a target label \( t \) (typically \( \mathsf {Benign} \)); the model prediction function \( C \); the optimization objective \( G \); maximum allowed perturbation \( d_{max} \); the subset of features \( F_S \) that can be modified; the features that have dependencies \( F_D \subset F_S \); the maximum number of iterations \( M; \) and a learning rate \( \alpha \) for gradient descent. The set of dependent features is split into families of features. A family is defined as a subset of \( F_D \) such that features within the family need to be updated simultaneously, whereas features outside the family can be updated independently. In our malicious network traffic classification application, a family of features is defined for each port by including all features extracted for that particular port. All these features are dependent and they are modified jointly during the adversarial optimization procedure.

The algorithm proceeds iteratively. The goal is to update the data point in the direction of the gradient (to minimize the optimization objective) while preserving the domain-specific and mathematical dependencies between features. In each iteration, the gradients of all modifiable features are computed, and the feature of the maximum gradient is selected. The update of the data point \( x \) in the direction of the gradient is performed as follows:

(1)

If the feature of maximum gradient belongs to a family with other dependent features, function \( \mathsf {UPDATE\_FAMILY} \) is called (Algorithm 1, line 10). Inside the function, the representative feature for the family is computed (this needs to be defined for each application). In the malicious network traffic classification example, the representative feature for a port’s family is the number of sent packets on that port. The representative feature is updated first, according to its gradient value, followed by updates to other dependent features using the function \( \mathsf {UPDATE\_DEP} \) (Algorithm 2, line 11). We need to define the function \( \mathsf {UPDATE\_DEP} \) for each application, but we use a set of building blocks that support common operations in feature space and are reusable across applications. We refer the reader to Section 3.4 for a set of dependencies supported by our framework. Once all features in the family have been updated, there is a possibility that the updated data point exceeds the allowed distance threshold from the original point. If that is the case, the algorithm backtracks and performs a binary search for the amount of perturbation added to the representative feature (until it finds a value for which the modified data point is inside the allowed region).

(2)

If the feature of maximum gradient does not belong to any feature family, then it can be updated independently from other features. The feature is updated using the standard gradient update rule (Algorithm 1, line 13). This is followed by a projection \( \Pi _2 \) within the feasible \( L_2 \) ball.

Finally, if the attacker is successful at identifying an adversarial example in feature space, it is projected back to the raw input space representation (using function PROJECT_TO_RAW). With the FENCE framework, we guarantee that modifications in feature space always result in feasible regions of the feature space, meaning that they can be projected back to the raw input space. Of course, if the raw data representation is used directly as features for the ML classifier, the projection is not necessary.

FENCE currently supports two optimization objectives:

  • Objective for Projected attack. We set the objective \( G(x)=Z_1(x) \), where \( Z_1 \) is the logit for the \( \mathsf {Malicious} \) class, and \( Z_0 = 1-Z_1 \) for the \( \mathsf {Benign} \) class:

    \( \delta = \operatorname{arg\,min}Z_1(x+\delta) \),

    s.t. \( ||\delta ||_2 \le d_{max} \),

    \( x+\delta \in \mathsf {Feasible\_Set}(R,\mathcal {O}) \)

  • Objective for Penalty attack. The penalty objective for binary classification is equivalent to:

    \( \delta = \operatorname{arg\,min}||\delta ||_2+ c \cdot \max (0, Z_1(x + \delta)) \),

    \( x+\delta \in \mathsf {Feasible\_Set}(R,\mathcal {O}) \)

Our general FENCE evasion framework can be used for different classifiers, with multiple features representations and constraints. The components that need to be defined for each application are: (1) the optimization objective \( G \) for computing adversarial examples; (2) the families of dependent features and family representatives; (3) the \( \mathsf {UPDATE\_DEP} \) function that performs feature updates per family; and (4) the projection operation PROJECT_TO_RAW that transforms adversarial examples from feature space to the raw data input.

3.4 Dependencies in Feature Space

In this section, we describe the dependencies in the feature space that FENCE supports. For each of these, there is a corresponding \( \mathsf {UPDATE\_DEP} \) algorithm used in the FENCE optimization framework. Once the representative feature in a family is updated according to the gradient value in Algorithm 1, the dependent features are updated with the \( \mathsf {UPDATE\_DEP} \) algorithm.

Domain-Specific Dependencies. The supported domain-specific dependencies are illustrated in Table 3. These dependencies might occur naturally in the raw data space. The Range dependency ensures that feature values are in a particular numerical range, while the Ratio dependency ensures that the ratio of two features is in a particular interval. The one-hot encoded feature dependency is a structural dependency of the input vector representation, encountered when categorical data is represented by creating a binary feature for each value. Algorithms 3 and 4 describe how to preserve the Range and Ratio dependencies, respectively.

Table 3.
Type of dependencyFormula
\( Range \)\( x_i: x_i \in [a, b] \)
\( Ratio \)\( x_i, x_j: x_i/x_j \in [a,b] \)
One-hot encoding (\( OHE \))\( \lbrace x_i\rbrace _1^N: x_i \in \lbrace 0, 1\rbrace , \sum _{i = 1}^N{x_i} = 1 \)

Table 3. Domain-specific Feature Dependencies

Algorithm 3 illustrates the procedure for updating dependent features to satisfy the Ratio relationship. If the dependency between two features \( x \) and \( y \) is such that \( x/y \in [a, b] \), then feature \( x \) is modified according to the gradient value, but the final range is restricted to the interval \( [a \cdot y,b \cdot y] \).

Algorithm 4 gives the update function for Range. It ensures that input \( x \) is projected to interval \( [a,b] \). It returns the projected value of \( x \), as well as the absolute value of the difference between \( x \) and its projection.

Mathematical Feature Dependencies. Mathematical dependencies resulting from feature engineering supported by FENCE are illustrated in Table 4. These include statistical dependencies, linear and non-linear dependencies between multiple features, as well as combinations of these. To provide some insight, Algorithms 5 and 6 illustrate how to preserve \( NonLin \) and \( Stat \) dependencies.

Table 4.
Type of dependencyFormula
Statistical (\( Stat \))\( x_{min} \le x_{avg}\le x_{max} \)
Linear (\( Lin \))\( \sum _i^M (w_i * x_i) = Ct \)
Non-linear (\( NonLin \))\( x_i - x_j/x_k = 0 \)
Combinations of \( Lin \),\( x_{min} \le (x_j/x_k)_{avg}\le x_{max} \)
\( Stat \), and \( NonLin \)\( \sum _i^M (w_i * x_i/x_k) = Ct \)

Table 4. Mathematical Feature Dependencies

Algorithm 5 shows the NonLin update feature dependency procedure. Here, we need to ensure that the constraint \( x_i - x_j / x_k = 0 \) is satisfied for three features \( x_i,x_j \), and \( x_k \). Gradient update is performed first for \( x_j \), after which the value of \( x_i \) is modified to ensure the equality constraint, while feature \( x_k \) is kept constant.

Algorithm 6 gives the update method for satisfying the Stat dependency. This is done for a family of features that includes the minimum \( x_{min} \), the average \( x_{avg} \), the maximum \( x_{max} \), and the total number \( x_{tot} \) from some events from the raw data. After the update of feature \( x_{tot} \) (by increasing, for example, the total number of network connections), we need to adjust the average value \( x_{avg} \) and the corresponding minimum and maximum values. The input to \( \mathsf {Update\_Stat} \) also includes a value \( v \) that is the new value added to the raw data, which could impact the minimum or maximum values.

Skip 4Concrete Applications of FENCE Section

4 Concrete Applications of FENCE

In this section we describe the application of FENCE to two classification problems for threat detection: malicious network traffic classification, and malicious domain classification. We highlight that FENCE can be applied to other domains with feature constraints such as healthcare and finance, but our focus in the paper is on cyber security applications.

4.1 Malicious Network Traffic Classification

Network traffic includes important information about communication patterns between source and destination IP addresses. Classification methods have been applied to labeled network connections to determine malicious infections, such as those generated by botnets [7, 10, 34, 50]. Network data comes in a variety of formats, but the most common include net flows, Zeek logs, and packet captures.

Dataset. We leverage a public dataset of botnet traffic that was captured at the CTU University in the Czech Republic, called the CTU-13 dataset [25]. We consider the three scenarios for detecting the Neris botnet in this dataset. The dataset includes Zeek connection logs with communications between internal IP addresses (on the campus network) and external ones. The dataset has the advantage of providing ground truth, i.e., labels of \( \mathsf {Malicious} \) and \( \mathsf {Benign} \) IP addresses. The goal of the classifier is to distinguish \( \mathsf {Malicious} \) and \( \mathsf {Benign} \) IP addresses on the internal network.

The fields available in Zeek connection logs are given in Figure 3. They include: the timestamp of the connection start; the source IP address; the source port; the destination IP address; the destination port; the number of packets sent and received; the number of bytes sent and received; and the connection duration (the time difference between when the last packet and first packets are sent).

Fig. 3.

Fig. 3. Zeek logs (top), raw data representation (left), and feature families per port (right) for network traffic classifier.

In this application, we can use either the raw connection representation or leverage domain knowledge to create aggregated features. We describe existing feature relationships and apply our FENCE framework against both representations.

Raw Data Representation. This consists of the following fields: one-hot encoded port number, one-hot encoded connection type, duration, original bytes, received bytes, original packets, and received packets. The feature vector is illustrated in Figure 3 on the left. The raw data representation includes no mathematical dependencies, but has the following domain-specific constraints:

The TCP and UDP packet sizes are capped at 1,500 bytes. We create range intervals for these values, resulting in a \( Ratio \) dependency between the number of packets and their sizes.

The connection duration is the interval between the last and the first packet. If the connection is idle for some time interval (e.g., 30 seconds), it is closed by default by Zeek. The attacker can thus control the duration of the connection by sending packets at certain time intervals to avoid closing the connection. We generate a range of valid durations from the distribution of connection duration in the training dataset. This creates again a \( Ratio \) dependency between the number of packets and their duration.

Each continuous feature has its related minimum and maximum values, which are obtained from the training data distribution, thus forming \( Range \) relationships.

The port number and connection type have one-hot encoded \( OHE \) dependencies.

Attack algorithm on raw data representation. The attacker’s goal is to have a connection log classified as \( \mathsf {Benign} \) instead of \( \mathsf {Malicious} \). We assume that the attacker communicates with an external IP under its control (for instance, the command-and-control IP), and thus has full control of the malicious traffic in that connection. We assume that the attacker can only add traffic to network connections, by increasing the number of bytes, packets, and connection duration, to preserve the malicious functionality. For simplicity, we set the number of received packets and bytes to 0, assuming that the external IP does not respond to these connections. We assume that the attacker does not have access to the security monitor that collects the logs and cannot modify directly the log data.

The attack algorithm follows the framework from Algorithm 1. There is only one family of dependent features, including the packets and bytes sent, and connection duration. The representative feature is the number of sent packets, which is updated with the gradient value, following a binary search for perturbation \( \delta \), as specified in Algorithm \( \mathsf {UPDATE\_FAMILY} \). The dependent number of bytes sent and duration features are updated using the update dependency functions (\( \mathsf {Update\_Ratio} \), Update_Range and Update_OHE), thus preserving the \( Ratio \), \( Range \) and \( OHE \) dependencies.

Engineered Features. Another possibility is to use domain knowledge to create features that improve classification accuracy. A standard method for creating network traffic features is aggregation by destination port to capture relevant traffic statistics per port (e.g., [25, 49]). This is motivated by the fact that different network services and protocols run on different ports, and we expect ports to have different traffic patterns. We select a list of 17 ports for popular applications, including HTTP (80), SSH (22), and DNS (53). We also add a category called OTHER for connections on other ports. We aggregate the communication on a port based on a fixed time window (the length of which is a hyper-parameter set at one minute). For each port, we compute traffic statistics using the \( \mathsf {Max} \), \( \mathsf {Min} \), and \( \mathsf {Total} \) operators for outgoing and incoming connections. See the example in Figure 3 on the right, in which features extracted for each port define a family of dependent features. We obtain a total of 756 aggregated traffic features on these 17 ports. Table 5 includes the feature description. The resulting feature vector includes both types of dependencies. The domain-specific relationships are the same as for the raw data representation except for the one-hot encoding relationship. There are additional \( Stat \) mathematical dependencies between features: the minimum and the maximum number of packets, bytes, and duration per connection must be updated after a change in the total number of packets, bytes, or connections.

Table 5.
CategoryFeatureDescription
BytesTotal_Sent_BytesTotal number of bytes sent
Min_Sent_BytesMinimum number of bytes sent per connection
Max_Sent_bytesMaximum of bytes sent per connection
PacketsTotal_Sent_PktsTotal number of packets sent
Min_Sent_PktsMinimum number of packets sent per connection
Max_Sent_PktsMaximum of packets sent per connection
DurationTotal_DurationTotal duration of all connections
Min_DurationMinimum duration of a connection
Max_DurationMaximum duration of a connection
Connection typeTotal_TCPTotal number of TCP connections
Total_UDPTotal number of UDP connections
  • These features are defined for each port by aggregating over all connections on that port in a fixed time window.

Table 5. Features Definition for Malicious Connection Classification

  • These features are defined for each port by aggregating over all connections on that port in a fixed time window.

Attack algorithm on engineered features. The goal of the attacker here is to change the prediction of a feature vector aggregated over time from \( \mathsf {Benign} \) to \( \mathsf {Malicious} \). Therefore, in this attack model, the attacker has the ability to insert network connections during the targeted time window to achieve his goal. Similar to the above scenario, the attacker controls a victim IP and can send traffic to external IPs under its control. The adversary has a lot of options in mounting the attack by selecting the protocol, port, and connection features. Here we have 17 families of dependent features, one for the features on each port.

The attack algorithm against the Neris botnet classification task called the Neris attack follows the framework from Algorithm 1. First, the feature of the maximum gradient is determined and the corresponding port is identified. The family of dependent features is all the features computed for that port. The attacker attempts to add a fixed number of connections on that port (which is a hyper-parameter of our system). This is done in the \( \mathsf {INIT\_FAMILY} \) function. The attacker can add either TCP, UDP, or both types of connections, according to the gradient sign for these features and also respecting network-level constraints. The representative feature for a port’s family is the number of packets that the attacker sends in a connection. This feature is updated by the gradient value, following a binary search for perturbation \( \delta \), as specified in Algorithm \( \mathsf {UPDATE\_FAMILY} \).

In the \( \mathsf {UPDATE\_DEP} \) function an update to the aggregated port features is performed. First, we ensure that the feature corresponding to the number of packets sent satisfies the \( Range \) and \( Stat \) constraints. Then the difference in the total number of bytes sent by the attacker and duration is determined from the gradient, followed by the \( \mathsf {Update\_Ratio} \) function to keep the resulting values inside the feasible domain. The port family also includes features such as \( \mathsf {Min} \) and \( \mathsf {Max} \) sent bytes and connection duration. These features are updated by the \( \mathsf {Update\_Stat} \) function. The detailed algorithm for \( \mathsf {INIT\_FAMILY} \) and \( \mathsf {UPDATE\_DEP} \) functions of the Neris attack are illustrated in Algorithms 7 and 8.

4.2 Malicious Domain Classifier

The second threat detection application is to classify FQDN domain names contacted by enterprise hosts as \( \mathsf {Malicious} \) or \( \mathsf {Benign} \). This security application has multiple types of feature-space constraints, including linear, non-linear, and statistical dependencies, and therefore can be used to test our FENCE framework for supporting multiple constraints.

Dataset. We obtained access to a proprietary dataset collected by a company that includes 89 domain features extracted from HTTP proxy logs collected at the border of an enterprise network. This is the same dataset used for the design of MADE system for detecting malicious activity in enterprise networks and prioritizing the detected activities according to their risk described in detail by Oprea et al. [50]. This dataset enables us to experiment with the variety of constraints in feature space, representative of security applications. Features are aggregated over multiple HTTP connections to the same external FQDN domain and are defined with the help of security experts. Each external FQDN is labeled as \( \mathsf {Malicious} \) or \( \mathsf {Benign} \). We group the modifiable features of MADE into a set of seven families, included in Table 6. More details on all the features used in MADE are provided in the original paper [50] (we preserved the feature ID for the MADE dataset in Table 6).

Table 6.
FamilyFeature IDFeatureDescription
Connections1Num_ConnNumber of established connections
2Avg_ConnAverage number of connections per host
Bytes3Total_Recv_BytesTotal number of received bytes
4Total_Sent_BytesTotal number of sent bytes
5Avg_Ratio_BytesAverage ratio of received bytes
over sent bytes per IP
6Min_Ratio_BytesMaximum ratio of received bytes
over sent bytes per IP
7Max_Ratio_BytesMinimum ratio of received bytes over sent bytes per IP
HTTP8Num_POSTTotal number of POST requests
Method9Num_GETTotal number of GET requests
10Avg_POSTAverage number of POST requests
over GET requests per IP
11Min_POSTMinimum number of POST requests
over GET requests per IP
12Max_POSTMaximum number of POST requests over GET requests per IP
Content46Frac_emptyFraction of connections with empty content type
47Frac_jsFraction of connections with js content type
48Frac_htmlFraction of connections with html content type
49Frac_imgFraction of connections with image content type
50Frac_videoFraction of connections with video content type
51Frac_textFraction of connections with text content type
52Frac_appFraction of connections with app content type
Result59Num_200Number of connections with result code 200
Code60Num_300Number of connections with result code 300
61Num_400Number of connections with result code 400
62Num_500Number of connections with result code 500
63Frac_200Fraction of connections with result code 200
64Frac_300Fraction of connections with result code 300
65Frac_400Fraction of connections with result code 400
66Frac_500Fraction of connections with result code 500
Independent43Avg_OSAverage number operating systems
extracted from user-agent
44Avg_BrowserAverage number of browsers used
68Dom_LevelsNumber of levels
69Sub_DomainsNumber of sub-domains
70Dom_LengthLength of domain
71Reg_AgeWHOIS registration age
72Reg_ValidityWHOIS registration validity
73Update_AgeWHOIS update age
74Update_ValidityWHOIS update validity
75Num_ASNsNumber of ASNs
76Num_CountriesNumber of countries contacted the domain

Table 6. Feature Set for Malicious Domain Classification that can be Modified by the Evasion Attack

In this application, we do not have access to the raw HTTP traffic, only to features extracted from it and domain labels. Thus, the constraints are mathematical constraints in feature space, for instance:

  • For the Connection family, we have \( NonLin \) dependence: computing average value over a number of events.

  • For the Bytes family, we need to update the ratio of two values and then update minimum, maximum and average values, thus, we have the combination of \( Stat \) and \( NonLin \) dependencies.

  • For the HTTP Method, we have the same combination of \( Stat \) and \( NonLin \) dependencies as for Bytes family.

  • For the Content family, we need to ensure that the sum of all ratio values equals 1. This is a combination of \( Lin \) and \( NonLin \) dependencies.

  • For the Result Code, we also need to ensure that the sum of all fraction values equals to 1. Additionally, the number of connections with different codes must sum up to the total number of connections. This is a combination of \( Lin \) and \( NonLin \) dependencies.

Attack algorithm. We assume that we add events to the logs, and never delete or modify existing events. For instance, we can insert more connections, as in the malicious connection classifier. The attack algorithm against the malicious domain classifier called the MADE attack follows the framework from Algorithm 1. If the feature of the maximum gradient has no dependencies, it is just updated with the gradient value. Otherwise, every dependency family has a specific representative feature and is updated following one of the specified \( \mathsf {UPDATE\_DEP} \) functions. For example, for the Connection family, the representative feature is Num_Conn, which is updated with the gradient value, and other features in this family are updated by calling the \( \mathsf {Update\_Stat} \) function. The detailed algorithm for the \( \mathsf {UPDATE\_DEP} \) function of the MADE attack is illustrated in Algorithm 9. The functions for updating dependencies (e.g., \( \mathsf {Update\_Stat} \), \( \mathsf {Update\_Lin} \), \( \mathsf {Update\_NonLin} \)) are the same defined in the FENCE framework and discussed in Section 3.4.

Skip 5EXPERIMENTAL EVALUATION FOR NETWORK TRAFFIC CLASSIFIER Section

5 EXPERIMENTAL EVALUATION FOR NETWORK TRAFFIC CLASSIFIER

We evaluate FENCE for the malicious network traffic classifier trained with both the raw data and engineered feature representations. We show feasible attacks that insert a small number of network connections to change the \( \mathsf {Malicious} \) prediction to \( \mathsf {Benign} \). We only analyze the FENCE attack with the Projected optimization objective here. In the following section, we analyze our FENCE framework for the malicious domain classifier for both the Projected and Penalty attacks.

5.1 Experimental Setup

CTU-13 is a collection of 13 scenarios including both legitimate traffic from a university campus network, as well as labeled connections of malicious botnets [25]. We restrict to three scenarios for the Neris botnet (1, 2, and 9). We choose to train on two of the scenarios and test the models on the third, to guarantee independence between training and testing data.

The raw data representation has 3,712,935 data points, from which 151,625 are labeled as botnets. The attacker can modify three features per connection: bytes and packets sent, and duration. The training data in the engineered features representation has 3,869 \( \mathsf {Malicious} \) examples, and 194,259 \( \mathsf {Benign} \) examples, and an imbalance ratio of 1:50. There is a set of 432 statistical features that the attacker can modify (the ones that correspond to the characteristics of sent traffic on 17 ports). The physical constraints and statistical dependencies in both scenarios have been detailed in Section 4.1.

We considered two baseline attacks: Baseline 1 (in which the features that are modified iteratively are selected at random), and Baseline 2 (in which, additionally, the amount of perturbation is sampled from a standard normal distribution \( N(0,1) \)).

5.2 Attack Results for Raw Data Representation

For training we have used FFNN with two layers and a sigmoid activation function. The architecture that corresponds to the best performance has 12 neurons in the first layer, and one neuron in the second layer. We have trained it using Adam optimizer with a learning rate equal to 0.0001 for 20 epochs with batch size 64. The best results are for training on scenarios 2 and 9, and testing on scenario 1, with an F1 score of 0.70.

We consider an attack on testing scenario 1, and the success rate of our attack is 100% already at a small \( L_2 \) distance of 2. Intuitively, an attacker can add a few packets and bytes to a connection and change its classification easily. We compare its performance to Baseline 2, which achieves only 73% success rate at \( L_2 \) distance of 2.

5.3 Attack Results for Engineered Features

We perform model selection and training for a number of FFNN architectures on all combinations of two scenarios, and test the models for generality on the third scenario. The best architecture consists of three layers with 256, 128 and 64 hidden layers. We used the Adam optimizer, 50 epochs for training, a mini-batch of 64, and a learning rate of 0.00026. The F1 and AUC scores are much better than the FFNN based on raw data representation. For instance, the best scenario is training on 1, 9, and testing on 2, which achieve an F1 score of 0.97, compared to 0.70 for raw data.

We thus perform a more extensive analysis of the attack against engineered features in this scenario. The testing data for the attack is 407 \( \mathsf {Malicious} \) examples from scenario 2, among which 397 were predicted correctly by the classifier.

Evasion attack performance. First, we analyze the attack success rate with respect to the allowed perturbation, shown in Figure 4(a). The attack reaches 99% success rate at \( L_2 \) distance 16. Interestingly, in this case the two baselines perform poorly, demonstrating again the clear advantages of our framework. We plot next the ROC curves under evasion attack in Figure 4(b) (using the 407 \( \mathsf {Malicious} \) examples and 407 \( \mathsf {Benign} \) examples from testing scenario 2). At distance 8, the AUC score is 0.93 (compared to 0.98 without adversarial examples), but there is a sudden change at distance 10, with the AUC score dropping to 0.77. Moreover, at distance 12, the AUC reaches 0.12, showing the model’s degradation under evasion attack with relatively small distance.

Fig. 4.

Fig. 4. Projected attack results on network traffic classifier.

The average number of port families updated during the attack is shown in Figure 4(c). The maximum number is three ports, but it decreases to one port at distance higher than 12. While counter-intuitive, at larger distances the attacker can add larger perturbation to the aggregated statistics of one port, crossing the decision boundary. The ports most frequently modified are 443 and 80, which are the ports with most network traffic.

Adversarial examples. We show an adversarial example generated by the Projected attack at distance 14. The attacker adds only 12 TCP connections on port 443, including 87 packets, each of size 1,292 bytes, with connection duration of 432.47 seconds. Table 7 shows one of the 12 attacker-generated connections to create the adversarial example. The destination IP can be selected by the attacker so that it is under its control and does not send any bytes or packets. These new connections are added to the activity the attacker already does inside the network, so the malicious functionality of the attack is preserved. Interestingly, all adversarial attacks succeed with at most 12 new connections at distances higher than 10. In Table 8 we show statistics for the duration, sent bytes and sent packets features in the training data. We make the observation that the feature values in the resulting adversarial example are below the average feature values of the training data. In particular, the Orig_Bytes feature has an average value 150KB and a very high standard deviation (12.9MB), while the adversarial example only uses 112,404 bytes, which is very little communication (112KB).

Table 7.
tsportprotdurationo_btsr_btso_pktsr_pktsstate
153UDP2.266386755822SF
213363TCP444.3347076711411SF
31035TCP276.0842182076801100OTH
4443TCP432.471124040870OTH

Table 7. Example of Zeek Logs Records (top 3 rows), and One of the 12 Attacker-generated Connection Logs Added for the Adversarial Example Perturbation (4th row)

Table 8.
FeatureAverageStd. Dev.25%50%75%95%Maximum
Duration552.207872.840.135.7179.381578.821060383.73
Orig_Bytes150756.5912976082.7696415246651117.32091734081
Orig_Pkts342.755221.952934522405463

Table 8. Feature Statistics in Training Data

Furthermore, we illustrate the \( L_2 \) norm statistics of samples in the training data along with \( L_2 \) norms of perturbations added to create adversarial examples at distance 8 in Table 9. We observe that the average \( L_2 \) perturbation norm (7.66) is 4.97 times lower than the training data average \( L_2 \) norm (38.12). Given that the standard deviation (0.57) and maximum value (7.99) of the perturbation \( L_2 \) norm are small compared to the standard deviation (98.85) and the maximum value (890.56) of the \( L_2 \) norm of training samples, we consider the resulting adversarial attacks stealthy.

Table 9.
SettingAverageStd. Dev.25%50%75%95%Maximum
Training data38.1298.852.424.5224.53195.58890.56
Projected perturbation7.660.577.667.857.937.977.99

Table 9. \( L_2 \) Norm Statistics for Training Data (top row) and Perturbation Added by the Projected Attack (bottom row)

Skip 6EXPERIMENTAL EVALUATION FOR MALICIOUS DOMAIN CLASSIFIER Section

6 EXPERIMENTAL EVALUATION FOR MALICIOUS DOMAIN CLASSIFIER

In this section we perform a detailed evaluation of the FENCE attack on the MADE malicious domain classifier trained on the enterprise dataset [50]. We compare the Projected and Penalty optimization methods and analyze the impact of imbalanced training datasets. We also test the transferability of the FENCE attacks across other models and architectures and evaluate the potential of adversarial training as mitigation against FENCE evasion attacks.

6.1 Experimental Setup

The data for training and testing the models was extracted from security logs collected by web proxies at the border of a large enterprise network with over 100,000 hosts. The number of monitored external domains in the training set is 227,033, among which 1,730 are classified as \( \mathsf {Malicious} \) and 225,303 are \( \mathsf {Benign} \). For training, we sampled a subset of training data to include 1,230 \( \mathsf {Malicious} \) domains, and a different number of \( \mathsf {Benign} \) domains to get several imbalance ratios between the two classes (1, 5, 15, 25, and 50). We used the remaining 500 \( \mathsf {Malicious} \) domains and sampled 500 \( \mathsf {Benign} \) domains for testing the evasion attack. Overall, the dataset includes 89 features from seven categories.

Among the features included in the dataset, we determined a set of 31 features that can be modified by an attacker (see Table 6 for their description). These include communication-related features (e.g., number of connections, number of bytes sent and received, etc.), as well as some independent features (e.g., number of levels in the domain or domain registration age). Other features in the dataset (for example, those using URL parameters or values) are more difficult to change, and we consider them immutable during the evasion attack.

This dataset is extremely imbalanced, and we sample a different number of \( \mathsf {Benign} \) domains from the data, to control the imbalance ratio. We are interested in how the imbalance affects the attack success rate. On this dataset, we also compare the Projected and Penalty attack objectives.

6.2 FENCE Attack Evaluation

We experimented with several models for training classifiers, including logistic regression, random forest, and different FFNN architectures. The best performance was achieved by a two-layer FFNN with 80 neurons in the first layer, and 50 neurons in the second layer. ReLU activation function is used after all hidden layers except for the last layer, which uses sigmoid. We used the Adam optimizer and SGD with different learning rates. The best results were obtained with Adam and a learning rate of 0.0003. We trained for 75 epochs with a mini-batch size of 32. The resulting model had an AUC score of 89% with cross-validation, in the balanced case. These results were comparable to the best random forest model we trained and better than logistic regression.

The ROC curves for training logistic regression, random forest and FFNN are given in Figure 5 (a), while the results for FFNN with different imbalanced ratios are in Figure 5 (b). Interestingly, the performance of the model increases to 93% AUC for an imbalance ratio up to 25, after which it starts to decrease (with AUC of 83% at a ratio of 50). Our intuition is that the FFNN model achieves better performance when more training data is available (up to a ratio of 25). But once the \( \mathsf {Benign} \) class dominates the \( \mathsf {Malicious} \) one (at ratio of 50), the model performance starts to degrade. FENCE Projected attack results. We evaluate the success rate of the attack with Projected objective first for balanced classes (1:1 ratio). We compare in Figure 6(a) the attack against the two baselines. The attacks are run on 412 \( \mathsf {Malicious} \) testing examples classified correctly by the FFNN. The Projected attack improves both baselines, with Baseline 2 performing much worse, reaching success rate of 57% at a distance of 20, and Baseline 1 has a success of 91.7% compared to our attack (98.3% success). This shows that the attacks are still performing reasonably if feature selection is done randomly, but it is very important to add perturbation to features consistent with the optimization objective.

Fig. 5.

Fig. 5. Training results for malicious domain classifier.

Fig. 6.

Fig. 6. FENCE Projected attack results for the malicious domain classifier.

We also measure in Figure 6(b) the decrease in the model’s performance before and after the evasion attack at different perturbations (using 500 \( \mathsf {Malicious} \) and 500 \( \mathsf {Benign} \) examples not used in training). While the AUC score is 0.87 originally, it drastically decreases to 0.52 under evasion attack at perturbation 7. This shows the significant degradation of the model’s performance under evasion attack.

Finally, we run the attack at different imbalance ratios and measure its success for different perturbations. In this experiment, we select 62 test examples that all models (trained for different imbalance ratios) classified correctly before the evasion attack. The results are illustrated in Figure 6(c). At \( L_2 \) distance 20, the evasion attack achieves a 100% success rate for all ratios except 1. Additionally, we observe that with a higher imbalance, it is easier for the attacker to find adversarial examples (at a fixed distance). One reason is that models that have lower performance (such as the model trained with 1:50) are easier to attack. Second, we believe that as the imbalance gets higher the model becomes more biased towards the majority class (\( \mathsf {Benign} \)), which is the target class of the attacker, making it easier to cross the decision boundary between classes.

We include an adversarial example in Table 10. We only show the features that are modified by the attack and their original value. As we observe, the attack preserves the feature dependencies: the average ratio of received bytes over sent bytes (Avg_Ratio_Bytes) is consistent with the number of received (Total_Recv_Bytes) and sent (Total_Sent_Bytes) bytes. In addition, the attack modifies the domain registration age, an independent feature, relevant to malicious domain classification [45]. However, there is a higher cost to change this feature: the attacker should register a malicious domain and wait to get a larger registration age. If this cost is prohibitive, we can easily modify our framework to make this feature immutable.

Table 10.
FeatureOriginalAdversarial
NIP11
Total_Recv_Bytes32.3243653.50
Total_Sent_Bytes2.02702.62
Avg_Ratio_Bytes16.1516.15
Registration_Age3493616

Table 10. Adversarial Example for the FENCE Projected Attack at Distance 10

We constructed 45 adversarial examples at \( L_2 \) distance 20 and calculated the average perturbation for every feature that was modified to show that the adversarial examples generated at this distance are unnoticeable. The results can be found in Table 11. Additionally, we include statistics for the features on the training dataset in Table 12. We observe that the generated adversarial examples have stealthy perturbations, given the feature distribution and meaning. For example, the number of sent bytes is increased by 17195.3 (17.19KB), the registration age of a domain is increased on average by 1.14 days, while the number of sub domains is increased by 0.28 on average. Moreover, all average perturbations are much smaller than the standard deviation of features in the training data. For instance, the average perturbation of the sub domains number is 0.28 compared to the corresponding standard deviation of 2867.53 in the training data. This additionally confirms the fact that the generated perturbations for adversarial examples can be considered unnoticeable.

Table 11.
FeatureAverage perturbation
Total_Recv_Bytes96340
Total_Sent_Bytes17195.3
Min_Ratio_Bytes23.26
Sub_Domains0.28
Reg_Age1.14
Reg_Validity158.91
Update_Age24.98
Update_Validity59.53

Table 11. Average Modified Features Perturbation for the FENCE Projected Attack at Distance 20

Table 12.
FeatureAverageStd. Dev.25%50%75%95%Maximum
Total_Recv_Bytes3332.484148.4996.09433.171420.077092.4024854191.98
Total_Sent_Bytes213.7652373.226.1614.5431.48120.4625133008.74
Min_Ratio_Bytes94.563298.472.4317.5848.42181.731190540.7
Num_GET77.221704.09123062198746776
Sub_Domains485.662867.53112124754632
Reg_Age2377.441880.0382024043179613416649
Reg_Validity2914.572158.44109629274017694037114
Update_Age348.68486.2996295385112842215
Update_Validity839.28888.90365605908292843587

Table 12. Feature Statistics in Training Data

Finally, we show the \( L_2 \) norm statistics of samples in the training data along with the \( L_2 \) norm of perturbations added to create adversarial examples by the Projected attack at distance 5 in the first two rows of Table 13. Even if we used a distance of 5 in the Projected attack, many of the generated perturbations have lower norm (in particular, half of the adversarial examples at distance 5 have norm lower than 3.17). The average \( L_2 \) norm of perturbation (3.20) is much smaller than the standard deviation of the training samples \( L_2 \) norm (26.9), resulting in relatively stealthy perturbations. Additionally, the maximum \( L_2 \) norm of perturbation (5) is only a 0.005 fraction of the maximum possible \( L_2 \) norm of the training samples (890).

Table 13.
SettingAverageStd. Dev.25%50%75%95%Maximum
Training data5.7926.93.894.525.6310.4890
Projected perturbation3.201.202.173.174.214.995
Penalty perturbation2.761.031.922.563.704.444.92

Table 13. \( L_2 \) Norm Statistics for Feature Vectors in Training Data (first row), Perturbation Added by the Projected Attack (second row, 89% success rate), and Perturbation for the Penalty Attack (third row, 70% success rate) at Distance 5

FENCE Penalty attack results. We now discuss the results achieved by applying our attack with the Penalty objective on the testing examples. Similar to the Projected attack, we compare the success rate of the Penalty attack to the two types of baseline attacks for balanced classes, in Figure 7(a) (using the 412 \( \mathsf {Malicious} \) testing examples classified correctly). Overall, the Penalty objective performs worse than the Projected one, reaching 79% success rate at \( L_2 \) distance of 20. We observe that in this case both baselines perform worse, and the attack improves upon both baselines significantly. The decrease of the model’s performance under the Penalty attack is illustrated in Figure 7(b) (for 500 \( \mathsf {Malicious} \) and 500 \( \mathsf {Benign} \) testing examples). While AUC is 0.87 originally on the testing dataset, it decreases to 0.59 under the evasion attacks at the maximum allowed perturbation of 7. Furthermore, we measure the attack success rate at different imbalance ratios in Figure 7(c) (using the 62 testing examples classified correctly by all models). For each ratio value we searched for the best hyper-parameter \( c \) between 0 and 1 with step 0.05. Here, as with the Projected attack, we see the same trend: as the imbalance ratio gets higher, the attack performs better, and it works best at imbalance ratio of 50.

Fig. 7.

Fig. 7. FENCE Penalty attack results for malicious domain classifier.

We include an adversarial example generated by the Penalty attack in Table 14. We only show the features that were updated by the attack, which modifies only the amount of bytes sent (by 4.7KB) and the bytes received (by 73KB). The attack is able to preserve the \( Ratio \) dependency: Avg_Ratio_Bytes = Total_Recv_Bytes/Total_Sent_Bytes/NIP.

Table 14.
FeatureOriginalAdversarial
NIP11
Total_Recv_Bytes146.7773016.06
Total_Sent_Bytes9.554752.92
Avg_Ratio_Bytes15.3615.36

Table 14. Adversarial Example for the FENCE Penalty Attack at Distance 10

We constructed 45 adversarial examples at \( L_2 \) distance 20 for the Penalty attack and calculated the average perturbation for every modified feature. The results are illustrated in Table 15. Given the meaning of the features, we can conclude that an \( L_2 \) distance of 20 can be considered reasonable to generate undetectable attacks. For instance: the Num_GET feature is increased only by 2.19 on average, meaning that the attacker needs to add only two or three additional GET requests to the domain; the number of sub domains is increased by 0.21; the registration age of the domain is increased by 1.23 days, and the update age is increased by 17.84 days on average. Lastly, we include \( L_2 \) norm statistics of perturbations added to create adversarial examples by the Penalty attack at distance 5 in the third row of Table 13. We notice that the average \( L_2 \) norm of perturbation (2.76) is much smaller than the standard deviation of the training samples \( L_2 \) norm (26.9). The maximum \( L_2 \) norm of perturbation is only a 0.005 fraction of the maximum possible \( L_2 \) norm of samples in the training data, confirming the fact that the resulting adversarial examples can be considered stealthy. Perturbations with the Penalty attacks are slightly lower than those generated by the Projected attack.

Table 15.
FeatureAverage perturbation
Total_Recv_Bytes76616.96
Total_Sent_Bytes8437.57
Min_Ratio_Bytes21.04
Num_GET2.19
Sub_Domains0.21
Reg_Age1.23
Reg_Validity113.51
Update_Age17.84
Update_Validity48.93

Table 15. Average Modified Features Perturbation for the FENCE Penalty Attack at Distance 20

Attack comparison. We compare the success rate of our Projected and Penalty FENCE attacks with the C&W attack, as well as an attack we call Post-processing. The Post-processing attack runs directly the original C&W developed for continuous domains, after which it projects the adversarial example to the raw input space to enforce the constraints. For each family of dependent features, the attack retains the value of the representative feature but then modifies the dependent features using the \( \mathsf {UPDATE\_DEP} \) function. The success rate of all these attacks is shown in Figure 8, using the 412 \( \mathsf {Malicious} \) testing examples classified correctly. The attacks based on our FENCE framework (with Projected and Penalty objectives) perform best, as they account for feature dependencies during the adversarial example generation. The attack with the Projected objective has the highest performance. The vanilla C&W has slightly worse performance at small perturbation values, even though it does not take into consideration the feature constraints and works in an enlarged feature space. Interestingly, the Post-processing attack performs worse (reaching only 0.005% success at a distance of 20 – can generate two out of 412 adversarial examples). This demonstrates that it is not sufficient to run state-of-the-art attacks for continuous domains and then adjust the feature dependencies, but more sophisticated attack strategies are needed. Number and importance of features modified. We compare how many features were modified in order to generate each of the three attacks: Projected, Penalty, and C&W Figure 9(a).

Fig. 8.

Fig. 8. Malicious domain classifier attacks. FENCE Projected attacks perform best. C&W does not generate feasible adversarial examples.

Fig. 9.

Fig. 9. Feature modification statistics for malicious domain classifier.

It is not surprising that the C&W attack modifies almost all features, as it works in \( L_2 \) norms without any restriction in feature space. Both the Projected and the Penalty attacks modify a much smaller number of features (4 on average).

We are interested in determining if there is a relationship between feature importance and choice of feature by the optimization algorithm. The number of feature updates along with features importance is shown in Figure 9(b). For additional details on feature description, we include the list of features that can be modified in Table 6.

We observe that features of higher importance are chosen more frequently by the optimization attack. However, since we are modifying the representative feature in each family, the number of modifications on the representative feature is usually higher (it accumulates all the importance of the features in that family). For the Bytes family, feature 3 (number of received bytes) is the representative feature and it is updated more than 350 times. However, for features that have no dependencies (e.g., 68 – number of levels in the domain, 69 – number of sub-domains, 71 – domain registration age, and 72 – domain registration validity), the number of updates corresponds to the feature importance.

6.3 Misclassification of Benign Samples as Malicious

We ran the Projected and Penalty attacks for changing the classification of \( \mathsf {Benign} \) inputs to \( \mathsf {Malicious} \) to show the generality of the FENCE framework. The success rate of the attacks compared to two baseline attacks for balanced classes is illustrated in Figure 10. Both attacks achieve a high success rate: At \( L_2 = 20 \) the Projected attack reaches 92% success rate and the Penalty attack reaches 95% success rate. The FENCE attack performs significantly better compared to both baseline attacks.

Fig. 10.

Fig. 10. FENCE attack results for misclassification of \( \mathsf {Benign} \) samples to \( \mathsf {Malicious} \) .

6.4 Attack Transferability

We consider here a threat model in which the adversary only knows the feature representation, but not the exact ML model or the training data. One approach to generate adversarial examples is through transferability [20, 44, 51, 67, 71]. We perform several experiments to test the transferability of the Projected attacks against FFNN to logistic regression (LR) and random forest (RF). Models were trained with different data and we vary the imbalance ratio. The results are in Table 16. We observe that the largest transferability rate to both LR and RF is for the highest imbalanced ratio of 50 (98.2% adversarial examples transfer to LR and 94.8% to RF). As we increase the imbalance ratio, the transfer rate increases and the transferability rate to LR is lower than to RF.

Table 16.
RatioFFNNLRRF
1100%40%51.7%
593.3%66.5%82.9%
1599%60.9%90.2%
25100%47.6%68.8%
50100%98.2%94.8%
  • We vary the imbalance ratio in training. Column FFNN shows the white-box attack success rate.

Table 16. Transferability of Adversarial Examples from FFNN to LR and RF

  • We vary the imbalance ratio in training. Column FFNN shows the white-box attack success rate.

We also look at the transferability between different FFNN architectures trained on different datasets (results in Table 17). The attacks transfer best at the highest imbalance ratio (with a success rate higher than 96%), confirming that weaker models are easier to attack.

Table 17.
RatioDNN1DNN2DNN3
[80, 50][160, 80][100, 50, 25]
1100%57.6%42.3%
593.3%73.6%58.6%
1599%78.6%52.4%
25100%51.4%45.3%
50100%96%97.1%
  • Adversarial examples computed for DNN1 are transferred to DNN2 and DNN3.

Table 17. Transferability Between Architectures (number of neurons per layer in the second row)

  • Adversarial examples computed for DNN1 are transferred to DNN2 and DNN3.

6.5 Mitigations

Finally, we looked at defensive approaches to increase the FFNN robustness against the FENCE evasion attack. A well-known defensive technique is adversarial training [27, 46]. We trained FFNN using adversarial training with the Projected attack at \( L_2 \) distance 20. We trained the model adversarially for 11 epochs and obtained the AUC score of 89% (each epoch takes approximately 7 hours). We measured the Projected attack’s success rate for the balanced case against the standard and adversarially training models in Figure 11. Interestingly, the success rate of the evasion attacks significantly drops for the adversarially-trained model and reaches only 16.5% at 20 \( L_2 \) distance. This demonstrates that adversarial training is a promising direction for designing robust ML models for security. We plan to investigate it further and optimize its design in future work.

Fig. 11.

Fig. 11. Success rate of the FENCE Projected attack against adversarially and standard trained model.

Skip 7RELATED WORK Section

7 RELATED WORK

Adversarial machine learning studies ML vulnerabilities against attacks [35]. Research on the robustness of DNNs at testing time started with the work of Biggio et al. [8] and Szegedy et al. [68]. They showed that classifiers are vulnerable to adversarial examples generated with minimal perturbation to testing inputs. Since then, the area of adversarial ML has received a lot of attention, with the majority of work focused on evasion attacks (at testing time), e.g., [6, 11, 27, 39, 52, 53, 62]. Other classes of attacks include poisoning (e.g., [9, 77]) and privacy attacks (e.g., [23, 63]), but we focus here on evasion attacks.

Evasion attacks in security. Several evasion attacks have been proposed against models with discrete and constrained input vectors, as encountered in security. The majority of these use datasets with binary features, not considering dependencies in feature space. Biggio et al. [8] use a gradient-based attack to construct adversarial examples for malicious PDF detection by only adding new keywords to PDFs. Grosse et al. [30] leverage the JSMA attack by Papernot et al. [53] for a malware classification application in which features can be added or removed. Suciu et al. [66] add bytes to malicious binaries either at the end or in slack regions to create adversarial examples. Kreuk [37] discover regions in executables that would not affect the intended malware behavior. Kolosnjaji et al. [36] create gradient-based attack against malware detection DNNs that learn from raw bytes, and can create adversarial examples by only changing a few specific bytes at the end of each malware sample. Xu et al. [78] propose a black-box attack based on genetic algorithms for manipulating PDF files while maintaining the required format. Dang et al. [18] propose a black-box attack against PDF malware classifiers that uses hill-climbing over a set of feasible transformations. Anderson et al. [4] construct a general black-box framework based on reinforcement learning for attacking static portable executable anti-malware engines. Kulynych et al. [38] propose a graphical framework for discrete domains with guarantees of minimal adversarial cost. Recently, Pierazzi et al. [55] define a formalization for the domain-space attacks, along with a new white-box attack against Android malware classification. The authors use automated software transplantation to extract slices of bytecode from benign applications and inject them into a malicious host to mimic the benign activity and evade the classifier. Chen et al. [14] proposed an evasion attack called EvnAttack on malware present in portable Windows executable files, where input vectors are binary features each representing an API call to Windows.

Evasion attacks for network traffic classifiers include: Apruzesse et al. [5] analyzing the robustness of random forest for botnet classification; Clements et al. [16] evaluating the robustness of an anomaly detection method [48] against existing attacks; and De Lucia et al. [19] attacking an SVM for network scanning detection. A number of papers perform attacks against intrusion detection systems (IDS). Among them, Warzynski and Kołaczek [74] consider an FGSM attack under L1 norm against IDS and illustrate the ability of the generated adversarial example to evade the classifier. Rigaki et al. [57] generate targeted attacks by using FGSM and JSMA to evade decision tree, random forest, linear SVM, voting ensembles of the previous three classifiers, and a multi-layer perceptron (MLP) neural network IDS. Similarly, Wang et al. [73] leveraged FGSM, JSMA, Deepfool, and Carlini-Wagner to attack an MLP neural network for intrusion detection. Yang et al. [81] used Carlini-Wagner, a GAN attack, and black-box ZOO attacks against IDS DNNs. In their work, Martins et al. [47] tested the performance of FGSM, JSMA, Deepfool, and Carlini-Wagner attacks against decision tree, random forest, SVM, Naive Bayes, neural networks, and denoising autoencoders intrusion detection models. Wu et al. [76] applied deep reinforcement learning to generate adversarial attacks on botnet attacks. Yan et al. [80] generate adversarial examples for denial of service attacks. Lin et al. [43] use a modification of GAN called IDSGAN to produce adversarial examples while retaining functional features of the attack.

Evasion attacks with dependency constraints. A number of attacks that preserve constraints between extracted features in security exist in the literature. We survey these papers in greater detail and add a comparison to FENCE in Table 18.

Table 18.
PaperClassifiersAlgorithmDomainMathematical
DependenciesDependencies
FENCENetwork Traffic,IterativeRatioStat
Malicious DomainsoptimizationRangeLin
ProjectedOHENon-Lin
PenaltyCombination
Alhajjar et al. [2]Network TrafficGAN/PSOOHELinear
Granados et al. [28]Network TrafficIterativeRangeStat
optimization
Sadeghzadeh et al. [59]PacketsUAPRange-
Network Flows
Network Bursts
Han et al. [31]BotnetGAN/PSORange-
Abusnaina et al. [1]DDoS DetectionSampleRange-
injectionRatio-
Chen et al. [13]Network TrafficIterativeRange-
optimization-
GAN
  • We only include methods which respect dependencies in feature space. For each method, we mention the adversarial attack algorithm and the supported feature dependencies.

Table 18. Comparison to Existing Work on Evasion Attacks in Cybersecurity Domains

  • We only include methods which respect dependencies in feature space. For each method, we mention the adversarial attack algorithm and the supported feature dependencies.

Alhajjar et al. [2] explore the use of evolutionary computations and generative adversarial networks as a tool for crafting adversarial examples that aim to evade machine learning models used for network traffic classification. These strategies were applied to the NSL-KDD and UNSW-NB15 datasets. The paper operates only in features space, and the following dependencies are preserved: binary features, linear dependencies and features that can only be increased. There is no limitation on the amount of perturbations, as the authors claim that large changes in data are not subject to easy recognition by human observers. The percent of successful evasion for multi-layer perceptron (MLP) is much worse for both datasets than in FENCE: maximum 86.18% for NSL-KDD using GANs, and 55.10% for the UNSW-NB15 dataset. In contrast, FENCE minimizes the perturbation needed for evasion, preserves a much larger number of dependencies, such as non-linear, statistical, or combination of them, achieving a higher success rate.

Granados et al. [28] introduce Restricted Traffic Distribution Attack (RTDA) against network traffic classifiers, which is based on the Carlini and Wagner attack. In order to ensure the feasibility of adversarial examples, the attack only modifies the features corresponding to different percentiles for packet sizes by increasing their values and preserving the monotonic non-decreasing property of generated packet-size distribution. The attacker operates only in feature space, and there is no limit on bytes added to the packet, which may result in the packet sizes that are larger than allowed by network protocols. In contrast, FENCE generates more realistic attacks in the real-world scenario, by adding new connections of different types (UDP/TCP) and on different ports. Moreover, FENCE ensures feasibility by preserving packet sizes, connection durations, and other network traffic characteristics according to the constraints of network protocols.

Abusnaina et al. [1] present MergeFlow attack against DDoS detection models. They create adversarial examples by combining features from existing network flow with a representative mask flow from the target class. The features are combined either through averaging for ratio features or accumulating for count-based features. This attack preserves dependencies between features but generates large perturbations. Abusnaina et al. present similar ideas against graph-based IoT Malware Detection System that creates realistic adversarial examples by combining the original graph with a selected targeted class.

Sadeghzadeh et al. [59] introduce an adversarial network traffic attack (ANT) that uses a universal adversarial perturbation (UAP) generating method. To generate ANT they introduce the following three types of attack: AdvPad injects UAP to the content of packets to evaluate the robustness of packet classifiers, AdvPay injects UAP into payload if a dummy packet to evaluate flow-based classifiers, and AdvBurst modifies a burst of the flow to test the robustness of flow time-series classifiers. In order to generate the UAP, a set of flows or packets from a particular class is leveraged and the UAP is inserted into new incoming traffic of that class. This attack works in the domain space by inserting payload, dummy packet with payload, or the sequence of packets with statistical features. All perturbations are calculated by inserting the randomly initialized perturbation vector into examples from the target class and optimizing the loss function towards the selected class for some number of iterations. Sequences of bytes were used for training the classifiers, thus, there are no dependencies in the domain space. The only thing that needs to be preserved is the range for the number of bytes achieved by performing the clipping operation while computing UAP.

Han et al. [31] propose a practical traffic-space evasion attack against Network Intrusion Detection System (NIDS). The attacker can modify the original traffic generated from devices he controls at an affordable overhead. It has two main steps: finding the adversarial feature vector which can be classified as benign but is close to the malicious feature vector in terms of features’ values (lies in the low-confidence region of the classifier) using the GAN model and transforming malicious original traffic to transfer its features to the closest adversarial ones preserving its functionality using particle swarm optimization (PSO). While performing the second step, authors allow only to modify the interarrival time of packets in the original traffic, and for the injected crafted traffic the attacker is able to alter the interarrival time of packets, protocol layer of packets, and payload size. The budget overhead for the number of buckets and time elapsed is controlled by the rate of the original traffic. Feature dependencies are preserved in the domain space only; in contrast, FENCE allows to additionally preserve complex mathematical dependencies in feature space.

Chen et al. [13] introduce two types of evasion attacks against machine learning-based intrusion detection systems. The first attack called \( Opt \) uses an iterative optimization approach to find the adversarial examples that maximize the probability of malicious output while remaining stealthy and preserving the Range dependency. The second attack detaches the malicious payload from the input vector, and then optimizes the modifiable features based on a GAN under the Range dependency, keeping some features as integers. Thus, both attacks preserve only domain-specific dependencies, while FENCE is capable of satisfying a larger amount of domain dependencies as well as a list of mathematical constraints.

None of these previous works can handle the same number of domain and mathematical dependencies supported by our FENCE framework. In Table 18 we compare FENCE with these attacks against network traffic classifiers and include the list of supported constraints.

Evasion attacks in other domains. There is work on designing attacks in other domains, such as audio: [12, 15, 26, 56, 60, 64, 79, 85]; text: [3, 21, 24, 42, 54]; and video: [33, 40, 75]. Physically realizable attacks have been designed for face recognition [61] and vision [22].

Defenses against evasion attacks. Standard methods to defend against adversarial evasion attacks in continuous domains include: adversarial training [46], randomized smoothing [17], and defenses based on the detection of adversarial examples [41, 58, 82, 83, 84]. Randomized smoothing provides certifiable guarantees, but unfortunately, it cannot be directly applied in discrete domains because it requires the classifier to be evaluated on inputs perturbed with Gaussian noise. Therefore, we are not aware of any uses of randomized smoothing and similar randomization-based mechanism for cyber security defenses.

Defenses based on adversarial training have been used in cyber security. For instance, Abusnaina et al. [1] validate whether adversarial training improves the robustness of the DDoS detection system. They show that adversarial training is able to improve the robustness of the anomaly classification model, but only for certain types of adversarial examples included in the adversarial training. Hashem et al. [32] develop a network intrusion detection system that utilizes a novel reconstruction from partial observation method to build a more accurate anomaly detection model. The method also improves robustness to adversarial examples.

Methods for the detection of adversarial examples in security have been proposed. For example, Wang et al. [72] propose a defense mechanism against adversarial examples for NIDS, which exploits the inconsistency between manifold evaluation and model inference along with the closeness of the resulting adversarial example to the manifold. This allows them to train a logistic regression that predicts whether the input may be considered adversarial or not. In general, a known limitation of detection methods is that they fail in the face of adaptive attacks. Tramer et al. [70] showed that multiple detection-based methods (e.g., [58]) are not resilient against adaptive attacks, and thus designing detection-based defenses in cyber security which are resilient to adaptive attacks remains an open problem.

Skip 8CONCLUSIONS Section

8 CONCLUSIONS

We showed that evasion attacks against DNNs can be designed to preserve the dependencies in feature space in constrained domains. We proposed a general framework FENCE for generating adversarial examples that respects mathematical dependencies and domain-specific constraints imposed by these applications. We demonstrated evasion attacks that insert a small number of network connections (12 records in Zeek connection logs) to misclassify \( \mathsf {Malicious} \) activity as \( \mathsf {Benign} \) in a malicious connection classifier. We also showed that adversarial training has the potential to increase the robustness of classifiers in the malicious domain setting.

Our FENCE framework is not restricted to security applications, and we plan to apply it to healthcare and financial scenarios. An important open problem in this space is how to increase the resilience of DNN classifiers used in critical, constrained applications.

Skip ACKNOWLEDGMENTS Section

ACKNOWLEDGMENTS

We thank Simona Boboila and Talha Ongun for generating the features used for the malicious network traffic classifier.

REFERENCES

  1. [1] Abusnaina Ahmed, Khormali Aminollah, Nyang DaeHun, Yuksel Murat, and Mohaisen Aziz. 2019. Examining the robustness of learning-based DDoS detection in software defined networks. In 2019 IEEE Conference on Dependable and Secure Computing (DSC). IEEE, 18.Google ScholarGoogle ScholarCross RefCross Ref
  2. [2] Alhajjar Elie, Maxwell Paul, and Bastian Nathaniel D.. 2020. Adversarial machine learning in network intrusion detection systems. arXiv preprint arXiv:2004.11898 (2020).Google ScholarGoogle Scholar
  3. [3] Alzantot Moustafa, Sharma Yash, Elgohary Ahmed, Ho Bo-Jhang, Srivastava Mani, and Chang Kai-Wei. 2018. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998 (2018).Google ScholarGoogle Scholar
  4. [4] Anderson Hyrum S., Kharkar Anant, Filar Bobby, Evans David, and Roth Phil. 2018. Learning to evade static PE machine learning malware models via reinforcement learning. arXiv preprint arXiv:1801.08917 (2018).Google ScholarGoogle Scholar
  5. [5] Apruzzese Giovanni and Colajanni Michele. 2018. Evading botnet detectors based on flows and random forest with adversarial samples. 18. Google ScholarGoogle ScholarCross RefCross Ref
  6. [6] Athalye Anish, Carlini Nicholas, and Wagner David. 2018. Obfuscated gradients give a false sense of security: Circumventing defenses to adversarial examples. arXiv preprint arXiv:1802.00420 (2018).Google ScholarGoogle Scholar
  7. [7] Bartos Karel, Sofka Michal, and Franc Vojtech. 2016. Optimized invariant representation of network traffic for detecting unseen malware variants. In 25th USENIX Security Symposium (USENIX Security 16). USENIX Association, 807822.Google ScholarGoogle Scholar
  8. [8] Biggio Battista, Corona Igino, Maiorca Davide, Nelson Blaine, Srndic Nedim, Laskov Pavel, Giacinto Giorgio, and Roli Fabio. 2013. Evasion attacks against machine learning at test time. In Proc. Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD).Google ScholarGoogle ScholarDigital LibraryDigital Library
  9. [9] Biggio Battista, Nelson Blaine, and Laskov Pavel. 2012. Poisoning attacks against support vector machines. In ICML.Google ScholarGoogle Scholar
  10. [10] Bilge Leyla, Kirda Engin, Christopher Kruegel, and Balduzzi Marco. 2011. EXPOSURE: Finding malicious domains using passive DNS analysis. In Proc. 18th Symposium on Network and Distributed System Security (NDSS).Google ScholarGoogle Scholar
  11. [11] Carlini Nicholas and Wagner David. 2017. Towards evaluating the robustness of neural networks. In Proc. IEEE Security and Privacy Symposium.Google ScholarGoogle Scholar
  12. [12] Carlini Nicholas and Wagner David A.. 2018. Audio adversarial examples: Targeted attacks on speech-to-text. CoRR abs/1801.01944 (2018). arxiv:1801.01944 http://arxiv.org/abs/1801.01944.Google ScholarGoogle Scholar
  13. [13] Chen Jiming, Gao Xiangshan, Deng Ruilong, He Yang, Fang Chongrong, and Cheng Peng. 2020. Generating adversarial examples against machine learning based intrusion detector in industrial control systems. IEEE Transactions on Dependable and Secure Computing (2020).Google ScholarGoogle Scholar
  14. [14] Chen Lingwei, Ye Yanfang, and Bourlai Thirimachos. 2017. Adversarial machine learning in malware detection: Arms race between evasion attack and defense. In 2017 European Intelligence and Security Informatics Conference (EISIC). IEEE, 99106.Google ScholarGoogle ScholarCross RefCross Ref
  15. [15] Cisse Moustapha, Adi Yossi, Neverova Natalia, and Keshet Joseph. 2017. Houdini: Fooling deep structured prediction models. arXiv preprint arXiv:1707.05373 (2017).Google ScholarGoogle Scholar
  16. [16] Clements Joseph, Yang Yuzhe, Sharma Ankur, Hu Hongxin, and Lao Yingjie. 2019. Rallying adversarial techniques against deep learning for network security. arXiv preprint arXiv:1903.11688 (2019).Google ScholarGoogle Scholar
  17. [17] Cohen Jeremy, Rosenfeld Elan, and Kolter Zico. 2019. Certified adversarial robustness via randomized smoothing. In International Conference on Machine Learning. PMLR, 13101320.Google ScholarGoogle Scholar
  18. [18] Dang Hung, Huang Yue, and Chang Ee-Chien. 2017. Evading classifiers by morphing in the dark. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 119133.Google ScholarGoogle ScholarDigital LibraryDigital Library
  19. [19] Lucia Michael J. De and Cotton Chase. 2019. Adversarial machine learning for cyber security. Journal Of Information Systems Applied Research (2019).Google ScholarGoogle Scholar
  20. [20] Demontis Ambra, Melis Marco, Pintor Maura, Jagielski Matthew, Biggio Battista, Oprea Alina, Nita-Rotaru Cristina, and Roli Fabio. 2019. Why do adversarial attacks transfer? Explaining transferability of evasion and poisoning attacks. In 28th USENIX Security Symposium (USENIX Security 19). 321338.Google ScholarGoogle Scholar
  21. [21] Ebrahimi Javid, Rao Anyi, Lowd Daniel, and Dou Dejing. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751 (2017).Google ScholarGoogle Scholar
  22. [22] Eykholt Kevin, Evtimov Ivan, Fernandes Earlence, Li Bo, Rahmati Amir, Xiao Chaowei, Prakash Atul, Kohno Tadayoshi, and Song Dawn Xiaodong. 2018. Robust physical-world attacks on deep learning visual classification. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (2018), 16251634.Google ScholarGoogle ScholarCross RefCross Ref
  23. [23] Fredrikson Matt, Jha Somesh, and Ristenpart Thomas. 2015. Model inversion attacks that exploit confidence information and basic countermeasures. In Proceedings of the 22nd ACM Conference on Computer and Communications Security (CCS).Google ScholarGoogle ScholarDigital LibraryDigital Library
  24. [24] Gao Ji, Lanchantin Jack, Soffa Mary Lou, and Qi Yanjun. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW). IEEE, 5056.Google ScholarGoogle ScholarCross RefCross Ref
  25. [25] Garcia Sebastian, Grill Martin, Stiborek Jan, and Zunino Alejandro. 2014. An empirical comparison of botnet detection methods. Computers and Security 45 (2014), 100123.Google ScholarGoogle ScholarDigital LibraryDigital Library
  26. [26] Gong Yuan and Poellabauer Christian. 2017. Crafting adversarial examples for speech paralinguistics applications. arXiv preprint arXiv:1711.03280 (2017).Google ScholarGoogle Scholar
  27. [27] Goodfellow Ian J., Shlens Jonathon, and Szegedy Christian. 2014. Explaining and Harnessing Adversarial Examples. arXiv:1412.6572.Google ScholarGoogle Scholar
  28. [28] Granados Alonso, Miah Mohammad Sujan, Ortiz Anthony, and Kiekintveld Christopher. 2020. A realistic approach for network traffic obfuscation using adversarial machine learning. In International Conference on Decision and Game Theory for Security. Springer, 4557.Google ScholarGoogle ScholarDigital LibraryDigital Library
  29. [29] Grosse Kathrin, Manoharan Praveen, Papernot Nicolas, Backes Michael, and McDaniel Patrick. 2017. On the (Statistical) Detection of Adversarial Examples. arXiv:1702.06280.Google ScholarGoogle Scholar
  30. [30] Grosse Kathrin, Papernot Nicolas, Manoharan Praveen, Backes Michael, and McDaniel Patrick. 2016. Adversarial perturbations against deep neural networks for malware classification. arXiv preprint arXiv:1606.04435 (2016).Google ScholarGoogle Scholar
  31. [31] Han Dongqi, Wang Zhiliang, Zhong Ying, Chen Wenqi, Yang Jiahai, Lu Shuqiang, Shi Xingang, and Yin Xia. 2020. Practical traffic-space adversarial attacks on learning-based NIDSs. arXiv preprint arXiv:2005.07519 (2020).Google ScholarGoogle Scholar
  32. [32] Hashemi Mohammad J. and Keller Eric. 2020. Enhancing robustness against adversarial examples in network intrusion detection systems. In 2020 IEEE Conference on Network Function Virtualization and Software Defined Networks (NFV-SDN). IEEE, 3743.Google ScholarGoogle ScholarCross RefCross Ref
  33. [33] Hosseini Hossein, Xiao Baicen, Clark Andrew, and Poovendran Radha. 2017. Attacking automatic video analysis algorithms: A case study of Google cloud video intelligence API. In Proceedings of the 2017 on Multimedia Privacy and Security. ACM, 2132.Google ScholarGoogle ScholarDigital LibraryDigital Library
  34. [34] Hu Xin, Jang Jiyong, Stoecklin Marc Ph., Wang Ting, Schales Douglas Lee, Kirat Dhilung, and Rao Josyula R.. 2016. BAYWATCH: Robust beaconing detection to identify infected hosts in large-scale enterprise networks. In DSN. IEEE Computer Society, 479490.Google ScholarGoogle Scholar
  35. [35] Huang Ling, Joseph Anthony D., Nelson Blaine, Rubinstein Benjamin I. P., and Tygar J. D.. 2011. Adversarial machine learning. In Proceedings of the 4th ACM Workshop on Security and Artificial Intelligence. ACM, 4358.Google ScholarGoogle ScholarDigital LibraryDigital Library
  36. [36] Kolosnjaji Bojan, Demontis Ambra, Biggio Battista, Maiorca Davide, Giacinto Giorgio, Eckert Claudia, and Roli Fabio. 2018. Adversarial malware binaries: Evading deep learning for malware detection in executables. In 2018 26th European Signal Processing Conference (EUSIPCO). IEEE, 533537.Google ScholarGoogle ScholarCross RefCross Ref
  37. [37] Kreuk Felix, Barak Assi, Aviv-Reuven Shir, Baruch Moran, Pinkas Benny, and Keshet Joseph. 2018. Deceiving end-to-end deep learning malware detectors using adversarial examples. arXiv preprint arXiv:1802.04528 (2018).Google ScholarGoogle Scholar
  38. [38] Kulynych Bogdan, Hayes Jamie, Samarin Nikita, and Troncoso Carmela. 2018. Evading classifiers in discrete domains with provable optimality guarantees. arXiv preprint arXiv:1810.10939 (2018).Google ScholarGoogle Scholar
  39. [39] Kurakin Alexey, Goodfellow Ian, and Bengio Samy. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533 (2016).Google ScholarGoogle Scholar
  40. [40] Li Shasha, Neupane Ajaya, Paul Sujoy, Song Chengyu, Krishnamurthy Srikanth V., Roy-Chowdhury Amit K., and Swami Ananthram. 2019. Stealthy adversarial perturbations against real-time video classification systems. In NDSS.Google ScholarGoogle Scholar
  41. [41] Li Yingzhen, Bradshaw John, and Sharma Yash. 2019. Are generative classifiers more robust to adversarial attacks? In International Conference on Machine Learning. PMLR, 38043814.Google ScholarGoogle Scholar
  42. [42] Liang Bin, Li Hongcheng, Su Miaoqiang, Bian Pan, Li Xirong, and Shi Wenchang. 2017. Deep text classification can be fooled. arXiv preprint arXiv:1704.08006 (2017).Google ScholarGoogle Scholar
  43. [43] Lin Zilong, Shi Yong, and Xue Zhi. 2018. IDSGAN: Generative adversarial networks for attack generation against intrusion detection. arXiv preprint arXiv:1809.02077 (2018).Google ScholarGoogle Scholar
  44. [44] Liu Yanpei, Chen Xinyun, Liu Chang, and Song Dawn. 2016. Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv:1611.02770 (2016).Google ScholarGoogle Scholar
  45. [45] Ma Justin, Saul Lawrence K., Savage Stefan, and Voelker Geoffrey M.. 2009. Beyond blacklists: Learning to detect malicious web sites from suspicious URLs. In Proc. 15th ACM International Conference on Knowledge Discovery and Data Mining (KDD).Google ScholarGoogle ScholarDigital LibraryDigital Library
  46. [46] Madry Aleksander, Makelov Aleksandar, Schmidt Ludwig, Tsipras Dimitris, and Vladu Adrian. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083 (2017).Google ScholarGoogle Scholar
  47. [47] Martins Nuno, Cruz José Magalhães, Cruz Tiago, and Abreu Pedro Henriques. 2019. Analyzing the footprint of classifiers in adversarial denial of service contexts. In EPIA Conference on Artificial Intelligence. Springer, 256267.Google ScholarGoogle ScholarDigital LibraryDigital Library
  48. [48] Mirsky Yisroel, Doitshman Tomer, Elovici Yuval, and Shabtai Asaf. 2018. Kitsune: An ensemble of autoencoders for online network intrusion detection. arXiv preprint arXiv:1802.09089 (2018).Google ScholarGoogle Scholar
  49. [49] Ongun Talha, Sakharaov Timothy, Boboila Simona, Oprea Alina, and Eliassi-Rad Tina. 2019. On designing machine learning models for malicious network traffic classification. arXiv preprint arXiv:1907.04846 (2019).Google ScholarGoogle Scholar
  50. [50] Oprea Alina, Li Zhou, Norris Robin, and Bowers Kevin. 2018. MADE: Security analytics for enterprise threat detection. In Proceedings of the 34th Annual Computer Security Applications Conference. ACM, 124136.Google ScholarGoogle ScholarDigital LibraryDigital Library
  51. [51] Papernot Nicolas, McDaniel Patrick, and Goodfellow Ian. 2016. Transferability in machine learning: From phenomena to black-box attacks using adversarial samples. arXiv preprint arXiv:1605.07277 (2016).Google ScholarGoogle Scholar
  52. [52] Papernot Nicolas, McDaniel Patrick, Goodfellow Ian, Jha Somesh, Celik Z. Berkay, and Swami Ananthram. 2017. Practical black-box attacks against machine learning. In Proceedings of the ACM SIGSAC Asia Conference on Computer and Communications Security (AsiaCCS).Google ScholarGoogle ScholarDigital LibraryDigital Library
  53. [53] Papernot Nicolas, McDaniel Patrick, Jha Somesh, Fredrikson Matt, Celik Z. Berkay, and Swami Ananthram. 2017. The limitations of deep learning in adversarial settings. In Proc. IEEE European Security and Privacy Symposium (Euro S&P).Google ScholarGoogle Scholar
  54. [54] Papernot Nicolas, McDaniel Patrick, Swami Ananthram, and Harang Richard. 2016. Crafting adversarial input sequences for recurrent neural networks. In MILCOM 2016-2016 IEEE Military Communications Conference. IEEE, 4954.Google ScholarGoogle ScholarDigital LibraryDigital Library
  55. [55] Pierazzi Fabio, Pendlebury Feargus, Cortellazzi Jacopo, and Cavallaro Lorenzo. 2019. Intriguing properties of adversarial ML attacks in the problem space. arXiv preprint arXiv:1911.02142 (2019).Google ScholarGoogle Scholar
  56. [56] Qin Yao, Carlini Nicholas, Goodfellow Ian, Cottrell Garrison, and Raffel Colin. 2019. Imperceptible, robust, and targeted adversarial examples for automatic speech recognition. arXiv preprint arXiv:1903.10346 (2019).Google ScholarGoogle Scholar
  57. [57] Rigaki Maria. 2017. Adversarial deep learning against intrusion detection classifiers.Google ScholarGoogle Scholar
  58. [58] Roth Kevin, Kilcher Yannic, and Hofmann Thomas. 2019. The odds are odd: A statistical test for detecting adversarial examples. arXiv preprint arXiv:1902.04818 (2019).Google ScholarGoogle Scholar
  59. [59] Sadeghzadeh Amir Mahdi, Shiravi Saeed, and Jalili Rasool. 2021. Adversarial network traffic: Towards evaluating the robustness of deep-learning-based network traffic classification. IEEE Transactions on Network and Service Management 18, 2 (2021), 19621976.Google ScholarGoogle ScholarCross RefCross Ref
  60. [60] Schonherr Lea, Kohls Katharina, Zeiler Steffen, Holz Thorsten, and Kolossa Dorothea. 2018. Adversarial attacks against automatic speech recognition systems via psychoacoustic hiding. arXiv preprint arXiv:1808.05665 (2018).Google ScholarGoogle Scholar
  61. [61] Sharif Mahmood, Bhagavatula Sruti, Bauer Lujo, and Reiter Michael K.. 2016. Accessorize to a crime: Real and stealthy attacks on state-of-the-art face recognition. In Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security (Vienna, Austria) (CCS’16). ACM, New York, NY, USA, 15281540. Google ScholarGoogle ScholarDigital LibraryDigital Library
  62. [62] Sharif Mahmood, Bhagavatula Sruti, Bauer Lujo, and Reiter Michael K.. 2019. A general framework for adversarial examples with objectives. ACM Transactions on Privacy and Security (TOPS) 22, 3 (2019), 130.Google ScholarGoogle ScholarDigital LibraryDigital Library
  63. [63] Shokri Reza, Stronati Marco, Song Congzheng, and Shmatikov Vitaly. 2017. Membership inference attacks against machine learning models. In Proc. IEEE Security and Privacy Symposium (S&P).Google ScholarGoogle Scholar
  64. [64] Song Liwei and Mittal Prateek. 2017. Inaudible voice commands. arXiv preprint arXiv:1708.07238 (2017).Google ScholarGoogle Scholar
  65. [65] Srndic Nedim and Laskov Pavel. 2014. Practical evasion of a learning-based classifier: A case study. In Proc. IEEE Security and Privacy Symposium.Google ScholarGoogle Scholar
  66. [66] Suciu Octavian, Coull Scott E., and Johns Jeffrey. 2018. Exploring adversarial examples in malware detection. arXiv preprint arXiv:1810.08280 (2018).Google ScholarGoogle Scholar
  67. [67] Suciu Octavian, Marginean Radu, Kaya Yigitcan, III Hal Daume, and Dumitras Tudor. 2018. When does machine learning {\( FAIL \)}? generalized transferability for evasion and poisoning attacks. In 27th USENIX Security Symposium (USENIX Security 18). 12991316.Google ScholarGoogle Scholar
  68. [68] Szegedy Christian, Zaremba Wojciech, Sutskever Ilya, Bruna Joan, Erhan Dumitru, Goodfellow Ian, and Fergus Rob. 2014. Intriguing properties of neural networks. arXiv:1312.6199.Google ScholarGoogle Scholar
  69. [69] Tong Liang, Li Bo, Hajaj Chen, Xiao Chaowei, Zhang Ning, and Vorobeychik Yevgeniy. 2019. Improving robustness of ML classifiers against realizable evasion attacks using conserved features. In 28th USENIX Security Symposium (USENIX Security 19). USENIX Association, Santa Clara, CA, 285302. https://www.usenix.org/conference/usenixsecurity19/presentation/tong.Google ScholarGoogle Scholar
  70. [70] Tramer Florian, Carlini Nicholas, Brendel Wieland, and Madry Aleksander. 2020. On adaptive attacks to adversarial example defenses. arXiv preprint arXiv:2002.08347 (2020).Google ScholarGoogle Scholar
  71. [71] Tramèr Florian, Papernot Nicolas, Goodfellow Ian, Boneh Dan, and McDaniel Patrick. 2017. The space of transferable adversarial examples. arXiv preprint arXiv:1704.03453 (2017).Google ScholarGoogle Scholar
  72. [72] Wang Ning, Chen Yimin, Hu Yang, Lou Wenjing, and Hou Y. Thomas. 2021. MANDA: On adversarial example detection for network intrusion detection system. In IEEE INFOCOM 2021-IEEE Conference on Computer Communications. IEEE, 110.Google ScholarGoogle ScholarDigital LibraryDigital Library
  73. [73] Wang Zheng. 2018. Deep learning-based intrusion detection with adversaries. IEEE Access 6 (2018), 3836738384.Google ScholarGoogle ScholarCross RefCross Ref
  74. [74] Warzyński Arkadiusz and Kołaczek Grzegorz. 2018. Intrusion detection systems vulnerability on adversarial examples. In 2018 Innovations in Intelligent Systems and Applications (INISTA). IEEE, 14.Google ScholarGoogle Scholar
  75. [75] Wei Xingxing, Zhu Jun, and Su Hang. 2018. Sparse adversarial perturbations for videos. arXiv preprint arXiv:1803.02536 (2018).Google ScholarGoogle Scholar
  76. [76] Wu Di, Fang Binxing, Wang Junnan, Liu Qixu, and Cui Xiang. 2019. Evading machine learning botnet detection models via deep reinforcement learning. In ICC 2019-2019 IEEE International Conference on Communications (ICC). IEEE, 16.Google ScholarGoogle ScholarCross RefCross Ref
  77. [77] Xiao Huang, Biggio Battista, Brown Gavin, Fumera Giorgio, Eckert Claudia, and Roli Fabio. 2015. Is feature selection secure against training data poisoning? In Proc. 32nd International Conference on Machine Learning(ICML, Vol. 37). 16891698.Google ScholarGoogle Scholar
  78. [78] Xu Weilin, Qi Yanjun, and Evans David. 2016. Automatically evading classifiers. In Proceedings of the 2016 Network and Distributed Systems Symposium. 2124.Google ScholarGoogle ScholarCross RefCross Ref
  79. [79] Yakura Hiromu and Sakuma Jun. 2018. Robust audio adversarial example for a physical attack. arXiv preprint arXiv:1810.11793 (2018).Google ScholarGoogle Scholar
  80. [80] Yan Qiao, Wang Mingde, Huang Wenyao, Luo Xupeng, and Yu F. Richard. 2019. Automatically synthesizing DoS attack traces using generative adversarial networks. International Journal of Machine Learning and Cybernetics 10, 12 (2019), 33873396.Google ScholarGoogle ScholarCross RefCross Ref
  81. [81] Yang Kaichen, Liu Jianqing, Zhang Chi, and Fang Yuguang. 2018. Adversarial examples against the deep learning based network intrusion detection systems. In MILCOM 2018-2018 IEEE Military Communications Conference (MILCOM). IEEE, 559564.Google ScholarGoogle ScholarDigital LibraryDigital Library
  82. [82] Yang Zhuolin, Li Bo, Chen Pin-Yu, and Song Dawn. 2018. Characterizing audio adversarial examples using temporal dependency. arXiv preprint arXiv:1809.10875 (2018).Google ScholarGoogle Scholar
  83. [83] Yin Xuwang, Kolouri Soheil, and Rohde Gustavo K.. 2019. Adversarial example detection and classification with asymmetrical adversarial training. arXiv preprint arXiv:1905.11475 (2019).Google ScholarGoogle Scholar
  84. [84] Yu Tao, Hu Shengyuan, Guo Chuan, Chao Wei-Lun, and Weinberger Kilian Q.. 2019. A new defense against adversarial images: Turning a weakness into a strength. arXiv preprint arXiv:1910.07629 (2019).Google ScholarGoogle Scholar
  85. [85] Zhang Guoming, Yan Chen, Ji Xiaoyu, Zhang Tianchen, Zhang Taimin, and Xu Wenyuan. 2017. DolphinAttack: Inaudible voice commands. In Proceedings of the 2017 ACM SIGSAC Conference on Computer and Communications Security. ACM, 103117.Google ScholarGoogle ScholarDigital LibraryDigital Library

Index Terms

  1. FENCE: Feasible Evasion Attacks on Neural Networks in Constrained Environments

      Recommendations

      Comments

      Login options

      Check if you have access through your login credentials or your institution to get full access on this article.

      Sign in

      Full Access

      • Published in

        cover image ACM Transactions on Privacy and Security
        ACM Transactions on Privacy and Security  Volume 25, Issue 4
        November 2022
        330 pages
        ISSN:2471-2566
        EISSN:2471-2574
        DOI:10.1145/3544004
        Issue’s Table of Contents

        Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected].

        Publisher

        Association for Computing Machinery

        New York, NY, United States

        Publication History

        • Published: 21 July 2022
        • Online AM: 21 June 2022
        • Accepted: 1 May 2022
        • Revised: 1 January 2022
        • Received: 1 November 2020
        Published in tops Volume 25, Issue 4

        Permissions

        Request permissions about this article.

        Request Permissions

        Check for updates

        Qualifiers

        • research-article
        • Refereed
      • Article Metrics

        • Downloads (Last 12 months)480
        • Downloads (Last 6 weeks)62

        Other Metrics

      PDF Format

      View or Download as a PDF file.

      PDF

      eReader

      View online with eReader.

      eReader

      HTML Format

      View this article in HTML Format .

      View HTML Format
      About Cookies On This Site

      We use cookies to ensure that we give you the best experience on our website.

      Learn more

      Got it!