Evading Botnet Detection

Botnet detection remains a challenging task due to many botnet families with different communication strategies, traffic encryption, and hiding techniques. Machine learning-based methods have been successfully applied, but have proven to be vulnerable to evasion attacks. In this paper, we show how an attacker can evade the detection of botnet traffic by manipulating selected features in the attack flows. We first build two well-performing machine learning models - based on Random Forest and Support Vector Machine classifiers - trained using only features that are also available in encrypted traffic. We then show with two different datasets how the detection rate (recall) decreases significantly for both classifiers if only a few basic features are manipulated by the attacker. We apply two state-of-the-art evasion attacks: Hop Skip Jump and Fast Gradient Sign. For all manipulated attack vectors we perform a plausibility check to ensure consistency with traffic statistics and protocol rules, as well as a bot check to ensure the manipulated attack vectors are still valid bot samples. We show, that for both Hop Skip Jump and Fast Gradient Sign, it is possible to craft plausible network traffic samples, but for Fast Gradient Sign, the feature values of the manipulated samples are far outside the normal range for botnet traffic. Our results show that the models can easily be fooled if the attacker is able to test the black-box models multiple times. Since in a real setting attackers may not have access to the model and training data, we implement a local substitute model to generate the attack samples and then check if they are transferable to other machine learning models trained with different training data. Our results show that samples generated with Hop Skip Jump generally do not transfer well while Fast Gradient Sign samples also evade the detection of models other than the substitute model.


INTRODUCTION
A botnet provides a flexible infrastructure with access to compromised resources that can be used by an attacker for a wide variety of malicious activities.Typical activities performed by botnets are attacking other devices with distributed Denial of Service attacks, information ex-filtration, distribution of SPAM emails or just using the bots as distributed computing platforms for cracking passwords or mining crypto-currencies [13] [25].Operating a botnet has many advantages for the attacker.The attacker can use many resources and the real origin of the attack is concealed [19].For botnets it is possible to prepare and schedule attacks and control intensity and type [30].Botnets have evolved from just using classical network devices to compromising mobile phones and IoT devices.The detection of malware communication and especially the detection of botnet traffic remains a challenging task.Communication patterns of different botnet command and control (c&c) structures differ and can be quite complex [27].In addition, botnet operators try to avoid detection and use more and more methods for hiding c&c communication.Modern botnet software often uses encryption and authentication based on standard protocols (such as IPsec, TLS) and therefore it is very difficult to distinguish the resulting malicious traffic from benign traffic [27].Traffic encryption drastically reduces the possibilities for attack detection methods.Payload inspection is not possible and only a limited set of features from some header fields is available for analysis.This makes deep packet inspection (DPI) without further adjustments infeasible [24].In the scientific literature, machine learning (ML) methods are increasingly used for attack detection and can help to detect sophisticated botnet communication.However, attack detection methods always operate in an adversarial environment and ML itself provides a further point of attack.This means, attackers can try to craft their attacks in a way to evade detection by the ML model.In our framework, we focus on the detection of two botnets: Ares and Murlo.Ares -a newer version of the Kronos banking trojan -is an IoT botnet based on the well-known Mirai botnet [17].Murlo is an IRC botnet that contacts lots of c&c hosts and receives large amounts of encrypted traffic [11].The reasons for choosing these two botnets are that A) they are contained in two well-known publicly available benchmark datasets that have an adequate number of benign and botnet samples.This also makes our results comparable to other scientific work.B) both attacks are well-detected by our chosen detection model with high recall and F1 scores -which form our baseline to measure the evasion performance.C) the traffic is partially encrypted which constitutes a realistic and highly relevant scenario.In this paper, we investigate methods to manipulate botnet traffic in a way that the attacks are no longer detected by an ML-based network intrusion detection system (NIDS).In our scenario, we assume that all traffic (benign and malicious) is TLS or IPsec encrypted.Therefore we are very restricted regarding features that we can extract from the data as input for the ML.As a basis for our NIDS, we use four different well-performing ML classifiers: Random Forest (RF), Decision Tree (DT), Support Vector Machine (SVM), and Multi-Layer Perceptron (MLP).All four have performed very well in the past for attack detection [16] and indicate simpleand for DT (and RF partially) explainable -supervised ML methods.We test our suggested framework with two different network traffic datasets and achieve a very good detection performance for both.Then we show how an attacker can manipulate the attack traffic in a way that the attack communication remains undetected.We investigate in-depth which features can be manipulated, how the traffic can be modified to not violate traffic statistics or protocol rules while the manipulated attack flows still remain valid attacks, and show how the evasion success depends on the amount and type of the manipulated features.We assume that the adversary does not have any access to or knowledge about the NIDS.The adversary trains their own local substitute model with a different training dataset and customizes their attack vector on that model.In our proposed framework, we analyze the detection performance of the NIDS on the adversarial samples -per model and also as an ensemble vote of all four proposed detection models.The following points summarize our contributions: ML models for botnet detection: We train four different ML models with data from two different datasets.We use only features available also in encrypted traffic.All models perform very well in detecting botnet traffic.Evasion attacks: From the test samples we generate a set of manipulated feature vectors using the Hop Skip Jump attack (HSJ) [9] and the Fast Gradient Sign Method (FGS) [12].We show that with simple changes to a few basic features, the attacker can successfully hide the malware communication if they can test the model an infinite number of times.Plausibility Check: All manipulated samples are passed through an advanced post-processor where their plausibility is checked and the implausible feature values are manipulated further -and hence sanitized -if possible.With this it is ensured that no unusual or impossible flow statistics are caused by the manipulation or any protocol rules are violated.We further check if the manipulated feature values are still valid values for operatable botnets.

Transferable evasion attacks:
We further analyze the detection performance of adversarial samples if the attacker has no knowledge of the model and training data and cannot test the NIDS in advance and hence has to generate the adversarial samples using their own local substitute model -different from the NIDS.

THREAT MODEL
In our scenario, the attacker (A) implements a black box ML model (model A) that is able to detect botnet attacks with high detection performance scores.Model A is trained using dataset A. The attacker queries their own model an unlimited number of times to generate adversarial samples (that evade the detection by model A) by manipulating chosen features.The second player in our scenario is the network operator who tries to defend the network from the adversary.The defender (D) has an NIDS deployed that is based on four different ML techniques (RF, DT, SVM, and MLP).These victim models are trained with botnet traffic different from dataset A. Training dataset D is however similar to dataset A with regard to composition (relation of benign flows to attack flows) and attack types contained.The attacker has the goal of fooling the NIDS without any access to the model.They can only attack once with the adversarial samples generated and customized for their substitute model A. Our assumptions are: • The attacker knows that the NIDS works with an ML-based detection method.The attacker does not know which method is used.• The attacker assumes that all traffic in the network is encrypted and therefore no DPI can be used and the set of features that can be used for the attack detection is limited.in the beginning and it is not updated during the period when manipulated samples are tested on the NIDS.
We further discuss the implications of some of those assumptions in the following sections.

RELATED WORK
ML is widely used for ML-based network intrusion detection [16], [22] and has also proven to be highly effective for the classification of botnet traffic [28].
Evasion Attacks on ML-based NIDS: Recently the application of adversarial ML has also spread to the network domain.In Table 1, we compare our work to contributions that are the closest to our proposal with regard to six key points we established as most important.Especially evasion attacks show high rewards for possible adversaries because adversaries do not need access to the specifics of the model or training data and evasion attacks still prove to be highly effective [18].[15].They constrain the manipulation in order to maintain the malware's executability and show promising transferability results.The above-mentioned frameworks do not consider network traffic, botnet attacks, encryption, and the manipulation of only a few features.
Our Contribution: In Table 1, it can be seen, that, unlike all of the above-mentioned publications, we consider encryption, which makes our framework more forthcoming as modern botnet software often uses TLS or IPsec encryption.This makes the ML-based classification more challenging as there are fewer features available for training and testing and there are fewer features to which the perturbation can be added.In our framework, we propose applying HSJ and FGS attacks on ML-based botnet detection models.
We achieve competitive success rates while keeping the samples plausible and further, we assess evasion performance differences with regard to variations in the number of manipulated features.Furthermore, we test the transferability of the adversarial examples to four black-box ML models with different architectures.

EVASION ATTACKS
We propose a method to effectively evade the detection of botnet traffic when using an ML-based NIDS.Our approach is to add minor changes to the feature vectors of botnet flows so that those flows fool the model and hence lead to a misclassification of the flows as benign flows.We propose a setup where the attacker only needs to modify features that they have easy access to.To test the robustness of the ML-based NIDS, we assess detection performance differences with regard to the number of features that are manipulated.We take special care to ensure, that all manipulated flows are still valid and form realistic attack flows with regard to network constraints, statistical analysis, and general limitations.As the attacker usually cannot test the NIDS an infinite amount of times without getting caught, we test if the adversarial samples are transferable to other ML models different from the substitute model they were designed for.

Botnet Detection
Lately, the amount of encrypted traffic in network traffic has been increasing.Malware communication and especially botnet c&c communication often uses encryption [1], [6], [14], [27].To create a realistic and forthcoming scenario, we assume IPsec or TLS encryption for all traffic including the botnet samples.This makes the detection of an attack more challenging as previously applied techniques like DPIs are infeasible.When assuming encrypted traffic, ML has proven to be a promising method for intrusion detection systems to correctly flag attack traffic.We specifically select four different ML techniques: RF, DT, SVM, and MLP which all show promising classification performances for our selected botnets.

Adversarial Examples Generation
We apply two state-of-the-art evasion attacks for the generation of adversarial examples: HSJ [9] and FGS [12].Compared to the more commonly used boundary attack [8], HSJ attack is more queryefficient and still manages to be highly effective as an evasion attack while generating samples that differ only minimally from the original samples.HSJ attack first estimates the gradient direction and then searches the step size using geometric progression.Finally, a binary search is applied to search the boundary.However, the authors in [9] propose their method for image recognition (a field where evasion attacks are commonly applied) and not for network traffic classification.The FGS attack is also a well-established evasion attack.First, the gradients of the loss function are computed with respect to the input data.The loss function typically measures the discrepancy between the predicted output of the model and the true label.Small perturbations are generated based on the values and signs of the gradients in order to modify the original input in a way to fool the classifier.As our application field (NIDS) highly differs from the field these evasion attacks were first proposed for (image recognition), we need to take further measures to ensure the plausibility of the results.For our experiments, we define the following conditions: A) The manipulation is restricted only to forward features.In a real scenario, an attacker can only manipulate features from packets that are sent by them.Features extracted from packets in the backward direction (packets received by the adversary) cannot easily be accessed by the attacker and are hence left unaltered in our scenario.B) To assess the attack's effectiveness when manipulating fewer features, we first manipulate only one feature and then increase the amount of manipulated features in consecutive experiments.As feature importances can vary distinctly for different ML classifier architectures, we set the order in which we manipulate the features randomly -which would imply a random guess of the adversary.However, we also tried out experiments with other feature orderings and evaluated the results for the attacker's best possible guess: the attacker guessing the exact feature importances of the defender's model.These results do not differ greatly from the random guess.C) All manipulated feature values need to remain valid with regard to statistical properties, network, and protocol constraints and the botnet attacks need to remain feasible.For the generation of the adversarial samples, we use the Python implementation of the above-mentioned evasion attacks in the Adversarial Robustness Toolbox [20].When applying evasion attacks on image recognition, the main goal is to effectively fool the classifier while keeping the modifications on the image small enough so that the manipulated image is indistinguishable from the original one for the human eye.However, in our domain, there can be no human eye that checks if flows are indistinguishable and it is not sufficient to just keep the modifications minimal.In network traffic, small changes can have a major impact and can lead to inconsistencies in traffic statistics or protocol rules.Since flow features result from aggregation, rules for the statistics apply, and manipulated flows could be mathematically incorrect (e.g., the minimal value of a feature is higher than the maximal value of the same feature).Also, manipulated flows could violate basic network rules (e.g., the maximum packet length is higher than the protocol allows or the inter-arrival time of two packets is too low).This makes it necessary to apply further processing on the manipulated flows in order to make the feature values realistic and consistent with other flow statistics and protocol behavior.We propose an advanced post-processor where plausibility checks are performed.The exact checks are described further in subsection 5.3 part III).These checks ensure that all manipulated samples are plausible and realistic.All samples, that fail the plausibility check are passed through the post-processor where they are -if possibleadjusted in order to sanitize them in a way to pass the plausibility check while still fooling the classifier.Furthermore, the general feature range of the samples is checked in order to make sure the samples are still valid bot samples and not outliers.We note that all features far outside of the normal feature distribution for botnet samples are suspicious and possibly not valid bot samples.In addition, samples that deviate from typical feature distributions in benign traffic are usually easier to detect.

Transferability
Adversarial examples tend to have the same effect not only on one model but also on other models, even if those models have different architectures and were trained on separate datasets as long as the models were trained to perform the same task [21].Consequently, an attacker can train a substitute model and generate adversarial examples specifically for that model.These samples are then transferred to a victim model, all while having minimal information about that model and very limited access to that model.In our framework, we evaluate the detection performance of four different victim NIDS models on adversarial samples generated with two state-of-the-art evasion attacks.

EXPERIMENTS
With our experiments, we want to show the following: Detection: Botnet traffic can be detected with several ML techniques (RF, DT, SVM, and MLP) even if trained only with features available also for encrypted traffic.Evasion: Botnet flows can be manipulated in a way that the ML model no longer detects them if the attacker has an unlimited number of queries to test the model.Plausibility: It is possible to find manipulated feature vectors containing only valid bot samples that comply with protocol rules and statistical principles.Manipulated Features: It is sufficient to manipulate features that can easily be accessed by an attacker (as opposed to manipulating all features of a flow).There is no need to manipulate many features -even when only manipulating a few features the model can be fooled.Transferability: Evasion attacks are also feasible without the possibility of repeatedly querying the model.

Datasets
In Table 2, the datasets and their composition are shown.Each flow is distinguished by its unique 5-tuple (source and destination IP, source and destination port, and protocol).For both datasets, we select all benign flows and the attack flows of one specific botnet attack (Ares for the CIC-IDS-17 dataset [23] and Murlo for the ISCX-Bot-14 dataset [7]).As can be seen, both datasets are highly imbalanced.This is not unusual for IDS benchmark datasets and also reflects the reality in which usually only a small percentage of flows belong to attack traffic.

Features
In Table 3

Experiment setup
In the following subsections, the exact processes in steps I) to IV) shown in Figures 1 to 4 are described in detail.I) Feature generation, pre-processing, and training.Figure 1 shows the exact steps for extraction, pre-processing, and training of the models for the attacker (A) and defender (D).First, we generate flows from the raw pcap datasets.We then extract the corresponding feature vectors.These flow and feature extraction steps are performed using go-flows [26] as a flow and feature extractor.Flows that have missing values are dropped.Features with a co-correlation higher than 0.99 to other features are dropped.This reduction of the dataset is possible without impacting the results because we generate our own detection model (and not use one from the literature) as a baseline.Therefore we just use the reduced dataset as a basis for all experiments.The dataset is then split 50/50.The first half of the dataset (dataset A) is used by the attacker to train and test their own substitute model (model A) with which they will later generate adversarial examples.The second half (dataset D) is used by the defender to train and test four different ML models which we will consider as the NIDS of the victim.The adversary does not have any knowledge about the NIDS or dataset D and is not able to manipulate either.All steps for datasets A and D are separated completely.Both feature vectors A and D are split into 80 % training data and 20 % testing data making sure that the original distribution of the dataset is matched for both subsets.We perform the following pre-processing steps individually for both training datasets A and D: first, the scaler is fitted on the training data.We choose min-max scaling for the normalization of the data which scales all features to a range of [0,1].All values of the training set and of the test set are normalized feature-wise based on the statistics of the training set.When testing model A, we use the scaling parameters of training set A. When testing the NIDS, we use the scaling parameters of the training set D. This is done to ensure that the attacker has no information about the data used by the defender.II) Baseline: Testing and evaluation of models.In Figure 2, the steps for testing and evaluating the models are shown.These results form the baseline for our further experiments.a) Model A (trained with training set A) is evaluated using test set A based on selected classification performance metrics.This is the attacker's local substitute model to which they have full access and an unlimited amount of trials to test the model.b) The NIDS (trained with training set D) is based on four different ML classifiers (RF, DT, SVM, MLP).These models are -equal to model A -evaluated using test set A to form a valid baseline for the evasion attacks.This is the victim model to which the attacker has no access and only one trial per sample to test.We achieve very good performance metrics on the clean test set for all models.That means without any manipulation most of the botnet flows are well detected.These performance metrics are used as a baseline for part IV) where we evaluate the classification performance of the (same) models on manipulated samples.III) Manipulation.Figure 3 shows the manipulation steps performed by the attacker.From the attacker's test set A, we select all botnet samples (76 flows for the CIC-IDS-17 dataset and 216 flows for the ISCX-Bot-14 dataset) and manipulate them using HSJ and FGS attacks.All benign samples in the test set are left unaltered.We only manipulate the features that can be easily accessed by an attacker (see Table 3).With each manipulation step, the number of manipulated features is increased by one (starting by manipulating one feature).The order in which the features are selected for the manipulation is guessed randomly by the attacker because they do not know anything about the victim model and hence can only guess which features have a high importance.We performed our experiments with various differing random sequences and all results were similar.All manipulated samples are passed through a postprocessor where basic checks are performed in order to make sure that the packets cannot be flagged as obvious adversarial samples due to inconsistencies.This plausibility check includes thorough feature value checks to ensure the feature values could be extracted from realistic packets of botnet attack flows.All unrealistic samples are processed further for sanitization.The post-processor includes a check of basic protocol rules and network traffic constraints as well as a bot check where it is checked if all samples can still be viewed as valid botnet samples.The following steps are performed after the manipulation: A) The scaled adversarial samples are transformed back to their real values.B) Basic plausibility checks as well as bot checks are performed on the unscaled feature values.The exact plausibility checks are described in Table 4.
C) If the samples fail the plausibility checks (which implies inconsistencies), they are sanitized if possible: if there are floating point numbers instead of integers for features like the number of packets, the values are rounded.If the statistics are unrealistic, the values are modified to make sense while keeping the changes minimal.If

RESULTS AND DISCUSSION
In this section, we present the results of our experiments as well as our most important findings.The performance metrics for the experiments without feature manipulation (step II) and experiments with different manipulated feature sets tested on the substitute models A (step IV a) are shown in Table 5.The results for the evaluation of the detection performance of the four NIDS models with manipulated samples (step IV b) are presented in Tables 6 (HSJ) and 7 (FGS).As metrics, we show precision, recall, and F1 Score to guarantee a good summary of the classification performance.Please note that the precision is N/A if there are no true positive and no false positive samples.The low precision values (due to a high number of false positives) for SVM do not matter in our case, as we only interfere with the attack samples and leave the benign samples unaltered.For the generation of the adversarial samples with HSJ attack, we chose a random forest-based substitute model A because the RF classification results on the clean data outperformed all other models.For the generation of attack samples with FGS, we chose SVM because FGS requires a gradient-based classifier.

Classification performance of model A before and after feature manipulation
We first check if an attacker can manipulate samples to evade detection of his own model A, which can be seen in Table 5.The precision, recall, and F1 scores can be reduced with feature manipulation.Adversarial samples generated with HSJ attacks are not able to evade detection when only one feature value is manipulated.When five features are manipulated, the recall and F1 scores drop drastically for the CIC-IDS-17 dataset.The manipulated features and the order in which they are manipulated can be seen in Figures 5 to 8 (x-axis).For both datasets, the performance metrics drop to zero, when all features are manipulated.When applying the FGS method it is sufficient to manipulate one single feature and the 0.00 0.00 0.00 0.58 0.07 0.12 10 features 0.00 0.00 0.00 0.00 0.00 0.00 Table 7: Performance metrics of defender models D for clean data and manipulated attack samples generated with FGS attack.
previously high precision, recall and F1 scores drop drastically for both datasets.The classification accuracy is very high for both models and for both datasets, but this is biased due to the imbalanced datasets (see Table 2) -only 0.04% of the CIC-IDS-17 dataset and 1.55% of the ISCX-Bot-14 dataset are botnet samples.The accuracy score is -obviously -an unsuitable metric for such highly imbalanced datasets, and hence not included in our table.Finding 1: Even with encrypted traffic a high detection performance can be achieved with different ML models.Without feature manipulation, all ML models show a very good detection performance even when restricted to the few features that are available with encrypted traffic.The high recall values (for the case of no manipulation -Tables 5-7, Manipulation: None) show that most of the botnet flows are detected by the models.Our SVM models have low precision and F1 scores for the clean test sets -this means that they have a high false positive rate (a high number of benign flows are classified as attack samples) -this does however not impact our baseline which is the recall score (number of botnet samples that are correctly classified).Finding 2: An attacker can evade detection by manipulating more than five forward features when using HSJ attack.The attacker can evade the detection of their own substitute model (to which they have an unlimited number of queries).Many flows that belong to the botnet traffic are classified as benign traffic by model A for both datasets.Finding 3: An attacker can evade detection by manipulating only one feature if they are using the FGS method.The attacker can evade the detection of their own substitute model (to which they have an unlimited number of queries).Many flows that belong to the botnet traffic are classified as benign traffic by model A for both datasets.However, all of these manipulated samples have feature values far outside of the normal range of botnet attacks and it can hence not be guaranteed, that the botnet attack would still be feasible.

Transferability for adversarial HSJ samples
We now check if the manipulated samples also evade the detection of the defender model D (only one single query possible).The decrease in the performance metrics can generally not be guaranteed when applying HSJ attack.This can easily be seen in Table 6, where the performance metrics remain high for most NIDS models and both datasets.Especially SVMs and MLPs seem hard to fool for both datasets.The curve progression of the recall values of all four NIDS models and the attackers model A can further be seen in Figure 5 for the CIC-IDS-17 dataset and in Figure 6 for the ISCX-Bot-14 dataset.On the x-axis, the manipulated features are shown.Each step means one additional feature is manipulated.The ideal would be to see a decrease as soon as possible with few features manipulated.The percentage of samples with feature values inside the feature range of clean botnet samples is also marked.In addition, the recall values  sample if two or more classifiers flag it as an attack sample.When applying a majority vote for HSJ, the ensemble NIDS outperforms all other classifiers (except for SVM in the ISCX-Bot-14 dataset) with regard to the detection performance of adversarial samples.In Figure 6, we want to draw attention to the unexpectedly bad results for the DT model for the ISCX-Bot-14 dataset.This DT can be fooled by manipulating only one feature and even seems to be easier to fool than the attacker's model A when only a few features are manipulated.Even though the decision tree shows good classification results on the clean data, its decision is based on very few feature values.DTs are hence very susceptible to minor changes in these feature values (if the right change is applied) but on the other hand also very robust if the change is in the wrong direction or on the wrong feature.The behavior of DTs is therefore hard to anticipate for an attacker.Finding 4: The attacker cannot generally evade the detection by a defender model when using HSJ.The performance metrics of all four victim models drop slightly for the first dataset (Figure 5) and vary more distinctly for the second dataset (Figure 6).Still, the detection rates of the victim models are way higher than the detection rate of the model the attacker has access to.Also, the high variance in detection performances (with a varying number of manipulated features) means, that if the attacker is lucky they could find samples that evade the detection but they could also fail with the evasion attempt without prior knowledge or guarantee.This means, as far as we can say with our two datasets, HSJ samples generally do not transfer well to other models.

Transferability for adversarial FGS samples
We now check if adversarial FGS samples would also evade the detection of the defender model D. The decrease in the performance metrics can be achieved when using FGS for the generation of the samples.This can be seen in Table 7, where the recall values drop drastically for all models except for DT in the CIC-IDS-17 dataset.
The decrease in the recall values of all four NIDS models and the attackers model A can further be seen in Figure 7 for the CIC-IDS-17 dataset and in Figure 8 for the ISCX-Bot-14 dataset.Please notice the MLP-and SVM-based NIDS in the CIC-IDS-17 dataset, which are fooled when only one feature value is manipulated whereas the RF-based NIDS is only fooled when a higher number of feature values is manipulated.We want to again draw attention to the unexpected behavior of the DT in Figure 7.This behavior can be explained by the same reasons mentioned in subsection 6.2.For the ISCX-Bot-14 dataset, the recall values for all defender models (except for RF) drop drastically when manipulating three features.When looking at the recall values of the ensemble vote of all four defender NIDS models in Figures 7 and 8, it is obvious, that the ensemble NIDS also fails at the detection of FGS samples, although it outperforms most of the other classifiers.The dashed line denotes the percentage of samples, that are inside the normal range of the botnet samples.This means that although the adversarial FGS samples transfer to other models, it cannot be guaranteed, that the botnet attacks would still be feasible.Finding 5: The attacker can evade the detection of a victim model when using FGS but the feature values of the manipulated samples lie far outside the normal ranges for botnet attacks.As can be seen in Figure 7, except for the DT all NIDS models fail to correctly detect the adversarial samples (the RF only with a higher number of manipulated features).Also, the recall scores of all four victim models approach zero for the second dataset (RF only when ten features are manipulated), which can be seen in Figure 8.This means the detection rates of the defender models can be reduced significantly without access (without any additional queries) to the NIDS.This means that the FGS samples generally transfer well to other models.This is only possible when altering the feature values drastically and hence moving the feature values far outside the normal distribution of botnet samples.We want to  further show the results for a variation in the order in which the features are manipulated.Figure 9 shows the recall value curve for model A and RF for the CIC-IDS-17 dataset and FGS attack when the order of the manipulation of features is guessed randomly (as in Figure 7).Furthermore, the same recall values are shown, if the attacker manages to guess the RF GINI feature importance of the defender model (which would be the best possible choice as the more important features should have more impact on the classifier's decision).As can be seen, the results do not differ much.This plot is exemplary for both evasion attacks and datasets.

Discussion and Limitations
Our experiments show, that highly effective evasion attacks against ML-based botnet detection are possible even without any preknowledge about the NIDS model or data and without previous tests or queries of the victim's NIDS.As a modern intrusion detection system could be alarmed after a few failed trials, we developed a framework where the attacker can use samples they manipulated before only by querying their own substitute model.Especially for the FGS Method, we show that adversarial samples can be transferred to victim models and reduce detection performance metrics significantly with the drawback of needing to alter the samples drastically.This hinders real-world deployment because a too drastic manipulation of features may contradict the botnet communication demands and also may be easy to detect.Furthermore, if the defender knows that an attacker is trying to evade the botnet detection with our model, this could lead to a never-ending cat-and-mouse game.Whenever the defender notices the evasion attack, they can adjust the model to detect the modified samples (e.g., through adversarial training), making it necessary for the attacker to modify the samples further to evade detection.

CONCLUSION
In this paper, we introduced a novel framework to evade the detection of botnet traffic in NIDS.We trained several ML-based models on two publicly available datasets (CIC-IDS-17 and ISCX-Bot-14) using flow feature vectors consisting only of features available in encrypted traffic.Our main objective was to test the model's classification performance on adversarial samples generated by applying two modern evasion attacks.We furthermore introduced a postprocessor where all manipulated samples are checked to ensure their plausibility and sanitized if needed.Our experiments show that even though all models have high detection rates on botnet traffic, the recall scores drop drastically after manipulating only a few features that an adversary can easily access.We generally prove that an attacker who applies the FGS method could even fool a victim model to which they have no access.In this case, the attacker can evade detection without knowing anything about the defender's model or data.Our proposed framework shows, that evasion attacks can be a real threat to ML-based NIDS as we efficiently generated adversarial samples that fooled the models with a high success rate while remaining valid attack samples.

Figure 5 :
Figure 5: Recall score for the detection of botnet Ares traffic in CIC-IDS-17 dataset with a varying number of features manipulated using HSJ attack.Each subsequent feature on the x-axis is manipulated in addition to all previous features.

Figure 6 :
Figure 6: Recall score for the detection of botnet Murlo traffic in ISCX-Bot-14 dataset with a varying number of features manipulated using HSJ attack.Each subsequent feature on the x-axis is manipulated in addition to all previous features.

Figure 7 :
Figure 7: Recall score for detecting botnet Ares traffic in the CIC-IDS-17 dataset with varying features manipulated using FGS attack.Each subsequent feature on the x-axis is manipulated in addition to all previous features.

Figure 8 :
Figure 8: Recall score for detecting botnet Murlo traffic in the ISCX-Bot-14 dataset with varying features manipulated using FGS attack.Each subsequent feature on the x-axis is manipulated in addition to all previous features.

Figure 9 :
Figure 9: Variation of recall value, when feature order is changed.The solid line remarks a random feature order, and the dashed line shows the recall value when using the feature importance of the defender's RF model (note that the attacker could not know this FI) for the CIC-IDS-17 dataset and FGS attack.
• The attacker can use multiple queries on their substitute model A but can only test the attack samples on the NIDS once.Without in-depth knowledge about the model and training data, the attacker does not know beforehand if their attack traffic will be detected or get through.• The attacker is not able to modify the training data for the NIDS and cannot interfere in the NIDS training process in any way.• The NIDS is not adjusted during the experiment.It is trained

Table 1 :
Comparison of Related Work 4.3 Post-Processor, Plausibility, and Bot Check

Table 2 :
, the extracted features are shown.For some features we extracted statistical properties of the feature values: min, max, mean, var, std, and mode.Those features are marked in the Stat.Description of datasets and the number of flows contained in the datasets.(statistics)column of the table.The features that can easily be accessed by an attacker -and hence manipulated -are marked in the Man.(manipulated) column.For our experiments, we only extract features, that are also available with IPsec and TLS encryption and restrict them further to timing and packet length-related features.

Table 3 :
Description of extracted features.Stat.means, that the statistics of the features are extracted, Man. marks, if the features are manipulated in our experiments.

Table 6 :
Performance metrics of defender models D for clean data and manipulated attack samples generated with HSJ attack.