Abstract
Anomaly detection involves identifying observations that deviate from the normal behavior of a system. One of the ways to achieve this is by identifying the phenomena that characterize “normal” observations. Subsequently, based on the characteristics of data learned from the “normal” observations, new observations are classified as being either “normal” or not. Most state-of-the-art approaches, especially those which belong to the family of parameterized statistical schemes, work under the assumption that the underlying distributions of the observations are stationary. That is, they assume that the distributions that are learned during the training (or learning) phase, though unknown, are not time-varying. They further assume that the same distributions are relevant even as new observations are encountered. Although such a “stationarity” assumption is relevant for many applications, there are some anomaly detection problems where stationarity cannot be assumed. For example, in network monitoring, the patterns which are learned to represent normal behavior may change over time due to several factors such as network infrastructure expansion, new services, growth of user population, and so on. Similarly, in meteorology, identifying anomalous temperature patterns involves taking into account seasonal changes of normal observations. Detecting anomalies or outliers under these circumstances introduces several challenges. Indeed, the ability to adapt to changes in nonstationary environments is necessary so that anomalous observations can be identified even with changes in what would otherwise be classified as “normal” behavior. In this article we propose to apply a family of weak estimators for anomaly detection in dynamic environments. In particular, we apply this theory to spam email detection. Our experimental results demonstrate that our proposal is both feasible and effective for the detection of such anomalous emails.
- Androutsopoulos, I., Koutsias, J., Chandrinos, K., and Spyropoulos, C. 2000. An experimental comparison of naive Bayesian and keyword-based anti-spam filtering with personal e-mail messages. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM New York, 160--167. Google Scholar
Digital Library
- Chandola, V., Banerjee, A., and Kumar, V. 2009. Anomaly detection: A survey. ACM Comput. Surv. To appear. Google Scholar
Digital Library
- Chopra, M., Martin, M., Rueda, L., and Hung, P. 2006. Toward new paradigms to combating Internet child pornography. In Proceedings of the Canadian Conference on Electrical and Computer Engineering (CCECE’06). 1012--1015.Google Scholar
- Didion, J. 2004. The Java WordNet Library. http://jwordnet.sourceforge.net/.Google Scholar
- Enron. 2006. Enron-spam dataset. http://www.aueb.gr/ users/ion/data/enron-spam/.Google Scholar
- Guzella, T. and Caminhas, W. 2009. A review of machine learning approaches to spam filtering. Expert Syst. Appl. To appear. Google Scholar
Digital Library
- Kong, J., Rezaei, B., Sarshar, N., Roychowdhury, V., and Boykin, P. 2006. Collaborative spam filtering using e-mail networks. Computer 39, 8, 67--73. Google Scholar
Digital Library
- Kushner, H. and Yin, G. 2003. Stochastic Approximation and Recursive Algorithms and Applications 2nd Ed. Springer, Berlin.Google Scholar
- McGregor, C. 2007. Controlling spam with spamassassin. Linux J. 153, 9. Google Scholar
Digital Library
- Metsis, V., Androutsopoulos, I., and Paliouras, G. 2006. Spam filtering with naive Bayes -- Which naive Bayes. In Proceedings of the 3rd Conference on Email and Anti-Spam (CEAS). 125--134.Google Scholar
- Miller, A. 1995. Wordnet: A lexical database for English. Comm. ACM 38, 11, 39--41. Google Scholar
Digital Library
- Mladenić, D., Brank, J., Grobelnik, M., and Milic-Frayling, N. 2004. Feature selection using linear classifier weights: Interaction with classification models. In Proceedings of the 27th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval. ACM, New York, 234--241. Google Scholar
Digital Library
- Narendra, K. and Thathachar, M. 1989. Learning Automata. An Introduction. Prentice Hall, Englewood Cliffs, NJ. Google Scholar
Digital Library
- Norris, J. 1999. Markov Chains. Springer, Berlin.Google Scholar
- Oommen, B. and Misra, S. 2006. A fault-tolerant routing algorithm for mobile ad hoc networks using a stochastic learning-based weak estimation procedure. In Proceedings of the IEEE International Conference on Wireless and Mobile Computing, Networking and Communications. IEEE, Los Alamitos, CA, 31--37. Google Scholar
Digital Library
- Oommen, B. and Rueda, L. 2006. Stochastic learning-based weak estimation of multinomial random variables and its applications to pattern recognition in non-stationary environments. Pattern Recogn. 39, 3, 328--341. Google Scholar
Digital Library
- Open NLP. 2008. Open NLP. http://opennlp.sourceforge.net.Google Scholar
- Rueda, L. and Oommen, B. 2006. Stochastic automata-based estimators for adaptively compressing files with nostationary distributions. IEEE Trans, Syst. Man, Cybern. Part B 36, 5, 1196--1200. Google Scholar
Digital Library
- Sebastiani, F. 2002. Machine learning in automated text categorization. ACM Comput. Surv. 34, 1, 1--47. Google Scholar
Digital Library
- Wang, B., Jones, G., and Pan, W. 2006. Using online linear classifiers to filter spam emails. Pattern Anal. Appl. 9, 4, 339--351. Google Scholar
Digital Library
- Watkins, C. 1989. Learning from delayed rewards. Ph.D. dissertation, University of Cambridge, UK.Google Scholar
- Yang, Y. and Pedersen, J. O. 1997. A comparative study on feature selection in text categorization. In Proceedings of the 14th International Conference on Machine Learning (ICML’97). D. H. Fisher Ed., Morgan Kaufmann, San Francisco, CA, 412--420. Google Scholar
Digital Library
- Zhan, J., Oommen, J., and Crisostmo, J. 2009. Anomaly detection in dynamic social email systems. In Proceedings of the IEEE International Conference on Social Computing. IEEE, Los Alamitos, CA.Google Scholar
- Zhang, L., Zhu, J., and Yao, T. 2004. An evaluation of statistical spam filtering techniques. ACM Trans. Asian Lang. Inf. Process. 3, 4, 243--269. Google Scholar
Digital Library
Index Terms
Anomaly Detection in Dynamic Systems Using Weak Estimators
Recommendations
Anomaly Detection in Dynamic Social Systems Using Weak Estimators
CSE '09: Proceedings of the 2009 International Conference on Computational Science and Engineering - Volume 04Anomaly detection involves identifying observationsthat deviate from the normal behavior of a system. One ofthe ways to achieve this is by identifying the phenomena thatcharacterize “normal” observations. Subsequently, based on thecharacteristics of ...
Two-stage anomaly detection algorithm via dynamic community evolution in temporal graph
AbstractDetecting anomalies from a massive amount of user behavioral data is often liken to finding a needle in a haystack. While tremendous efforts have been devoted to anomaly detection from temporal graphs, existing studies rarely consider community ...
Anomaly Detection in Embedded Systems
Special issue on fault-tolerant embedded systemsBy employing fault tolerance, embedded systems can withstand both intentional and unintentional faults. Many fault-tolerance mechanisms are invoked only after a fault has been detected by whatever fault-detection mechanism is used, hence, the process of ...






Comments