Most existing network intrusion detection systems use signature-based methods which depend on labeled training data. This training data is usually expensive to produce due to cost of laboratory set up, experienced or knowledge person and non availability of ready software tool. Above all, these methods have difficulty in detecting new or unknown types of attacks. Using unsupervised anomaly detection techniques, however, the system is capable of detecting previously unknown attacks without labeled training data. In this paper, we have discussed anomaly based network intrusion detection and proposed two unsupervised clustering algorithms for anomaly detection. The algorithms are evaluated with our generated real life intrusion dataset. The dataset is created with extracted features of captured network packet as well as flow traffic. The algorithm is also tested and validated with standard KDD Cup 1999 dataset and NSL-KDD dataset. The results are compared with results of similar algorithms and have been found excellent.

1. Bace, R. and Mell, P. (2001). Intrusion detection systems. NIST Special Publications SP 800, U S Department of Defence, 31 November 2001.
2. Lee, W. and Stolfo, S. J. (1998) Data mining approaches for intrusion detection. Proceedings of the 7th conference on USENIX Security Symposium-Volume 7, San Antonio, Texas, USA, Jan., pp. 6–6. USENIX.
3. Roesch, M. (1999) Snort-lightweight intrusion detection for networks. Proceedings of the 13th USENIX conference on System administration, Seattle, Washington, Nov., pp. 229–238. USENIX.
4. Portnoy, L., Eskin, E., and Stolfo, S. J. (2001) Intrusion detection with unlabeled data using clustering. In Proc. of the ACM CSS workshop DMSA-2001, Philadelphia PA, November 8, pp. 5–8. ACM.
5. Daniel, B., Julia, C., Sushil, J., and Ningning, W. (2001) Adam: a testbed for exploring the use of data mining in intrusion detection. SIGMOD Rec., 30, 15–24.
6. Eskin, E., Arnold, A., Prerau, M., Portnoy, L., and Stolfo, S. (2002) A geometric framework for unsupervised anomaly detection. Applications of Data Mining in Computer Security, Norwell, MA, USA, Dec., pp. 78–100. Kluwer Academic Publishers.
7. Smith, R., Bivens, A., Embrechts, M., Palagiri, C., and Szymanski, B. (2002) Clustering approaches for anomaly based intrusion detection. Proc. of Walter Lincoln Hawkins Graduate Research Conference 2002, New York, USA, October.
8. Leung, K. and Leckie, C. (2005) Unsupervised anomaly detection in network intrusion detection using clusters. Proc. of 28 Australasian conference on Computer Science - Volume 38, Newcastle, NSW, Australia, January/February, pp. 333–342. Australian Computer Society, Inc. Darlinghurst.
9. Leon, E., Nasraoui, O., and Gomez, J. (2004) Anomaly detection based on unsupervised niche clustering with application to network intrusion detection. IEEE Congres on Evolutionary Computation, 1, 502–508.
10. Chimphlee, W., Abdullah, A. H., Sap, M. N. M.,Chimphlee, S., and Srinoy, S. (2005) Unsupervised clustering methods for identifying rare events in anomaly detection. Proc. of World Academy of Science, Engineering and Technology, October.
11. Zhong, S., Khoshgoftaar, T., and Seliya, N. Clustering-based network intrusion detection. Int’nl J of Reliability, Quality and Safety Engineering,14.
12. Zhang, C., Zhang, G., and Sun, S. (2009) A mixed unsupervised clustering-based intrusion detection. Proc. of 3rd International Conference on Genetic and Evolutionary Computing, WGEC 2009, Gulin, China, 14-17 October. IEEE Computer Society.
13. Hettich, S. and Bay, S. D. (1999). The uci kdd archive. Irvine, CA:University of California, Department of Information and Computer Science, http://kdd.ics.uci.edu.
14. Tavallaee, M., Bagheri, E., Lu, W., and Ghorbani, A. A. (2009). A detailed analysis of the kdd cup 99 data set. Availble on: http://nsl.cs.unb.ca/NSL-KDD/.
15. Eskin, E. (2000) Anomaly detection over noisy data using learned probability distributions. Proceedings of the 7th International Conference on Machine Learning, Stanford University, Stanford, CA, USA, June 29-July 2, pp. 255–262. Morgan Kaufmann Publishers Inc.
16. 16. Oldmeadow, J., Ravinutala, S., and Leckie, C. (2004) Adaptive clustering for network intrusion detection. In Proc. of the PAKDD 2004, Sydney, Australia, May 26-28, pp. 255–259. LNCS 3056 Springer 2004, ISBN 3-540-22064-X.
17. Guan, Y., Ghorbani, A., and Belacel, N. (2003) Ymeans: A clustering method for intrusion detection. Proc. of the Canadian Conference on Electrical and Computer Engineering, Montreal, Quebec, Canada, May 4-7.
18. Lu,W. and Traore, I. Unsupervised anomaly detection using an evolutionary extension of k-means algorithm. Int. J. Information and Computer Security, 2.
19. Nagesh, H. S., Goil, S., and Choudhary, A. N. (2000) A scalable parallel subspace clustering algorithm for massive data sets. Proc. of the ICPP 2000, Toronto, Canada, 21-24 August 477. IEEE Computer Society.
20. Burbeck, K. and Nadjm-Tehrani, S. (2005) Adwice -anomaly detection with real-time incremental clustering. Proceedings of Information Security and Cryptology -ICISC 2004, Berlin , Germany, May, pp. 407–424. Springer Berlin / Heidelberg.
21. Zhang, T., Ramakrishnan, R., and Livny, M. (1996) Birch: an efficient data clustering method for very large databases. Proc. of the 1996 ACM SIGMOD, Montreal, Quebec, Canada, June 4-6, pp. 103–114. ACM Press.
22. Proto, A., Alexandre, L. A., Batista, M. L., Oliveira, I. L., and Cansian, A. M. Statistical model applied to netflow for network intrusion detection. LNCS, 6480.
23. Diaz-Verdejo, P. G.-T. J., Macia-Fernandez, G., and Vazquez, E. Anomaly-based network intrusion detection: Techniques, systems and challenges. Computers & Security, 28.
24. Gogoi, P., Borah, B., and Bhattacharyya, D. K. Anomaly detection analysis of intrusion data using supervised & unsupervised approach. Journal of Convergence Information Technology, 5.
25. Gogoi, P., Bhattacharyya, D. K., Borah, B., and Kalita, J. K. A survey of outlier detection methods in network anomaly identification. The Computer Journal, 54.
26. Cha, S.-H. (2007) Comprehensive survey on distance/ similarity measures between probability density functions. nternational Journal of Mathematical Models and Methods in Applied Science, 1 (4), 300–307.
27. Boriah, S., Chandola, V., and Kumar, V. (2008) Similarity measures for categorical data: A comparative evaluation. Proceedings of the 8th SIAM International Conference on Data Mining, Atlanta, Georgia, USA, Apr., pp. 243–254. Society for Industrial and Applied Mathematics(SIAM).
28. T. S. Chou and L. J. K. K. Yen, “Network intrusion detection design using feature selection of soft computing paradigms,” International Journal of computational Intelligence, vol. 4, pp. 196–208, 2008.
29. S. Mukkamala and A. H. Sung, “Significant feature selection using computational intelligent techniques for intrusion detection,” Advanced Method for Knowledge Discovery from Complex Data, vol. Part II, pp. 285–306, 2005.
30. F. Amiri, M. M. R. Yousefi, C. Lucas, A. Shakery, and N. Yazdani, “Mutual information-based feature selection for intrusion detection systems,” Journal of Network and Computer Applications, vol. 34, pp. 1184-1199, 2011.
31. H.G. Kayacik, A.N. Zincir-Heywood, and M.I. Heywood, “Selecting Features for Intrusion Detection: A Feature Relevance Analysis on KDD 99 Intrusion Detection Datasets”, In Proceedings of the 3rd Annual Conference on Privacy, Security and Trust (PST-2005), Oct., 2005.
32. Mixter (2003). Attacks tools and information. http://packetstormsecurity.nl/index.html.
33. 33 Satten, C. (2008). Lossless gigabit remote packet capture with linux. http://staff.washington.edu/corey/gulp/, University of Washington Network Systems.
34. 34 (2009). Wireshark. http://www.wireshark.org/.
35. 35 Osterman, S. (2009). Tcptrace. http://www.tcptrace.org.
36. 36 Quittek, J., Zseby, T., Claise, B., and Zender, S. (2004). Rfc 3917: Requirements for ipflow information export: Ipfix, hawthorn victoria. http://www.ietf.org/rfc/rfc3917.txt.
37. 37 Claise, B. (2004). Rfc 3954: Cisco systems netflow services export version 9. http://www.ietf.org/rfc/rfc3954.txt.
38. 38 Haag, P. (2010). Nfdump & nfsen. http://nfdump.sourceforge.net/.
39. 39 Cisco.com (2010). Cisco ios netflow configuration guide, release 12.4. http://www.cisco.com.
40. 40 Tan, P.-N., Steinbach, M., and Kumar, V. (2009) Introduction to Data Mining. Pearson Education, Inc., Newyork, USA.

Intrusion Unsupervised Supervised Anomaly Clustering TPR FPR Supervised Anomaly Clustering TPR FPR