Skip to main content
SpringerLink
Log in

A hybrid approach for mismatch data reduction in datasets and guide data mining

  • Published:
Cluster Computing Aims and scope Submit manuscript

Abstract

An outlier is a set of data that distinctly differ from rest of the data in a dataset defined as normal. Detection of outlier is an active area of research in data mining. If clustering methods are used, the elements that are lying outside the clusters are focused and detected as outliers. But it is not true few unknown elements will become a part of the cluster. So to ignore the irrelevant data completely from the data set, it becomes necessary to identify and eliminate these data merged with the clusters. An efficient hybrid approach is proposed to reduce the number of outliers. Two algorithms namely multilayer neural networks (MLN) and weighted-K means adopted for datamining are employed in proposed approach to identify outliers in a data group. This approach guides and results in better cluster formation. Each element in the dataset provided as input to MLN after assigning weights by weighted K-means. MLN is trained to reproduce the normal input data (inliers) and ensures that groups formed by weighted K-means are consisting of inliers only. Among the outlier detection methods presented in literature for outlier detection in data mining, the proposed method is based on Integrating Semantic Knowledge. This method relates the data point is an outlier by identifying the behaviour of the data elements that differ from other data elements belonging to the same cluster or class. The principle intention of this research work is to reduce the amount of outliers by enhancing the performance of clustering or classification techniques that guides to improve accuracy and reduce the mean square error. The test results provides evident to supremacy of the proposed strategy in reducing the outlier.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Han, J., Kamber, M.: Data Mining—Concepts & Techniques. Morgan Kaufmann Publishers, Academic Press (2001)

  2. Sankar Rajagopal, D.R.: Customer data clustering using data mining technique. Int. J. Database Manag. Syst. 3(4) (2011)

  3. Yabing, J.: Research of an improved apriori algorithm in data mining association rule. Int. J. Comput. Commun. Eng. 2(1), 25 (2013)

    Article  Google Scholar 

  4. Fournier-Viger, P., Lin, J.C.W., Kiran, R.U., Koh, Y.S., Thomas, R.: A survey of sequential pattern mining. Data Sci. Pattern Recognit. Ubiquitous Int. 1(1) (2017)

  5. Lin, L., Ye, J., Deng, F., Xiong, S., Zhong, L.: A comparison study of clustering algorithms for microblog posts. Cluster Comput. 19(3), 1333–1345 (2016)

    Article  Google Scholar 

  6. Kamila, N.K., Jena, L., Bhuyan, H.K.: Pareto-based multi-objective optimization for classification in data mining. Cluster Comput. 19(4), 1723–1745 (2016)

    Article  Google Scholar 

  7. Wang, J., Su, X.: An improved K-means clustering algorithm. In: 2011 IEEE 3rd International Conference on Communication Software and Networks, Xi’an, pp. 44–46 (2011)

  8. Fawcett, T., Provost, F.: Adaptive fraud detection. Data Min. Knowl. Discov. J. 1(3), 291–316 (1997)

    Article  Google Scholar 

  9. DuMouchel, W., Schonlau, M.: A fast computer intrusion detection algorithm based on hypothesis testing of command transition probabilities. In: Proceedings of the 4th International Conference on Knowledge Discovery and Data Mining, pp. 189–193 (1998)

  10. Williams, G., Huang, Z.: Advanced topics in artificial intelligence. In: Sattar, A. (ed.) Mining the Knowledge Mine: The Hot Spots Methodology for Mining Large Real World Databases. Lecture Notes in Artificial Intelligence, vol. 1342, pp. 340–348. Springer, Berlin (1997)

    Google Scholar 

  11. Yamanishi, K., Takeuchi, J., Williams, G., Milne, P.: On-line unsupervised outlier detection using finite mixtures with discounting learning algorithm. In: Proceedings of KDD2000, pp. 320–324 (2000)

  12. Breunig, M., Kriegel, H., Ng, R., Sander, J.: Lof: identifying density-based local outliers. In: Proceedings of ACM SIGMOD, International Conference on Management of Data (2000)

  13. Ramaswamy, S., Rastogi, R., Shim K.: Efficient algorithms for mining outliers from large data sets. In: Proceedings of International Conference on Management of Data, ACM-SIGMOD, Dallas (2000)

  14. Knorr, E., Ng, R.: Algorithms for mining distance-based outliers in large datasets. In: Proceedings of 24rd International Conference on Very Large Data Bases (VLDB), pp. 392–403 (1998)

  15. Atkinson, A.C.: Fast very robust methods for the detection of multiple outliers. J. Am. Stat. Assoc. 89, 1329–1339 (1994)

    Article  Google Scholar 

  16. Kosinksi, A.S.: A procedure for the detection of multivariate outliers. Comput. Stat. Data Anal. 29 (1999)

  17. Knorr, E., Ng, R.: A unified approach for mining outliers. In: Proceedings of KDD, pp. 219–222 (1997)

  18. Knorr, E., Ng, R.: Algorithms for mining distance-based outliers in large datasets. In: Proceedings of 24th International Conference on Very Large Data Bases, VLDB, pp. 392– 403, 24–27 (1998)

  19. Huang, J.Z., et al.: Automated variable weighting in k-means type clustering. IEEE Trans. Pattern Anal. Mach. Intell. 27(5), 657–668 (2005)

    Article  Google Scholar 

  20. Chan, E.Y., et al.: An optimization algorithm for clustering using weighted dissimilarity measures. Pattern Recognit. 37(5), 943–952 (2004)

    Article  Google Scholar 

  21. Huang, J.Z., et al.: Weighting method for feature selection in K-means. In: Computational Methods of Feature Selection , pp. 193–210 (2008)

  22. de Amorim, R.C., Mirkin, B.: Minkowski metric, feature weighting and anomalous cluster initializing in K-means clustering. Pattern Recognit. 45(3), 1061–1075 (2012)

    Article  Google Scholar 

  23. Hung, E., Cheung, D.W.: Parallel mining of outliers in large database. Distrib. Parallel Databases 12(1), 5–26 (2002)

    Article  Google Scholar 

  24. Lozano, E., Acuna, E.: Parallel algorithms for distance-based and density-based outliers. In: Proceedings of Fifth IEEE International Conference on Data Mining (ICDM), pp. 729–732 (2005)

  25. Bay, S.D., Schwabacher, M.: Mining distance-based outliers in near linear time with randomization and a simple pruning rule. In: Proceedings of Ninth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD) (2003)

  26. Ng, R., Han, J.: Efficient and effective clustering methods for spatial data mining. In: Proceedings of 20th VLDB, pp. 144–155 (1994)

  27. Ester, M., Kriegel, H.P., Sander, J., Xu, X.: A density-based algorithm for discovering clusters in large spatial databases with noise. In: Proceedings of KDD, pp. 226–231 (1999)

  28. Zhang, T., Ramakrishnan, R., Livny M.: An efficient data clustering method for very large databases. In: Proceedings of ACM SIGMOD, pp. 103–114 (1996)

  29. Kollios, G., Gunopoulos, D., Koudas, N., Berchtold, S.: An efficient approximation scheme for data mining tasks. In: ICDE (2001)

  30. Bartkowiak, A., Szustalewicz, A.: Detecting multivariate outliers by a grand tour. Mach. Graph. Vis. 6(4), 487–505 (1997)

    Google Scholar 

  31. Williams, G., Altas, I., Bakin, S., Christen, P., Hegland, Markus, Marquez, Alonso, Milne, Peter, Nagappan, Rajehndra, Roberts, Stephen: Large-scale parallel data mining, LNAI state-of-the art survey. In: Zaki, M.J., Ho, C.-T. (eds.) The Integrated Delivery of Large-Scale Data Mining: The ACSys Data Mining Project, pp. 24–54. Springer, Berlin (2000)

    Google Scholar 

  32. Swayne, D.F., Cook, D., Buja A.: XGobi: interactive dynamic graphics in the X window system with a link to S. In: Proceedings of the ASA Section on Statistical Graphics, pp. 1–8, Alexandria, VA. American Statistical Association (1991)

  33. Sykacek, P.: Equivalent error bars for neural network classifiers trained by Bayesian inference. In: Proceedings of ESANN (1997)

  34. Ackley, D.H., Hinton, G.E., Sejinowski, T.J.: A learning algorithm for boltzmann machines. Cognit. Sci. 9, 147–169 (1985)

    Article  Google Scholar 

  35. Hecht-Nielsen, R.: Replicator neural networks for universal optimal source coding. Science 269, 1860–1863 (1995)

    Article  Google Scholar 

  36. Hampel, F.R.: The influence curve and its role in robust estimation. J. Am. Stat. Assoc. 69, 383–393 (1974)

    Article  MathSciNet  Google Scholar 

  37. Hawkins, S., He, H., Williams, G.J., Baxter, R.A.: DaWaK 2002. In: Kambayashi, Y., Winiwarter, W., Arikawa, M. (eds.) Outlier detection using replicator neural networks. LNCS, vol. 2454, pp. 170–180. Springer, Heidelberg (2002)

    MATH  Google Scholar 

  38. Zhao, X., Liang, J., Cao, F.: A simple and effective outlier detection algorithm for categorical data. Int. J. Mach. Learn. Cyber. 5, 469–477 (2014)

  39. Zengyou, H., Shengchun, D., Xiaofei, X., Huang, J.Z.: A fast greedy algorithm for outlier mining. Applications of Evolutionary Computing. In: Proceedings of the EvoWorkshops 2006: EvoBIO, EvoCOMNET, EvoHOT EvoIASP, EvoINTERACTION, EvoMUSART, and EvoSTOC. LNCS, vol. 3907, pp. 567–576 (2006)

  40. Zhang, W., Wu, J., Yu, J.: An improved method of outlier detection based on frequent pattern. In: Proceeding of WASE International Conference on Information Engineering (2010)

  41. Otey, M.E., Ghoting, A., Parthasarathy, A.: Fast distributed outlier detection in mixed-attribute data sets. Data Min. Knowl. Discov. (2006)

Download references

Author information

Authors and Affiliations

  1. Department of Computer Science and Engineering, RMK Engineering College, Chennai, India

    R. Dhanalakshmi & T. Sethukarasi

Authors
  1. R. Dhanalakshmi
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. T. Sethukarasi
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to R. Dhanalakshmi.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Dhanalakshmi, R., Sethukarasi, T. A hybrid approach for mismatch data reduction in datasets and guide data mining. Cluster Comput 22 (Suppl 5), 10605–10614 (2019). https://doi.org/10.1007/s10586-017-1137-4

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10586-017-1137-4

Keywords

Search

Navigation