Skip to main content
SpringerLink
Log in

IAMonSense: multi-level handwriting classification using spatiotemporal information

  • Special Issue Paper
  • Published:
International Journal on Document Analysis and Recognition (IJDAR) Aims and scope Submit manuscript

Abstract

Online handwriting classification has become an open research problem as it serves as a preliminary step for handwriting recognition systems and applications in several other fields. This paper aims to extend the current trends and knowledge with multiple contributions in handwriting classification using spatiotemporal information. Firstly, it enriches the annotations of several publicly available online handwriting datasets, SenseThePen, IAM-onDB, and IAMonDo, for online handwriting classification and recognition tasks. The said datasets are updated with three distinguished levels of annotations, i.e., stroke, sequence, and line levels. The enriched annotations of these datasets extend their functionality for online handwriting classification at different levels for further research analysis. In addition to enrichment, it also unifies the annotation levels across the datasets, which enables the research community to benchmark proposed methods for comparative analysis using multiple datasets. All the datasets with enriched annotations are made publicly available for the research community as part of the IAMonSense dataset. Moreover, this paper presents a comprehensive benchmark of these datasets using multiple deep neural networks such as traditional convolutional neural networks (CNNs), graph convolutional networks(GCNs), attention-based neural networks, and transformers. These benchmarks can be used later on for further development in online handwriting classification.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9
Fig. 10
Fig. 11
Fig. 12
Fig. 13

Similar content being viewed by others

References

  1. Liwicki, M., Bunke, H.: Recognition of Whiteboard Notes: Online, Offline and Combination. Series in machine perception and artificial intelligence. World Scientific, Singapore (2008). https://doi.org/10.1142/6854

  2. Liwicki, M., Bunke, H.: Feature selection for hmm and BLSTM based handwriting recognition of whiteboard notes. IJPRAI 23, 907–923 (2009). https://doi.org/10.1142/S0218001409007417

    Article  Google Scholar 

  3. Mandal, S., Prasanna, S.R.M., Sundaram, S.: GMM posterior features for improving online handwriting recognition. Expert Syst. Appl. 97, 421–433 (2018). https://doi.org/10.1016/j.eswa.2017.12.047

    Article  Google Scholar 

  4. Van Phan, T., Nakagawa, M.: Text/non-text classification in online handwritten documents with recurrent neural networks. In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 23–28 (2014). https://doi.org/10.1109/ICFHR.2014.12

  5. Ye, J.-Y., Zhang, Y.-M., Yang, Q., Liu, C.-L.: Contextual stroke classification in online handwritten documents with edge graph attention networks. SN Comput. Sci. 1(3), 1–13 (2020)

    Article  Google Scholar 

  6. Weber, M., Liwicki, M., Schelske, Y.T.H., Schoelzel, C., Strauß, F., Dengel, A.: MCS for Online Mode Detection: Evaluation on Pen-Enabled Multi-touch Interfaces. In: 2011 International Conference on Document Analysis and Recognition, pp. 957–961 (2011). https://doi.org/10.1109/ICDAR.2011.194

  7. Kholmatov, A., Yanikoglu, B.: Identity authentication using improved online signature verification method. Pattern Recogn. Lett. 26(15), 2400–2408 (2005)

    Article  Google Scholar 

  8. Liwicki, M., Schlapbach, A., Bunke, H., Bengio, S., Mariéthoz, J., Richiardi, J.: Writer Identification for Smart Meeting Room Systems. Technical report (2005). Published in Seventh IAPR Workshop on Document Analysis Systems, DAS, 2006. http://publications.idiap.ch/downloads/reports/2005/liwiki-idiap-rr-05-70.pdf

  9. Said, H.E.S., Baker, K.D., Tan, T.N.: Personal identification based on handwriting. In: Proceedings. Fourteenth International Conference on Pattern Recognition (Cat. No.98EX170), vol. 2, pp. 1761–17642 (1998). https://doi.org/10.1109/ICPR.1998.712068

  10. Sesa-Nogueras, E., Faundez-Zanuy, M., Roure-Alcobé, J.: Gender classification by means of online uppercase handwriting: a text-dependent allographic approach. Cogn. Comput. 8(1), 15–29 (2016). https://doi.org/10.1007/s12559-015-9332-1

    Article  Google Scholar 

  11. Younas, J., Malik, M.I., Ahmed, S., Shafait, F., Lukowicz, P.: Sense the pen: classification of online handwritten sequences (text, mathematical expression, plot/graph). Expert Syst. Appl. 172, 114588 (2021). https://doi.org/10.1016/j.eswa.2021.114588

    Article  Google Scholar 

  12. Younas, J., Fritsch, S., Pirkl, G., Ahmed, S., Malik, M.I., Shafait, F., Lukowicz, P.: What am i writing: Classification of on-line handwritten sequences. In: Intelligent Environments (Workshops). Ambient Intelligence and Smart Environments, vol. 23, pp. 417–426. IOS Press, Amsterdam (2018)

  13. Delaye, A., Liu, C.-L.: Contextual text/non-text stroke classification in online handwritten notes with conditional random fields. Pattern Recognit. 47(3), 959–968 (2014). https://doi.org/10.1016/j.patcog.2013.04.017

    Article  Google Scholar 

  14. Van Phan, T., Nakagawa, M.: Combination of global and local contexts for text/non-text classification in heterogeneous online handwritten documents. Pattern Recogn. 51, 112–124 (2016). https://doi.org/10.1016/j.patcog.2015.07.012

    Article  Google Scholar 

  15. Bresler, M., Průša, D., Hlaváč, V.: Online recognition of sketched arrow-connected diagrams. Int. J. Doc. Anal. Recognit. (IJDAR) 19(3), 253–267 (2016). https://doi.org/10.1007/s10032-016-0269-z

    Article  Google Scholar 

  16. Indermühle, E., Liwicki, M., Bunke, H.: IAMonDo-database: an online handwritten document database with non-uniform contents. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems. DAS ’10, pp. 97–104. Association for Computing Machinery, New York (2010). https://doi.org/10.1145/1815330.1815343

  17. Liwicki, M., Bunke, H.: IAM-OnDB - an On-Line English Sentence Database Acquired from Handwritten Text on a Whiteboard. In: Proceedings of the Eighth International Conference on Document Analysis and Recognition. ICDAR ’05, pp. 956–961. IEEE Computer Society, USA (2005). https://doi.org/10.1109/ICDAR.2005.132

  18. Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural Comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735

    Article  Google Scholar 

  19. Graves, A., Schmidhuber, J.: Framewise phoneme classification with bidirectional lstm networks. In: Proceedings of 2005 IEEE International Joint Conference on Neural Networks, 2005., vol. 4, pp. 2047–2052 (2005). IEEE

  20. Kipf, T.N., Welling, M.: Semi-Supervised Classification with Graph Convolutional Networks. arXiv (2016). arXiv:1609.02907

  21. Veličković, P., Cucurull, G., Casanova, A., Romero, A., Liò, P., Bengio, Y.: Graph Attention Networks (2017). arXiv arXiv:1710.10903

  22. Brody, S., Alon, U., Yahav, E.: How Attentive are Graph Attention Networks? (2021). arXiv arXiv:2105.14491

  23. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., Polosukhin, I.: Attention Is All You Need (2017). arXiv arXiv:1706.03762

  24. Livingston, O.B.: A handwriting and pen-printing classification system for identifying law violators. J. Criminal Law Criminol. 49, 487 (1959)

    Google Scholar 

  25. Moore, J.: Handwriting classification. Police J. 18(1), 39–61 (1945). https://doi.org/10.1177/0032258X4501800107

    Article  MathSciNet  Google Scholar 

  26. Tappert, C.C., Suen, C.Y., Wakahara, T.: The state of the art in online handwriting recognition. IEEE Trans. Pattern Anal. Mach. Intell. 12(8), 787–808 (1990). https://doi.org/10.1109/34.57669

    Article  Google Scholar 

  27. Plamondon, R., Srihari, S.N.: Online and off-line handwriting recognition: a comprehensive survey. IEEE Trans. Pattern Anal. Mach. Intell. 22(1), 63–84 (2000). https://doi.org/10.1109/34.824821

    Article  Google Scholar 

  28. Pham, V., Bluche, T., Kermorvant, C., Louradour, J.: Dropout Improves Recurrent Neural Networks for Handwriting Recognition. In: 2014 14th International Conference on Frontiers in Handwriting Recognition, pp. 285–290 (2014). https://doi.org/10.1109/ICFHR.2014.55

  29. Michal, Kozielski, M., Doetsch, P., Ney, H.: Improvements in RWTH’s System for Off-Line Handwriting Recognition. In: 2013 12th International Conference on Document Analysis and Recognition, pp. 935–939 (2013). https://doi.org/10.1109/ICDAR.2013.190

  30. Carbune, V., Gonnet, P., Deselaers, T., Rowley, H.A., Daryin, A., Calvo, M., Wang, L.-L., Keysers, D., Feuz, S., Gervais, P.: Fast multi-language LSTM-based online handwriting recognition. Int. J. Doc. Anal. Recognit. (IJDAR) 23(2), 89–102 (2020)

    Article  Google Scholar 

  31. Nguyen, C.T., Nakagawa, M.: Finite state machine based decoding of handwritten text using recurrent neural networks. In: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), pp. 246–251 (2016). https://doi.org/10.1109/ICFHR.2016.0055

  32. Shivram, A., Ramaiah, C., Govindaraju, V.: A hierarchical Bayesian approach to online writer identification. Iet Biomet. 2(4), 191–198 (2013)

    Article  Google Scholar 

  33. Shivram, A., Ramaiah, C., Govindaraju, V.: Data sufficiency for online writer identification: A comparative study of writer-style space vs. feature space models. In: 2014 22nd International Conference on Pattern Recognition, pp. 3121–3125 (2014). https://doi.org/10.1109/ICPR.2014.538

  34. Ahmed, M., Rasool, A.G., Afzal, H., Siddiqi, I.: Improving handwriting based gender classification using ensemble classifiers. Expert Syst. Appl. 85, 158–168 (2017). https://doi.org/10.1016/j.eswa.2017.05.033

    Article  Google Scholar 

  35. Illouz, E., Netanyahu, N.S., et al.: Handwriting-based gender classification using end-to-end deep neural networks. In: International Conference on Artificial Neural Networks, pp. 613–621. Springer (2018)

  36. Ye, J.-Y., Zhang, Y.-M., Liu, C.-L.: Joint training of conditional random fields and neural networks for stroke classification in online handwritten documents. In: 2016 23rd International Conference on Pattern Recognition (ICPR), pp. 3264–3269 (2016). https://doi.org/10.1109/ICPR.2016.7900138

  37. Degtyarenko, I., Deriuga, I., Grygoriev, A., Polotskyi, S., Melnyk, V., Zakharchuk, D., Radyvonenko, O.: Hierarchical recurrent neural network for handwritten strokes classification. In: ICASSP 2021 - 2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 2865–2869 (2021). https://doi.org/10.1109/ICASSP39728.2021.9413412

  38. Ott, F., Wehbi, M., Hamann, T., Barth, J., Eskofier, B., Mutschler, C.: The OnHW Dataset: Online Handwriting Recognition from IMU-Enhanced Ballpoint Pens with Machine Learning. Proc. ACM Interact. Mob. Wearable Ubiquitous Technol. 4(3) (2020). https://doi.org/10.1145/3411842

  39. Hamdi, Y., Boubaker, H., Alimi, A.M.: Data Augmentation Using Geometric, Frequency, and Beta Modeling Approaches for Improving Multi-Lingual Online Handwriting Recognition. Int. J. Doc. Anal. Recognit. 24(3), 283–298 (2021). https://doi.org/10.1007/s10032-021-00376-2

    Article  Google Scholar 

  40. Tang, W., Long, G., Liu, L., Zhou, T., Jiang, J., Blumenstein, M.: Rethinking 1d-cnn for time series classification: a stronger baseline (2020). arXiv preprint arXiv:2002.10061

  41. Liu, M., Ren, S., Ma, S., Jiao, J., Chen, Y., Wang, Z., Song, W.: Gated Transformer Networks for Multivariate Time Series Classification (2021). arXiv arXiv:2103.14438

  42. Wang, M., Zheng, D., Ye, Z., Gan, Q., Li, M., Song, X., Zhou, J., Ma, C., Yu, L., Gai, Y., Xiao, T., He, T., Karypis, G., Li, J., Zhang, Z.: Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks (2019). arXiv:1909.01315

Download references

Author information

Author notes

  1. Ahmad Mustafid and Junaid Younas have contributed equally to this work.

Authors and Affiliations

  1. Computer Science Department, Technische Universität (TU), Kaiserslautern, 67663, Germany

    Ahmad Mustafid, Junaid Younas & Paul Lukowicz

  2. German Research Centre for Artificial Intelligence (DFKI), Kaiserslautern, 67663, Germany

    Junaid Younas, Paul Lukowicz & Sheraz Ahmed

  3. School of Electrical Engineering and Computer Science, NUST, Islamabad, Pakistan

    Junaid Younas

Authors
  1. Ahmad Mustafid
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Junaid Younas
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Paul Lukowicz
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Sheraz Ahmed
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Junaid Younas.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Appendix A: Classification result error

Appendix A: Classification result error

Figures 12 and 13 highlight the classification result error. The blue color indicates text, the green color indicates math, and the black color indicates graph. In Fig. 13, red color was added to see the error location easily.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Mustafid, A., Younas, J., Lukowicz, P. et al. IAMonSense: multi-level handwriting classification using spatiotemporal information. IJDAR 26, 303–319 (2023). https://doi.org/10.1007/s10032-023-00433-y

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10032-023-00433-y

Keywords

Search

Navigation