Despite receiving a lot of attention from the document analysis community, defining a robust handwriting representation remains to this day an unsolved problem. Such representation cannot be defined explicitly using a set of rules, and should instead be obtained from the intelligent extraction of high level features from document images. In this thesis, deep learning models are investigated to learn a representation for handwriting. The learned representations are used to define a system for recognizing and spotting handwritten words. The choice of handling segmented words is motivated by the fact that any written text can be segmented into a set of separated word images.
In our first contribution, an unsupervised deep handwriting representation is introduced for the task of word spotting. This representation is based on the spherical k-means algorithm, which is used to build a hierarchy of parametric functions encoding document images. The advantages of this representation are diverse. First, it is defined automatically in unsupervised manner, thereby avoiding the need for labeled data for training. It can also be quickly calculated and has a compact size, which enables efficient word spotting.
In our second contribution, an end-to-end model for handwritten word recognition is presented. This model is composed of a deep convolutional neural network (CNNs), which takes as input a word image and produces as output a representation for the recognized text. This text representation, called pyramid of bidirectional character sequences, encodes sub-sequences of characters in a hierarchical manner, considering both forward and backward directions. It has several advantages compared to existing approaches. It is binary and has a fixed size, making it robust to the text length. Moreover, it captures the distribution of character sub-sequences in the training document corpus, which allows the trained model to transfer this knowledge to new words containing the same sub-sequences.
In our third and last contribution, an end-to-end model is proposed to solve the word recognition and the word spotting tasks simultaneously. This model is used for the joint embedding of handwritten word texts and images. Word images are embedded via a convolution neural network (CNN), which is trained to predict a representation modeling character-level information. Likewise, a recurrent neural network (RNN) is used to map a sequence of characters to the joint subspace representation. This embedding model is trained such that a word image and its text are projected at the same point in the embedding space, where the tasks of word spotting and word recognition are carried out as a nearest neighbor search problem.
Mhiri, M. (Author),
Cheriet (Supervisor) &
Desrosiers (Co-supervisor),
29 Oct 2018Student thesis: Doctoral thesis › Doctorate in Engineering: Engineering