Visual object tracking (VOT) and person re-identification (ReID) go hand in hand for a wide range of monitoring and video surveillance, like multi-camera target tracking, pedestrian tracking for autonomous driving, access control in biometrics, etc. Once a person has been detected, VOT produces continuous bounding boxes for that person over a video sequence given an initial location. At the same time, ReID identifies lost tracks of a person or identifies persons or objects seen in different cameras. They suffer from common challenges, particularly in scenarios with occlusion and changing target appearances. Learning objectives for tracking and ReID are similar to similarity-matching tasks using Siamese network architectures. The update of a tracker is performed online as object appearance or prototype needs to be learned over time with changes in appearance. ReID does not involve explicit online learning; it is more about using video sequences to represent an object best and match it with a database of objects.
Challenges related to online learning in tracking, where methods like classical approaches or deep convolutional neural network (CNN) methods aim to learn target appearance during tracking to prevent target drift over time. However, model adaptation using samples from a tracker can be hindered by noise introduced through drifting. Detecting and managing drifts becomes crucial, and improper sample selection can impact model robustness. For instance, inadequate diversity in training samples during adaptation can significantly lead to model drift when the target’s appearance changes due to variations in, e.g., lighting and viewpoint. In video-based person ReID, leveraging person tracklets in queries can address challenges like occlusion, inaccurate bounding box positioning, and variations in appearance due to lighting and viewpoint changes. Improvements are sought in feature aggregation methods for video sequences, considering limitations in sequence length and potential overfitting. Occlusion poses a significant challenge in online learning for tracking and person ReID, causing feature extractors to focus on non-object regions, leading to false matches. Developing computationally efficient solutions for handling occlusion is crucial for real-time applications. This thesis focused on three main problems: online learning with dynamic template/sample selection for tracking, video representation, and occlusion-aware representation learning.
The first contribution in this thesis primarily concentrates on enhancing online learning in VOT models, with a specific focus on addressing challenges related to concept drift and occlusion. Concept drift encompasses changes in the appearance of the tracked object, categorized as gradual, abrupt, and recurring. Our research highlights the adaptability benefits of gradual concept drift while acknowledging that abrupt changes often result from distractions like occlusion, necessitating cautious handling of model updates. Moreover, recurring drifts occur when the object’s previous appearance reoccurs, and we found that maintaining a sample buffer with high variance proves effective for online tracking.
The second contribution in this thesis focuses on video person ReID within video analytics and surveillance, aiming to overcome existing limitations such as changing appearance, viewpoint variation across cameras, and occlusion. To address this, incorporate the motion patterns of individuals as an additional cue for ReID. Our proposed solution introduces the Flow-Guided Mutual Attention network, merging bounding box and optical flow sequences over tracklets. This utilizes a 2D-CNN backbone to encode both temporal and spatial appearance information. Furthermore, we present a novel method for aggregating features from extended input streams to enhance video sequence-level representation. Experimental results show a significant improvement in ReID accuracy compared to traditional gated-attention networks and current state-of-the-art methods in video-based person ReID. This research highlights the potential of motion-pattern-guided attention mechanisms in enhancing the capabilities of deep learning models for robust video ReID applications.
Our next introduces a novel Holistic-Generative (HG) student-teacher model designed for occluded person ReID, eliminating the need for image identity labels and resource-intensive processes focused solely on visible parts of occluded regions. The proposed HG teacher uses the Distribution of CLass Distances (DCD) across samples in a comprehensive dataset to train a student model, allowing the generation of attention maps and addressing challenges posed by occlusion. Unlike prevalent approaches in the literature that use external supervision like pose for visibility cues, our method relies on holistic data distribution during training, treating it as a soft label. Consequently, during testing, our model seamlessly operates without needing external cues like pose, with the overall parameters comprising only the backbone Encoder and a compact embedding for attention map generation during feature extraction. Additionally, joint learning of a denoising autoencoder enhances the model’s ability to self-recover from occlusion. Empirical evaluations on diverse and challenging datasets demonstrate the superior performance of our HG methodology, surpassing state-of-the-art models in Occluded-ReID and Holistic ReID tasks.
Our final contribution explores the space for dissimilarity in Person ReID to solve the problem of overlap between classes caused by computationally efficient models with relatively smaller capacities and limited training data. We propose applying dichotomy transformation to feature space and classifying sample pairs as similar or dissimilar with a max-margin classifier. Endto- end learning of a classifier in the dissimilarity space has been observed to improve retrieval accuracy for Person ReID tasks.
Results in this thesis indicate that effective online learning with careful sample selection based on techniques for drift detection can allow long-term tracking with minimal updates to the tracker, thereby low overall complexity. In addition, it has also been shown that a good representation can be learned by choosing long video sequences. Occlusion is a challenge in both tracking and ReID applications. It has been shown that with occlusion-aware learning, there can be an overall improvement in both tracking and ReID performances. It solves this problem for the practical scenario with minimalistic occlusion labeled data.