SURVEILLANCE VIDEO-TO-TEXT GENERATOR

Abstract

Video-to-text generation is a relatively new field which is recently gaining a popularity as other generative models. It is particularly beneficial for surveillance to retrieve textual descriptions from CCTV cameras. Thus, this thesis aims to design an efficient video-to-text generator for surveillance. The research established that encoder-decoder neural networks are the most up-to-date technique. Specifically, pretrained CNN models and the LSTM based encoder and decoder models are the most prominent neural networks. Though the research was initially intended for surveillance videos, the largest MSR-VTT and MSVD datasets with general videos were used for training due to the absence of available surveillance dataset with captions. The models were designed in four parts: feature extraction, model training, test caption generation, and model evaluation. Video features, which were extracted using pretrained VGG16 CNN, were fed into the encoder using one LSTM layer. Then, the decoder in the form of another LSTM layer was implemented for video caption generation. In general, 12 models were trained for two datasets with various number of frames per video and vocabulary size. The best performing model, which was trained on MSVD dataset with 16 frames per video and 2000 vocabulary size, scored 12.8, 32.2, 32.9, and 44.0 on METEOR, BLEU1, ROUGE, and CIDEr evaluation metrics respectively. Therefore, MSVD dataset is the most suitable for the designed architecture. Furthermore, it was found that increasing number of frames per video was not legitimate in terms of computational resources for short videos between 10 to 30 seconds. Finally, 2000 vocabulary size is an excellent size for MSVD dataset. Though the proposed model generated captions, it performed worse than the past research in terms of evaluation metrics. This might be caused by the computational limitation, inaccurate caption datasets, and improper selection of the search algorithm.