Lifelong Learning Architecture of Video Surveillance System

It is crucial to develop a model not largely dependent on a particular layer to deploy a CNN model to a camera (e.g., depthwise separable convolution layer). The ResBlock used in the original model of PeleeNet applied only one (3×3) size kernel, so that it is difficult to robustly detect various sizes of people especially in occluded and blurred areas shown in Fig. 3. It is important to consider both the surrounding information and the object itself when determining whether objects are obscured by other objects or the image borders. The inability to extract and examine the pixel data surrounding the objects while employing a common kernel on the feature map. This problem is more severe when the resolution of image is lower, which is usually the case in cloud-based video surveillance services.

Fig. 3. The results of PeleeNet and our CNN model (MSRB) in the difficult cases of (left) the occluded person and (right) the blurred person.

Download Original Figure

In general, some object detectors like Yolo v5 [10] and ssFPN [11] were proposed to detect objects more accurately. However, a novel object detection model which efficiently captures important objects in real-time has not yet been systematically developed especially for video surveillance camera. Thus, we suggest a Multi-Scale ResBlock (MSRB)- PeleeNet for our object detection model in the intelligent front-end camera motivated and use the ResBlock of the residual structure at each feature map, as shown in Fig. 4. The MSRB proposed is made up of different sized kernels (3×3, 5×5, and 7×7) to handle the surrounding context information more concretely. Multi-scale kernels can be used to evaluate each of the layer characteristics, yielding statistically better performance at various scenes. It is also noticeable that the performance is maintained by using the retrieved context information even in severe blurring cases.

Fig. 4. Network architecture of object detection model.

Download Original Figure

Fig. 3 shows the qualitative results on two video surveillance scenes, where the proposed object detector detects a person more accurately even in the occluded or blurred cases compared to the original PeleeNet. Furthermore, we disregard the last three feature maps (5×5, 3×3, and 1×1) in the original design to lower the expensive computational expenditures in PeleeNet. Finally, we can identify that MSRB-PeleeNet is appropriate to detect objects for front-end surveillance camera due of its high accuracy and simplicity of computation.

For intelligent back-end architecture, new domain adaption scheme is what we suggest on MSRB-PeleeNet at each camera to become familiar with the appropriate personal surroundings. Denote data of the target during the adaptation in frame duration N as $x_A^t=\left\{x_0^t,x_1^t,\mathrm{...},x_N^t\right\},$ where n is the frame number within 0≤n≤N. Even if there isn’t any labeled data that corresponds to the bounding box coordinates for training, we may use it as a negative set when there is no one present on the image.

Here, we provide a novel technique for sampling the negative background images automatically during the adaptation period. First, we choose the first frame $x_0^t$ as an element of the final selected set $x_S^t\left(x_S^t\subset x_A^t\right).$ In this step, even there are some foreground objects in the scene, even after many rounds, we can still provide effective outcomes for domain adaptation. The following step is to acquire the gray-scale frame difference image, ${\widehat{x}}_{D_n}^t={\widehat{x}}_n^t-{\widehat{x}}_{n-1}^t,$ where ${\widehat{x}}_n^t$ is the gray-scale image in the n^th frame. Hence, if the average of difference across the most recent few frames (5 frames in this paper) is more than a specific threshold ε, it means moving objects appear on the n^th frame. Avoiding the repetition problem is what we choose a (n−1)^th frame, $x_{n-1}^t$ as an alternative for the consecutive step. It is preferable to choose only one frame as a representative image instead of choosing from all the candidates in a surveillance video since the next frames are nearly similar.

The third phase is where we define a new metric (M) of $x_{n-1}^t$ through all frames in $x_S^t$ to avoid the repetition in this set by considering the luminance, contrast, structure, and texture dissimilarity:

where ssim(·) is the structural similarity index metric (SSIM) [12], which is a remarkable human visual system-aware objective image quality assessment method that it is calculated with all images in $x_S^t,$ and $\daleth \left({\widehat{x}}_{n-1}^t\right)$ is the absolute difference of texture entropy on (n−1)^th frame, $E_{\left({\widehat{x}}_{n-1}^t\right)},$ which means the dissimilarity of visual entropy of texture information based on the information theory [13].

We only use the negative set $x_S^t$ when refining the first object detector for training without any positive samples for detection. As a result, if the initial detector discovers bounding boxes that are considered as false positives, since this area is devoid of any thing in the frame, according to the confidence score of, loss value is penalized on the predicted bounding box as follows.

where N_p is the quantity of expected bounding box l and c is the confidence score.

In our research, we took use of two publicly accessible data sets of Performance Evaluation in Tracking for Surveillance (PETS) 2009 [14] and Oxford Town Centre [15]. In the PETS 2009 data set, we used the ‘S2-L1’ video of a scenario outside with several people crossing each other, totaling 795 frames. Furthermore, we conducted the experiments on the Oxford Town Centre data set displaying people strolling through a busy street having 25 fps with a Full HD resolution. However, there is a limited amount of real-world video surveillance data available to use due to the privacy regulations. Thus, we utilized a private data set made up of abount 130K training images (Private_T) and 6K evaluation images (Private_E) as shown in Fig. 5.

Fig. 5. Examples of private dataset.

Download Original Figure

In the training and domain adaptation of the object detector, the batch size and learning rate were specified as 64 and 0.001, respectively. When trained with the private data set of Private_T, the network was trained over 210K iterations having the learning rate of 10⁻³, the weight decay of 5×10⁻⁴, and the learning rate decay value of 0.1 at 90k and 120K. A particular size was applied to the inputs for those benchmark techniques, and training was done with the default options utilized in each algorithm’s official, publicly available code. We adopted SSD [16] as a baseline backbone in the case of MobileNet v3 [17] and PeleeNet [18]. K-means clustering was used to obtain the anchor size for each training data sets.

The tests were carried out using a Qualcomm Visual Intelligence Platform with QCS603, which is intended for effective machine learning on Internet of Things (IoT) devices. Mean Average Precision (mAP), extensively utilized in earlier research on object detection, is employed with an Intersection over Union (IoU) threshold of 0.5 to compare accuracy.

To show the performance of MSRB-PeleeNet, we evaluated its effectiveness with several earlier object detection models of YOLO v4 [19], MobileNet v3, PeleeNet. We me-asured mAP values with the processing time on the camera after training with same data sets for detecting a people in video surveillance sequences.

Table 1 summarizes the performance of mAP and frame per seconds (fps) on two publics and one private data set on Digital Signal Processor chip. Our suggested model exhibits the greatest mAP values and a suitable processing time by a significant margin on an edge camera. Compared to the original PeleeNet, our model performs more accurately across all test data sets but at a slightly more computation time about 2−3 ms. In addition, we would like to underline that, when compared to the outcomes of the original object detector, our suggested domain adaption method’s efficacy is noteworthy. The fps value is decreased in all models due to the short adaptation time required for understanding context information, however it is insignificant for inference in the actual world. The updating of the original model requires the most work, which is caused by fine-tuning.

We conducted POC testing on a commercial video surveillance service, which manages about 130K video cameras, to validate if the suggested technique functions in a real-world context. Three hundred video cameras were used (for a week) among those to validate our method.

All the implementations of the proposed back-end system were deployed on Microsoft’s Azure services. Every camera has a container-based client to interact with the server that records an image and its associated meta data (object detection results), together with their identity data, were transmitted to the storage server.

Fig. 6 plots the exemplar result frames for four representative test sequences that were validated from the test sequences. The results show several false positives alongside a person or at the backgrounds, as indicated by the first column in Fig. 6. In addition, false positives were more frequent than true negatives, and those errors were more likely to occur in the infrared mode of the camera. The following column in Fig. 6 displays the outcomes of domain adaptation, where the accuracy of person detection was maintained but the number of false positives was significantly decreased.

Fig. 6. POC results on four different examples.

Download Original Figure