Mobile Palmprint Segmentation Based on Improved Active Shape Model

With the rapid development of telecommunication and network technologies, mobile devices have become integrated information processing platforms with plenty of functions. It follows that mobile devices play a significant role for almost everyone nowadays, and hence it is vital to protect users’ permissions and privacy on mobile devices [1]. Unfortunately, users are often racked by the missing or theft of mobile devices. Thus accurate user authentication and authorization control are crucial functions of mobile device. However, current authentication techniques are not secure enough as they are claimed, especially if only one factor, i.e., possession, knowledge, or biometric, is employed [2].

Biometric refers to the metrics related to human physiologic and behavioral characteristics. Users do not need to memorize their biometric data. Furthermore, biometric data have high-entropy, so they are not easy to crack [3].

A lot of authentication protocols were consequently designed to guarantee the security and privacy in mobile environments [4]. Some schemes combined multiple factors, namely possession, knowledge, and biometric, to further enhance security and privacy [5].

Palmprint refers to the features on the palm region of a hand, which has several superiorities to other biometric modalities for the recognition on mobile devices, including plenty of discriminant features, few restrictive conditions, low cost, difficult leakage, high acceptance, and so on [6]. According to the above comparison, palmprint is a favorable authentication technique on mobile devices.

Palmprint recognition can be briefly categorized into contact and contactless modes [7].

In contact systems, users’ hands and equipment surface are contacted, even some pegs are used to fix hand position. Both the background and illumination are stable in contact acquisition, so it is easy to segment hand region and locate ROI. Although contact palmprint recognition systems can achieve high accuracy performance, their practical applications incur several problems, including the infection risks, the lack of acquisition flexibility, the contamination of equipment surface, the resistance of traditional cultures in some countries and regions, etc.

Users’ hands do not need to contact any equipment surface in contactless systems, so the acceptance is improved. Unfortunately, it is impractical to directly transplant the contact preprocessing methods onto contactless systems due to the severe challenges, including the uncontrolled hand poses and positions, the interference of complex backgrounds, the illumination variance and so forth [8].

Since palmprint images can be captured by the built-in cameras of mobile device, hands do not need to contact the equipment surface, and accordingly mobile mode can be considered as one special case of contactless mode.

Palmprint recognition on mobile devices has tremendous economic and market potential; however, far more technical challenges impede its development and promotion. Besides the uncontrolled hand poses and positions, the interference of complex backgrounds and illumination variance are more severe. Furthermore, the computation power and storage capacity of mobile devices are not comparable to those of desktop computers, which illustrates that both computation complexity and storage cost of palmprint recognition on mobile devices must be low.

Enrollment and authentication are two stages in the framework of palmprint recognition. Four steps, including acquirement, preprocessing, feature extraction, storage (in enrollment) / matching ( in authentication), are performed in sequence.

Preprocessing is critical to palmprint recognition. The aim of palmprint preprocessing is to accurately locate and crop the region of interest (ROI). It is relatively easy to segment hand region from background in contact mode because the background and illumination can be carefully and rigidly controlled. Contact palmprint preprocessing could not be used for contactless mode directly, but it is very helpful to the design of practical preprocessing in contactless mode and mobile environments, where more restrictions and complexities have to be fully considered.

Due to the imperfect segmentation in contactless mode, robust ROI localization is a knotty problem. Gaussian skin-color model [9], 2D-Otsu [10], ellipse skin-color [11], twice adaptive skin-color [12], traditional ASM [13], active appearance model + nonlinear regression [14] are some examples of frequently-used palmprint preprocessing methods. Nevertheless, it is still difficult to accurately segment the hands from the quite complex backgrounds and illuminations in mobile mode, so some assistant acquirement techniques are significant for palmprint acquirement sometimes.

Root regions between three fingers were aligned with the frame top border in [15]. Red guide rectangle guided users to place their palms in [16]. Hand-shaped guide curves helped users to place their four fingers, i.e., index finger, middle finger, ring finger, and little finger, in [17]. Hand locations are not strictly restricted in the above assistant acquirement techniques, so preprocessing is crucial to accurate localize hand position, which requires additional computation complexity. Vertical peak + horizontal peak was designed in [18]. The far distance between two peak points leads to low comfort, in other words, this scheme is not easy for users to manipulate. Besides, the accuracy of ROI localization depends on the stretch content of fingers, which is not easy to control.

Since binary texture codes need little storage capacity, low computational complexity of matching, and are free of training, they have become popular palmprint features for recognition [19]. The dissimilarity between two texture codes is commonly measured with normalized Hamming distance.

Skin-color information is not sufficient for palmprint segmentation in complex scenes, especially in mobile environments. Traditional active shape model (ASM) combines gray information and shape information, but its performance is not good in complex scenes. An improved ASM method is developed for palmprint segmentation, in which Perux method normalizes the shape of palm. Then the shape model of the palm is calculated with principal component analysis (PCA). Finally, the color likelihood degree is used to replace the gray information for target fitting. The improved ASM method reduces the complexity, while improves the accuracy and robustness.

The statistical principle is used in ASM to combine the shape information of the target with the texture information, so the boundary contour of the target is fitted for the target localization. Its advantages include high localization accuracy, low sensitivity to illumination, and strong robustness.

The contour boundary points constitute the shape of the target. First, the boundary points on the target contour are manually marked, which are required to effectively reflect the target shape. The marked boundary points are represented by a shape vector of the length 2 × n, i.e., x_i=(x_i1, y_i1, x_i2, y_i2, x_i3,…x_ik, y_ik, x_in, y_in)^T.

The above process is repeated to mark the contour boundary points on each sample in the training set to obtain the training sample set Ω = {x₁,x₂,…x_L}, where x_i represents the shape vector of the object in the i-th training image.

There is a difference in the shape and position of each sample in the training sample set, and these training samples need to be aligned, so Perux method is used for alignment. The commonly used alignment methods include scaling, translation and rotation. The specific steps are as follows.

Step 1: Calculate the coordinates of the center points of each sample contour.

Step 2: Use the zoom operation to unify the dimensions of the target.

Step 3: Align the center coordinates of the sample contour.

Step 4: Determine the direction of the palm in the sample and rotate the sample for alignment.

The coordinate of the center point of the sample contour and the calculation equation for the dimensions are:

Due to the large number of samples and the large number of feature points on the contour, the complexity of direct processing is high, so the data are usually reduced by PCA.

Assume that there are n training samples (x₁, x₂, …, x_n). The model in (3) can be obtained by PCA. The average shape x̅ is computed by (4). Φ is obtained by the feature vector matrix composed of the feature vectors corresponding to the first k largest eigenvalues. b is a feature vector composed of the first k eigenvalues.

The essence of target contour search based on ASM is an iterative problem. The positions of the boundary points are adjusted according to the gray information around them, and then the prior model is gradually approached to the target object until the change degree before and after the iteration is less than a certain threshold. When the contour does not change significantly, the final searched contour is obtained. The specific process is as follows.

Step 1: Place the average sample model obtained in the alignment operation on the test sample.

Step 2: Use the gray level information as the search condition of the boundary point, look for the best matching position of each boundary point at its normal direction, and move the boundary point to the new position.

Step 3: Compute the displacement Δx at each boundary point.

Step 4: Compute Δb to satisfy

Please note that the size of Δb needs to be limited by a certain threshold.

Step 5: Jump to Step 2 until Δb does not change significantly.

In order to overcome the uncontrollable problems in contactless mode, especially in mobile environments, “Dual-restriction-box assistance” is designed for the acquisition and localization of palmprint image [12], as shown in Fig. 1.

Fig. 1. Dual-restriction-box assistance.

Download Original Figure

During capture stage, the two bottom-points of the valleys, which are between the index finger and the middle finger, the ring finger and the little finger, are located on two “assistance points”, i.e., the centers of the two “restriction boxes”. “Dual-restriction-box assistance” can effectively restrain the location and posture of the acquired hand, reduce the complexity of the follow-up image preprocessing, and improve the execution efficiency and accuracy.

The sub-image in each restriction-box is a sample. Select 10 samples shown in Fig. 2, and then use manual marking and interpolation to mark the points on the contour.

Fig. 2. Samples for contour marking.

Download Original Figure

The boundary points of the selected samples are shown in Fig. 3. 11 feature points on the valley boundary between two fingers are roughly manually marked, and then 31 feature points are obtained with piece-wise linear interpolation. The Euclidean distances between the pairs of adjacent feature points are approximately equal.

Fig. 3. Boundary point set. (a) Sub-image in restriction box, (b) Manual marking (11 points), (c) Piece-wise linear interpolation (31 points).

Download Original Figure

There are 31 boundary points in each sample, and these boundary points are represented by two-dimensional coordinates. The boundary point set of each image is represented by x_i=(x_i1, y_i1, x_i2, y_i2, x_i3,…x_i30, y_i30, y_i31y_i31)^T.

After training the feature points of the 10 selected samples, the sample shape is normalized using the Perux method described in Section 2.2.

Each training sample x_i is a feature vector with a dimension of 62. There is a high correlation between these vectors, for example, the shape of the valley between two fingers is close to U-shape. To eliminate this correlation, a model that reflects this correlation needs to be built. This correlation can be reflected by PCA. The specific steps are as follows.

Step 1: Use x_i (i = 1, 2, 3, …, 10) to represent the normalized vector of the valley between the two fingers in the restriction box, and compute a new average shape vector $\overline{x}=\frac{1}{n}{\displaystyle \sum _{i=1}^n{x}_i}$.

Step 2: dx_i = x_i - x̅.

Step 3: Compute the covariance $S=\frac{1}{n}{\displaystyle \sum _{i=1}^{n}d{x}_{i}d}{x}_{\text{i}}^T$ of dx_i (i = 1,2,3,…,10).

Step 4: Compute all the eigenvalues in the covariance matrix S and their corresponding eigenvectors.

The larger the eigenvalue is, the more the corresponding eigenvector can mainly reflect the target shape. Therefore, after sorting, the feature vectors corresponding to the first k largest eigenvalues are selected to form a feature vector matrix Φ=(p₁, p₂,…p_k), and k eigenvalues constitute a vector b = (λ₁, λ₂,…, λ_k). Any shape vector in the final training sample can be expressed as:

Please note that the relationship criterion ${\displaystyle \sum _{i=1}^k{\lambda }_i\ge }V{\displaystyle \sum _{i=1}^{10}{\lambda }_i}$ should be satisfied, that is, the ratio of the target object deformation determined by the first k eigenvalues to the total deformation of all objects is not less than V (V is generally 0.98).

Since the k feature vectors in Φ are mutually orthogonal, b can be expressed as:

By adjusting b, a new model instance can be generated, and the variation of b needs to be limited to a certain range. Otherwise, the model generated in the iterative process is prone to deformation, resulting in failure of the target fitting. Usually limit each feature value in b :

We use the twice adaptive skin-color model in [12] for target search. The specific steps are as follows.

Step 1: Place the initial shape model on the center of the test image as the initial position for the target search, and set the model parameter b to 0, then the model is x=x̅.

Step 2: According to a certain search criterion, look for the best matching point (x_i, y_i)(i = 1,2,3,…,31) of each boundary point (x_i, y_i) along its bidirectional normal directions.

Step 3: Update the position of each boundary point to get a new shape. Adjust b so that the newly generated model can get close to the actual shape to the maximum extent, and finally set the newly generated model as the current model.

Step 4: Repeat Steps 2 and 3 until b does not significantly change.

In the above iterative process, Step 2 is significant. The bidirectional normal directions of the current boundary point are its search directions. After the search directions of the boundary points are determined, the search criteria directly affect the accuracy of the target localization. The developed method in this paper adopts a more accurate skin-color likelihood-based search scheme.

Based on the “twice adaptive skin-color model” in [12], the skin-color likelihood is computed. Then, along the search directions of the boundary points, the likelihood statistics are performed on the pixels around them. Suppose that for the i-th boundary point, take k (here k=7) points along bidirectional normal directions, and use the skin-color likelihood of 2k+1 points to form a vector g_i = {v₁, v₂, v₃,…,v_2k+1}. Perform the same operation for each boundary point to form a skin-color statistical model of the target contour S = {g₁, g₂, g₃,…,g₃₁}.

For a contour boundary point to be processed during accurate segmentation, since the first k points and the last k points are located in the palm area and the background area, respectively, the matching function is defined as:

f(g_i) is maximum at the best split position.

After using the above search scheme to determine the shape position of the valley between the fingers, the detection of the key point between the fingers is designed.

As shown in Fig. 4, the edge of the valley between the fingers is U-like shaped. It is observed that the first 1/3 feature points and the last 1/3 feature points are approximately two straight lines that are not parallel. The first 10 feature points and the last 10 feature points are selected for the fitting of two straight lines with the least square method. The intersection of the angle bisector between the two fitted straight lines and the inter-finger valley edge is detected as the key point.

Fig. 4. Key point detection. (a) Feature points, (b) Two fitted lines, (c) Angle bisector, (d) Key point.

Download Original Figure