Access Control Training
Resume:
SUBMITTED FOR PUBLICATION TO:, JUNE **, **** *
Multi-Camera Face Recognition by
Reliability-Based Selection
Binglong Xie1, Terry Boult2, Visvanathan Ramesh1, Ying Zhu1
1
Real-Time Vision and Modeling Dept.,
Siemens Corporate Research,
Princeton, NJ 08540
E-mail: {binglong.xie,visvanathan.ramesh,yingzhu}@siemens.com
2
Department of Computer Science,
University of Colorado at Colorado Springs,
Colorado Springs, CO 80933
E-mail: ******@**.****.***
Abstract
Automatic face recognition has a lot of application areas and current single-camera face recogni-
tion has severe limitations when the subject is not cooperative, or there are pose changes and different
illumination conditions. A face recognition system using multiple cameras overcomes these limita-
tions. In each channel, real-time component-based face detection detects the face with moderate
pose and illumination changes with fusion of individual component detectors for eyes and mouth,
and the normalized face is recognized using an LDA recognizer. A reliability measure is trained
using the features extracted from both face detection and recognition processes, to evaluate the in-
herent quality of channel recognition. The recognition from the most reliable channel is selected as
the nal recognition results. The recognition rate is far better than that of either single channel, and
consistently better than common classi er fusion rules.
Keywords
Multi-Camera Face Recognition, Reliability Measure.
I . I NTRODUCTION
Face recognition has a lot of application areas, such as biometrics, information security, law
enforcement, smart cards, access control and surveillance etc., and has seen much improvement
in recent years [1]. However, current face recognition still has some severe limitations in typical
applications like surveillance and access control, for example, when the subject is not cooperative
and turns away from the camera, the accuracy of face recognition can be marred signi cantly [1].
Traditionally face recognition was performed on 2D images, mostly frontal or near-frontal
view faces, without recovering 3D shape and albedo. These include landmark points/geometric
feature-based methods, template matching/correlation,PCA (Principal Component Analysis, or
Eigenfaces), LDA (Linear Discriminant Analysis, or Fisherfaces) [2], neural networks, EBGM
(Elastic Bunch Graph Matching), etc [3] [4]. In general 2D face recognition methods suffer from
pose and illumination changes, because they rely on seen image instances while the same face can
generate novel image instances by varying the pose or lighting conditions.
3D face recognition methods, include range-based recognition,stereo reconstruction,SFS (Shape
From Shading),3D morphable model [5], etc[3] [4]. The 3D reconstruction used in these methods
is often either intrusive, slow, or inaccurate, or requiring manual initialization, and is not appropri-
ate for real-time applications.
In this paper, we present a face recognition system using two cameras. In each channel,
component-based face detector detects faces with pose and illumination changes and LDA-based
face recognition is performed to recognize the normalized faces. The recognitions from the two
channels are fused to get the nal results, using a selection scheme based on a channel reliability
measure trained inherent to the individual channel performance. The architecture of the system is
shown in Figure 1 and explained in the following sections.
Fig. 1. Reliability based selection of multiple channel face recognition.
II. C OMPONENT-BASED FACE D ETECTION AND R ECOGNITION
A. Component-Based AdaBoost Face Detection
Face detection must be carried out before face recognition. We roughly classify face detection
algorithms into two camps: the holistic approaches and the component-based approaches. The
former treats the face as a complete pattern, and tries to model it in a global way. The latter
decomposes the face into smaller components, for example, eyes and mouth, and model them
speci cally. It is known that component-based approaches are more robust than global ones for
face detection with pose variations, illumination variations, and occlusions of facial parts [6].
AdaBoost learning [7] has been very popular in face detection since Viola et al s effective
usage to achieve both fast and accurate face detection with Haar wavelet features quickly calculated
from the integral image [8]. AdaBoost does not automatically overcome the dif culties faced by an
holistic approach, however, we can combine it with component-based approach and bene t from
both.
We use a component model shown in Figure 2 Left. The three face component detectors, left
eye, right eye and mouth, are trained independently using Haar wavelets and AdaBoost learning
technique. The individual component detections are fused and t to a component face model sta-
tistically, to decide if they can composite into a valid face. For details of component fusion, please
Fig. 2. Left: Three face components de ned on a standard face template. Right: Real world detection examples.
see [9]. Our face detection allows exible component con guration, covers wide pose, illumina-
tion and expression changes, while running in real time. Some real world detection examples are
shown in Figure 2 Right.
B. LDA-Based Face Recognition
We use LDA-based face recognition. One nearest neighbor for each class is found when the
unknown face is transformed into the LDA subspace. The matches are sorted by its distance to
the probe face in ascending order. An important bene t from component-based face detection is
better registration of detected face, which is essential for recognition performance. The complete
detection and recognition system typically runs at 25fps for 352x288, and 15fps for 640x480 pixel
videos on a P4 1.8GHz PC.
III. S ELECTION FROM T RAINED R ELIABILITY M EASURE
The component-based face detection and recognition framework works only with moderate
pose changes near frontal view. To cover even wider pose changes, we use two cameras setting up
with a large baseline, so one camera provides complementary coverage to the other.
TABLE I
C OMMON COMBINING RULES FOR MULTIPLE CLASSIFIERS USING DISTANCES
Method Rule
k = argmin i N N d(xj, i )
Minimal geometric mean j =1
1
k = argmin i N N d(xj, i )
Minimal arithmetic mean j =1
k = argmin i medj {d(xj, i ), j = 1, N }
Minimal median
k = argmin i minj {d(xj, i ), j = 1, N }
Minimal minimum
k = argmin i maxj {d(xj, i ), j = 1, N }
Minimal maximum
k = argmax i N 1d(xj, i )=minm {d(xm, i ),m=1 N }
Majority voting j =1
A. Data Fusion
When multiple face recognizers yield individual recognitions, fusion can be performed to
improve the performance. Consider we have N classi ers, and each compares its input xj, j =
1, N to C known classes { 1, C } to get the distance metric {d(xj, i )}. By constraining
the joint probability with assumptions such as statistical independence, etc,the common combining
rules [10] are summarized in Table I. I.
The common combining rules are simple and proved useful in some applications, but they
assume strong statistical constraints for them to apply. Moreover, these rules are rigid. Even when
training examples are available, which should allow better combination the classi ers, the rules are
not possible to be tuned by the examples and trained for better performance.
B. Reliability Measure from Training
With labeled training examples on hand we can train a classi er to predict the correctness of
channel recognition. When a channel correctly recognizes the face in the top match, we label the
data sample x as positive y = +1, otherwise as negative y = 1. Friedman [11] proved that in
an additive logistic regression model, when the AdaBoost error bound is minimized by choosing
appropriate f (x) in boosting, the channel reliability P (y = +1 x) is a monotone function of the
AdaBoost strong classi er response f (x):
ef (x) e2f (x)
P (y = +1 x) = = 2f (x) (1)
ef (x) + e f (x) +1
e
Therefore, we can train f (x) to represent the channel reliability equivalently using AdaBoost.
C. Data Representation and Feature Design
The common combining rules only use the recognition matching distances for fusion. How-
ever, in a channel, the face detection performance affects the overall channel reliability as well. Our
reliability measure f takes both detection and recognition data into account as shown in Figure 1.
Speci cally, we design 5 categories of features for the weak classi ers to boost f : face detec-
tion geometric features checking the component sizes, locations, con dences, overall face detec-
tion con dence, and the coherence among the component geometric con guration; face detection
Haar wavelets, which are the plain features used in the low-level face component detectors; face
recognition features derived from recognition matching distances, e.g., the slope from the rst dis-
tance to second distance and so on; consecutive time features checking smoothness over time; and
joint channel features checking cross-channel properties. In total we have 1011 features and 1921
weak classi ers used for boosting, and 200 weak classi ers are selected in the reliability measure.
IV. E XPERIMENTS AND P ERFORMANCE E VALUATION
A. Experiment Settings
We set up two cameras with a baseline of 42cm pointing to the subjects at 50cm depth. 33
synchronous videos are collected for 33 different subjects, with yaw in ( 23, 23 ) and pitch in
( 17, 17 ). Each video has about 683 synchronous frames, about 481 are used for training and
202 for testing. There is little overlapping in pose coverage between the training and testing frames.
B. Performance Evaluation
When testing the system, a threshold is imposed on the selected reliability. Detection is de ned
as the selected reliability meets threshold, and recognition is that the top match corresponds to the
true identity. The detection rate is de ned as number of detection divided by number of testing
frames. The absolute recognition rate is de ned as number of recognition divided by number of
testing frames. The reliability threshold is varied to obtain the performance curve.
TABLE II
B REAKDOWN OF FUSED FACE RECOGNITION .
ground truth frames fusion detection fusion recognition
correct/correct 164*-****-****
correct/wrong 198*-****-****
wrong/wrong 204 101 0
correct/NA 149*-****-****
wrong/NA 442 56 0
comparison of selection and common rules
comparison of selection and common rules
0.8
0.78 perfect selection
perfect selection
fusion by selection
fusion by selection
minimal minimum, 3
minimal minimum
0.7 minimal geo mean, 3
0.76
minimal geo mean
minimal mean, 3
minimal mean
absolute recognition rate
absolute recognition rate
channel 0
0.74
minimal maximum
0.6
channel 1
0.72
0.5
0.7
0.4
0.68
0.3
0.66
0.3 0.4 0.5 0.6 0.7 0.8 0.9 0.7 0.75 0.8 0.85
detection rate detection rate
Fig. 3. Performance of different fusions. Perfect selection is performed manually for reference.
Table II shows the breakdown according to the ground truth of channel recognition, e.g., in the
correct/wrong case (one channel is correct but not the other), it takes the correct channel at 92.7%.
Figure 3 Left shows that the reliability-based selection is far better than either individual channel
and the minimal maximum rule. We use leave-one-out strategy to sample the 202 testing frames
and compute the con dence of the recognition rate. As shown in Figure 3 Right, our fusion by
selection outperforms the best common fusion rule, the minimal minimum, with high con dence.
The curves are well separated with 3, which corresponds to con dences larger than 99.7%.
Figure 4 shows a real world example that fusion selects the more reliable channel.
Fig. 4. Real world example of fusion by reliability-based selection, left channel selected.
V. C ONCLUSION
We present a two-camera face recognition system that uses fusion by selection from trained
reliability measure. The experiments shows that the system performs far better than either channel
and is consistently better than common fusion rules. The real-time component-based face detection
and recognition is just an example; the methodology is open to use other single-channel face
detection/recognition technologies, only feature design needs to adapt to that change. It can be
easily extended to use more cameras to cover wider pose range and/or illumination conditions.
[1] P.J. Phillips, P. Grother, R.J Micheals, D.M. Blackburn, E Tabassi, and J.M. Bone, Frvt 2002:
Overview and summary, http://www.frvt.org/FRVT2002/documents.htm, March 2003.
[2] Peter N. Belhumeur, Joao Hespanha, and David J. Kriegman, Eigenfaces vs. sherfaces:
Recognition using class speci c linear projection, in ECCV (1), 1996, pp. 45 58.
[3] W. Zhao, R. Chellappa, A. Rosenfeld, and P.J. Phillips, Face recognition: A literature survey,
in UMD, 2000.
[4] R. Chellappa, C.L. Wilson, and S. Sirohey, Human and machine recognition of faces: A
survey, PIEEE, vol. 83, no. 5, pp. 705 740, May 1995.
[5] V. Blanz, S. Romdhani, and T. Vetter, Face identi cation across different poses and illumina-
tions with a 3d morphable model, in AFGR02, 2002, pp. 192 197.
[6] B. Heisele, T. Serre, M. Pontil, and T. Poggio, Component-based face detection, in CVPR01,
2001, pp. I:657 662.
[7] Y. Freund and R. E. Schapire, A desicion-theoretic generalization of on-line learning and
an application to boosting, in Computational Learning Theory: Second European Confer-
ence(EuroCOLT 95), P. Vitanyi, Ed., pp. 23 37. Springer, Berlin,, 1995.
[8] P. Viola and M. Jones, Robust real-time face detection, in ICCV01, 2001, p. II: 747.
[9] Binglong Xie, Dorin Comaniciu, Visvanathan Ramesh, Terry Boult, and Markus Simon,
Component fusion for face detection in the presence of heteroscedastic noise, in Annual
Conf. of the German Society for Pattern Recognition (DAGM 03), Magdeburg, Germany, 2003.
[10] Shaohua Zhou, Rama Chellappa, and Wenyi Zhao, Unconstrained Face Recognition,
Springer, 2006.
[11] J. Friedman, T. Hastie, and R. Tibshirani, Additive logistic regression: a statistical view of
boosting, 1998.
ps.dvi
Contact this candidate