BMC 2012
1 like537 views
The document discusses foreground detection through robust low-rank matrix decomposition combined with spatio-temporal constraints, focusing on its application in video surveillance and motion capture. It elaborates on the techniques used, including iterative reweighted least squares (IRLS) and different formulations for robust PCA, supplemented with experimental results validating the method's effectiveness. Key challenges and future works, such as the need for real-time processing improvements, are also addressed.
![Various RPCA formulation (only for α = 1)
PCA with fixed rank is : min ||S||F
L,S
s.t. Rank(L) = k (7)
A=L+S
R(obust)PCA is (Non convex and NP-hard ) :
min ||σ(L)||0 + λ||S||0
L,S (8)
s.t. A=L+S
Convex relaxed problem of (2) is RPCA-PCP proposed by Candès et al. [1] :
min ||σ(L)||1 + λ||S||1
L,S (9)
s.t. A=L+S
Where σ(L) means singular values of L.
A mix is Stable PCP of Zhou et al. [2] (both entry-wise and sparse noise) :
min ||σ(L)||1 + λ||S||1
L,S (10)
s.t. ||A − L − S||F < δ
All of them could be solved by Augmented Lagrangian Multipliers (ALM).
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 10 / 27
La Roche](https://image.slidesharecdn.com/bmc2012-121119005431-phpapp02/85/BMC-2012-10-320.jpg)
![Experimental Protocol
Optimal threshold is chosen for maximizing F-measure criterion
which is based 2 × 2 histogram of True/false/positive/negative :
TP TP 2 DR Prec
DR = , Prec = , F =
TP + FN TP + F P DR + Prec
Good performance is then obtained when the F-measure is closed to 1
Time consumption is not take into account in the evaluation process.
RPCA-LBD is compared with the following two Robust methods :
Robust Subspace Learning (RSL) De La Torre et al. [4]
Principal Component Pursuit (RPCA-PCP) [1]
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 22 / 27
La Roche](https://image.slidesharecdn.com/bmc2012-121119005431-phpapp02/85/BMC-2012-22-320.jpg)
![Future Works & References
Future Works
Lack in computation time : Further research consists in developping an
incremental version to update the model at every frame and to achieve
the real-time requirements.
References
[1] E. Candes, X. Li, Y. Ma, and J. Wright, Robust principal component
analysis, International Journal of ACM, vol. 58, no. 3, May 2011.
[2] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma, Stable principal
component pursuit,IEEE ISIT Proceedings, pp. 1518-1522, Jun. 2010.
[3] G. Tang and A. Nehorai, Robust principal component analysis based
on low-rank and block-sparse matrix decomposition, CISS 2011, 2011.
C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati
(MIA Laboratory (Mathematics Images & Applications), 5, 2012
November University of 27 / 27
La Roche](https://image.slidesharecdn.com/bmc2012-121119005431-phpapp02/85/BMC-2012-27-320.jpg)
BMC 2012
- 1. Foreground Detection via Robust Low Rank Matrix Decomposition including spatio-temporal Constraint C. Guyon, T. Bouwmans and E. Zahzah MIA Laboratory (Mathematics Images & Applications), University of La Rochelle, France — Workshop BMC, Daejong Korea November 5, 2012 C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 1 Roche
- 2. Summary 1 Introduction and motivations on IRLS 2 Temporal constraint with an adapted Norm 3 Diagram flow and Spatial constraint 4 Experimental Results 5 Conclusion C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 2 Roche
- 3. Introduction and motivations Purpose Foreground detection : Segmentation of moving objects in video sequence acquired by a fixed camera. Background modeling : Modelization of all that is not moving object. Involved applications Surveillance camera Motion capture On the importance Crucial Task : Often the first step of a full video surveillance system. Strategy used Eigenbackground decomposition. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 3 Roche
- 4. Eigenbackgrounds Find an « ideal » subspace of the video sequence, which describes the best as possible the (dynamic) background. Fig.1 The common process of background subtraction via PCA (Principal Component Analysis). At final step, an adaptative threshold is used to get a binary image. Without a robust framework, the moving object may be absorbed in the model ! C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 4 Roche
- 5. Data Structure Transformation First, we consider a video sequence as a matrix A ∈ Rn×m n is the amount of pixels in a frame (∼ 106 ) m is the number of frames considered (∼ 200) C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 5 Roche
- 6. IRLS : Vector version (1) The usual IRLS (Iteratively reweighted least squares) scheme for solve argmin ||Ax − b||α is given by x D (i) = diag((ε + |b − Ax (i) |)α−2 ) (1) x (i+1) = (At D (i) A)−1 At D (i) b that a suitable IRLS method is convergent for 1 ≤ α < 3 Other formulation, r (i) = b − Ax (i) D = diag((ε + |r (i) |)α−2 ) (2) y (i) = (A DA)−1 A Dr (i) x (i+1) = x (i) + (1 + λopt )y (i) λopt computed in a second inner loop and convergent for 1 ≤ α < +∞ C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 6 Roche
- 7. IRLS : Vector version (2) For Spatio/temporal RPCA, needs to solve the following general problem : argmin ||Ax − b||α + λ||Cx − d ||β (3) x By derivation, the associated IRLS scheme is, r1 = b − Ax (i ) , r2 = d − Cx (i ) , e1 = ε + |r1 |, e2 = ε + |r2 | α 1 β 1 −1 β−2 D1 = ( e1 ) α −1 diag(e1 ), D2 = λ( e2 ) β diag(e2 ) α−2 (i ) −1 (4) y = (A D1 A + C D2 C ) (A D1 r1 + C D2 r2 ) x (i +1) = x (i ) + (1 + λopt )y (i ) Good news : Just few line of matlab code ! C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 7 Roche
- 8. Matrix Version More generally, we consider the following matrix regression problem with two parameters norm (α, β) and a weighted matrix (W ), n m α 1 min ||AX − B||α,β with ||Mij ||α,β = ( ( Wij |Mij |β ) β ) α (5) X W W i=1 j=1 The problem is solved in the same manner on matrices with a reweighted regression strategy, Until X is stable, repeat on each k-columns R ← B − AX S ← ε + |R| (6) α −1 β−2 β Dk ← diag(Sik ◦ ( j (Sij ◦ Wij )) β ◦ Wik )k Xik ← Xik +(1+Λ(max(α, β)))(At Dk A)−1 At Dk Rik C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 8 Roche
- 9. Exemple : Geometric median (convex problem) Fig.1 Find the x-axis points which minimize the ||.||α distance between the 7 others points (black stars). Minimum of Curves is corresponding to this optimal value for α = 2, 1.66, 1.33 and 1 (blue, green, red, black). Fig.1 Same in Two Dimension. By continuously changing the minization problem (i.e. varying α) during each iterations, this trick accelerates the convergence of IRLS. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), University of La / 27 November 5, 2012 9 Roche
- 10. Various RPCA formulation (only for α = 1) PCA with fixed rank is : min ||S||F L,S s.t. Rank(L) = k (7) A=L+S R(obust)PCA is (Non convex and NP-hard ) : min ||σ(L)||0 + λ||S||0 L,S (8) s.t. A=L+S Convex relaxed problem of (2) is RPCA-PCP proposed by Candès et al. [1] : min ||σ(L)||1 + λ||S||1 L,S (9) s.t. A=L+S Where σ(L) means singular values of L. A mix is Stable PCP of Zhou et al. [2] (both entry-wise and sparse noise) : min ||σ(L)||1 + λ||S||1 L,S (10) s.t. ||A − L − S||F < δ All of them could be solved by Augmented Lagrangian Multipliers (ALM). C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 10 / 27 La Roche
- 11. Video examples Some examples, temporal RPCA and ideal RPCA with groundtruth fitting and optimal parameters fitting. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 11 / 27 La Roche
- 12. Summary 1 Introduction and motivations on IRLS 2 Temporal constraint with an adapted Norm 3 Diagram flow and Spatial constraint 4 Experimental Results 5 Conclusion C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 12 / 27 La Roche
- 13. Sparse solution In RPCA, residual error is sparse. Using the RPCA decomposition on a synthetic low-rank random matrix plus noise, the error looks like : Same principle with video. Sparse noise (or outliers) are the moving objects. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 13 / 27 La Roche
- 14. Let’s play with norms Varying the α, β norm → Different kind of recovering pattern error. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 14 / 27 La Roche
- 15. Let’s play with norms...(2) Some issues What is the best specific norm for temporal constrain ? Initial assumption is ||.||2,1 . Confirmed experimentally ? C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 15 / 27 La Roche
- 16. Validation If ideal eigenbakgrounds are that, best norm must be ... Let’s denote Lopt , the ideal low-rank subspace which outliers do not contribute to PCA C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 16 / 27 La Roche
- 17. Experimental results Let’s denote Lα,β , the low-rank recovered matrix with a ||.||α,β -PCA. The plot show the error between ||Lopt − Lα,β ||F for parameters α and β chosen freely. The darkest value means that the error is the smallest here. ||S||2,1 is not optimal, but for convenience we use it. The benefit of the ad hoc block-sparse hypothesis is confirmed by testing its efficiency directly on video dataset. Experimentation done on dynamic category of dataset change detection workshop 2012 : http://www.changedetection.net/ C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 17 / 27 La Roche
- 18. Summary 1 Introduction and motivations on IRLS 2 Temporal constraint with an adapted Norm 3 Diagram flow and Spatial constraint 4 Experimental Results 5 Conclusion C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 18 / 27 La Roche
- 19. Overview and addition of the spatial constraint Figure: Overview of the learning and evaluation process. Learning process needs GT (Groundtruth) for better fits the eigenbackground components. Spatial Constrain Suppose A = L + S and L and S are computed via some kind of RPCA technique with the addtion of Total Variation penalty on S. This increase connected (or connexe) shapes. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 19 / 27 La Roche
- 20. Exemple with a synthetic 1-D signal C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 20 / 27 La Roche
- 21. Summary 1 Introduction and motivations on IRLS 2 Temporal constraint with an adapted Norm 3 Diagram flow and Spatial constraint 4 Experimental Results 5 Conclusion C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 21 / 27 La Roche
- 22. Experimental Protocol Optimal threshold is chosen for maximizing F-measure criterion which is based 2 × 2 histogram of True/false/positive/negative : TP TP 2 DR Prec DR = , Prec = , F = TP + FN TP + F P DR + Prec Good performance is then obtained when the F-measure is closed to 1 Time consumption is not take into account in the evaluation process. RPCA-LBD is compared with the following two Robust methods : Robust Subspace Learning (RSL) De La Torre et al. [4] Principal Component Pursuit (RPCA-PCP) [1] C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 22 / 27 La Roche
- 23. Quantitative Results Here, we show experimental results on the real dataset of BMC, Video Recall Precision F-measure PSNR Visual Results 1 0.9139 0.7170 0.8036 38.2425 2 0.8785 0.8656 0.8720 26.7721 3 0.9658 0.8120 0.8822 37.7053 4 0.9550 0.7187 0.8202 39.3699 5 0.9102 0.5589 0.6925 30.5876 6 0.9002 0.7727 0.8316 29.9994 7 0.9116 0.8401 0.8744 26.8350 8 0.8651 0.6710 0.7558 30.5040 9 0.9309 0.8239 0.8741 55.1163 Table: Quantitative results with common criterions. Last column show the original, GT and result of the first four real video sequences. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 23 / 27 La Roche
- 24. Other results Figure: First eigenBackground of the fifths sequence of Rotary (BMC) with the norms ||.||opt , ||.||1,1 and ||.||2,1 . Last row shows the first five eigenBackground on real dataset with ||.||2,1 C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 24 / 27 La Roche
- 25. Summary 1 Introduction and motivations on IRLS 2 Temporal constraint with an adapted Norm 3 Diagram flow and Spatial constraint 4 Experimental Results 5 Conclusion C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 25 / 27 La Roche
- 26. Qualitative analysis Advantages Experiments on video surveillance datasets show that this approach is more robust than RSL and RPCA-PCP in presence of dynamic backgrounds and illumination changes. Well suited for video with spatially spread and temporarily sparse outliers. Disadvantages Not efficient on sequences with outliers always presents. For small local variations, like « wind tree » are not (yet) well modelized by this kind global PCA. Probably needs more eigen components ! C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 26 / 27 La Roche
- 27. Future Works & References Future Works Lack in computation time : Further research consists in developping an incremental version to update the model at every frame and to achieve the real-time requirements. References [1] E. Candes, X. Li, Y. Ma, and J. Wright, Robust principal component analysis, International Journal of ACM, vol. 58, no. 3, May 2011. [2] Z. Zhou, X. Li, J. Wright, E. Candes, and Y. Ma, Stable principal component pursuit,IEEE ISIT Proceedings, pp. 1518-1522, Jun. 2010. [3] G. Tang and A. Nehorai, Robust principal component analysis based on low-rank and block-sparse matrix decomposition, CISS 2011, 2011. C. Guyon, T. Bouwmans and E. Zahzah Foreground Detection via Robust Low Rank Matrix Decomposition including spati (MIA Laboratory (Mathematics Images & Applications), 5, 2012 November University of 27 / 27 La Roche