Conic Clustering
- 1. Size Matters: Cardinality-Constrained Clustering & Outlier Detection Napat RUJEERAPAIBOON Department of Industrial Systems Engineering & Management National University of Singapore
- 2. k–Means Clustering Ÿ k–means clustering is NP-hard1 . min °n i1 °k j1 πij }ξi ¡ζj }2 s. t. πij € t0, 1u, ζj € Rd °k j1 πij 1 di 1, . . . , n Ÿ In practice, k–means heuristic2 produce solutions quickly. Fix πij ùñ ζj average of ξi with πij 1 Fix ζj ùñ πij 1 if ζj is the closest center to ξi 1 Aloise et al. (2009) 2 Arthur Vassilvitskii (2007)
- 3. Challenges Ÿ k–means heuristics suffers from several shortcomings. Technical challenges 1 Slow runtime.3 2 Unknown suboptimality. Practical challenges 1 Skewed clustering.4 2 Sensitivity to outliers.5 Ÿ Skewed clustering is unfavorable in many applications. 3 Arthur Vassilvitskii (2006) 4 Bennett et al. (2000) 5 Chawla Gionis (2013)
- 4. Cardinality Constraints Implicit Cdn. Market Segmentation Distributed Computing Explicit Cdn. Category Management Vehicle Routing Outlier Det. Fraud Detection Medical Diagnosis
- 5. Outlier Detection Ÿ k–means (25.21) Ÿ Balanced k–means (54.27) Ÿ Balanced k–means + Outlier detection (1.97)
- 6. Cardinality Constraints Ÿ Introduce dummy (0th cluster) for outliers. Ÿ tnj uk j1 and n0 denote sizes of regular and dummy clusters. Ÿ Cardinality–constrained k–means clustering. min °n i1 °k j1 πij }ξi ¡ζj }2 s. t. πij € t0, 1u, ζj € Rd °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k
- 7. Cardinality Constraints Ÿ Introduce dummy (0th cluster) for outliers. Ÿ tnj uk j1 and n0 denote sizes of regular and dummy clusters. Ÿ Cardinality–constrained k–means clustering. min °n i1 °k j1 πij }ξi ¡ζj }2 s. t. πij € t0, 1u, ζj € Rd °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k Ÿ A remedy for the practical challenges .
- 8. Cardinality Constraints Ÿ Introduce dummy (0th cluster) for outliers. Ÿ tnj uk j1 and n0 denote sizes of regular and dummy clusters. Ÿ Cardinality–constrained k–means clustering. min °n i1 °k j1 πij }ξi ¡ζj }2 s. t. πij € t0, 1u, ζj € Rd °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k Ÿ A remedy for the practical challenges . Ÿ What about the technical challenges?
- 9. Linearization Convexification min °n i1 °k j1 πij }ξi ¡ζj }2 s. t. πij € t0, 1u, ζj € Rd °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k Ÿ The problem is NP-hard. Ÿ Heuristics for biconvex optimization can still be used.6 Ÿ No runtime/optimality guarantees. 6 Bennett et al. (2000)
- 10. Linearization Convexification Conic Relaxations MILP Feasible Solution enlarge feasible set rounding algorithm recovery guarantee Ÿ The problem is NP-hard. Ÿ Heuristics for biconvex optimization can still be used.6 Ÿ No runtime/optimality guarantees. Ÿ We propose a convex relaxation that comes with guarantees. 6 Bennett et al. (2000)
- 11. Linearization Ÿ Equivalent MINLP reformulation. min °k j1 °n i,i1 1 1 2nj πij πi1 j }ξi ¡ξi1 }2 r costpπqs s. t. πij € t0, 1u °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k (P) Ÿ The products πij πi1 j can be linearized, resulting in an MILP. Zha et al. (2001)
- 12. Convex Relaxation Ÿ Apply the following variable transformations: xj : 2πj ¡1, D rdii1 s : }ξi ¡ξi1 }2 . Ÿ The MILP admits an equivalent non-linear reformulation. min °k j1 1 8nj p1 xj qp1 xj q , D s. t. xj € t¡1, 1un °k j0 xj p1 ¡kq1 1 xj 2nj ¡n dj 0, . . . , k Ÿ Introduce Mj xj xj to linearize the objective function °k j1 1 8nj 11 1xj xj 1 Mj , D .
- 13. Convex Relaxation Ÿ In doing so, non-convexity is relegated to the constraints xj € t¡1, 1un , Mj xj xj . Ÿ The resulting MINLP can be relaxed to an SDP (RSDP) min °k j1 1 8nj 11 1xj xj 1 Mj , D s. t. xj € Rn , Mj € Sn °k j0 xj p1 ¡kq1 1 xj 2nj ¡n dj 0, . . . , k diagpMj q 1, Mj © xj xj dj 0, . . . , k (RSDP) Goemans Williamson (1995)
- 14. Convex Relaxation Ÿ In doing so, non-convexity is relegated to the constraints xj € t¡1, 1un , Mj xj xj . Ÿ The resulting MINLP can be relaxed to an SDP (RSDP) min °k j1 1 8nj 11 1xj xj 1 Mj , D s. t. xj € Rn , Mj € Sn °k j0 xj p1 ¡kq1 1 xj 2nj ¡n dj 0, . . . , k diagpMj q 1, Mj © xj xj dj 0, . . . , k (RSDP) Ÿ Unfortunately, this SDP relaxation is very weak. Goemans Williamson (1995)
- 15. Valid Inequalities Ÿ Strengthen RSDP with the valid cuts. Mj 1 γ γ 1 , xj α β Ex. γ 0.4 Mj xj xj Mj © xj xj Mj © xj xj + VCs Anstreicher (2009)
- 16. Valid Inequalities Ÿ Strengthen RSDP with the valid cuts. Mj 1 γ γ 1 , xj α β Ex. γ 0.4 Mj xj xj Mj © xj xj Mj © xj xj + VCs Ÿ These cuts are instrumental to proving optimality guarantees. Ÿ As a by-product, we also have an LP relaxation RLP. Anstreicher (2009)
- 17. Convex Relaxation Theorem 1 We have min RLP ¤ min RSDP ¤ min P. Ÿ RLP RSDP are polynomial-time solvable, whereas P is not. Next Steps: Ÿ How to construct ˜π ij € t0, 1u feasible in P from x ij ? Ÿ How to gauge the quality of the obtained ˜π ij ?
- 18. Rounding Algorithm Ÿ Recall that x ij € r¡1, 1s: 1 2 p1 x ij q P pξi € CLj q Ÿ Solve the following linear assignment problem to retrieve ˜π. max °n i1 °k j1 πij 1 2 p1 x ij q s. t. πij € t0, 1u °k j0 πij 1 di 1, . . . , n °n i1 πij nj dj 0, . . . , k Ÿ LAP7 is an MILP with totally unimodular matrix (4 tractable). 7 Burkard et al. (2009)
- 19. Optimality Gap Ÿ Our approach gives an a posteriori estimate of optimality gap. min RLP ¤ min RSDP ¤ min P costpπ q ¤ costp˜π q Ÿ Under perfect separation condition, the optimality gap vanishes. Elhamifar et al. (2012)
- 20. Recovery Guarantee Theorem 2 (Perfect Separation) We have tightness hkkkkkkkkkkkkkkkkkkikkkkkkkkkkkkkkkkkkj min RLP min RSDP min P costpπ q costp˜π qloooooooooooooooooomoooooooooooooooooon LAP-recovery . Proof Sketch (Tightness): 1 Distinguish outlier (M 0) and regular (M j ) clusters. 2 The RLP/RSDP can be solved analytically.
- 21. Recovery Guarantee Theorem 2 (Perfect Separation) We have tightness hkkkkkkkkkkkkkkkkkkikkkkkkkkkkkkkkkkkkj min RLP min RSDP min P costpπ q costp˜π qloooooooooooooooooomoooooooooooooooooon LAP-recovery . Proof Sketch (Tightness): M 0 M j
- 22. Recovery Guarantee Theorem 2 (Perfect Separation) We have tightness hkkkkkkkkkkkkkkkkkkikkkkkkkkkkkkkkkkkkj min RLP min RSDP min P costpπ q costp˜π qloooooooooooooooooomoooooooooooooooooon LAP-recovery . Proof Sketch (Recovery): 1 Distinguish outlier (M 0) and regular (M j ) clusters. 2 The RLP/RSDP can be solved analytically. 3 The LAP exploits strong signal in M 0 and M j .
- 23. Recovery Guarantee Theorem 2 (Perfect Separation) We have tightness hkkkkkkkkkkkkkkkkkkikkkkkkkkkkkkkkkkkkj min RLP min RSDP min P costpπ q costp˜π qloooooooooooooooooomoooooooooooooooooon LAP-recovery . Proof Sketch (Recovery): M 0 M j
- 24. Numerical Experiments I Ÿ Perform cardinality-constrained clustering on classification datasets8 . Ÿ tnj uk j1 : the number of true class occurrences. Ÿ Compare our SDP/LP+LAP approach with biconvex heuristic9 . Ÿ Optimality gaps yielded by the LP+LAP approach are À 20.6%. Ÿ Optimality gaps yielded by the SDP+LAP approach are À 2.9%. Ÿ The SDP approach is competitive with the biconvex heuristic. 8 UCI repository 9 Bennett, K. et al. (2000)
- 25. Numerical Experiments II Ÿ Perform outlier detection on the breast cancer dataset10 . Ÿ Varying the number of malignant cancers n0. Ÿ Calculate prediction accuracy, false positives false negatives. 0 50 100 150 200 250 300 350 400 0 10 20 30 40 50 60 70 80 90 100 Á 80% prediction accuracy, À 3.3% optimality gap. 10 UCI repository
- 26. References Ÿ Arthur, D. and Vassilvitskii, S. K-means++: the advantages of careful seeding. Proceedings of ACM-SIAM Symposium on Discrete Algorithms, 2007. Ÿ Bennett, K., Bradley, P., Demiriz, A. Constrained k–Means Clustering. Microsoft Technical Report, 2000. Ÿ Rujeerapaiboon, N., Schindler, K., Kuhn, D., Wiesemann, W. Size matters: Cardinality-constrained clustering and outlier detection via conic optimization. SIAM Journal on Optimization 29(2), 2019. napat.rujeerapaiboon@nus.edu.sg Special thanks to artwork from tPopcorns Arts, Maxim Basinski, Business strategy, Freepik, Prosymbols, Vectors Market, Madebyoliver, Alfredo Hernandez, Devil, Roundiconsu@Flaticon.