Least Square with L0, L1, and L2 Constraint
0 likes1,728 views
The document discusses least square regression with L0, L1, and L2 constraints. It presents the cost functions for each constraint and derives the solutions. For L0 constraint, the solution is a hard threshold determined by comparing the squared value to the regularization parameter. For L1 constraint, the solution is a soft threshold. For L2 constraint, the solution is obtained by setting the derivative of the cost function to zero. It also formulates L0 and L1 constraints as proximal operators.
Least Square with L0, L1, and L2 Constraint
- 1. Masayuki Tanaka Jun. 22, 2016 Least Square with L0, L1, and L2 Constraint
- 2. Cost Functions 𝐸2 𝑥 = 𝑦 − 𝑥 2 + 𝜆2 𝑥 2 𝑥 2 = 𝑥2 𝐸1 𝑥 = 𝑦 − 𝑥 2 + 𝜆1 𝑥 1 𝑥 1 = 𝑥 (𝑥 ≥ 0) −𝑥 (𝑥 < 0) 𝐸0 𝑥 = 𝑦 − 𝑥 2 + 𝜆0 𝑥 0 𝑥 0 = 0 (𝑥 = 0) 1 (𝑥 ≠ 0)
- 3. Least Square with L0 constraint 𝐸0 𝑥 = 𝑦 − 𝑥 2 + 𝜆0 𝑥 0 𝑥 0 = 0 (𝑥 = 0) 1 (𝑥 ≠ 0) 𝑥0 = 0 (𝑦2 < 𝜆0) 𝑦 (𝑦2 ≥ 𝜆0) Hard threshold Solution:
- 4. Least Square with L1 constraint 𝑥1 = sign 𝑦 𝑦 − 𝜆1/2 + sign 𝜉 = −1 (𝜉 < 0) 0 (𝜉 = 0) 1 (𝜉 > 0) 𝜉 + = max 𝜉, 0 = 𝜉 (𝜉 > 0) 0 (𝜉 ≤ 0) Soft threshold 𝐸1 𝑥 = 𝑦 − 𝑥 2 + 𝜆1 𝑥 1 𝑥 1 = 𝑥 (𝑥 ≥ 0) −𝑥 (𝑥 < 0) Solution:
- 5. Least Square with L2 constraint 𝜕𝐸2 𝜕𝑥 = 𝑥 − 𝑦 + 𝜆2 𝑥 = 0 𝑥2 = 𝑦 1 + 𝜆2 𝐸2 𝑥 = 𝑦 − 𝑥 2 + 𝜆2 𝑥 2 𝑥 2 = 𝑥2 Solution:
- 6. Derivation of Least Square with L0 constraint 𝐸0 𝑥 = 𝑦 − 𝑥 2 + 𝜆0 𝑥 0 𝑥 0 = 0 (𝑥 = 0) 1 (𝑥 ≠ 0) 𝑥 = 0 𝑥 ≠ 0 𝐸0 0 = y2 𝐸0 𝑥 = 𝑦 − 𝑥 2 + 𝜆0 min 𝑥≠0 𝐸0 𝑥 = 𝜆0 𝑦 𝜆0− 𝜆0 𝑥0 = 0 𝐸0 𝑥0 = 𝑦2 𝑥0 = 𝑦 𝐸0 𝑥0 = 𝜆0 𝑥0 = 𝑦 𝐸0 𝑥0 = 𝜆0 𝑥0 = 0 (𝑦2 < 𝜆0) 𝑦 (𝑦2 ≥ 𝜆0) Hard threshold
- 7. Formulation as a Proximal Operator prox 𝐿0,𝜌 𝑦 = arg min 𝑥 𝑥 0 + 𝜌 2 𝑦 − 𝑥 2 𝑥 0 = 0 (𝑥 = 0) 1 (𝑥 ≠ 0) = 0, 𝑦2 < 2 𝜌 1, 𝑦2 ≥ 2 𝜌
- 8. Formulation as a Proximal Operator prox 𝐿1,𝜌 𝑦 = arg min 𝑥 𝑥 1 + 𝜌 2 𝑦 − 𝑥 2 𝑥 1 = 𝑥 (𝑥 ≥ 0) −𝑥 (𝑥 < 0) = sign 𝑦 𝑦 − 1/𝜌 + sign 𝜉 = −1 (𝜉 < 0) 0 (𝜉 = 0) 1 (𝜉 > 0) 𝜉 + = max 𝜉, 0 = 𝜉 (𝜉 > 0) 0 (𝜉 ≤ 0)