Application of Chebyshev and Markov Inequality in Machine Learning
- 1. Application of Chebyshev and Markov Inequality in Supervised Machine Learning Domain: Application of Supervised Machine Learning Dr. Varun Kumar Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 1 / 9
- 2. Outlines 1 Introduction to Chebyshev Inequality 2 Introduction to Markov Inequality 3 Introduction to Supervised Learning 4 Application of these Inequalities in Supervised Machine Learning 5 References Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 2 / 9
- 3. Introduction to Chebyshev Inequality Mathematical Description: General mathematics for continuous random variable: ⇒ Mean E(x) = µ = ∞ −∞ xfX (x)dx (1) ⇒ Variance E((x − µ)2 ) = σ2 = ∞ −∞ (x − µ)2 fX (x)dx (2) Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 3 / 9
- 4. Chebyshev inequality ∞ −∞ (x − µ)2 fX (x)dx ≥ |x−µ|≥ (x − µ)2 fX (x)dx (3) Taking the minimum value, i.e |x − µ| = → Finite deviation |x−µ|≥ (x − µ)2 fX (x)dx = |x−µ|≥ 2 fX (x)dx = 2 P(X − µ ≥ ) (4) From (2) and (4) 2 P(X − µ ≥ ) ≤ σ2 ⇒ P(X − µ ≥ ) ≤ σ2 2 (5) Case 1: when = nσ P(X − µ ≥ ) = P(X − µ ≥ nσ) ≤ 1 n2 (6) Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 4 / 9
- 5. Continued– As per the properties of probability, P(X ≤ µ) + P(X ≥ µ) = 1. Hence, P(X − µ ≤ ) ≤ 1 − σ2 2 ⇒ P(X − µ ≤ nσ) ≤ 1 − 1 n2 (7) For discrete random variable: Mean E(x) = µ = ∞ i=−∞ xi PX (xi ) (8) Variance Var(x) = σ2 = E (x − µ)2 = ∞ i=−∞ (xi − E(x))2 PX (xi ) (9) PX (.) → Probability mass function. Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 5 / 9
- 6. Markov inequality P(X − µ ≤ ) ≤ 1 − σ2 2 ⇒ P(X − µ ≤ nσ) ≤ 1 − 1 n2 (10) Markov inequality Statement: If X is a positive random variable, i.e X > 0, having probability density function fX (x). Let a is an positive arbitrary constant, then P(X ≥ a) ≤ E(x) a (11) Proof: As per the properties of random variable, E(x) = ∞ 0 xfX (x)dx ≥ ∞ a xfX (x)dx Let x = a, then ⇒ E(x) ≥ ∞ a xfX (x)dx ≥ ∞ a afX (x)dx = aP(X > a) Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 6 / 9
- 7. Introduction to supervised learning Supervised learning 1 It is a method of learning, where some set of predefined training data is available. 2 Based on these training data or sequence, a mathematical or logical model is developed. 3 This training data sequence or developed model through these data acts as a supervisor. 4 When new data comes then it is expected that the data will follow the developed model. 5 For developing a model through these training data, we may utilize some well defined statistical, mathematical or logical model. 6 Those model gives a minimum mean square error value, that may be selected as a most suitable model. Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 7 / 9
- 8. Relation between supervised learning and inequality 1. Decision action plays an important role in machine learning. 2. Inequality relation helps for making a decision favorable or non-favorable region. 3. Statistical framework helps for modeling the synthetic data that is nothing but the theoretical bound. 4. Applying Chebyshev inequality, there is requirement of variance of the data sequence. It is independent from the type of distribution. 5. From relation (7) and (10), we can predict or find the probability of any real world new data that is above or below some threshold value. 6. Applying Markov inequality, only mean value is required for finding probability. It also independent from density function. Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 8 / 9
- 9. References J. Navarro, “A very simple proof of the multivariate chebyshev’s inequality,” Communications in Statistics-Theory and Methods, vol. 45, no. 12, pp. 3458–3463, 2016. M. I. Jordan and T. M. Mitchell, “Machine learning: Trends, perspectives, and prospects,” Science, vol. 349, no. 6245, pp. 255–260, 2015. Domain: Application of Supervised Machine Learning Dr. Varun Kumar (IIIT Surat)Lecture 9 9 / 9