C++ Corehard Autumn 2018. Обучаем на Python, применяем на C++ - Павел Филонов

Train with Python,
predict with C++
Pavel Filonov
+
+ +
+
+
+
+

About me
Pavel Filonov
Research Development Team Lead at
Kaspersky Lab
01
CoreHard. Train with Python, predict with C++

Machine Learning everywhere!
02
• Mobile
• Embedded
• Automotive
• Desktops
• Games
• Finance
• Etc.
Image from [1]

Machine Learning face the real world
03
Image from [2]

CRISP-DM
04
Image from [3]

Dream team
05
Developer Data Scientist

Dream team – synergy way
06
Developer Data Scientist Research Developer

Dream team – process way
07
Developer Data Scientist
Communications

Machine learning sample cases
08
1. Energy efficiency prediction
2. Intrusion detection system
3. Image classification

Buildings Energy Efficiency
09
ref: [4]
● Input attributes
○ Relative Compactness
○ Surface Area
○ Wall Area
○ etc.
● Outcomes
○ Heating Load

Regression problem
10

Regression problem
11

Regression problem
12

Quality metric
13
●

Baseline model
14
●
class Predictor {
public:
using features = std::vector<double>;
virtual ~Predictor() {};
virtual double predict(const features&) const = 0;
};
class MeanPredictor: public Predictor {
public:
MeanPredictor(double mean);
double predict(const features&) const override { return mean_; }
protected:
double mean_;
};

Linear regression
15
●
class LinregPredictor: public Predictor {
public:
LinregPredictor(const std::vector<double>&);
double predict(const features& feat) const override {
assert(feat.size() + 1 == coef_.size());
return std::inner_product(feat.begin(), feat.end(), ++coef_.begin(),
coef_.front());
}
protected:
std::vector<double> coef_;
};

Polynomial regression
16
●
class PolyPredictor: public LinregPredictor {
public:
using LinregPredictor::LinregPredictor;
double predict(const features& feat) const override {
features poly_feat{feat};
const auto m = feat.size();
poly_feat.reserve(m*(m+1)/2);
for (size_t i = 0; i < m; ++i) {
for (size_t j = i; j < m; ++j) {
poly_feat.push_back(feat[i]*feat[j]);
}
}
return LinregPredictor::predict(poly_feat);
}
};

Integration testing
17
● you always have a lot of data for testing
● use python model output as expected values
● beware of floating point arithmetic problems
TEST(LinregPredictor, compare_to_python) {
auto predictor = LinregPredictor{coef};
double y_pred_expected = 0.0;
std::ifstream test_data{"../train/test_data_linreg.csv"};
while (read_features(test_data, features)) {
test_data >> y_pred_expected;
auto y_pred = predictor.predict(features);
EXPECT_NEAR(y_pred_expected, y_pred, 1e-4);
}
}

Intrusion detection system
18
● input - network traffic features
○ protocol_type
○ connection duration
○ src_bytes
○ dst_bytes
○ etc.
● Output
○ normal
○ network attack
ref: [5]

Classification problem
19

Quality metrics
20
● Receive operation characteristics (ROC) curve

Baseline model
21
● always predict most frequent class
● ROC area under the curve = 0.5

Logistic regression
22
●

Logistic regression
23
● easy to implement
template<typename T>
auto sigma(T z) {
return 1/(1 + std::exp(-z));
}
class LogregClassifier: public BinaryClassifier {
public:
float predict_proba(const features_t& feat) const override {
auto z = std::inner_product(feat.begin(), feat.end(),
++coef_.begin(), coef_.front());
return sigma(z);
}
protected:
std::vector<float> coef_;
};

Gradient boosting
24
● de facto standard universal method
● multiple well known C++ implementations with python bindings
○ XGBoost
○ LigthGBM
○ CatBoost
● each implementation has its own custom model format

CatBoost
25
● C API and C++ wrapper
● own build system (ymake)
class CatboostClassifier: public BinaryClassifier {
public:
CatboostClassifier(const std::string& modepath);
~CatboostClassifier() override;
double predict_proba(const features_t& feat) const override {
double result = 0.0;
if (!CalcModelPredictionSingle(model_, feat.data(), feat.size(),
nullptr, 0, &result, 1)) {
throw std::runtime_error{"CalcModelPredictionFlat error message:" +
GetErrorString()};
}
return result;
}
private:
ModelCalcerHandle* model_;
}

CatBoost
26
● ROC-AUC = 0.9999

Image classification
27
● Handwritten digits recognizer – MNIST
● input – gray-scale pixels 28x28
● output – digit on picture (0, 1, … 9)

Multilayer perceptron
28
Image from: [6]

Quality metrics
29
●

30
●
auto MlpClassifier::predict_proba(const features_t& feat) const {
VectorXf x{feat.size()};
auto o1 = sigmav(w1_ * x);
auto o2 = softmax(w2_ * o1);
return o2;
}

31
●
auto MlpClassifier::predict_proba(const features_t& feat) const {
VectorXf x{feat.size()};
auto o1 = sigmav(w1_ * x);
auto o2 = softmax(w2_ * o1);
return o2;
}

Convolutional networks
31
● State of the Art algorithms in image processing
● a lot of C++ implementation with python bindings
○ TensorFlow
○ Caffe
○ MXNet
○ CNTK

Tensorflow
31
● C++ API
● Bazel build system
● Hint – prebuild C API

Conclusion
31
● Don’t be fear of the ML
● Try simpler things first
● Get benefits from different languages

References
02
1. Andrew Ng, Machine Learning – coursera
2. How to Get the Right Creative Requirements From Your Client
3. The Forgotten Step in CRISP-DM and ASUM-DM Methodologies
4. Energy efficiency Data Set
5. KDD Cup 1999
6. MNIST training with Multi Layer Perceptron
7. Code samples

+
+
+
+
+
+
+
Many thanks!
Let’s talk
Pavel Filonov Pavel.Filonov@kaspersky.com +7(966)077-32-80
+

C++ Corehard Autumn 2018. Обучаем на Python, применяем на C++ - Павел Филонов

More Related Content

What's hot (20)

Similar to C++ Corehard Autumn 2018. Обучаем на Python, применяем на C++ - Павел Филонов (20)

More from corehard_by (20)

Recently uploaded (20)

C++ Corehard Autumn 2018. Обучаем на Python, применяем на C++ - Павел Филонов