![Q-learning
action discrete ïŹnite set
a : epsilon greedy a =
(
random, prob(â)
arg max
x
Q(s, x), prob(1 â)
new old learning
rate
state s, action a, state sâ, reward r
aâ: state sâ, action
Q(s, a) Q(s, a) + (â”) †[r + max
a0
Q(s0
, a0
) Q(s, a)]
, 0r + max
a0
Q(s0
, a0
) Q(s, a)
Value Iteration](https://image.slidesharecdn.com/mlfortradig180328-180911051602/85/Machine-Learning-for-Trading-52-320.jpg)
![RNN(LSTM)
label: [1,0,0] ( /long day 6 close - day 5 close > threshold)
label: [0,0,1] ( /short day 6 close - day 5 close < - threshold)
K
D1
D2
DD3
D4
D5
D1: (K )
D2: (K )
D5: (K )
.
.
LSTM
Input
LSTM
(LSTM)
fully connected
Ëy0, Ëy1, Ëy2,
X
= 1
y,y2, y3
cross entropy
one-hot
(Keras: 10](https://image.slidesharecdn.com/mlfortradig180328-180911051602/85/Machine-Learning-for-Trading-57-320.jpg)
![Using CNN
Input
CNN
(CNN)
fully connected
Ëy0, Ëy1, Ëy2,
X
= 1
y,y2, y3
cross entropy
K
D1
D2
DD3
D4
D5
D1: (K )
D2: (K )
D5: (K )
.
.
CNN Sliding window
Self DeïŹne Features
label: [1,0,0] ( /long day 6 close - day 5 close > threshold)
label: [0,0,1] ( /short day 6 close - day 5 close < - threshold)](https://image.slidesharecdn.com/mlfortradig180328-180911051602/85/Machine-Learning-for-Trading-60-320.jpg)
![2: Using RL for Portfolio
Optimization
Asset : [S1, S2]
S1, S2 : Stocks with predicted growth
Use RL to Optimize Portfolio
A = w1S1 + w2S2
w1 + w2 = 1
Consider action: w1 : {0.10, 0.25, 0.5, 0.75, 0.9}
Reward: Rt = log
At
At 1
How about n-stocks ?](https://image.slidesharecdn.com/mlfortradig180328-180911051602/85/Machine-Learning-for-Trading-64-320.jpg)
Machine Learning for Trading
- 1. ML for Trading ML/DL / BU (BU Head) Larry Guo larryguo5078@gmail.com, FB: ( ) May 16th, 2018 : Marketing, Sales, Technical Marketing, Legal / , BU Head, .
- 2. Disclaimer âą talk, ... or ..... âą âą / ( , ) âą / Google image
- 4. Agenda âąMachine Learning/Deep Learning ( / âą ML for Trading âą ,, the path to AI
- 5. vs AI + + IfâŠ. buy/ hold/ sell data arg min â Loss(f(X)) learning fËâ(X) model rule base leanring base
- 6. = =
- 7. AI@ Finance
- 8. ML Basic DL âą ML âą AI
- 9. ? âą ?? ⎠( , ) âŽ(vs , , label, ground truth) ⎠( ) (Loss/objective function) ⎠⹠⹠( ) âą Lear from Error ( ) â†f(X, â), vs.{(x1, y1), (x2, y2), ..(xn, yn)} s.t., fâ†(xi) ! yi
- 10. Types of Learning Learn from Paired data (with Label)Supervised Learning Learn from Un-Paired data (without Label) Un-Supervised Learning Learn from Action, reward) Reinforcement Learning
- 11. Supervised Learning ClassiïŹcation : (1,2,3) vs. (1,5,6) Regression vs: %
- 13. Reinforcement Learning State Action Reward Environment Agent Stateâ Agentâs Objective: Find best policy max E " X i i Ri # i Ri
- 14. (Supervised Learning) âąClassiïŹcation ( / , ,Buy/Hold/Sell) P(y|X) (X, y) pair data set f(X) = Ëy ! y X = features y = label âąRegression ( ) eg. p(Ëy1 = cat|x1) > p(Ëy1 = dog|x1) ! cat
- 16. GAN celebA 1024
- 17. Learn from Loss min(loss) y: loss: : â dy dâdy dâ â =: â â †dy dâ â =: â dy dâ r : Gradient Gradient Decent , â =: â â †rây
- 18. Robust Under ïŹtting Over ïŹtting â â << > >Generalization Train/Test Split, No Data Leak !
- 19. Mean Squared Error Ëy = ax1 + bx2 + cx3 L = 1 N X kËy yk2
- 20. Cross Entropy H(p, p) = X x p(x) log p(x) 0 = X x p(x) log 1 p(x)entropy DKL(p||q) = X x p(x) log p(x) q(x) q p , ?? p = q, =0
- 21. Cross Entropy DKL(p||q) = X x p(x) log 1 q(x) ( X x p(x) log 1 p(x) ) DKL(p||q) = H(p, q) H(p) (if given p) min DKL(p||q) â min H(p, q) cross entropy H(p, q) = X x p(x) log q(x) or X i yi log Ëyi or log Ëy for binary i= X n X i yn i log Ëyn i n=#sample min (- log likelihood) ~ max (likelihood)
- 22. X: Feature( ) K_D RSI ROE Day1 Day2 .. DayN { ) Time dependency To predict Day(N+1) ?? Domain Knowledge!!
- 23. y: labeling âąCategory Data into Numeric Encoding, (or even, One_Hot Encoding) yi 2 D âą {A, B, C, D, E, F} , A = 0 B = 1 C = 2 D = 3 E = 4 F = 5
- 24. y:One-Hot 0 0 1 0 0 0 0 yËy 0.1 0.05 0.05 0.748 0.05 0.001 0.001 maximize minimize minimize X Model fËâ(X) sum=1 p Ëy to estimate y loss = X yi log(Ëyi) = log(0.748) Softmax!
- 25. ML Frameworks âąRegression (Logistic Regression) âąSVM (Kernel Trick.) , LibSVM âąDecision Tree, Random Forest, XGBoost âąDeep Learning
- 26. : Decision Tree
- 27. âąimport numpy as np âąfrom sklearn.model_selection import train_test_split ⎠âąX_train, X_test, y_train,y_test = train_test_split (X,y, test_size=0.2,shufïŹe=True) âŽ# X (feature), y (label) , # 80% training 20% test . âąfrom sklearn.tree import DecisionTreeClassiïŹer âąclf = DecisionTreeClassiïŹer() âąclf.ïŹt(X_train, y_train) ⎠Deciion tree ; / ; clf = classiïŹer âąclf.score(X_test, y_test) # test set âąclf.predict (new_data) #
- 28. Tree to Forest !
- 29. Deep Learning
- 30. Deep Learning W1 W2 W3 X1 X2 X3 Neural Network Ëy yËy = ( X i WiXi) ËËy = ( X i Ui Ëyi) y activation eg 0,1
- 31. FCN, fully connected, dense Neural Network 2 4 x1 x2 x3 3 5 2 6 6 6 4 h (1) 1 h (1) 2 h (1) 3 h (1) 4 3 7 7 7 5 = 2 6 6 4 w11 w12 w13 w21 . . w31 . . w41 w42 w43 3 7 7 5 h (1) i = 0 @ X j wijxj 1 A Neural Network, â â(tensor) ( : ) !
- 33. Back Propagation Forward Pass Backward Pass loss
- 35. Dot Product ??? ~u ~vi ~vi âą ~u = kvkkuk cos â cosine similarity target
- 37. 2D Convolution 1 0 1 0 1 0 1 0 1 ??? Kernel , ïŹlter,
- 39. Filter on Image Filter ? detect detect patternâš detecte
- 42. ImageNet Competition 100 1000 â12 AlexNet, DL VGG 19, Google Inception (NiN) , ResNet 152 : Transfer Learning,
- 44. Convolution Auto Encoder dim: 1024 x 768 dim: 300 input output Representation Learning DNA Neural Network
- 45. Train your own classiïŹer Neural style Transfer Face Recognition
- 46. Recurrent Neural Network RNN, LSTM, GRU This movie is good, really not bad This movie is bad, really not good
- 47. Sequence Modeling When the recurrent network is trained to perform a task that requires predicting the future from the past, the network typically learns to use h(t) as a kind of lossy summary of the task-relevant aspects of the past sequence of inputs up to t. This summary is in general necessarily lossy, since it maps an arbitrary length sequence (x(t), x(t 1), x(t 2), . . . , x(2), x(1)) to a ïŹxed length vector h(t). Depending on the training criterion, this summary might selectively keep some aspects of the past sequence with more precision than other aspects. For example, if the RNN is used in statistical language modeling, typically to predict the next word given previous words, it may not be necessary to store all of the information in the input sequence up to time t, but rather only enough information to predict the rest of the sentence. The most demanding situation is when we ask h(t) to be rich enough to allow one to approximately recover the input sequence, as in autoencoder frameworks (chapter 14). ff hh xx h(t 1) h(t 1) h(t) h(t) h(t+1) h(t+1) x(t 1) x(t 1) x(t) x(t) x(t+1) x(t+1) h(... ) h(... ) h(... ) h(... ) ff Unfold ff ff f Figure 10.2: A recurrent network with no outputs. This recurrent network just processes information from the input x by incorporating it into the state h that is passed forward through time. (Left)Circuit diagram. The black square indicates a delay of a single time step. (Right)The same network seen as an unfolded computational graph, where each node is now associated with one particular time instance. Equation 10.5 can be drawn in two diïŹerent ways. One way to draw the RNN is with a diagram containing one node for every component that might exist in a 376 h(t) = f(h(t 1) , x(t) , â) input xt 1 ht 1 Ëyt 1 Ëyt Ëyt+1 ht+1 xt+1xt ht P(Ëyt = buy|x1, x2, ...xt 1, xt) L = X t Lt = X i log P(yt|x1, x2, ...xt)
- 48. RNN Various Type Image ClassiïŹcation Sentiment Analysis , ( ) Image Captioning MNIST A Baby Eating a piece of paper
- 49. RNN Various Type Character RNN Deep Shakespear Deep Math Project Magenta NLP, Translation Seq2Seq Google Translation representation
- 50. âDeep Math â source: The Unreasonable Effectiveness of Recurrent Neural Networks, Andrej Karpathy
- 51. Reinforcement Learning State Action Reward Environment Agent Stateâ Agentâs Objective: Find best policy max E " X i i Ri # i Ri
- 52. Q-learning action discrete ïŹnite set a : epsilon greedy a = ( random, prob(â) arg max x Q(s, x), prob(1 â) new old learning rate state s, action a, state sâ, reward r aâ: state sâ, action Q(s, a) Q(s, a) + (â”) †[r + max a0 Q(s0 , a0 ) Q(s, a)] , 0r + max a0 Q(s0 , a0 ) Q(s, a) Value Iteration
- 53. Deep Q-Learning state action Q- leanring Q(s,a) w =: w â †rwLoss(w) state Q(s,a0) Q(s,a1) Q(s,a2) Neural Network as Q function a0:sell a1:hold a2:buy , 0r + max a0 Q(s0 , a0 ) Q(s, a) Loss(w) = (r + max a0 Qw(s0 , a0 ) Qw(s, a))2 w
- 54. Modeling Trading
- 55. Modeling Price Based/ Technical Analysis Factor Model Event/Text
- 56. K D1 D2 D3 D4 D5 K P(SD6|(DK 1 , DK 2 , DK 3 ..DK 5 ) { 1 0 1 K D1 D2 DD3 D4 D5 LSTM or GRU (RNN) D6 ? ? buy/sell signal
- 57. RNN(LSTM) label: [1,0,0] ( /long day 6 close - day 5 close > threshold) label: [0,0,1] ( /short day 6 close - day 5 close < - threshold) K D1 D2 DD3 D4 D5 D1: (K ) D2: (K ) D5: (K ) . . LSTM Input LSTM (LSTM) fully connected Ëy0, Ëy1, Ëy2, X = 1 y,y2, y3 cross entropy one-hot (Keras: 10
- 58. Technical Index as Features K D RSV MA5 MA20 OBV MACD D1 D2 D3 D4 D5
- 59. Factor Model as Features Rev G% EPS PE ROE M1 M2 M3 M4 M5 {Preprocess /
- 60. Using CNN Input CNN (CNN) fully connected Ëy0, Ëy1, Ëy2, X = 1 y,y2, y3 cross entropy K D1 D2 DD3 D4 D5 D1: (K ) D2: (K ) D5: (K ) . . CNN Sliding window Self DeïŹne Features label: [1,0,0] ( /long day 6 close - day 5 close > threshold) label: [0,0,1] ( /short day 6 close - day 5 close < - threshold)
- 61. Consider Candle Chart as Image (Using CNN) source: DEEP STOCK REPRESENTATION LEARNING: FROM CANDLESTICK CHARTS TO INVESTMENT DECISIONS Candlestick image Convolution Auto Encoder, VGG 512D Representation Clustering Method: Network Modularity Find Best Stock in each cluster ¯”
- 62. Using Reinforcement Learning state Q(s,a0) Q(s,a1) Q(s,a2) Neural Network as Q function a0:sell a1:hold a2:buy K D1 D2 DD3 D4 D5 Position Sizing, Asset + {LSTM/CNN get representation Or, Reward ? Fix a time T as a episode for all t <=T R_t = log return of day t Rt = log Rt Rt 1
- 63. 1: Day Trading âą Long/Short features âąNo perfect answer âą Features ? âą Label ? loss ? âą
- 64. 2: Using RL for Portfolio Optimization Asset : [S1, S2] S1, S2 : Stocks with predicted growth Use RL to Optimize Portfolio A = w1S1 + w2S2 w1 + w2 = 1 Consider action: w1 : {0.10, 0.25, 0.5, 0.75, 0.9} Reward: Rt = log At At 1 How about n-stocks ?
- 65. 3: Inspired from Project Magenta source: Tuning Recurrent Neural Networks with Reinforcement Learning : note , as input of character RNN RL: (reward) Character RNN RL ?âš (min DD, ..or ....)
- 66. Path Two roads diverged in a yellow wood, And sorry I could not travel both And be one traveler, long I stood And looked down one as far as I could ~ Robert Frost
- 68. : Overwhelming âą MOOC, CS231n, CS224n, ⊠Arxiv, Youtube Udacity, Coursera, âąOpen Source PackageâŠgithub, âą âą
- 69. ML
- 70. , âą Excel âąLearn to be focused: Leverage, to be Focused, for at least 45 minutes , Flora, ) âą 30 ⎠30 10 ... ⎠30 , ) âą / .
- 71. Correct Path ?
- 72. Thank you!
- 74. âąML/DL: Coursera: Andrew NG, Udacity âąML/DL: , âąCNN: CS231n âąNLP: CS224n âąRL: CS294, CS234 (no video), David Silver âąbook: Hands on machine Learning & deep learning with tensor ïŹow âąbook: Deep Learning ( ) âąRL: David Silver