Estimating Financial Risk with Apache Spark
2 likes2,936 views
This document discusses estimating financial risk using Apache Spark. It introduces the value-at-risk (VaR) metric, which estimates the maximum potential loss of a portfolio over a given time period and probability. It describes approaches for VaR estimation including variance-covariance, historical simulation, and Monte Carlo simulation. It then explains how to predict instrument returns from market risk factors using linear regression models in Spark. The document outlines sampling factor returns from a multivariate normal distribution and running simulations across a Spark cluster to estimate VaR and expected shortfall. It argues that Spark enables easier, more powerful financial risk analysis by allowing joint processing, simulation matrices to be saved in memory, and calling GPUs for matrix operations.
![import org.apache.commons.math3.stat.regression.OLSMultipleLinearRegression
// Load the instruments and factors
val factorReturns: Array[Array[Double]] = ...
val instrumentReturns: RDD[Array[Double]] = ...
// Fit a model to each instrument
val models: Array[Array[Double]] =
instrumentReturns.map { instrument =>
val regression = new OLSMultipleLinearRegression()
regression.newSampleData(instrument, factorReturns)
regression.estimateRegressionParameters()
}.collect()](https://image.slidesharecdn.com/slidesharerisktalk-150911155504-lva1-app6891/85/Estimating-Financial-Risk-with-Apache-Spark-15-320.jpg)
![import org.apache.commons.math3.stat.correlation.Covariance
// Compute means
val factorMeans: Array[Double] = transpose(factorReturns)
.map(factor => factor.sum / factor.size)
// Compute covariances
val factorCovs: Array[Array[Double]] = new Covariance(factorReturns)
.getCovarianceMatrix().getData()](https://image.slidesharecdn.com/slidesharerisktalk-150911155504-lva1-app6891/85/Estimating-Financial-Risk-with-Apache-Spark-21-320.jpg)
![def trialReturn(factorDist: MultivariateNormalDistribution, models: Seq[Array[Double]]): Double = {
val trialFactorReturns = factorDist.sample()
var totalReturn = 0.0
for (model <- models) {
// Add the returns from the instrument to the total trial return
for (i <- until trialFactorsReturns.length) {
totalReturn += trialFactorReturns(i) * model(i)
}
}
totalReturn
}](https://image.slidesharecdn.com/slidesharerisktalk-150911155504-lva1-app6891/85/Estimating-Financial-Risk-with-Apache-Spark-24-320.jpg)
![// Broadcast the factor return -> instrument return models
val bModels = sc.broadcast(models)
// Generate a seed for each task
val seeds = (baseSeed until baseSeed + parallelism)
val seedRdd = sc.parallelize(seeds, parallelism)
// Create an RDD of trials
val trialReturns: RDD[Double] = seedRdd.flatMap { seed =>
trialReturns(seed, trialsPerTask, bModels.value, factorMeans, factorCovs)
}](https://image.slidesharecdn.com/slidesharerisktalk-150911155504-lva1-app6891/85/Estimating-Financial-Risk-with-Apache-Spark-25-320.jpg)
Estimating Financial Risk with Apache Spark
- 1. Estimating Financial Risk with Spark Sandy Ryza Senior Data Scientist, Cloudera
- 4. In reasonable circumstances, what’s the most you can expect to lose?
- 7. def valueAtRisk( portfolio, 2 weeks, .05 ): = $1,000,000
- 8. Probability density Portfolio return ($) over time period
- 9. VaR estimation approaches • Variance-covariance • Historical • Monte Carlo
- 12. Market Risk Factors • Indexes (S&P 500, NASDAQ) • Prices of commodities • Currency exchange rates • Treasury bonds
- 13. Predicting Instrument Returns from Factor Returns • Train a linear model on the factors for each instrument
- 14. Fancier • Add features that are non-linear transformations of the market risk factors • Decision trees • For options, use Black-Scholes
- 15. import org.apache.commons.math3.stat.regression.OLSMultipleLinearRegression // Load the instruments and factors val factorReturns: Array[Array[Double]] = ... val instrumentReturns: RDD[Array[Double]] = ... // Fit a model to each instrument val models: Array[Array[Double]] = instrumentReturns.map { instrument => val regression = new OLSMultipleLinearRegression() regression.newSampleData(instrument, factorReturns) regression.estimateRegressionParameters() }.collect()
- 16. How to sample factor returns? • Need to be able to generate sample vectors where each component is a factor return. • Factors returns are usually correlated.
- 17. Distribution of US treasury bond two- week returns
- 18. Distribution of crude oil two-week returns
- 19. The Multivariate Normal Distribution • Probability distribution over vectors of length N • Given all the variables but one, that variable is distributed according to a univariate normal distribution • Models correlations between variables
- 21. import org.apache.commons.math3.stat.correlation.Covariance // Compute means val factorMeans: Array[Double] = transpose(factorReturns) .map(factor => factor.sum / factor.size) // Compute covariances val factorCovs: Array[Array[Double]] = new Covariance(factorReturns) .getCovarianceMatrix().getData()
- 22. Fancier • Multivariate normal often a poor choice compared to more sophisticated options • Fatter tails: Multivariate T Distribution • Filtered historical simulation • ARMA • GARCH
- 23. Running the Simulations • Create an RDD of seeds • Use each seed to generate a set of simulations • Aggregate results
- 24. def trialReturn(factorDist: MultivariateNormalDistribution, models: Seq[Array[Double]]): Double = { val trialFactorReturns = factorDist.sample() var totalReturn = 0.0 for (model <- models) { // Add the returns from the instrument to the total trial return for (i <- until trialFactorsReturns.length) { totalReturn += trialFactorReturns(i) * model(i) } } totalReturn }
- 25. // Broadcast the factor return -> instrument return models val bModels = sc.broadcast(models) // Generate a seed for each task val seeds = (baseSeed until baseSeed + parallelism) val seedRdd = sc.parallelize(seeds, parallelism) // Create an RDD of trials val trialReturns: RDD[Double] = seedRdd.flatMap { seed => trialReturns(seed, trialsPerTask, bModels.value, factorMeans, factorCovs) }
- 27. Executor Task Task Trial Trial TrialTrial Task Task Trial Trial Trial Trial Executor Task Task Trial Trial TrialTrial Task Task Trial Trial Trial Trial Time Model parameters Model parameters
- 28. // Cache for reuse trialReturns.cache() val numTrialReturns = trialReturns.count().toInt // Compute value at risk val valueAtRisk = trials.takeOrdered(numTrialReturns / 20).last // Compute expected shortfall val expectedShortfall = trials.takeOrdered(numTrialReturns / 20).sum / (numTrialReturns / 20)
- 30. VaR
- 32. So why Spark?
- 33. Easier to use • Scala and Python REPLs • Single platform for • Cleaning data • Fitting models • Running simulations • Analyzing results
- 34. New powers • Save full simulation-loss matrix in memory (or disk) • Run deeper analyses • Join with other datasets
- 35. But it’s CPU bound and we’re using Java? • Computational bottlenecks are normally in matrix operations, which can be BLAS-ified • Can call out to GPUs just like in C++ • Memory access patterns aren’t high-GC inducing
- 36. Want to do this yourself?
- 37. spark-finance • https://github.com/cloudera/spark-finance • Everything here + the fancier stuff • Patches welcome!