Improving the solution time of TIMES by playing with CPLEX/Barrier
0 likes1,028 views
The document discusses improving the solution time of TIMES models by optimizing settings for the CPLEX/Barrier interior point method. It provides information on how to configure options like lpmethod, threads, barcolnz, barorder, and baralg to improve performance by reducing matrix sizes, treating dense columns specially, and choosing better ordering and starting point algorithms. It also discusses interpreting log files to diagnose issues like numerical difficulties and suggests actions like changing algorithms, tolerances, or emphasis parameters. Finally, it covers whether to perform crossover to obtain a basic solution or not.
Improving the solution time of TIMES by playing with CPLEX/Barrier
- 1. WIR SCHAFFEN WISSEN – HEUTE FÜR MORGEN Improving the solution time of TIMES by playing with CPLEX/Barrier Evangelos Panos 73d Semi – Annual ETSAP meeting, Gothenburg, 17 – 18th June 2018
- 2. • Questions to be answered become more complex, and models more detailed • Solution times can hinder the efficiency in performing e.g. uncertainty analysis Examples of the complexity of TIMES models Page 2 Matrix sizes for different TIMES-based models
- 3. How to setup options in CPLEX Page 3
- 4. • Simplex starts along the edge of the feasible region, searching for an optimal vertex • Barrier method starts somewhere inside the feasible region and burrows through it to find the optimal solution; the path to optimality is adjusted by a predictor-corrector algorithm The CPLEXBarrier – Interior point method Page 4https://ibmdecisionoptimization.github.io/tutorials/html/Linear_Programming.html Barrier «likes» dense matrices To enable Barrier we set the CPLEX option lpmethod=4 and threads=-1
- 5. The CPLEXBarrier log file Page 5
- 6. The CPLEX log file – Initial matrix size Page 6 --- Test.RUN(305807) 1604 Mb --- 4,465,185 rows 3,406,036 columns 19,096,151 non-zeroes --- Executing CPLEX: elapsed 0:01:17.802 --- Test.RUN(305807) 1604 Mb 3 secs IBM ILOG CPLEX 25.1.1 r66732 Released May 19, 2018 WEI x86 64bit/MS Windows --- GAMS/Cplex licensed for continuous and discrete problems. Cplex 12.8.0.0 When CPLEX starts the initial size of the model is reported The model matrix is usually not dense: (number of non-zeros)/(total number of elements)
- 7. The CPLEX log file – Preprocessing Page 7 Aggregator has done 637425 substitutions... Tried aggregator 4 times. Aggregator has done 637425 substitutions... LP Presolve eliminated 1549211 rows and 1348918 columns. Aggregator did 637425 substitutions. Reduced LP has 1435395 rows, 2415792 columns, and 11493571 nonzeros. Presolve time = 20.86 sec. (41286.55 ticks) The CPLEX presolver and the aggregator may use more memory, but they examine the problem for logical reduction opportunities and ultimately reduce the model matrix seen by Barrier They are enabled by default and it is recommended not to disable them It is also a good idea to turn on the CPLEX dependency checker option depind = 1 or 2 or 3
- 8. The CPLEX/Barrier log file – Dense Columns Page 8 ***NOTE: Found 6136 dense columns. Number of nonzeros in lower triangle of A*A' = 18081132 Total time for nested dissection ordering = 15.17 sec. (23675.57 ticks) A dense column has a given variable appearing in many rows, and it can degrade performance Barrier treats dense columns in a special way to reduce their impact on the solution time If in a large problem there are plenty of columns e.g. with less than 10 entries and some with more than 50 entries, then the performance will increase if we instruct Barrier to handle those columns with more than 50 entries as dense columns The option barcolnz set the number of non-zeros in a column above which the column is treated as a dense column
- 9. Histogram of column counts helps in detecting dense columns Page 9 0 500,000 1,000,000 1,500,000 2,000,000 2,500,000 3,000,000 3,500,000 0 1000 2000 3000 4000 5000 6000 Histogram of the column count 0 200,000 400,000 600,000 800,000 1,000,000 2 4 6 8 10 12 14 16 18 20 22 24 26 28 Of which more than 99% of them have <30 non-zeros The option writemps writes a text file (MPS) which can be easily processed to generate histogram like the ones above
- 10. The CPLEX/Barrier log file – Number of nonzeros Page 10 ***NOTE: Found 6136 dense columns. Number of nonzeros in lower triangle of A*A' = 18081132 Total time for nested dissection ordering = 15.17 sec. (23675.57 ticks) The number of nonzeros in the lower triangle of A*AT gives an early indication of how long each barrier iteration will take (the larger the number the more time is needed) It also gives an indication if the CPLEX/Barrier will perform faster than the Simplex: if the non- zeros in the lower triangle of A*AT are less than the non-zeros of A, then Barrier will solve faster If the number of nonzeros is too large then we may consider to treat more columns as dense columns with the barcolnz
- 11. The CPLEX/Barrier log file – Non-zeros in factor Page 11 ***NOTE: Found 6136 dense columns. Number of nonzeros in lower triangle of A*A' = 18081132 Total time for nested dissection ordering = 15.17 sec. Summary statistics for Cholesky factor: Threads = 19 Rows in Factor = 1441531 Integer space required = 13513150 Total non-zeros in factor = 489517779 Total FP ops to factor = 798619521347 Total non-zeros in factor: the difference with the nonzeros in A*AT indicates the fill-level of the Cholesky factor If this difference is large we may consider a different ordering algorithm by setting the option barorder = 0, 1, 2, 3
- 12. The CPLEX/Barrier log file – FP ops to factor Page 12 ***NOTE: Found 6136 dense columns. Number of nonzeros in lower triangle of A*A' = 18081132 Total time for nested dissection ordering = 15.17 sec. (23675.57 ticks) Summary statistics for Cholesky factor: Threads = 19 Rows in Factor = 1441531 Integer space required = 13513150 Total non-zeros in factor = 489517779 Total FP ops to factor = 798619521347 Total FP ops to factor is a predictor of the time that will be required to perform each iteration The barcolnz , barorder , and a selection of a different barrier algorithm via baralg = 1, 2, 3 can help in reducing FP ops
- 13. The CPLEX/Barrier log file – Iteration log Page 13 CPLEX Barrier treats inequality constraints as equalities with added slack and surplus variables - The primal constraints are written as: Ax=b and x+s=u - The dual constraints are written as: ATy+z-w=c Primal inf: │b – Ax │ Upper inf: │ u – (x+s) │ Dual inf: │ c – (ATy+z-w) │ When using one of the barrier infeasibility algorithms baralg = 1, 2 then there is an additional column Inf Ratio; this should increase to a large number when the problem is optimal Itn Primal Obj Dual Obj Prim Inf Upper Inf Dual Inf Inf Ratio 0 -2.5880e+07 3.5297e+07 1.49e+09 7.70e+04 3.55e+09 1.00e+00 1 -2.8485e+07 3.4183e+07 1.38e+09 7.13e+04 3.51e+09 1.53e-03 106 1.25915e+06 1.2591e+06 5.63e-01 5.79e-08 2.19e-03 8.80e+06
- 14. The CPLEX/Barrier log file – Start point heuristic Page 14 CPLEX Barrier supports different heuristics to compute the staring point For most problems the default heuristic works well, but it is worthy to also check the alternative heuristics if they provide a better starting point at iteration 0 The starting point heuristic can be set via the option barstartalg = 1, 2, 3, 4 Itn Primal Obj Dual Obj Prim Inf Upper Inf Dual Inf Inf Ratio 0 -2.5880e+07 3.5297e+07 1.49e+09 7.70e+04 3.55e+09 1.00e+00 1 -2.8485e+07 3.4183e+07 1.38e+09 7.13e+04 3.51e+09 1.53e-03 106 1.25915e+06 1.2591e+06 5.63e-01 5.79e-08 2.19e-03 8.80e+06
- 15. Floating point calculations are of finite precision and LPs can be prone to innacurracies due to round- off errors or «unscaled infeasibilities» Possible actions need trial and error: 1. Change the barrier algorithm via baralg 2. Change the limit on barrier corrections barmaxcor to a value greater than 0 3. Choose a different starting-point heuristic via barstartalg 4. Increase the barcolnz but only if the removal of dense columns causes numerical instabilities 5. Tight the convergence tolerance barepcomp to a value greater than its default Issue 1: in some models a larger tolerance can save time in the crossover section Issue 2: if crossover takes a large part of the optimisation time then reduce the tolerance Dealing with numerical difficulties Page 15
- 16. Floating point calculations are of finite precision and LPs can be prone to innacurracies due to round- off errors or «unscaled infeasibilities» Possible actions need trial and error: 6. Set the numericalemphasis parameter to 1 7. Set the scaind parameter to 1 8. Set the epmrk parameter to a value greater than 0.01 and less than 0.99999 9. Increase the bargrowth parameter and/or the barobjrng parameter , but only if CPLEX reports the problem as unbounded before it actually finds the optimal solution Dealing with numerical difficulties Page 16
- 17. Putting all together in a heuristic
- 18. Simplex and Barrier differ with respect to the nature of the solutions: • Barrier solutions tend to be midface and not basic solutions Some variables can be e.g. 0.999999 in Barrier instead of 1 in Simplex, or 0.00001 in Barrier instead of 0 in Simplex • In case when multiple optima exist Barrier solutions tend to place the variables at values between the bounds, whereas in basic solutions from simplex when multiple optima exist have variables usually at lower or upper bounds Hence, while the objective values will be the same, the nature of the solution can be different between the Barrier and the Simplex algorithm To crossover or not to crossover? Page 18
- 19. The Barrier algorithm uses crossover after founding a solution, to produce a basis: Basis: the variables constitute the optimal solution are non-zero Without a corssover: • The solution is «ugly», having variables close to 0 but not 0, e.g. 1e-06 • We do not obtain range information for performing sensitivity analysis • We do not obtain a basic solution, which can be used to optimise the same or a similar problem with advanced start information (== this procedure uses Simplex and not Barrier) To crossover or not to crossover? Page 19 The crossover can be turned off by setting solutiontype=2 Having the crossover at the end increases the solution time Crossover does not benefit much from multiple cores
- 20. • In scaling the goal is to have coefficients in the constraint matrix of the model between 0.1 and 100 (although larger ranges will work too) • We can scale the model by: 1. Letting the CPLEX do it for us via «scaind 1» , or 2. Alter our data in the VEDA_FE excel input files , or 3. Instruct GAMS to scale our equations by specified factors • The 2nd option implies that our results will be also scaled • The 3d option is attractive, because the scaling does not affect our input data and the results obtained (i.e. both input and output remain at the original values) • To implement the 3d option useful information can be found in: https://www.gams.com/latest/docs/S_GAMSCHK.html?search=GAMSCHK Scaling the model to improve numerical stability (to be discussed in another presentation) Page 20
- 21. Page 21 Wir schaffen Wissen – heute für morgen Thank you very much for your attention Evangelos Panos Energy Economics Group Laboratory for Energy Systems Analysis evangelos.panos@psi.ch