OpenFOAM benchmark for EPYC server -Influence of coarsestLevelCorr in GAMG solver -
- 1. OpenFOAM benchmark for EPYC server - GAMG solver coarsestLevelCorr - Osaka Metropolitan University Takuya YAMAMOTO 2023/8/23
- 2. GAMG solver control p OpenFOAM v1906 https://www.openfoam.com/news/main- news/openfoam-v1906/numerics#numerics- adjoint Improved GAMG solver controls マルチグリッドで解く際に最も粗い格⼦レベルでの格⼦数が⾮常に少ない (数⼗セル) 並列計算時にこの粗い格⼦レベルでの通信量がボトルネックになることで、 並列化効率が悪化 これを改善するために、最も粗い格⼦レベルでの解法を選択できるように pFinal { solver GAMG; tolerance 1e-6; relTol 0; smoother GaussSeidel; coarsestLevelCorr { solver PCG; preconditioner DIC; relTol 0.05; } }
- 3. GAMG solver control p OpenFOAM v2206 https://www.openfoam.com/news/main- news/openfoam-v20-06/solvers-and- physics#solver-and-physics-pipelined-cg- solvers New pipelined Conjugate Gradient solvers 並列計算時に共役勾配系(CG) solverでは全てのプロセッサーで同じ探索⽅向に 解く必要があり、並列計算時のボトルネックになることがある パイプラインCG solverではオーバーヘッド、通信を減らすことができる 特に、GAMGの最も粗いレベルの格⼦にこの⼿法を取り⼊れると有効である PPCR, PPCGが選択可能 pFinal { $p; relTol 0; coarsestLevelCorr { solver PPCR; preconditioner DIC; relTol 0.05; } } これらの⼿法が果たしてどれだけEPYCサーバーで効くのか?
- 4. OpenFOAM benchmark p Benchmark of High Performance Computing (HPC) Technical Committee • Small, S (1M) • Medium, M (8M) • Extra-Large, XL (64M) Ø 3-D Lid Driven cavity flow Ø HPC Motorbike Ø Conical Diffuser Ø … Many benchmark has been prepared. In this study, we use 3-D Lid Driven Cavity Flow, S and M. The used OpenFOAM is ESI v2212 version. https://develop.openfoam.com/committees/hpc/-/tree/develop/
- 5. Server resource p Used server • Server 1: EPYC 7352 Dual CPU (2.3 GHz x 48 cores) RAM 128 GB (8GB x 16 channel) BW 187.7 GB/s (2933 MT/s x 8 channel x 8) L3 Cache 128 MB • Server 2: EPYC 7513 Dual CPU (2.6 GHz x 64 cores) RAM 128 GB (8GB x 16 channel) BW 204.8 GB/s (3200 MT/s x 8 channel x 8) L3 Cache 128 MB EPYC 3rd Gen EPYC 2nd Gen • Server 3: EPYC 7542 Dual CPU (2.9 GHz x 64 cores) RAM 128 GB (8GB x 16 channel) BW 187.7 GB/s (2933 MT/s x 8 channel x 8) L3 Cache 128 MB EPYC 2nd Gen
- 6. Server resource p Used server • Server 4: EPYC 7713 Dual CPU (2.0 GHz x 128 cores) RAM 256 GB (16GB x 16 channel) BW 204.8 GB/s (3200 MT/s x 8 channel x 8) L3 Cache 256 MB EPYC 3rd Gen • Server 5: EPYC 7763 Dual CPU (2.45 GHz x 128 cores) RAM 128 GB (8GB x 16 channel) BW 204.8 GB/s (3200 MT/s x 8 channel x 8) L3 Cache 256 MB EPYC 3rd Gen
- 7. Solver of algebraic matrix p solver • Solver 1: solver, p GAMG GAMG preconditioner, p GaussSeidel tolerance, p 1 x 10-4 solver, U smoothSolver preconditioner, U GaussSeidel tolerance, U 0 relTol, U 0 maxIter, U 5 • Solver 2: solver, p GAMG GAMG-PPCR preconditioner, p GaussSeidel tolerance, p 1 x 10-4 solver, U smoothSolver preconditioner, U GaussSeidel tolerance, U 0 relTol, U 0 maxIter, U 5 coarsestLevelCorr { solver PPCR; preconditioner DIC; relTol 0.05; }
- 9. 3D- Lid Driven cavity flow (S, 1M)
- 10. 3D- Lid Driven cavity flow (M, 8M)
- 12. 3D- Lid Driven cavity flow (S, 1M)
- 13. 3D- Lid Driven cavity flow (M, 8M)
- 15. 3D- Lid Driven cavity flow (S, 1M)
- 16. 3D- Lid Driven cavity flow (M, 8M)
- 18. 3D- Lid Driven cavity flow (S, 1M)
- 19. 3D- Lid Driven cavity flow (M, 8M)
- 21. 3D- Lid Driven cavity flow (S, 1M)
- 22. 3D- Lid Driven cavity flow (M, 8M)
- 24. 3D- Lid Driven cavity flow (S, 1M)
- 25. 3D- Lid Driven cavity flow (M, 8M)