Practical Xen Testing At Intel
1 like973 views
The document discusses practical Xen testing approaches at Intel. It outlines Intel's progressive testing of Xen features over time to increase test coverage and ensure release quality. The test infrastructure is described as being automated, extendable, and centered around a test result center to timely expose regressions. Best practices proposed include extending the test framework, using an automated build system, and a patch testing system.
Practical Xen Testing At Intel
- 1. Practical Xen Testing at Intel Nov. 2009 Haicheng Li <haicheng.li@intel.com> Jiajun Xu <jiajun.xu@intel.com> Xen Summit Asia 2009
- 2. Legal Information • INFORMATION IN THIS DOCUMENT IS PROVIDED IN CONNECTION WITH INTEL PRODUCTS. NO LICENSE, EXPRESS OR IMPLIED, BY ESTOPPEL OR OTHERWISE, TO ANY INTELLECTUAL PROPERTY RIGHTS IS GRANTED BY THIS DOCUMENT. EXCEPT AS PROVIDED IN INTEL'S TERMS AND CONDITIONS OF SALE FOR SUCH PRODUCTS, INTEL ASSUMES NO LIABILITY WHATSOEVER AND INTEL DISCLAIMS ANY EXPRESS OR IMPLIED WARRANTY, RELATING TO SALE AND/OR USE OF INTEL PRODUCTS INCLUDING LIABILITY OR WARRANTIES RELATING TO FITNESS FOR A PARTICULAR PURPOSE, MERCHANTABILITY, OR INFRINGEMENT OF ANY PATENT, COPYRIGHT OR OTHER INTELLECTUAL PROPERTY RIGHT. • Intel may make changes to specifications, product descriptions, and plans at any time, without notice. • All products, computer systems, dates, and figures specified are preliminary based on current expectations, and are subject to change without notice. • Performance tests and ratings are measured using specific computer systems and/or components and reflect the approximate performance of Intel products as measured by those tests. Any difference in system hardware or software design or configuration may affect actual performance. • Intel is a trademark of Intel Corporation in the U.S. and other countries. • *Other names and brands may be claimed as the property of others. • Copyright © 2009, Intel Corporation. All rights are protected. Software and Services Group 2 2
- 3. Agenda • The Overview – how test helps Xen? • Test Infrastructure – the world under automation • Best Practices – practical test approaches • Some Proposals – to improve Xen quality Software and Services Group 3 3
- 4. The Overview Software and Services Group 4 4
- 5. Progressive Xen Testing VT-x SR-IOV Intel VT VT- i VT-d Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009 Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0 XEN • Hundreds of Xen patches Monthly !!! XEN Development • How can TEST help? • Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS Coverage SMP VM Expose regression timely Live Migration TXT Real Mode Device Pass-Through SR-IOV • 3.0.x Release Ensure release quality 3.1.x Release 3.2.x Release 3.3.x Release 3.4.x Release Test Test Reports Test Reports Test Reports Test Reports Test Reports Deliverables Nightly Test Reports Weekly Test Reports Biweekly Test Reports Actively Bug Tracking Software and Services Group 5 5
- 6. Progressive Xen Testing VT-x SR-IOV Intel VT VT- i VT-d Dec. 2005 Oct. 2006 May. 2007 Jan. 2008 Aug. 2008 Apr. 2009 Xen-3.0.0 Xen-3.0.3 Xen-3.1.0 Xen-3.2.0 Xen-3.3.0 Xen-3.4.0 XEN XEN Development Test HVM x86/IA64 PV Guest Timers NUMA EPT Power Management RAS Coverage SMP VM Live Migration TXT Real Mode Device Pass-Through SR-IOV 3.0.x Release 3.1.x Release ~300 major Release reported, 3.2.x Release 3.3.x bugs 3.4.x Release Test Deliverables Test Reports Test Reports 20% of Xen Bugzilla Test Reports Test Reports Test Reports Nightly Test Reports Weekly Test Reports Biweekly Test Reports Actively Bug Tracking Software and Services Group 6 6
- 7. Test Infrastructure Software and Services Group 7 7
- 8. Solid Test Infrastructure • Test Automation in Everywhere – auto daily testing – auto release resting – auto patch testing • Test Infrastructure is The Basis • Extendable Test Framework • Auto Build System • Test Result Center • Patch Test System Software and Services Group 8 8
- 9. Extendable Test Framework Manual Testing Regular Testing Auto Build *XVS Result submitter to Result Center Common lib *XVS: eXtendable Validation System Management Module VT-d/ Controller Report Generator VT-d | -- VT-d lib | -- test1 | -- test2 Call with Scenario file SR-IOV Test Suite Pool … Harness SR-IOV/ Power | -- IOV lib Test Harness | -- test1 | -- test2 RAS Software and Services Group 9 9
- 10. Test Result Center Find Defects Out Timely! Software and Services Group 10 10
- 11. Patch Test System Recipe Submit UI Machine Status UI Result UI DB Tool Web UI Receive Core Management Runtime DB Recipe Test Schedule Test Box 1 Local Control Scheduler Machine Test Box 2 Recipe … Test Info Test Machine R-Queue Pool Test Info XVS Users Test Box N Test Resources OSD Results Results Processing Machines Software and Services Group 11 11
- 12. Patch Test System Ensure Patch Quality Before Submitting Out! Software and Services Group 12 12
- 13. Best Practices Software and Services Group 13 13
- 14. Power Management Testing • Challenge – How to achieve integrated Power Management testing? • Test Approach – Automated functionality testing. • Interface : xenpm, hypercall, sysfs interface • Coverage : idle, runtime, Dom0/HVM S3 – Performance measurement. • Scenario : idle, runtime, benchmarks • Platform : laptop & server • OS : Linux, Windows • Publish : web site Software and Services Group 14 14
- 15. Power Management Testing • Challenge – How to achieve integrated power testing? • Test Approach – Automated functionality testing. • Interface : xenpm, hypercall, sys i/f • Coverage : idle, runtime, Dom0/HVM S3 – Performance measurement. • Scenario : idle, runtime, benchmarks • Platform : laptop & server • OS : Linux, Windows • Publish : web site Software and Services Group 15 15
- 16. RAS Testing • Challenge – How to achieve good test coverage? • Test Approach – Add test stub to Xen to dump page types. – Add test stub to Xen to translate address from guest to host. – Use ACPI4.0/APEI to trigger CPU/Mem *MCE to specific pages. • To Do – How to stress? • Option: add more software test interface to *MCE handlers. – CPU Migration – MEM hot-plug *MCE: Machine Check Exception Software and Services Group 16 16
- 17. RAS Testing • Challenge – How to achieve good test coverage? • Test Approach – Add test stub to Xen to dump page types. – Add test stub to Xen to What MCE test i/f guest to host. translate address from Linux provides? – Use ACPI4.0/APEI to trigger CPU/Mem to trigger fake MCE • MCE injector *MCE to specific pages. • madvise syscall to poison page • To Do • HWPosion injector to poison page – How to stress? •…… • Option: add more software test interface to *MCE handlers. – CPU Migration – MEM hot-plug *MCE: Machine Check Error Software and Services Group 17 17
- 18. VT-d/SR-IOV Testing • Challenge – Hardware variety & complexity : platforms + devices • Test Approach – Select typical test configurations. • Platform : laptop -> desktop -> server • Device : NIC, USB, SATA • PCI type : PCIE MSI, PCIE INTx, PCI INTx • OS : Linux, Windows – Use NIC to achieve test automation. • Methods : ping, scp (file copy), ssh • Coverage : static assignment + VT-d hotplug • Multiple-devices assignment. Software and Services Group 18 18
- 19. Some Proposals Software and Services Group 19 19
- 20. Improve Xen Quality • Source Control – Synchronize source control for Ioemu, Dom0 and Xen. • E.g. VT-d is usually broken by ioemu check-in • Bug Tracking – Well track bug state in Xen Bugzilla • Current Xen BZ full of obsolete bug data • ~500 bugs , 30% Xen Bugzilla, are NEW! • Xen Testability – More built-in test features for Xen? • It’s a trend for Linux: memleak checker, fault injector, ... Software and Services Group 20 20
- 21. Questions? Software and Services Group 21 21
- 22. Backup Software and Services Group 22 22
- 23. Regular Testing To keep Xen from regressions Build Check Booting Tests Basic Functional Tests C/S Pull Auto Build Auto Install Regression Testing Bug Verification Daily Build *BAT Testing Daily *BAT Test Cycle *BAT: Build Acceptance Test Full Scope Functional Tests Functionality Testing Performance Testing Stress Testing Others System Testing Monthly System Test Cycle Software and Services Group 23 23
- 24. Release Testing To make qualified Xen release Build Check Booting Tests Basic Functional Tests C/S Pull Auto Build Auto Install Regression Testing Bug Verification Release Build *BAT Testing Release *BAT Test Cycle *BAT: Build Acceptance Test Full Scope Functional Tests Functionality Testing Performance Testing Stress Testing Others System Testing Release System Test Cycle Software and Services Group 24 24
- 25. Patch Testing To ensure patch quality for Xen Build Check Booting Tests Basic Functional Tests Auto Build Auto Install Regression Testing Unit Testing Patch Testing Patch Test Cycle Software and Services Group 25 25