Honorable Squires
- 1. Honorable Squires Supportive tools and environment for a high availability Smalltalk application
- 2. 2 Honorable Squires About … Authors Uwe Liebold, Factory Automation Engineer , AMD Torsten Happ, Factory Automation Engineer, AMD Taylan Kraus-Wippermann, Consultant from Georg Heeg eK Project “CEI Baseline” Relationship Georg Heeg eK is working for AMD since 2002 Consulting, design and implementation in “CEI Baseline” team Workshops and Smalltalk training for AMD developers
- 3. 3 Honorable Squires Agenda • AMD and Georg Heeg eK Company overview • Equipment Integration and CEI Baseline • Role of Testing • Support Tools Log File Viewer Remote Service Tool Replay Tool • Vision and Outlook
- 5. 5 Honorable Squires About AMD Advanced Micro Devices is a leading global provider of innovative processing solutions in the computing, graphics and consumer electronics markets. Founded: 1969 Headquarter Sunnyvale, California (Silicon Valley), USA Employees ~ 16,000 worldwide Sales Mix 79 % international 2006 Revenue > $ 5.6 billion (incl. ATI)
- 6. 6 Honorable Squires AMD in Dresden 1996 Future AMD Site Fab 36 DDC/OSRC Fab 30/38 Bump/Test Facility EVC 2 EVC 1 2007
- 7. 7 Honorable Squires Highly Successful 130nm 90nm 65nm Technology Transition High yield at start of volume production Builds on SOI success Increased thermal efficiencies Significant value addition Addition of strained silicon transistor technology 90nm130nm 90nm130nm 65nm 65nm90nm Dual-Core Quad-Core Dual-Core
- 8. 8 Honorable Squires Cross sections & dimensions 90 nm Structures Transistor for 90 nm
- 9. 9 Honorable Squires Company Overview Georg Heeg eK
- 10. 10 Honorable Squires About Georg Heeg eK Founded 1987, headquarter in Dortmund, since 1996 in Zurich, since 1999 in Koethen/Anhalt Consulting- and training company in Smalltalk Hotline support, maintenance, bug-fixes for ObjectStudio, VisualWorks and Visual Smalltalk VM-laboratory for VisualWorks and ObjectStudio Porting service of old VisualWorks applications to 5i/7 Technology-partner of Corporate Mission: Make Sophisticated Projects a Success for the Customer!
- 11. 11 Honorable Squires Equipment Integration
- 12. 12 Honorable Squires Equipment Interfaces Functional Scope Connection between production equipment and Factory Control Systems (FCS) Controls the processing of the equipment based on jobs from the Manufacturing Execution System (MES) Collects and reports data to various FCS components to – tune this or a dependent process – monitor process quality and tool health – allow other offline analysis (e.g. tool utilization) updates the MES with process status and material location Complexity > 50 Tool Types with different level of behavior specifics > 500 Entities running 7 x 24 little to no idle times
- 13. 13 Honorable Squires Equipment Interfaces (EI’s) in the Factory Control System (FCS) Equipment Interface 2 Equipment Interface 1 Equipment Interface n MES Setup System Data Mart APCSPC Equipment 2Equipment 1 Equipment n
- 14. 14 Honorable Squires CEI Baseline – what is it? CEI Baseline Framework Development Tools Standard Implementation Interfaces Configuration Event Management Scenario Management Error Handling Configuration Editors Test Runner Tool Model Validator Interface Tester Smalltalk Extensions 300 mm SEMI Standard Tool Fab Standard Scenarios Data Collection Interface Tester Smalltalk Extensions …etc.
- 15. 15 Honorable Squires Conflicting Objectives CEI Baseline Continuous Enhancements Increasing Complexity Automated Error Handling Minimize Restarts Minimize Failure Events Minimize Service CallsResource Constraints Minimize Production Tests Minimize Production Impact
- 16. 16 Honorable Squires The Role of Testing Testing • Minimize simple coding bugs • Avoid reintroduction of known errors (unit tests) • Ensure fully specified functionality • Ensure correct behavior in described error situations • Cover all code (usually) • Avoid unknown or unexpected errors situations • Ensure full functionality if these are not completely specified • Ensure correct behavior if number of influencing parameters is very high • address timing problems can can not or limited
- 17. 17 Honorable Squires The Application needs more Support…. • We want to understand and eliminate each issue after 1st occurrence we need support for the analysis of issues we want to be able to reproduce issues in development • Testing needs to be continuously improved • We have to accept that errors in production occur • Error will be more rare and more complex with improved testing • … but
- 18. 18 Honorable Squires … but what Support CEI Rapid Offline Analysis • simplify analysis of huge log data amounts Analyze living Instance • allow damage analysis in application with suspicious behavior Allow exact simulation • recording of production situation • precise recreation in development
- 19. 19 Honorable Squires Background for support reasons EI creates log there are many different log categories (interface level logging) support staff needs to look into more than one log file Solution Tool “Log File Viewer” to provide better overview (“merged” view) Features open/close files (even if zipped) merge files based on timestamp mark/un-mark entries in colours store sessions to file Analysis - Log File Viewer CEICEI SECS loglog MQ MQ logMQ log SECS logSECS log
- 20. 20 Honorable Squires Analysis - Log File Viewer (cont.) loglog MQ logMQ log SECS logSECS log
- 21. 21 Honorable Squires … but what Support CEI Rapid Offline Analysis • simplify analysis of huge log data amounts Analyze living Instance • allow damage analysis in application with suspicious behavior Allow exact simulation • recording of production situation • precise recreation in development
- 22. 22 Honorable Squires Analysis - Remote Service Tool CEI Remote Service Tool Message Bus • call defined services to generate reports • memory • process stack • instance reports • reports can be saved for later developer review • specific service opens port for passive inspection via STST • call defined services to generate reports • memory • process stack • instance reports • reports can be saved for later developer review • specific service opens port for passive inspection via STST • object inspection • manipulation suppressed • object inspection • manipulation suppressed
- 23. 23 Honorable Squires … but what Support CEI Rapid Offline Analysis • simplify analysis of huge log data amounts Analyze living Instance • allow damage analysis in application with suspicious behavior Allow exact simulation • recording of production situation • precise recreation in development
- 24. 24 Honorable Squires Exact Simulation – Replay Tool “Flight Recorder” for the Equipment Interface Analysis of crash and error situations Replay error situations many times with all development tools available Recording in production – Replay in development Recording all data traffic with external systems Deterministic behaviour Writing snapshots of application state Reading snapshot and replay recorded traffic Simulation of interfaces to external systems Application runs without changes No simulation of internal processes Simulation GUI – Replay Runner
- 25. 25 Honorable Squires Replay Tool – CEI Connections CEICEI Equipment APCAPC MESMES SPCSPC SAPSAPSetup System Setup System
- 26. 26 Honorable Squires Replay Tool – CEI Multiple Interfaces CEICEI M Q CORBA SECS H T T P
- 27. 27 Honorable Squires Replay Tool – Recording Low level interface extension Minimal impact to existing code Minimal performance impact Not changing application code All changes kept on interface level Invisible for application Persistence Binary format No message transformation One file for each interface M Q SECS Recorder Recorder
- 28. 28 Honorable Squires Replay Tool – Writing Snapshots Taking snapshots Long time recording leads to high data amounts Long time winding to find a suitable simulation position Simulation acceleration is limited Replay needs configuration and environments Different production and development configuration Different replay in development with production records
- 29. 29 Honorable Squires Replay Tool – Simulation Low level interface extension Disconnecting interface from media Moderate changes necessary Simulator Feeds recorded messages into the interface Receives messages from interface Virtualization of external interfaces Simulation of TCP/IP based communication – Virtual Sockets – Simulators for CORBA-IIOP, HTTP, SECS, … Simulation of queue communication – Virtual Queues (MQ series) M Q SECS Simulator Simulator
- 30. 30 Honorable Squires Replay Tool – Message Dispatching Replaying all messages in recording order Merge of all recording files Recording order is defined by timestamp Out of order messages are possible Event and Time driven dispatching Event dispatching for interface activity – Outgoing messages (Requests and Responses) Time dispatching for internal delays – Incoming messages (Requests and Responses)
- 31. 31 Honorable Squires Replay Tool – Simulation Components Corba SECS SECS Simulator Corba Simulator Dispatcher Corba Recorder SECS Recorder
- 32. 32 Honorable Squires CEI Replay Tool – Dispatching 1 1’2 2’ Time Recorded Message Stream Dispatcher 1 2 t t 2’ 1’ Simulator
- 33. 33 Honorable Squires Replay Tool – Message Order Problem Order of message sends vs. recorded message order Many processes using the same interface Execution speed differs from recording to replay machine Consequences Message matching strategy needed Allow out-of-order execution Time based message triggering need prerequisites
- 34. 34 Honorable Squires Replay Tool – Message Matching Strategy Replay messages differs from recorded messages Transport protocol using sequence ids Messages containing timestamps Messages containing timestamp derived identifiers Message matching strategies Use sequence id for request-response matching Remove sequence id from Replay-Recording matching Remove Timestamps from Replay-Recording matching Use recorded timestamps Allow out-of-order messages only within a specific window size
- 35. 35 Honorable Squires Replay-Tool GUI Replay control Selecting a replay scenario Starting and stopping replay Pause replay Stretching or squeezing Proceed immediately Set and remove breakpoints Visualization and inspection Preview of messages Inspection of raw and decoded messages Decoding recorded message format – CORBA objects – XML nodes
- 36. 36 Honorable Squires Replay Tool – Challenges Recognition of recorded messages Timestamps, Serial numbers Out-of-order messages Dispatching rules Finding the right prerequisites for message dispatching Finding the right window size for message lookup Snapshot of application state What is the minimum state? When is the right point in time to snapshot Stopping application input while writing a snapshot Memory consumption Keeping all recording messages in memory for simulation is expensive
- 37. 37 Honorable Squires Replay Tool – Status, Experiences, Outlook Status Recording and Replay with limited number of involved interfaces Simulation control GUI ready to use Snapshot is in development Experiences Number of code changes to enable recording differs between interfaces – CORBA needs seven code changes – MQ needs 10 changes – HTTP Opentalk does not need any changes Acceleration is limited! Detection of simulation parameter need some experiments Outlook Showing recorded messages in Log-Viewer Editing of recoding files
- 38. 38 Honorable Squires Support Time – Temporal Classification FuturePast Running Application Analysis of Error Situations Testing next Baseline Logging Log-ViewerReplay-Tool Replay-Tool Remote-Service Tool Time Recording
- 39. 39 Honorable Squires Visions – Optimized Updates CEI Control Center • lists instances • allows interactive / automatic updates CEI Control Center • lists instances • allows interactive / automatic updates CEICEI CEICEI CEICEI CEICEI • control center can inquire EI “busy” status • can request “idle” notification • can request to enforce “idle” situation • control center can inquire EI “busy” status • can request “idle” notification • can request to enforce “idle” situation • control center can initiate update • with restart • maybe without restart • control center can also change configuration • control center can initiate update • with restart • maybe without restart • control center can also change configuration
- 40. 40 Honorable Squires Copyright Information Clipart Page 17: – http://office.microsoft.com/en- us/clipart/results.aspx?qu=Knight&sc=20#12 Pictures Page 15, 18, 21, 23: – © Torsten Happ
- 41. AMD, the AMD Arrow logo, AMD Athlon, AMD Opteron, AMD Sempron, AMD Turion, AMD Virtualization, AMD-V, AMD LIVE!, AMD PowerNow!, AMD Cool’n’Quiet, ATI, the ATI logo, Imageon, Radeon, CrossFire, Avivo, TV Wonder, Theater, Xilleon, and combinations thereof are trademarks of Advanced Micro Devices, Inc. HyperTransport is a licensed trademark of the HyperTransport Technology Consortium. Microsoft and Windows are registered trademarks of Microsoft Corporation in the United States and/or other jurisdiction. Other names are for informational purposes only and may be trademarks of their respective owners. ©2007 Advanced Micro Devices, Inc. All rights reserved.
- 42. Georg Heeg eK Baroper Str. 337 44227 Dortmund Germany Tel: +49-231-97599-0 Fax: +49-231-97599-20 Email: georg@heeg.de http://www.heeg.de Georg Heeg AG Seestraße 135 8027 Zürich Switzerland Tel: +41(848) 43 34 24 Georg Heeg eK Mühlenstr. 19 06366 Köthen Germany Tel: +49-3496-214 328 Fax: +49-3496-214 712