BAM experiences in large scale deployments
Download as PPTX, PDF0 likes508 views
This document discusses setting up automated monitoring of a BizTalk environment using BAM (Business Activity Monitoring). It outlines several challenges faced with different implementation attempts, including issues with tools timing out or exceeding limits. Recommendations provided include using BAM in stable environments, scaling infrastructure appropriately, and knowing when to use tracking instead of BAM.
BAM experiences in large scale deployments
- 2. We’re gonna talk about...
- 3. • How is BAM working? Details.... • Challenges • AIMS BAM setup....in theory • Practial implementation, bugs and recommendations • Live experiences • Conclusion Agenda
- 4. • Data from ports & orchestrations • Realtime data (close to) • Not many prerequisites • Performance data • Low impact - > tracking / pipelines is not an option • Automated deployment • Autmated maintenance / updates • Supports BTS2006R2 and newer Challenges
- 5. • Integrated BizTalk functionality • Easy to enable, few prerequisites • Selectable tracking • Low overhead compared to tracking (unless you use it the same way as tracking) • Global tracking can be disabled • No big changes in different BizTalk versions • Tracking profiles can be deployed «in flight», adapts to changes We ended up with BAM
- 6. How is BAM working? MsgBox BAMPrimary (activities) Asynch TDDS Synch Eventstreams Tracking Tracking
- 7. • Scan BizTalk MgmtDb for artifacts & dependencies • Scan DTADb for orchestration XML to find shapes / orch ports • Dynamically deploy new activities / tracking based on changes How to automate the monitoring setup with BAM? That was the theory......sounds simple right?
- 8. • One activity for all orchestrations (ports & call / start shapes) • One activity for all receiveports • One activity for all sendports • Three trackingprofiles; receiveports / sendports / orchestrations • Used Microsoft.BizTalk.Bam.TrackingCompiler.TrackingCompiler / bm.exe ISSUES • Limit in number of fields in an activity definition • We had to create one unique field per shape per orchestration AIMS BAM – take 1
- 9. • One activity per each common orchestration shape • One activity for all receiveports • One activity for all sendports • One tracking profile receiveports, one for sendports, separate tracking profiles for each orchestration common shape activity • Used Microsoft.BizTalk.Bam.TrackingCompiler.TrackingCompiler / bm.exe ISSUES • Win32Exceptions on the TrackingCompiler (limit on open handles exceeded) due to large amount of ports in orchestrations • GUI handles in the TrackingCompiler, probably for TPE • Timeouts AIMS BAM – take 2
- 10. • One activity per each common orchestration shape • One activity for all receiveports • One activity for all sendports • One tracking profile receiveports, one for sendports, separate tracking profiles for each orchestration common shape activity • Used Microsoft.BizTalk.Bam.TrackingCompiler.TrackingCompiler / bm.exe ISSUES • Win32Exceptions on the TrackingCompiler (limit on open handles exceeded) due to large amount of ports in orchestrations • GUI handles in the TrackingCompiler, probably for TPE • Timeouts AIMS BAM – take 2
- 11. • One activity per each common orchestration shape • One activity for all receiveports • One activity for all sendports • One tracking profile receiveports, one for sendports, separate tracking profiles for each orchestration common shape activity • Bttdeploy.exe/ bm.exe ISSUES • Win32Exceptions on the bttdeploy.exe due to large amount of ports in orchestrations • GUI handles in the TrackingCompiler, probably for TPE. TPE dependencies in bttdeploy.exe (no logic, should been the other way around) • Timeouts AIMS BAM – take 3
- 12. • One activity per each common orchestration shape • One activity for all receiveports • One activity for all sendports • One tracking profile receiveports, one for sendports, separate tracking profiles for each orchestration common shape activity • Bttdeploy.exe/ bm.exe but this time more & smaller tracking profiles ISSUES • Win32Exceptions on the TrackingCompiler due to large amount of ports in orchestrations • GUI handles in the TrackingCompiler, probably for TPE • Timeouts AIMS BAM – take 4
- 13. • One activity for each orchestration shape • Multiple activities for all receiveports (100 per activity) • Multiple activities for all sendports (100 per activity) • Multiple tracking profiles for receiveports, multiple for sendports, separate tracking profiles for each orchestration shape activity • Bttdeploy.exe/ bm.exe ISSUES • Time consuming, thousands of trackingprofiles / activities • No changes to the BizTalk possible while deploying • Orphaned trackingprofiles AIMS BAM – take 5
- 14. • Good up to certain configs (6000+ components) / high throughput • TDDS not «cleaning» MsgBox quickly enough, leading to throttling • Condition worsening if tracking hosts & BAM are not properly scaled • Messaging has priority over TDDS if running TDDS on shared host (which still happens!) • Tested on multi MsgBox setups, scaled out BAM (dedicated server), single box setup, multi server setup, different clustering of BizTalk servers and SQL servers Performance
- 15. • Test & dev environments are subject to rapid changes • New TP deployments / removals necessary for each change • Takes time......and in the middle of this someone runs a deploy / undeploy • In many cases lead to orphaned tracking profiles Test / Dev environments
- 16. • BTS 2009 / 2010 with BAM and passthrough pipelines -> stopped processing of messages • No existing fixes for BTS2009 • BTS2010 CU 2-5 solved issue, CU 6-7 breaks support • DB locking during long deploys, caused uncomplete setup • XML encoding issues in tracking profiles (unknown chars) • Orchestrations without XML in DTADb Other bugs & issues XML declaration is removed at a receive location that uses BAM tracking and the PassThruReceive pipeline in BizTalk Server 2009 / 2010
- 17. • Use in stable environments with low frequency on changes and make sure you update your BAM tracking accordingly • Make sure you scale TDDS hosts, use dedicated tracking hosts • Use synch BAM if you want to bypass the MsgBox. However, synch «eats» cycles of the processing and can only be done from code (orchestrations, pipelines, externals etc) • Use asynch if you care about performance. Only option when using bttdeploy.exe. MsgBox is used for caching so monitor its size carefully • Ensure that your BAM SSIS packages are running • Know how to clean out orphaned tracking • Know when to use tracking instead of BAM. BAM is not necessary or good for everything Recommendations
- 18. AIMS conclusion
Editor's Notes
- #3: How is BAM working? Details.... Challenges AIMS BAM setup....in theory (automated, tracking profiles created, static activities) Practial implementation, bugs and recommendations (size limitations on tracking profiles, undeploy/ redeploy, timeouts, Live experiences (test environment challenges, orphaned, CU packs, performance, DB sizes, cleaning / archiving ) Conclusion
- #4: How is BAM working? Details.... Challenges AIMS BAM setup....in theory (automated, tracking profiles created, static activities) Practial implementation, bugs and recommendations (size limitations on tracking profiles, undeploy/ redeploy, timeouts, Live experiences (test environment challenges, orphaned, CU packs, performance, DB sizes, cleaning / archiving ) Conclusion