Dev Talk: Event Manipulation and Testing
- 1. Dev Talk: Event Manipulation and Testing Financial Services Group Jason Stanley Systems Engineer 4 at PNC jstanley734@gmail.com github.com/jstanley23 Zenoss Community Forums/irc: jstanley
- 2. © 2016 All Rights Reserved CONFIDENTIAL 3 Lets talk about: • Has/Get, Your two best friends • Debugging transforms • Warnings • Err on the side of caution • Take time to test and QA!!! • Automate testing/QA • Checking Triggers • Replaying Events • Trap mappings • zentrap replay function • Trap message format and MIB info • Event type traps Transforms Traps Tools and Testing
- 3. © 2016 All Rights Reserved CONFIDENTIAL 4 Best Friends • hasattr – Use to verify an object has an attribute before trying to call it. If hasattr(component, ‘speed’): stuff() • getattr – Use to get an attribute, an OR can be provided if the attribute does not exist. If ‘mysql’ in getattr(component, ‘displayName’, ‘’): stuff() Some attributes on evt can return None, be careful using operations/methods directly.
- 4. Debugging transforms • Add set_trace() to your transform to debug. • Use dir() to list out what attributes/objects are available to you. Will also show you attributes on an object. • device and component context objects will be set to the Zenoss object if possible. © 2016 All Rights Reserved CONFIDENTIAL 5 http://wiki.zenoss.org/Debugging_Event_Transforms
- 5. © 2016 All Rights Reserved CONFIDENTIAL 6 Warnings • Never assume an attribute will be present, always check • More logic to start, it is better to be explicit than to change events you did not account for. • Always verify and test your transforms. Looks for errors in zeneventd.log • Careful about dedup, think about the clears as well • Default to the original • Careful of changing severity of clears • Instead of cycling through components to find the proper one, try using ‘component’, or another field on the event. • Stay away from database commits • Speed is key. If a transform takes too long, it can bottleneck zeneventd workers. • If you want additional work to be done, use other methods. Triggers/Notification or a Job.
- 6. Careful adding pre-existing event classes in your ZenPacks. These Event Classes will be removed if your ZenPack is removed. Unless leaveObjects is set. Even upgrading, if the upgrade fails, your ZenPack can get removed. Verify your ‘ZenPack Provides’ in your ZenPack to make sure standard Classes have not been added. Use Sub Classes when possible to avoid removing standard classes. © 2016 All Rights Reserved CONFIDENTIAL 7 Adding Event Classes to ZenPacks
- 7. © 2016 All Rights Reserved CONFIDENTIAL 8 Transforms in ZenPack code PROS • Prevents users from changing logic • Easier version control • Easier testing and profiling • Better flow control CONS • Time consuming • Complicated to change logic on the fly • Lack of transparency
- 8. © 2016 All Rights Reserved CONFIDENTIAL 9 cProfile Profile your code to see where it is slowing down or what calls are taking the longest.
- 9. © 2016 All Rights Reserved CONFIDENTIAL 10 zeneventd debugging Add these options in zeneventd.conf if you are noticing issues processing events. usemetrology True logperfasinfo True This will allow you to see where zeneventd is having issues processing events. These settings currently only work with 4.2.5
- 10. © 2016 All Rights Reserved CONFIDENTIAL 11
- 11. © 2016 All Rights Reserved CONFIDENTIAL 12 • Load MIBs to allow zentrap to translate OIDs • Build mappings • evt.eventKey = random.getrandbits(128) • zentrap replay Starting with traps
- 12. © 2016 All Rights Reserved CONFIDENTIAL 13 zentrap replay Capture: zentrap run -v10 --cycle --captureAll --captureFilePrefix=/home/zenoss/traps/trap_ Replay: zentrap run --replayFilePrefix=/home/zenoss/traps/trap_-127.0.0.1-1 Do not leave running if you are receiving a lot of traps from devices.
- 13. © 2016 All Rights Reserved CONFIDENTIAL 14 Use the MIB to easily define event message and severity for traps MIB Information can be useful
- 14. © 2016 All Rights Reserved CONFIDENTIAL 15 Dedup and Event Traps Some traps can be considered an Event trap, where each trap that comes in can be a different event/alarm from the device. Without transform, these traps will roll into the previous. Depending on how verbose you want to be, use one of the trap fields to prevent this. • Use eventkey or summary to prevent dedup • Use a field like swEventIndex to make sure an event is created each time. • Or perhaps you want to dedup on the event type. In which case you could use swEventDescr Dedup fields Default: device, component, eventClass, eventKey, severity, ipRealm* No event key set: device, component, eventClass, severity, summary, ipRealm* * ipRealm is only used if you have the MultiRealm ZenPack installed
- 15. © 2016 All Rights Reserved CONFIDENTIAL 16 Testing and QA • With great power, comes great responsibility • Plan all your tasks with enough time to test and QA • Treat rolling out tasks in Zenoss like Software development • Don’t rush or skip steps to “produce” more • If a timeline cannot be met because of the extra time, tell management what you need to meet timeline.
- 16. © 2016 All Rights Reserved CONFIDENTIAL 17 Automate/Script Everything! Use automation and scripts to help development, test and QA • Generate events using zensendevent • Create mappings from MIBs from zendmd • Inject devices zenbatchload or jobs • Capture/Replay events using zentrap or other methods • Test triggers/notifications • Verify performance data using reports, zendmd or rrdtool
- 17. © 2016 All Rights Reserved CONFIDENTIAL 18 Check Triggers Use a lot of triggers? Add a ZenPack to check event(s) against one or all triggers to see which evaluate true. Utilize it for verifying your triggers do not pick up unwanted events.
- 18. © 2016 All Rights Reserved CONFIDENTIAL 19 Event Replay Allows for capturing of raw events to be replayed later. Test transforms and mappings Capture rare events Use local redis or remote Save raw events for later Import them into another QA/Test environment Export rawEvents to a pretty JSON format for review
- 19. DO NOT USE IN PRODUCTION ZenPacks: Event Replay https://github.com/jstanley23/ZenPacks.jstanley.EventReplay Check Trigger https://github.com/jstanley23/ZenPacks.jstanley.CheckTrigger
- 20. © 2016 All Rights Reserved CONFIDENTIAL 21 Questions
- 21. ZenPacks: Event Replay https://github.com/jstanley23/ZenPacks.jstanley.EventReplay Check Trigger https://github.com/jstanley23/ZenPacks.jstanley.CheckTrigger Jay Stanley jstanley734@gmail.com github.com/jstanley23 Zenoss Community Forums/IRC: jstanley
Editor's Notes
- #2: In this session we will be discussing event manipulation (transforms), trap handling and testing/QA Who am I? - Maintaining the health of the Zenoss environments (11 total, 5 production, 4 QA, 2 test) - Developing ZenPacks that: - Add functionality to Zenoss - Building out new monitoring for new devices and applications - And ZenPacks that extend the Zenoss API to work with our external automated systems Experience (4+ years) Two different companies that had different approaches to monitoring and how they used Zenoss
- #4: We will start out with some basics and slowly move into some more advance ideas. Advanced Zenoss users will already know a lot of what the items I will cover, but my hope is to provide some knowledge or ideas that some of you can use. I will cover common mistakes I that I have see (and made) over the years and some warnings on what to look out for or avoid doing. We will dive into traps, where I will share how I make working with traps easier and quicker. Then finally, we will talk about testing and qa. How important it is and how you can develop ZenPacks to provide additional functionality to Zenoss to ease the pain and lessen the amount of time needed.
- #5: Calling methods off of getattr(evt, ‘’)
- #6: Chet showed me this when I first writing my first zenpack. And I have to say, this tip has helped me out so much. Any issues you have with writing transforms, I would recommend using this.
- #7: This doesn’t mean you cannot do some crazy things in transforms, but it could affect performance and present problems in the long run. I would limit the number of events hitting those crazy ones as much as possible using event classes and logic. And be prepared to scale out zeneventd workers as needed.
- #17: Engineers and developers: Take extra time to test and QA more. Plan for it, when asked for a timeline, make sure you give enough time that includes QA. Managers: Don’t rush us! Expect for this QA time. ASK FOR IT. Zenoss is a great tool. You can pretty much do whatever you want with it, with some development time. It has this framework that allows for easy development. Using zenpacklib you can build out a ZenPack in 15 minutes. You can easily expand the API to allow for other teams to get information they need or automating common processes. But with this “great power” comes great responsibility. Zenoss gives you all the rope you need to do whatever you want, but it comes with a price. So, be careful, test and QA.
- #18: Engineers and developers: Take extra time to test and QA more. Plan for it, when asked for a timeline, make sure you give enough time that includes QA. Managers: Don’t rush us! Expect for this QA time. ASK FOR IT. Zenoss is a great tool. You can pretty much do whatever you want with it, with some development time. It has this framework that allows for easy development. Using zenpacklib you can build out a ZenPack in 15 minutes. You can easily expand the API to allow for other teams to get information they need or automating common processes. But with this “great power” comes great responsibility. Zenoss gives you all the rope you need to do whatever you want, but it comes with a price. So, be careful, test and QA.