Zero to Automated in Under a Year
- 1. Not that long ago in a galaxy near and dear to my heart. . . .
- 2. Zero to automated in under a year
- 4. Square one M O N I T O R I N G All monitoring, configuration backups, and alerting happen through Orion Solarwinds NPM and NCM. Alerts go to an email distribution list. Backups occur once every 24 hours and are stored on the Orion server. C H A N G E C O N T R O L There is no peer review and no QA process. Changes are made at any time of the day regardless of environment or impact. When changes are pushed, an email is sent to an email distribution list for historical reference, but only a select few people have access to read them. A S S E T M A N A G E M E N T Device inventory is kept in Solarwinds and is updated as new devices come into the network. Physical datacenter locations are kept in a separate DCIM tool called Nlyte. IP addresses are documented in yet another separate tool called 6Connect. All data entry is manual. T E M P L A T E S Configuration templates are held in text files on user desktops and shared via copy/paste when needed.
- 5. SERVICES Systems square one Device monitoring Configuration backups Alerting S O L A R W I N D S DCIM (Data center infrastructure management) N L Y T E IPAM (IP Address Management) 6 C O N N E C T Bandwidth monitoring C A C T I B M S
- 6. 01 Locate a template for the device and configure it accordingly 02 Send an email to our change control distribution list 03 Update all of our systems manually: Monitoring Backups DCIM IPAM BMS 04 Solarwinds backs up the device within 24 hours and stores it on its local server Configuration changes Square one
- 7. Major issues Square one Any time something in the environment changes, engineers must update a handful of systems with similar information in each system. M U L T I P L E S O U R C E S O F T R U T H If data is entered into one of our systems, or a change in the network occurs, none of the other systems know about it. U N C O O R D I N A T E D S Y S T E M S Peer review and QA are completely hidden from those who don’t participate in the peer review and QA. If a change is pushed and someone wasn’t involved in it, they don’t know about it. L A C K O F V I S I B I L I T Y Everything is done manually. Whether we’re deploying new gear or making configuration changes, everything is done by hand. M A N U A L C H A N G E S
- 9. S I N G L E S O U R C E O F T R U T H We can’t trust multiple systems because a disparity in one system renders the information in all other systems suspect. C E N T R A L I Z E D M A N A G E M E N T Once the single source of truth is established, update that one source and force the other systems to react to it. N O M I S T A K E S Nobody’s perfect, but we should strive to be and build systems around us that can get us as close to a 100% success rate as possible. Core concepts Planning phase
- 10. M O N I T O R I N G & D A T A G A T H E R I N G D A T A V I S U A L I Z A T I O N D C I M & I P A M P R O J E C T M A N A G E M E N T C O N F I G U R A T I O N B A C K U P S T E A M C O M M U N I C A T I O N S Systems Planning phase
- 11. Major issues Planning phase We had everything down on paper but theory are reality rarely line up perfectly. We weren’t sure if everything would work like we expected, we didn’t know if our code would break everything, and since this was our first time rolling out a project like this, we didn’t know what we didn’t know. U N C E R T A I N T Y This project was a massive undertaking. We constantly had to check ourselves to make sure we weren’t biting off more than we could chew. It was incredibly important for us to define the scope and stick to it. S C O P E Deploying something from scratch and maintaining something day-in and day- out are two very different things. We needed to make sure that not only could we deploy this in our infrastructure, but that we could also maintain it for years to come. M A I N T A I N A B I L I T Y It’s hard selling this to upper management when it’s never been done in the company before. This project was very cost effective but it still had cost. We addressed this by deploying our systems in parallel to the existing ones and proving that it would work and that cutover would be seamless. B U Y - I N
- 13. Autonet is our in-house automation platform that handles communications between all of our systems, pushes configurations to devices, audits devices for configuration drfit, and dynamically keeps track of the devices on our network. Everything we do with network automation goes through Autonet. I n t r o d u c i n g autonet
- 14. Autonet ecosystem Current design P R O M E T H E U S Metrics, monitoring, and alerting P A G E R D U T Y Incident resolution A U T O N E T Centralized automation server running on Ubuntu S L A C K Team communications B I T B U C K E T Code repository J I R A Software development and project management G R A F A N A Data visualization N E T B O X DCIM and IPAM C O N F L U E N C E Documentation U N I M U S Configuration backups
- 15. New device Workflow Current design A D D D E V I C E T O P R O M E T H E U S New devices in our network are added to Prometheus, our single source of truth. Devices can be added manually, or found automatically by Autonet scripts. A U T O N E T U P D A T E S A L L S Y S T E M S Once a device is in Prometheus, Autonet triggers updates across all of our systems via a set of API requests. G R A F A N A Grafana is updated in real-time as soon as Prometheus is updated. It has a direct connection to all of our Prometheus servers. U N I M U S Autonet updates the device list in Unimus and triggers device backups on newly added devices. N E T B O X Autonet updates the device list in Netbox, racks the devices in their physical datacenter location in DCIM, and scans the device for IP addresses to add to IPAM.
- 16. New configuration Workflow Current design A U T O N E T G E N E R A T E S C O N F I G Engineers select the appropriate script to generate a configuration, add any required arguments, and then run a script that outputs a configuration. E N G I N E E R R E V I E W S C O N F I G The engineer reviews the configuration on the spot and performs a QA. We’re validating that the configuration is correct and if there are any improvements that could be made to the automation. C O N F I G I S P U T I N T O J I R A The engineer either manually puts the configuration into Jira, or we automatically create a Jira issue if the script allows it. When a Jira issue is made for a configuration, the engineer has an option to request a peer review. If a configuration came from automation, peer reviews are optional. E N G I N E E R P U S H E S C O N F I G The engineer pushes the configuration to the device(s). This can be done either manually or via the script that generated the configuration depending on the use case. T E A M I S N O T I F I E D V I A S L A C K After an engineer pushes a configuration, the Jira issue is marked as “configuration pushed.” Jira automation sends a message to our team channel in Slack notifying the group that a change was just made. U N I M U S D E T E C T S C H A N G E Unimus scans all devices every hour and looks for changes. When a change is noticed, it triggers a full configuration backup and sends a message to the team channel in Slack that shows a diff between the previous configuration and the new one.
- 17. ISP connectivity issues B G P S E S S I O N F L A P S A BGP session with one of our upstream ISPs goes down. It can come back up or stay down, that part is irrelevant to our automation. R O U T E R A D D S P R E P E N D S Our router automatically prepends advertisements out to that provider. A U T O N E T V E R I F I C A T I O N Autonet keeps track of changes so that we can review and resolve them. P R O M E T H E U S N O T I F I E S U S I N S L A C K / P D Prometheus sends API requests to Slack and PagerDuty. C u r r e n t d e s i g n S q u a r e O N E B G P S E S S I O N F L A P S A BGP session with one of our upstream ISPs goes down. If the circuit stays down, we reach out to our ISP for assistance. If the circuit bounces, our engineers make a judgement call on whether or not to take further action. S H U T D O W N T H E B G P S E S S I O N If further action is deemed necessary, an engineer manually shuts down the BGP session until we can get a resolution from the ISP. V E R I F I C A T I O N Our engineers monitor the circuit status and bring up the circuit when all issues are perceived to be resolved.
- 18. A U T O N E T C O N F I G U R E S A S A V P N Lastly, our automation configures the remote end of the VPN tunnels which terminate on Cisco ASA firewalls. A U T O N E T C O N F I G U R E S N S X E D G E Once the NSX Edge has been deployed, our automation configures its firewall and NAT rules, and builds several VPN tunnels required for management and security. A U T O N E T D E P L O Y S V M W A R E C O M P O N E N T S With the network layer complete, our automation moves on to the VMware stack. It deploys a dvPortGroup in vCenter, external and org networks in vCloud Director, and an NSX Edge firewall in the appropriate vCD org. A U T O N E T D E P L O Y S R O U T E R A N D S W I T C H C O N F I G S Based on the engineer’s parameters from the previous step, our automation builds the necessary network configurations for our devices. R E Q U I R E M E N T S A R E D E F I N E D Things like IP addresses and vCloud organization IDs are provided so that our automation knows what it’s deploying and where. Vmware deployment Current design
- 19. Major issues Current design D O U B L E T H E S K I L L S , H A L F T H E F O C U S We don’t have a dedicated automation team, so we handle all of the programming ourselves. Not only do our engineers need to be progressing in their network skills, but now they also need to be progressing in their programming skills. We’ve doubled the amount of skills they need, but we haven’t doubled the amount of time they get to work on those skills. This effectively halves the amount of time we spend focusing on networking so that the team can progress with their programming skills, and vice versa. We’ve accepted that it takes time for us to get to a true “network as code” environment and for now, our answer to this problem is to lean on each other for help. We hold team meetings where we shadow someone on a network automation script, we teach each other the things we learn throughout the week, and we make sure that if we see someone struggling, we pick them up and help them. We move forward as a team and without that, I don’t think we would have succeeded like we did in such a short amount of time. D E V I A T I O N S A D D C O M P L E X I T Y Our goal is to standardize as much as possible. However, due to things like customer requirements, supply chain issues, technology advances, and shifting business requirements, it’s impossible for us to standardize all of our devices and infrastructure across all of our datacenters. This leads to one-offs and slight deviations throughout the infrastructure that add complexity to our automation. We account for this as high up the programming chain as possible so that it propagates down to all of our automation and reduces the amount of work done when our network requirements change. For example, if something in a specific datacenter changes, we’ll account for this at our Device class for that datacenter so that all of the scripts using that class get the update.
- 20. improvements Future design N E T W O R K A S C O D E We’re currently restructuring Autonet to become the single source of truth. All changes to the network will be defined as a configuration file on the Autonet server and our automation will convert it to a network configuration and push it to devices as requested. All configuration generation, peer review, network changes, QA, and change logs will live within Autonet. F U L L T E S T S U I T E All changes to the Autonet code repository will go through a full test suite before making it into production, and all aspects of Autonet will be automatically tested daily. By building a virtual lab that contains all of our current firmware versions, we’ll be able to make sure all of our authentication, authorization, syntax, and overall logic remains functional and performs how we expect.
- 21. LESSONS LEARNED DO, OR DO NOT. THERE IS NO TRY. FEAR IS THE PATH TO THE DARK SIDE. SIZE MATTERS NOT. PASS ON WHAT YOU HAVE LEARNED. STRENGTH, MASTERY… BUT WEAKNESS, FOLLY, FAILURE, ALSO. YES, FAILURE, MOST OF ALL. THE GREATEST TEACHER, FAILURE IS.
- 22. Questions F I N D M E A T T H E C O N F E R E N C E I’d love to meet you and talk to you more about my presentation or hear any feedback you have for me. E M A I L M E Garrett Nowak gnowak@1111systems.com B U Y M E A B E E R I like networking. I like automation. I like beer. 3 GREAT WAYS TO CONNECT
- 23. thanks N E T W O R K A U T O M A T I O N F O R U M Thank you to everyone at NAF for giving me the opportunity to speak at this conference, and thank you to everyone who came here and listened to me. I hope to be included in future conferences and to meet with you all again! 1 1 : 1 1 S Y S T E M S Thank you to the company for supporting me. Thank you to my mentors for providing me with a foundation with which I could build a fulfilling career. Thank you to my team for always being there for me; I couldn’t have accomplished these things without you. M Y W I F E Thank you for listening to this presentation 900 times over the past few months. Thank you for being an amazing wife and mother. Thank you for always believing in me. G A R R E T T N O W A K Senior Director of Network Architecture