AnsibleFest 2019 - Greenfielding Network and Systems Automation in a Large and Highly Dynamic Public Transit Network

Editor's Notes

• #3: - Programming for 21 years - Worked in the DC industry for 8 years (this week actually) - started in systems, graduated to network, and now I’m officially “DevOps” - using ansible for 5 years - fell into it as part of bootstrapping servers at my previous employer (public cloud, IaaS, private cloud, DR, Backups) - Ended up using it to auto build BGP peering configs, which then turned into config gen of BGP route-policies and customer provisioning (migrating perl to Ansible ugh)
• #6: Intro to TW - How many of you have been to NYC? - How many of you knew we have cellular/wifi in the stations? - How many of you have actually used it? Thoughts? I’m taking your feedback to my superiors ;) - 472 stations are covered (409 underground with cellular/wifi, rest are aboveground with just emergency services)
• #9: Who here has spotted red flags in this list so far?
• #10: I want people to start raising your hands when you start seeing red flags. Massive footprint of vendors, firmware versions, and platforms to cover NYC itself has almost 20,000 devices And if that wasn’t enough, Network_cli just isn’t enough sometimes. That doesn’t mean netconf or httpapi is used instead… Yes, that means some things rely on telnet. And the telnet module doesn’t always work that well. So that means expect scripts being executed by Ansible Those of you who haven’t raised their hands yet, try working with telnet on unsupported devices with Ansible and then let me know how you feel with your life. Now this also means there’s LOADS of underlying groundwork that is required to cover our network
• #11: So how do you even begin automating a massive network, largely using unsupported telnet with Ansible and expect scripts? Describe our low hanging fruit: Systems Proxmox/Netbox integration CIMC LDAP DNS DHCP Reporting
• #12: Source of truth, (not zabbix or any monitoring tool, and holy hell not excel sheets) Secrets Management CMDB Central Authentication. At the time of hiring everything was local authentication…. Everything. Self Service portal for less CLI proficient NetOps/NOC teams DEVELOPERS I knew coming in, that I would be the only one with programming experience and would be responsible for training the team or acquiring training for the team. And that’s ok!
• #13: - One teammate had already started dabbling with Ansible. - Luckily we had a full top to bottom network audit going on when I was hired, so it was fairly easy for me to latch onto that and start pulling reports in support of the audit. Stuff like “how many interfaces do we have in this datacenter that are used and unused”, or “What vlans are present in every station’s switches, and further more, what interfaces had what vlans on them and were they Admin Down/Up?” - Audit was completed over a month ahead of expectations. - This seems sort of like a no-brainer, but every company is different with a different budget. Coming in hot saying you need 4 people to immediately start going to several new cons that nobody there has ever attended certainly won’t go over well.
• #14: So let’s break this down into the Crawl Walk Run principle My number one goal in our automation journey is Team Education. Lay the groundwork to either build a robust monitoring solution or enhance the one you have. For us, we have one of the largest known Zabbix databases, currently at 13TB, that Zabbix is aware of. In fact, our monitoring is so important to our automation strategy that I tied it into our Education training and myself and 3 other teammates spent all last week in Zabbix training. Two of us, myself included, were there for the certifications as well and took and passed the Zabbix Certified Professional cert on Friday Luckily, our monitoring was already robust enough for us to get started. However, there’s always room for improvement and we’re on that road in parallel.
• #15: Introduce Netbox as the new SoT. This entails sanitizing and importing massive excel sheets, physically verifying information, physically mapping our rack elevations, TAGGING Someone in Network To Code’s slack mentioned a tagging strategy called Namespaced tagging, which I immediately loved the idea of and have started working on fitting that into our Netbox tagging Using facts from the network to populate config contexts in Netbox Setup internal DNS infrastructure to get away from IPs everywhere and to simplify our Ansible Inventory Introduce FreeIPA as new central auth, migrate local users to LDAP/Radius users, and setup service accounts for automation and other items that needs access (such as oxydized) Start small with jinja tempating and config modules to deploy these new AAA and DNS changes. Start with the Core IP network only. Don’t even think about touching stations yet LAB EVERYTHING! Luckily we have massive GNS3 hosts that our engineering team uses religiously to model changes to the network before applying them. This has drastically reduced error rates compared to previous years of not having that. Now the automation tests you run (you better be) can be tested against nearly identical GNS3 VMs.
• #16: Netbox implementation is complete and now you have an amazing dynamic inventory! Evolve your monitoring metrics to try to catch issues with automation changes before they cause a larger issue Implement rollback options! This means we have to integrate with Oxydized (which is our network backup solution of choice) to make a backup before the play runs. Drastically reduce field tech provisioning by implementing Zero Touch Provisioning in conjunction with Ansible. All they have to do is make sure the mgmt interface has DHCP running. We do the rest. If your org doesn’t already have good change mgmt and project mgmt culture, this is a must. We prefer Jira. You should’ve seen the drools from mgmt’s faces when I told them I was going to implement auto documenting change mgmt via ansible. NTC has a really great talk that Jason Edelman did on this very topic with ServiceNow. I’d highly recommend checking that out on youtube for an example of what this would look like. And of course, continue iterating your automation so that less and less manual changes are made to your devices
• #17: You may be wondering why I haven’t mentioned Stackstorm at all yet at this point. Basically Stackstorm is a workflow mgmt platform and is really a central part of our entire automation pipeline. Most of our workflow traverses through Stackstorm in some capacity to route specific actions to different tools all at the same time. In some cases, Stackstorm is listening for changes and acting on those, such as new Jira tickets, Zabbix alerts, Slack commands, or even Viewflow.io API calls from the self service portal.
• #20: So here’s a sample diagram of 4 train cars, not specifically NYC Subway related. When we’re considering putting Wifi onboard the trains, we have to assume the cars won’t stay together, but they have types of cars that form a pattern. A cars will always be at the front and back of the train and sometimes spreadout in the middle as well. C cars are everything in between. How in the world do you keep train networks consistent when you never know who your neighbor is going to be? Well I can’t say exactly how unfortunately but this is where ZTP plays a heavy role in our automation strategy, even more so than in the station level network. Essentially what could happen is a train will go into the yard at the end of it’s daily shift. That train could get dismantled and serviced in a variety of ways, again depending on the train operators involved. We have to assume that an A car and a C car will never see each other the next day when it’s placed into another train. By default our devices will reach out to ZTP on car power up to get it’s base golden config. Once it reboots into it’s new config, our onboard train automation will setup the dynamic peering with other cars (intentionally vague there) which ends up with a full train, completely networked and providing public wifi to it’s customers.