Learning from failures
1 like195 views
The document discusses the importance of learning from failures to improve network reliability. It provides examples of routing failures and port assignment mistakes. The author advocates creating a database of common mistakes by analyzing failure cases. This would help engineers identify frequent issues and their root causes to develop better solutions. The author also suggests the bdNOG community could conduct accident investigations to prevent repeated failures from occurring.
Learning from failures
- 1. Learning from failures Yoshinobu ‘maz’ Matsuzaki <maz@iij.ad.jp> bdNOG12 maz@iij.ad.jp 1
- 2. Reliability is getting important • More use of the Internet • COVID-19 has been pushing digitalization • Bandwidth is a key • When congestion occurs, the experience gets worse • But enough bandwidth just is not enough • Even if you have it set up wrong, you can still use it somehow • Reasonability, stability and resiliency is the other key bdNOG12 maz@iij.ad.jp 2
- 3. Risk prediction training 1. Understanding the situation • Discuss imaginable hazard scenario in the given situation. 2. Determining risks • Identify the hazards that need to be addressed 3. Establishing countermeasures • Discuss possible measures to solve the hazards 4. Setting goals • Selecting possible measures to implement bdNOG12 maz@iij.ad.jp 3
- 4. Example1: Routing • An ISP assigns /24 for a customer • ISP set up a static route for the link • The customer set up a default route to the uplink • The customer uses /28 out of the /24 10.0.0.0/24 10.0.0.0/28 static route static route default bdNOG12 maz@iij.ad.jp 4
- 5. Example1: Risks • If a packet comes to an address other than the /28 out of the /24, the packet will be looped • If the customer's LAN-side interface is down, all packets destined for the /24 will be looped. • Routing loop! 10.0.0.0/24 10.0.0.0/28 static route static route default A packet to: 10.0.0.99 bdNOG12 maz@iij.ad.jp 5
- 6. Example1: Measures • Implementing dynamic routing between ISP and the customer • Configuring a static route on the customer's router that directs the same /24 to null 10.0.0.0/24 10.0.0.0/28 static route static route default bdNOG12 maz@iij.ad.jp 6
- 7. Example1: Adopting • Configuring a static route on the customer's side router that directs the same /24 to null 10.0.0.0/24 10.0.0.0/28 static route static route default 10.0.0.0/24 static null route bdNOG12 maz@iij.ad.jp 7
- 8. Example2: Port assignments • Removing a cable from port X • Just to be safe, make sure the LED is off before pulling it out • But can you spot the right port for sure? bdNOG12 maz@iij.ad.jp 8
- 9. 1 2 3 4 5 6 Straight forward Starting from port 0The left LED is for LC status More efficient but confusable A little clearer Port 21 is the SFP now bdNOG12 maz@iij.ad.jp 9
- 10. And more... • We may see a different implementation in the future • Assumptions are the source of accidents! • Different products have different port/LED assignments • These caused confusion bdNOG12 maz@iij.ad.jp 10
- 11. The more you know, the more you can see • A variety of experience helps us to better consider the hazards • and to identify risks • Technical education and proper training are necessary to improve operational skills • bdNOG workshops and tutorials are helpful • There is always a need for appropriate educational materials bdNOG12 maz@iij.ad.jp 11
- 12. Mistakes! • Mistakes can be a very good teaching tool • There is a lot to learn from mistakes in the case studies • There are some special cases, but there are also many common failures and lessons to be learned by comparing them to your own situation • But as a business, we need to stop repeating failures in our service facilities • It damages reliability bdNOG12 maz@iij.ad.jp 12
- 13. Build a database of mistakes • It can be a great teaching tool for engineers! • not to reproduce the similar mistakes • You may find common and frequent mistakes • If you can find the root cause of the failure, you can come up with a more effective solution bdNOG12 maz@iij.ad.jp 13
- 14. Mistake trend analysis • Identify the high-impact mistakes • Minimize the bad effects • Reduce mistakes bdNOG12 maz@iij.ad.jp 14 effects of mistakes frequency of mistakes should not be happened problemsmatters problems
- 15. Accident investigation committee • In some industries, Accident Investigation Committees conduct detailed investigations and compile reports in order to prevent the repeating of serious accidents • Maybe bdNOG can do this as a community activity • For the healthy development of the Internet in Bangladesh • Regular reports of accident cases during bdNOG meetings bdNOG12 maz@iij.ad.jp 15
- 16. Summary • To have a reliable network, we need to continuously improve our operations • The use of failure cases allows for more effective risk analysis and countermeasures • As bdNOG community, I believe the following are worth considering • Collection of failure and mistake cases • Trials of accident analysis bdNOG12 maz@iij.ad.jp 16