How to create effective NOC in Poland
- 1. Kamil Grabowski, Paweł Nastachowski PLIX
- 2. Sylwester
- 3. What is PLIX? • PLIX is not only Sylwester Biernacki • 11 people, the chameleon, and some fish. • PLIX is the only neutral (and the biggest!) point of exchange of IP traffic in Poland. • We are in 3 locations, including the warmest place in Warsaw (LIM 42th floor).
- 4. PLIX in numbers The amount of Coustomers
- 5. PLIX in numbers The amount of traffic
- 6. PLIX in numbers January 2009 September 2009 Growth in % Customers ~90 108 20% Switches 11 11 0% Ports 1/10G 386 406 5% Traffic ~45 Gbps > 65 Gbps 44% BGP Session 200 > 225 12%
- 7. Network Operation Center • What is a NOC? • Requirements • Monitoring • Diagnosis • First Aid • Customer information • Events documentation
- 8. NOC PLIX • This is our NOC (a ‘potty’ in English means NOCNIK / NOC operator in Polish ((-;
- 9. NOC PLIX • ... And an "old school" NOC Manager
- 10. NOC PLIX What do we really monitor in PLIX NOC • BGP Session state • IP traffic level • Availability BGP neighbours • Port state (up/down)
- 11. NOC PLIX • It operates 24 hours a day, 7 days a week • You can contact us in many ways: • Phone: +48 22 427 39 99 • Mobile: +48 721 333 999 • E-Mail: noc@plix.pl • Customer panel my.plix.pl • It is always on the site
- 12. my.plix.pl Drawing irb(main):> rand(108) % 1 + 73 => 73
- 13. my.plix.pl
- 14. Disadvantages of the former NOC – Tools: nagios • No possibility of grouping events, • Resources, • Nagios must regularly check the all of the services
- 15. Disadvantages of the former NOC – Tools: CACTI • Problems when switching customers between ports/devices, • Lack of integration with CRM, • No possibility of traffic aggregation from selected ports.
- 16. Disadvantages of the former NOC – Tools: SMOKEPING • Poor notification, • a lot of e-mails.
- 17. Disadvantages of the former NOC – Tools: TRAP BROWSER • A large number of events to review, • No possibility of grouping events, • No filtering / searching.
- 18. Disadvantages of the former NOC - Tools. • Many utilities are : • many different APIs • many different user interfaces, • many different configuration files • many different types of notification…(e-mail, sms, www), • ... and lack of personalization (example: assigning the client to the port), counting SLA (agreement with the client), • All of this makes the monitoring system is very unreadable .
- 19. Disadvantages of the former NOC - Infrastructure • No redundancy, • Susceptibility to failure, • Difficulty in maintaining and updating, • Complex relations between system elements, • Long time "disaster recovery", • Low productivity, duplicating processes
- 20. Disadvantages of the former NOC - Infrastructure
- 21. Changes were needed • To cope with the demands we had to have eyes in the back of our heads, • The recruitment process won the chameleon called Strips. (His name takes roots from a type of a meal served in KFC restaurants), • We decided to hire him • It was a good step. • As you can see he does excellent cable work and does his best debugging our software (from any possible crickets )
- 22. PLIX LAB - objectives • The creation of efficient infrastructure and redundancy, • Creating a consolidated system of notification of anomalies occurring in the networks, • Intelligent monitoring - diagnosis, grouping and filtering of events, • Centralization (webservice) - uniform API for all tools, • Personalization of systems, in the direction of the SLA, and contract terms with customers
- 25. PLIX LAB - implementation • New infrastructure, • Admin.plix • Webservice - API • Trap Monitor, • Live Network Map, • Live Network Traffic Graph,
- 26. Implementation - infrastructure • Redundancy of services, • Virtualization, • Simplifying the process of deploying and maintenance, • Optimization processes (no tasks duplication)
- 29. PLIX LAB – admin.plix • Complete customer information (ports, sessions, statistics), • Infrastructure management and ticket system, • Configuration tools, which are still in use (nagios, smokeping)
- 30. Competition!!! • Guys probably forgot about the competition, so the slide will remind them:) • Do you remember how we called our chameleon, and what the origin of his name is? • All the time thinking about you ... Sylwester
- 31. Implementation - tools Trap Monitor • Visualization of events (live), • Centralization, • Grouping of events on the basis of the customer information system (session, port, ip), • Uniform notification, • Trap browser, • Precise counting time of failure.
- 32. Implementation - tools Trap Monitor
- 33. Implementation - tools Live Network Map • Visualization of network infrastructure, • State of the network infrastructure, • Saturation uplinks between the switches, • Visualization of events on devices.
- 34. Implementation - tools Live Network Map
- 35. Implementation - tools Live Network Traffic Graph • Drawing graphs with traffic on selected ports, (example: uplinks) in real time, • Sampling every 30-60 seconds (optional).
- 36. PLIX LAB – current projects Trap Monitor • Create tickets on the basis of grouped events, • Grouping and filtering of events using values such as BGP software, data transmission provider, location, • Intelligent diagnostics - the ability to detect a power outage at LIM, the supplier of transmission failures, errors in the software.
- 37. PLIX LAB – current projects Live Network Traffic Graph • Analysis of traffic (multicast, broadcast), • Anomalies (peaks and declines).
- 38. Brain storm • What an ideal NOC should be like? • How should NOC work? • What tools should it use? • Procedures
- 39. Questions?
- 40. ”