Embedded Webinar #17 "Low-level Network Testing in Embedded Devices Development"
0 likes495 views
The document outlines the importance of low-level network testing in embedded devices, highlighting their unique properties and requirements. It details the approach, tools, and various issues encountered during testing, such as TCP scans causing server downtimes and UDP packet fragmentation issues. The conclusion emphasizes the necessity of thorough low-level testing whenever hardware or configuration changes occur.
Embedded Webinar #17 "Low-level Network Testing in Embedded Devices Development"
- 1. Network Security in Embedded devices Serhii Korniienko - Embedded QA engineer
- 2. Examples of properties of typical embedded computers when compared with general-purpose counterparts are: • low power consumption • small size • rugged operating ranges • low per-unit cost • often ‘non-standard‘ solution
- 3. Why low-level network testing is a must in embedded: • Often custom protocols used to reduce CPU/RAM/ROM/bandwidth usage • Low cost • Timing properties of the program depend heavily on its environment • Environment = +Process or & Memory Hierarchy + Operating System, other processes/threads, … + Network + I/O Devices +…
- 4. Our experience: 1. Requirements 2. Approach 3. Tools 4. Test setup 5. Logs gathering and analysis 6. Issues 7. Test cases 8. Process
- 5. 1. Our requirements: • Low cost – cheapest HW components approved by customer (BoM < 20$) • Http and custom UDP support • Stack: freeRTOS + LwIP • Rigid requirements on synchronization jitter (PTP) • IC network switch to build daisy-chained network structure • Stable work in heavy loaded network (1Gb wireshark dump file per minute)
- 6. 2. Low –level testing approach: • List of all low-level network stuff: – Ports inventory: – 80/tcp – http – 554/tcp - rtsp – 67/udp - dhcps – 68/udp - dhcpc – 319/udp - ptp-event – 320/udp - ptp-general – 2467/udp - custom protocol – 17100/udp - custom protocol
- 8. 3. Tools: 1. NMAP – enumeration and packet sending - sS - sU - sO - sX - sN - sF - sX - sA - sY - --scanflags
- 9. 3. Tools: 2. Ostinato –packet editing and traffic generation - Set a value for any field of any protocol - Open and edit PCAP files, replay and save back - Support for the most common standard protocols - Free, Libre, Open-Source Software
- 10. 3. Tools: 3. LOIC – open-source network stress testing and denial-of-service attack application - IC Switch throughput test
- 11. 4. Test setup
- 12. 5. Logs gathering and analysis: The key thing is to gather and analyze all available info from all system components (switch and microcontroller) to recognize and eliminate bottlenecks and issues: - Buffer overflows and underflows - Queue overflows and underflows - Protocol/link state changes - Tx/Rx error counters - Switch counters - Error messages - Power dropdown - Unauthorized recording - Voltage levels - …
- 13. 6. Issues found: Issue Web server goes down during polite TCP scan Discovered/ Investigated NMAP scan Root cause LwIP configuration issue
- 14. Issues found: Issue Discovered/ investigated Root cause Hard-fault occurs if send fragmented UDP-packet NMAP/ Wireshark/ Ostinato LwIP incorrectly reassemble fragmented UDP packets
- 15. Issues found: Issue Discovered/ investigated Root cause Device can be shutdown by specially crafted control message Script/fuzzing Input validation issue Device can turn off it’s power supply and network chip power supply ☺
- 16. Issues found: Issue Device hard-fault if send abnormally large or small message on control port - Discovered/ investigated LOIC/ Script/fuzzing Root cause Input validation issue
- 17. Issues found: Issue Discovered/ investigated Root cause Sending short ptp packet cause ptp recalibration and packet loss NMAP scan Ostinato LwIP issue
- 18. Issues found: Issue Discovered/ investigated Root cause Network link goes down and Up during nmap scan NMAP/ Wireshark/ Ostinato PSU issue for new IC switch Network link goes down during nmap scan NMAP/ Wireshark/ Ostinato unauthorized command on switch management interface
- 19. Issues found: Issue Discovered/ investigated Root cause Device become unreachable if send short UDP packet on port 319/320 NMAP scan Ostinato IC switch issue All devices in chain become unavailable if perform UDP flood on port 319/320 of last device in chain LOIC/Ostinato IC switch issue
- 21. Test cases:
- 22. Test cases:
- 23. 8. Process: • Perform low-level testing to every hardware changes • Preform low-level testing for every configuration changes • Preform low-level testing for every low-level structures changes • Preform low-level testing periodically
- 24. Conclusion: If you still hesitate about the necessity to perform low-level network testing – go to the issues found chapter!