Real time data streaming and motion control over the internet
0 likes955 views
The document outlines the development of a real-time data streaming and motion control system for a vehicle connected to the Internet, focusing on quality of service (QoS) aware streaming. Key challenges include delivering consistent video streams and remote control commands despite variable network conditions. The implementation strategy involves using adaptive streaming techniques to monitor and react to network changes, optimizing both video quality and latency for effective vehicle control.
Real time data streaming and motion control over the internet
- 1. Motivation Our Results Summary Real-time data streaming and motion control over the Internet Implement real-time QoS-aware data streaming for Internet-connected things. Andrey Nechypurenko andreynech@gmail.com Maksym Parkachov mparkachov@me.com Ultracode-Munich, April 2014 veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 2. Motivation Our Results Summary Stream data between connected things Problem statement Let’s build the vehicle and control it over the Internet As a hobby project, we start developing small vehicle equipped with on-board computer connected to WLan adapter and web-camera. The idea was to control the car over Internet. Main challenges Find or build mechanical platform Build electronic which can: Compress live video stream to h264 format in real-time Support W-WLan connectivity Have enough IO channels to control motors Develop software which can: Deliver video stream and sensor data to the remote driver Receive user input such as steering and acceleration Deliver control commands from the driver to the vehicle software and drive actuators Support client and server NAT and firewall traversal Provide high quality video under variable network conditions veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 3. Motivation Our Results Summary Stream data between connected things Problem statement High level system overview The whole system software has two main tasks: Deliver video stream and sensor data from the vehicle to the remote driver and Deliver control commands from the driver to the vehicle software and drive vehicle actuators. Communication happens over the Internet where it is typical to have two firewalls (and/or NATs) on the client and server side. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 4. Motivation Our Results Summary Stream data between connected things Problem statement Mechanic and electronic veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 5. Motivation Our Results Summary Stream data between connected things Problem statement Need for adaptive video streaming It is important to provide constant frame rate with predictable latency to precisely control the vehicle. Otherwise, the wall may suddenly appears ahead of the car :-) . Negative effect of the changing network conditions leads to: 1 Delays in video stream (picture freezes). 2 Corrupted frames (because of dropped frames, etc.). 3 “Fast forward” effect when the next chunk of data arrives. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 6. Motivation Our Results Summary Implementation Strategy Adaptation Solution - QoS aware streaming The adaptive Quality of Service (QoS) aware streaming implementation is required to let the driver precisely control remote vehicle. To solve the problems mentioned above: We constantly observe network conditions. If sensor data could not be send fast enough, adapt to satisfy main goal. For example, reduce video frame-rate or increase compression rate (e.g. reduce video quality). The core adaptation logic resides in the vehicle on-board application. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 7. Motivation Our Results Summary Implementation Strategy Adaptation Main software modules To provide this functionality two main modules are required: Driver application which will be further reffered as cockpit. Vehicle on-board application which will be reffered as vehicle. In addition, to perform firewall traversal in the secure and efficient way, additional application is required on the server side. In this project, ZeroC Ice open-source middleware is used for all communication needs. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 8. Motivation Our Results Summary Implementation Strategy Adaptation Cockpit application. Cockpit application Cockpit application is responsible for receiving video stream, decoding, and visualizing it. In addition cockpit application receives control signals from input hardware and transmit commands to the vehicle. Implementation is heavily multithreaded and uses graphics hardware acceleration. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 9. Motivation Our Results Summary Implementation Strategy Adaptation Cockpit application An OpenGL-based application to remotely control the robot manually. Sensor data (including video from cameras) are rendered in real-time and control commands are sent back to the vehicle. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 10. Motivation Our Results Summary Implementation Strategy Adaptation On-board software The vehicle on-board application has the following responsibilities: Receive commands (such as steering and acceleration) and control actuators. Capture video, compress it in real-time and send to the cockpit application. Collect statistic about network bandwidth to perform adaptation in case of changed network conditions. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 11. Motivation Our Results Summary Implementation Strategy Adaptation QoS aware streaming Use TCP instead of UDP to get better feedback about packet delivery status. Permanently monitor the size of output queue with compressed frames to deduce the current QoS conditions. Define the set of states characterized by maximum and minimum queue size to transition to the better or worth state. In addition, the time-based weight is introduced which is calculated based on the overall time spent in certain state. It prevents the system from permanently jumping from one state to another. React on the state changes by dynamically adjusting frame size and codec target bitrate (if it is supported by codec). veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 12. Motivation Our Results Summary Implementation Strategy Adaptation Alternatives for streaming implementations Two implementation options were considered: Using GStreamer RTSP server. Custom transport protocol for RTP payload using Ice middleware. To make final decision, both variants were implemented and compared. Reasons why Ice was considered as communication solution: Reduce complexity when implementing complex bidirectional communication. Transparently handles cross-platform issues such as endianess. There are two versions of Ice. IceE which has reduced footprint and easier to cross-compile. The complete version provides the full set of functionality. Very easy to change the communication protocol. Changing between UDP, TCP or SSL is the matter of change endpoint description in the configuration file. There is service application Glacier which solves firewall/NAT related problems. Asynchronous Method Invocation (AMI) which makes possible to collect more information about bufferization and transmission performance. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 13. Motivation Our Results Summary Implementation Strategy Adaptation Test setup Stream video using two alternative implementations. Use Linux kernel trafic shaping capabilities to simulate variable network bandwidth. Use prerecorded video For test purposes, we feed the prerecorded video instead of live capturing. Original file is scaled down on the fly to make the test comparable with live streaming from web-camera. Decoding pipeline for cockpit application was not changed. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 14. Motivation Our Results Summary Implementation Strategy Adaptation Test setup Linux kernel network traffic shaping capabilities are used to restrict the available bandwidth and see how both streaming implementations would react on it. 1 MPlayer video output window. 2 Terminal window where traffic shaping commands are issued. 3 MPlayer console output. 4 KNemo network monitor window. 1 Cockpit application window. 2 Terminal window where traffic shaping commands are issued. 3 Vehicle application console output. 4 KNemo network monitor window. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 15. Motivation Our Results Summary Implementation Strategy Adaptation Important observations for RTSP server When available bandwidth decreased below the level required to transmit the video with current encoding parameters (resolution, frame rate, quality, etc.): Communication channel is saturated. Displayed frame rate is significantly reduced. No adaptation is occurred. “Old” frames might be also dropped which leads to lowered framerate compared to the original video. Concluion: Such behavior is unacceptable for real-time streaming required to control remote vehicle. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 16. Motivation Our Results Summary Implementation Strategy Adaptation Important observations for VETER infrastructure When available bandwidth decreased: Vehicle application detects changed networking condition. Reacts by reducing frame size (current implementation divides the original size by the power of two for each state). Keeps the same frame rate as original video. As a result, used bandwidth is below current limit and is not saturated. Conclusion: Quality of the video is reduced but frame rate remains the same, which is important for real-time remote control. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 17. Motivation Our Results Summary Implementation Strategy Adaptation Important observations for VETER infrastructure After preconfigured amount of time, attempts to return to the original video quality (frame size). If required bandwidth available, continues to increase frame size. Conclusion: Improved bandwidth usage and as a result best possible quality with constant frame rate is presented to the drivers. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 18. Motivation Our Results Summary Summary Conclusions: Currently there is no out of the box solution for adaptive real-time video streaming suited for IoT embedded systems. Using GStreamer and Ice it is possible to build such solution. To react on network QoS changes, frame size could be changed on the fly using videoscale or appropriate hardware accelerated elements like for example TIVidScale. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet
- 19. Appendix For Further Reading Source code and information availability I Web site: http://veterobot.org Main Veter-project web site. Blog: http://veter-project.blogspot.com Veter-project blog with regular updates about the project. GitHub: http://www.github.org/veter-team Complete source code and documentation repostiroy. veterobot.org, Andrey Nechypurenko, Maksym Parkachov Real-time data streaming and motion control over the Internet