![Memory Management
A video is a sequence of frames
A stream is a subsequence of the video with a fixed size
Implemented using the idea of a disruptor [Tho’11] :
using LMAX disruptors as streams
time
current frame
pointer
0
-1
-2
-3
…
8](https://image.slidesharecdn.com/2015-wscg-videomr-amapandreduceframeworkforreal-timevideoprocessing-220628124653-a3e471e3/85/VideoMR-A-Map-and-Reduce-Framework-for-Real-time-Video-Processing-11-320.jpg)
![– [Dean'04]
“Programs written in this functional style are automatically
parallelized and executed on a large cluster of
commodity machines.”
9
Map & Reduce](https://image.slidesharecdn.com/2015-wscg-videomr-amapandreduceframeworkforreal-timevideoprocessing-220628124653-a3e471e3/85/VideoMR-A-Map-and-Reduce-Framework-for-Real-time-Video-Processing-12-320.jpg)
![18
PAPER
[Dean'04]
Dean, J. & Ghemawat, S. MapReduce: Simplified Data
Processing on Large Clusters OSDI'04: Sixth Symposium on
Operating System Design and Implementation, 2004
[Tho'11]
Thompson, M.; Farley, D.; Barker, M.; Gee,
P. & Stewart, A. DISRUPTOR: High performance
alternative to bounded queues for exchanging data](https://image.slidesharecdn.com/2015-wscg-videomr-amapandreduceframeworkforreal-timevideoprocessing-220628124653-a3e471e3/85/VideoMR-A-Map-and-Reduce-Framework-for-Real-time-Video-Processing-45-320.jpg)
VideoMR - A Map and Reduce Framework for Real-time Video Processing
- 1. VideoMR: A Map and Reduce Framework for Real-time Video Processing Benjamin-Heinz Meier, Matthias Trapp, Jürgen Döllner Hasso Plattner Institute, University of Potsdam, Germany 1
- 2. Motivation For our research we were searching for a real-time video processing framework supporting us to develop GPU-based filter operation. 2
- 3. 3
- 7. GPU Program- ming Abstraction & Usability Video Processing GStreamer, Videotoolkit, Premiere, ... Plug- Ins Map & Reduce MARS, ... 4
- 9. VideoMR bounded memory concept, new definition of operators Map & Reduce transparent memory management using streams 6 GPU Program- ming Abstraction & Usability Video Processing
- 10. Agenda Memory management using LMAX disruptors as streams The concept of Map & Reduce A redefinition for video processing A small example Performance Evaluation Limitations & Improvements Conclusion 7
- 11. Memory Management A video is a sequence of frames A stream is a subsequence of the video with a fixed size Implemented using the idea of a disruptor [Tho’11] : using LMAX disruptors as streams time current frame pointer 0 -1 -2 -3 … 8
- 12. – [Dean'04] “Programs written in this functional style are automatically parallelized and executed on a large cluster of commodity machines.” 9 Map & Reduce
- 14. Map Operation map(key1,value1) → list(key2,value2) Map & Reduce concept 10
- 15. Map Operation map(key1,value1) → list(key2,value2) Map & Reduce concept for each word: if word is a noun: emit („noun“,1) else if word is a verb: emit („verb“,1) 10
- 16. Reduce Operation reduce(key2,list(value2)) → list(value2) Map Operation map(key1,value1) → list(key2,value2) Map & Reduce concept for each word: if word is a noun: emit („noun“,1) else if word is a verb: emit („verb“,1) 10
- 17. Reduce Operation reduce(key2,list(value2)) → list(value2) Map Operation map(key1,value1) → list(key2,value2) Map & Reduce concept for each word: if word is a noun: emit („noun“,1) else if word is a verb: emit („verb“,1) result = 0 for each value2: result += 1 emit („number of“+key2 +result) 10
- 18. Map & Reduce redefinition for video processing 11
- 19. Map & Reduce redefinition for video processing 11 Map map(streamin,list(streamside)) → streamout streamin
- 20. Map & Reduce redefinition for video processing 11 Map map(streamin,list(streamside)) → streamout streamin
- 21. Map & Reduce redefinition for video processing 11 Map map(streamin,list(streamside)) → streamout streamin Reduce reduce(streamin,list(streamside)) → streamout streamin
- 22. Map & Reduce redefinition for video processing 11 Map map(streamin,list(streamside)) → streamout streamin Reduce reduce(streamin,list(streamside)) → streamout streamin
- 23. Example 12
- 25. Example Program 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3);
- 26. Example Program 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // setup program and add the operation to it *prog << moveMap;
- 27. Example Program 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap;
- 28. Example Program 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run();
- 29. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run();
- 30. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run(); // get position of current pixel ivec2 pos = vmr_getPosition();
- 31. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run(); // get position of current pixel ivec2 pos = vmr_getPosition(); // get data from last three frames vec3 c; vec3 c1 = vmr_getStreamDataIn(pos, 0); vec3 c2 = vmr_getStreamDataIn(pos, -1); vec3 c3 = vmr_getStreamDataIn(pos, -2);
- 32. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run(); // get position of current pixel ivec2 pos = vmr_getPosition(); // get data from last three frames vec3 c; vec3 c1 = vmr_getStreamDataIn(pos, 0); vec3 c2 = vmr_getStreamDataIn(pos, -1); vec3 c3 = vmr_getStreamDataIn(pos, -2); // compute difference float diff1 = abs(c1.r-c2.r) + abs(c1.g-c2.g) + abs(c1.b-c2.b); float diff2 = abs(c1.r-c3.r) + abs(c1.g-c3.g) + abs(c1.b-c3.b);
- 33. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run(); // get position of current pixel ivec2 pos = vmr_getPosition(); // get data from last three frames vec3 c; vec3 c1 = vmr_getStreamDataIn(pos, 0); vec3 c2 = vmr_getStreamDataIn(pos, -1); vec3 c3 = vmr_getStreamDataIn(pos, -2); // compute difference float diff1 = abs(c1.r-c2.r) + abs(c1.g-c2.g) + abs(c1.b-c2.b); float diff2 = abs(c1.r-c3.r) + abs(c1.g-c3.g) + abs(c1.b-c3.b); // compare with threshold if (diff1>100 && diff2>100){ c = vec3(255, 255, 255); } else { c = vec3(0, 0, 0); }
- 34. Example Program Operation 12 // init program auto prog = std::make_shared <vmr::GlfwProgram >(); auto source = std::make_shared <vmr:: LibavLoader>("./example.mov"); source->init(3); // move detection auto move = std::make_shared<vmr::Display>(); auto moveMap = std::make_shared<vmr::Map> (source,move,“./move.map“); // setup program and add the operation to it *prog << moveMap; // run program prog->run(); // get position of current pixel ivec2 pos = vmr_getPosition(); // get data from last three frames vec3 c; vec3 c1 = vmr_getStreamDataIn(pos, 0); vec3 c2 = vmr_getStreamDataIn(pos, -1); vec3 c3 = vmr_getStreamDataIn(pos, -2); // compute difference float diff1 = abs(c1.r-c2.r) + abs(c1.g-c2.g) + abs(c1.b-c2.b); float diff2 = abs(c1.r-c3.r) + abs(c1.g-c3.g) + abs(c1.b-c3.b); // compare with threshold if (diff1>100 && diff2>100){ c = vec3(255, 255, 255); } else { c = vec3(0, 0, 0); } // write result to current position vmr_emitToStreamOut(pos,c);
- 35. Performance Evaluation 0 ms 10 ms 20 ms 30 ms 40 ms SD HD Full HD 4K map reduce complex 13 0 RLOC 125 RLOC 250 RLOC 375 RLOC 500 RLOC OpenGL VideoMR program operation
- 36. Improvements & Future Work 14
- 37. Improvements & Future Work 14 A general concept for n-dimensional buffers instead of frames
- 38. Improvements & Future Work 14 A general concept for n-dimensional buffers instead of frames Extension with template classes to decide which main data type to use
- 39. Improvements & Future Work 14 A general concept for n-dimensional buffers instead of frames Extension with template classes to decide which main data type to use Introduction of an explicit CPU operation with transparent memory management
- 40. Improvements & Future Work 14 A general concept for n-dimensional buffers instead of frames Extension with template classes to decide which main data type to use Introduction of an explicit CPU operation with transparent memory management Considering of OpenCL and CUDA as backend
- 41. Improvements & Future Work 14 A general concept for n-dimensional buffers instead of frames Extension with template classes to decide which main data type to use Introduction of an explicit CPU operation with transparent memory management Considering of OpenCL and CUDA as backend Comparison with other map and reduce frameworks will be part of future and therefore, a suitable benchmark has to be developed
- 42. Conclusion 15 GPU Program- ming Abstraction & Usability Video Processing VideoMR fulfills the requirements of real-time video processing and supports with the map and reduce concept the development of GPU-based filter operations.
- 43. Question & Comments 16 Contact: Benjamin-Heinz Meier/ benjamin-heinz.meier@student.hpi.de Matthias Trapp / matthias.trapp@hpi.de Jürgen Döllner / juergen.doellner@hpi.de Publications: www.4dndvis.de/publikationen.html This work was funded by the Federal Ministry of Education and Research (BMBF), Germany within the InnoProfile Transfer research group "4DnD-Vis".
- 44. VideoMR: A Map and Reduce Framework for Real-time Video Processing Benjamin-Heinz Meier, Matthias Trapp, Jürgen Döllner Hasso Plattner Institute, University of Potsdam, Germany 17
- 45. 18 PAPER [Dean'04] Dean, J. & Ghemawat, S. MapReduce: Simplified Data Processing on Large Clusters OSDI'04: Sixth Symposium on Operating System Design and Implementation, 2004 [Tho'11] Thompson, M.; Farley, D.; Barker, M.; Gee, P. & Stewart, A. DISRUPTOR: High performance alternative to bounded queues for exchanging data