2012 04-19 theory-of_operation
0 likes590 views
The document provides instructions for setting up the file structure and dependencies needed to run WebRTC on iOS. It involves creating a disk image, checking out various plugins and dependencies, and modifying files in specific WebRTC modules and third party libraries to add iOS compatibility and address iOS-specific issues like threading and input handling. Key changes include adding iOS-compatible audio and video capture, modifying tests to work without stdin, and addressing OpenGL rendering differences between iOS and other platforms.
2012 04-19 theory-of_operation
- 1. iOS Theory of Operation File structure setup Open “Disk Utility” and create a new disk image like so: Mount the image, it should appear as a disk called “GCI”
- 2. Check out the public webrtc plugin to the root as “webrtc_plugin”, check out the “dev” version as “webrtc_plugin_dev”, and “gocastmain” into root as well. “webrtc_plugin_ios” is a soft link to “gocastmain/projects/ios” Additionally, please put a copy of “depot_tools” in root as well. “cmptree.sh” is a script you can get at the link below, and the rest of the directories are less important. cmptree.sh: http://www.drdobbs.com/199103123 Doing the steps above is likely initially necessary as I’m sure there are hard- coded paths in the xcodeproject files. Also, some helpers, like yasm, won’t build under the iOS target and require that webrtc_plugin or webrtc_plugin_dev are built. I believe I used webrtc_plugin for everything.
- 3. Additional files So webrtc_plugin_ios is essentially webrtc_plugin with some changes. MacOS / BSD has a command called “ditto” that merges file trees together. I’d suggest doing the following then... 1. Duplicate webrtc_plugin as webrtc_plugin2 2. “ditto” “webrtc_plugin_ios” onto webrtc_plugin2 3. remove the original “webrtc_plugin_ios” 4. move webrtc_plugin2 into “webrtc_plugin_ios” I hadn’t tested this exact way of doing the setup, as I mentioned before really I wasn’t comfortable with writing a script to do all the overrides and merges like your group has done. This should be equivalent, for now.
- 4. Changes to src/examples/cmdline_audio_peer Everything here is properly quarantined with the macro MAC_IPHONE, so it should be safe to merge into the main source of gocastmain. • main.cpp – changes were made so that the “main” here isn’t the entry point on iOS. It’s now redefined as cmd_main and called as a function of one of the buttons in the iOS version • shell.cpp – an additional overloaded “parseline” method was added, so that you can call it directly with a string instead of just needing a stream. • TestClientShell.cpp – iOS does not respect cin or scanf, so this has been quarantined Some merging may be required here as it appears webrtc_plugin and the iOS versions are slightly out of sync. (Missing a few changes you guys have made.) Should be minor.
- 5. Changes to src/examples/cmdline_audio_peer/cmdline_audio_peer_ios • This is the closest thing to the “plugin” right now. It can actually connect with the plugin back when the plugin was version 1.11, your mileage may vary. • AppDelegate.h / mm – ignore • main.m – ignore • CLAPViewController – the ViewController for the project. The file name stands for “Command Line Audio Peer” and was renamed as there are bogus code completions generated by Xcode if you leave it named ViewController. This is because ios test apps also have a “ViewController” file name and class. • FrameBuffer – Internal class to support OGLView. I didn't write it, I took it from example code • OGLView – Class that implements a movable, non-full-screen, OpenGL “view”. This can be moved around or placed in Xcode's XIB editor, and is useful for that. From the same example code FrameBuffer's from. • GLError – helper macros / headers, etc. - From same example as FrameBuffer.
- 6. Changes to src/js_api/projects/Webrtcplugin/ • WebRTCPlugin.cpp – ignore, this is not used on iOS • WPLCall.cpp – gLocalStream is the local video stream. As there didn’t appear to be a clean way to include this, this was done as a global and where appropriate, set and unset in this file. • WPLPeerConnectionClient.cpp – “refreshpeerlist” is a method that’s called in order to update the on-screen peer list for the iOS client. Since in the command-line version on mac / linux you request the peer list via command only and are never notified, this is necessary as we don’t want to poll the command list every x seconds or something ridiculous like that. The WebRTC sample peerconnection_client posts messages so I had expected you might as well, but that wasn’t the case so this essentially does that. • WPLPeerConnectionObserver.cpp – gRemoteStream is the remote video stream, defined here. Like I had mentioned last week, currently we only support two streams as it was unclear to me how you wished for this particular subproject to proceed. • WPLVideoRenderer.cpp – This is based off of Manjesh’s Mac WPLVideoRenderer and has the same structure, except we use OpenGL ES 1.0 here. Setup for the drawing surface is defined here, and the major thing that happens is that any time you get a frame, a texture is either generated (first time) or regenerated. Textures in OpenGL are the bitmaps we blit to screen, and each has an ID. The ID is typically 1 for the first, 2 for the second, etc, however it can vary and so we keep a handle for this. (Which is the correct way to do things.) Since like I mentioned we need to perform all rendering on a single thread, every OpenGL-related call gets “performSelectorOnMainThread” treatment, which means there’s an ugly helper class that’s required. Some merging may be required here as it appears webrtc_plugin and the iOS versions are slightly out of sync. (Missing a few changes you guys have made.) Should be minor.
- 7. Changes to modules/audio_device • audio_device_impl.cc – Possibly add platform-specific reporting for iOS • mac/audio_device_iphone.cc – Gustavo(?)'s version of the audio driver, with some slight changes to make it correctly work with WebRTC @ 1080 • audio_device_utility_iphone.cc – Gustavo's implementation, which I had already done, essentially just some dummy functions that don't need to truly be implemented. • audio_mixer_manager_mac.h – minor changes to get compilation working for iOS
- 8. Changes to modules/video_capture • device_info_impl.cc – very minor change to add iOS support-- essentially just choosing one strncmp function over another. More likely the test should be “if not Win32” rather than “if case and case and case etc etc” • mac/video_capture_iphone.mm / h • This is based off of the Mac version, except it operates a little differently and cleaner. • You have a C++ VideoCaptureImpl class that's required, • VideoCaptureAVCaptureInfo which does some heavy lifting for the Impl class in the C++ side (reports “Info”, properties of the device, etc.) • VideoCaptureAVCapture C++ class that does some heavy lifting on the C++ side that's not handled by the Info class (actually controls the capture hardware directly.) • Objective C classes iOSCaptureClass (which does the capture on the Obj-C side), iOSCaptureInfoClass (which does property reporting on the Obj-C side) and VideoCaptureRecursiveLock which is a thread lock (same used by the Mac class.)
- 9. Changes to modules/video_render • Nothing in this source tree is complete, it's only used by vie_auto_test, and honestly is way to complicated to implement quickly for a new platform. Also it has no benefit to future code that I can see. • However if this situation changes, this is a starting point. The code is almost functional, but handles renderstreams and OpenGL incorrectly.
- 10. Changes to video_engine • test/auto_test/interface/vie_autotest_ios_cocoa_touch.h – Defines a “Window Manager” class for vie_auto_test. More fallout from modules/video_render • Again most of the code here is nearly functional for iOS, so if the situation changes, this can be used as a starting point for iOS. Some things done include skipping cin/scanf for input and adding macros for MAC_IPHONE. • vie_defines.h – One of the places where default resolution 352x288 is set.
- 11. Changes to voice_engine • This is where the two main test apps we've been running live • voe_auto_test_ios – An automated test suite • voe_cmd_test_ios – Equivalent to the cmdline test, only using buttons • main/test/auto_test/voe_standard_test.cc – starting with GetDocumentsDir, defines where to get file resources used in this app's testing suite. Also determines where we can write output files • Tests that fail (but don't matter) are commented out. • Overrides for cin/scanf since there's no terminal input • Adds parameters to some methods so that we can fake-call them with “command line” options • main/test/auto_test/voe_stress_test.cc – comments out the section for cin/scanf, since we can't do command line input for the test menu • voe_cmd_test.cc – Similar to voe_standard_test.cc, defines file methods, overloads methods to allow command line options, and overrides cin/scanf input • ViewController.mm in both voe_cmd_test_ios and voe_auto_test_ios define how interactions with UI (buttons, .xib, etc) work to call methods in the tests they belong to. Ignore AppDelegate.m/h and main.m • voe_cmd_test_ios is the one I demo'd in the YouTube video back in the day.
- 12. Changes to gtest • gtest-port.h – modify macro GTEST_HAS_DEATH_TEST to be 0. It's 1 by default and it messes up the testing interface for voe_*_test if left on.
- 13. Changes to third_party/libjingle • source/talk/session/phone/devicemanager.cc – Implemented GetVideoDevices, which is incredibly important for video capture to work. I discovered similar code for (I believe Android) in the official libjingle release, and thus back-ported a work-alike for iPhone here. • DeviceWatcher also needs to be available for iOS • source/talk/session/phone/webrtcvoiceengine.cc - “adm_sc_” is an additional instance of the audio device, which if available, will not work with Gustavo's audio driver. Gustavo's audio driver isn't friendly with multiple instances of itself, and will cause both instances to fail if this is enabled. Best I can tell this audio device is meant for “sound effects” (non-voice), and is not used anywhere from within WebRTC except for the test suites (which is why some of them are commented out) •
- 14. Changes to third_party/libjpegturbo • This was modified somehow to generate non-x86 code; the only hit I get on this is changes to the xcodeproj file. It's been too long so I can't remember if there's anything else, but otherwise we tend to compile this as “common” code.
- 15. Changes to third_party/libvpx • generate_gypi.sh – Modified to generate a generic source list for non-x86 code • libvpx.gyp – Modified to use the generated source list from the bash file above • libvpx_srcs_generic.gypi – The generated list (checked in for convenience.) • libvpx.xcodeproj – The final project generated from the gyp file • source/config/mac/ia32/vpx_config.cc / h – Even though this says “auto-generated”, I'm pretty sure I hand-tweaked this. Necessary to do so you're not generating assembly optimized code. • third_party/googletest/src/include/gtest/internal/gtest-port.h – Not a typo, this really exists and needs to be modified for the same “death test” reason as the previous gtest