Android Audio & OpenSL

Android Audio with OpenSL Yossi Cohen DSP-IP

Overview Android Audio APIs OpenSL Overview OpenSL Android implementation (code)

Playing Audio on Android MediaPlayer SoundPool AudioTrack / AudioRecord (OpenMAX) OpenSL

MediaPlayer Java Based, Rich API Good for playing files and streams Not Good for gaming sounds or audio conferencing duo to very large delay

OpenMAX OpenMAX, a Khronos multimedia integration API exposes AudioTrack() and AudioRecord() Those functions enables the creation of “channels” which play or capture audio Those API enables audio effects like reverb or distortion This API also has a significant delay

OpenSL OpenSL is a Khronos Open Sound Library OpenSL ES is OpenSL for embedded systems Android 2.3 provides PARTIAL implementation of the OpenSL ES More on the limitations of And More on the OpenSL ES functions supported by Android in this Article

OpenSL Advantages Enables Low level audio tweaking and control Device independent between android phone (Android 2.3+) Disadvantages Doesn’t improve the long audio delay of android phones

OpenSL Features OpenSL supports the following features Android OpenSL Implementation does not support most of those features Doppler MIDI messages Preset Reverb Equalizer Pitch Volume Buffer Queues Rate Metadata extraction Environmental Reverb Virtualization Stereo widening LED & Vibra 3D positioning

Android and OpenSL Open SL OpenSL ES Android OpenSL ES

OpenSL, OpenMAX and Android Base on Android logging we noticed that OpenSL calls ALSA and OpenMAX API functions Applications Media silicon (audio HW, CPUs, DSPs) OpenMax (AudioTrack / AudioRecord) ALSA

Profiles OpenSL has three profiles: Phone Music Game Phone Basic mobile phones Ring tone and alert tone playback (basic MIDI functionality), basic audio playback and record functionality, simple 2D audio games Game-centric mobile devices Advanced MIDI functionality, sophisticated audio capabilities such as 3D audio, audio effects, ability to handle buffers of audio, etc. Music-centric mobile devices High quality audio, ability to support multiple music audio codecs, audio streaming support Game Music

OpenSL API and Architecture OpenSL is a C API which implements a COM like API Each object must implements the SLObjectItf Interface (like Iunknown in COM) OpenSL is NOT a discrete component architecture like Directshow or OpenMax IL OpenSL is not a channel like architecture like OpenMAX AL

OpenSL Objects After creation, object should be realized in order to use them.

OpenSL Architecture OpenSL uses a few objects Engine – a singleton object which must be implemented so we could use OpenSL AudioPlayer – enables audio playing OutputMixer – enable mixing and audio effects (such as reverb) AudioRecorder – enable audio capturing from microphone. Since Android does NOT support OpenSL microphone enumeration & control functions so AudioRecorder is the only way to capture audio in OpenSL

Android OpenSL Sample To use the sample, you should install the Android NDK. Download Android NDK from http://developer.android.com/sdk/ndk/index.html no need to install just unzip The samples is under: Android-ndk-r5b/Samples/native-audio

Editing the project Select “Create new project” in eclipse select “android project” and follow the settings according to the screen capture:

Native Audio Overview Native Audio is a very simple (too simple) OpenSL sample. It enables Half duplex utilization of the OpenSL, this enables either capturing a SINGLE 5 second buffer or playing of a single buffer. What is missing? Most voice applications will require Full Duplex behevior. The best sample for that is to create a loopback from Microphone to speakers.

Loopback sample For RT loopback behavior we need to add: Small buffers instead of 5 seconds buffers A two threaded application one for capture and one for record Since this presentation is not about how to build native application on Android we will not create such app from scratch but use an existing one, HelloJNI Import it from samples as we did for Native-Audio

Building an OpenSL application Add the following include files: #include <SLES/OpenSLES.h> #include <SLES/OpenSLES_Android.h> Add to the makefile LOCAL_LDLIBS += -lOpenSLES

Add Audio Premissions Add the following user permissions for Audio capture and play in the Manifest permission tab: Internet is required for sending / receiving the audio packets

Application thread We will implement the recorder in the application thread. Recorder is create from the objects in the diagram, each object is created() and Realized()

Code Example – Create Engine // create engine __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "initEngine"); SLEngineOption EngineOption[] = {(SLuint32) SL_ENGINEOPTION_THREADSAFE, (SLuint32) SL_BOOLEAN_TRUE}; result = slCreateEngine( &engineObject, 1, EngineOption, 0, NULL, NULL); __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "create engine result %d", result); assert(SL_RESULT_SUCCESS == result); // realize the engine result = (*engineObject)->Realize(engineObject, SL_BOOLEAN_FALSE); assert(SL_RESULT_SUCCESS == result); // get the engine interface, which is needed in order to create other objects result = (*engineObject)->GetInterface(engineObject, SL_IID_ENGINE, &engineEngine); assert(SL_RESULT_SUCCESS == result); __android_log_print(ANDROID_LOG_INFO, "NativeAudio", "get engine result %d", result);

Play Thread creation result = pthread_create( &audioThread, NULL, AudioInOutThread, NULL); After creating the recorder like in the native-Audio sample we will create the player thread void *AudioInOutThread( void *ptr ) { SLresult result; initPlayer(); // set the player's state to playing result = (*bqPlayerPlay)->SetPlayState(bqPlayerPlay, SL_PLAYSTATE_PLAYING); assert(SL_RESULT_SUCCESS == result); }

Player CallBack Function void bqPlayerCallback(SLAndroidSimpleBufferQueueItf bq, void *context) { int playershift = playFrame*RECORDER_FRAME; SLresult result; // enqueue another buffer result = (*bqPlayerBufferQueue)->Enqueue(bqPlayerBufferQueue, &recorderBuffer[playershift++], RECORDER_FRAME*sizeof(short)); } We will enqueue the next sample in the callback (need to verify there is a next sample) Recorder Enqueue is the same but also preform the first enqueue for the player to start it

Player Thread The player implements the following objects

Resources WiseAndroid Blog Mobile Pearls Blog MindTheRobot Blog

DSP-IP Contact information Download slides at: www.dsp-ip.com For course & Development consulting contact www.dsp-ip.com Mail : [email_address] Phone: +972-545-313092 Fax : +972-50- 8962910 Yossi Cohen [email_address] +972-545-313092

Android Audio & OpenSL

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to Android Audio & OpenSL (20)

More from Yoss Cohen (20)

Recently uploaded (20)

Android Audio & OpenSL