SlideShare a Scribd company logo

Delivering Great WebRTC on Mobile Devices

Weemo, Inc., profile picture
Weemo, Inc.

The document discusses the integration, challenges, and solutions for implementing WebRTC on mobile devices, highlighting the evolution of real-time communication tools, device and network-related challenges, and various application integration options. It emphasizes key performance metrics including battery life, latency, and adaptability to varying network conditions, while also addressing interoperability issues with legacy systems. The latter part of the document details development considerations for mobile applications using WebRTC and provides different implementation strategies across platforms like iOS and Android.

TechnologyBusiness
Related topics:
Mobile App Development
1 of 64
Downloaded 52 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
Delivering Great WebRTC on Mobile Devices
Delivering  Great  WebRTC  on   Mobile  Devices   Amir  Zmora  
Workshop  Presenters   •  Saraj  Mudigonda   – ImaginaBon  Technologies     •  Thomas  Sheffler     – Weemo    
WebRTC  on  Mobile  -­‐     IntégraBon,  Challenges  &  SoluBons   Saraj  Mudigonda   ImaginaBon  Technologies   Saraj.mudigonda@imgtec.com  
Agenda   •  Background   •  IntegraBon  OpBons   •  V.VoIP  Building  Blocks     •  V.VoIP  Mobile  ApplicaBons   •  Challenges  &  SoluBons   •  Summary  
Background   Pre-WebRTC § Limited tools for developers § Real-time voice/video conferencing algorithms were limited to only few proprietary native and enterprise applications § Phone applications and web applications were two independent identities PC Modem Mobile IP Phone § Abundant Tools/APIs available for developers § Easy access to complex real-time voice/video algorithms § Enabling real-time communications on numerous platforms with different operating systems, sizes, functions and on browsers with no-plugins …. But it does not come free, it brings along multiple challenges Post-WebRTC
IntegraBon  OpBons   Over-­‐The-­‐Top   ApplicaBons   Pre-­‐Loaded   ApplicaBons   Embedded   ApplicaBons   NATIVE  APPLICATIONS   Web  Browser-­‐ Based  ApplicaBons   •  Best-­‐effort  services  offered   by  non-­‐operator   •  Can  be  downloaded  from   Apple  Store/Google  Play   •  Bundled  with  the  plaTorm   •  Cannot  be  deleted  or   disabled   •  Operator  driven   •  Deeply  integrated   applicaBons   •  Part  of  the  NaBve  Dialer    
V.VoIP  Building  Blocks   V.VoIP  Applica4on   WebRTC Media Engine Audio  Coding       Module   Video   Conferencing   Module   Voice/Video     QoS     Audio  Pre-­‐Processing  Module  –     AEC,  NC,  AGC     ICE STUN TURN Security TLS SRTP DTLS Audio/Video  Capture/Rendering   Signaling & User Interface Network  Interface  
Mobile  V.VoIP  ApplicaBons   SIP  Signaling   Voice  +  Video  App   SIP  +  TradiBonal   Voice  +  Video  Codecs   Enterprise  Telephony    IR.92/IR.94  +  AMR     Video  +Voice  over  LTE   RCS  5.0/5.1   Rich  CommunicaBon  Suite  
Challenges   I.  Device  Related   1.  Ba]ery  Life   2.  Latency  Variability   3.  PlaTorm  Diversity     II.  Network  Related   4.  Bandwidth  Variability   5.  Latency  Variability   6.  Packet  Losses     III.  Ecosystem  Related   7.  Network  Handoff   8.  Interoperability  
1.  Ba]ery  Life   •  Challenge   –  Improved  ba]ery  life  for  mobile  video/voice  conferencing     •  PotenBal  SoluBons   –  OpBmize  for  target  mobile  applicaBon  processors   •  MIPS  intensive  algorithms  such  as  echo  cancellaBon  and  codecs  should  be  opBmized  for  the  target   processors  thus  reducing  the  overall  mw/MHz  required   –  Leverage  hardware  accelerators  wherever  available   •  Not  all  plaTorms  expose  APIs  for  hardware  integraBon     –  Diminishing  returns  for  frame  rate   •  Significant  difference  in  perceivable  video  quality  between  10fps  and    15fps,  but  not  so  much  between   15fps  and  30fps  to  jusBfy  a  2x  increase  in  ba]ery  consumpBon   –  Principle  of  good  enough   •  ResoluBon  based  on  form  factor  of  the  devices.  No  need  for  HD  video  telephony  burning  up  CPU  and   your  hands  on  a  4  inch  screen.   •  If  user  really  wants  highest  possible  quality,  give  him  a  sehng  for  it  with  an  appropriate  warning  J  
0   10   20   30   40   50   60   70   80   90   100   Voice  NB  Call   Voice  WB  Call   Voice  +  SW  Video   Voice  +  HW  Video   C P U   U S A G E   %     1.1  Example  CPU  Usage  %   HD  Video   VGA  30fps  on  Single-­‐Core  CPU   Hardware  accelerators  not   only  offloads  CPU  and  reduces   power  consumpBon  but  also   enables  HD  resoluBon  Video    
2.  Latency  Variability   Propaga4on  +  Queuing  Delay  TX  Processing  Delay   RX  Processing  Delay   Network   Interface   Delay   Algorithmic   Delay  –   Encode/AEC   Audio/Video   Capture  Delay   Network   Interface   Delay   Algorithmic   Delay  –   Decode/AJB   Audio/Video   Rendering     Delay  
2.1  Latency  Variability   •  Challenge   –  Reduced  latency  for  superior  user  experience  and  to  avoid   talking  over  each  other   •  PotenBal  SoluBons   –  Reduced  algorithmic  delay  for  voice/video  encode/decode   and  DSP  algorithms  such  as  echo  cancellaBon   –  OpBmized  integraBon  with  audio,  video  and  network   interfaces   –  Dedicated  bearer  for  real-­‐Bme  applicaBons  -­‐  voice  and  video   conferencing  
2.2  Example  Latency  Measurements   0   50   100   150   200   250   300   350   400   1   2   3   Audio  Interface  Delay   Network  Interface  Delay   Algorithmic  Delay   Handset#   Handset#   Handset#   •  Audio  and  network  interface     delay  is  very  device  specific  and   contributes  to  major  chunk  of   the  latency  on  the  client   •  Algorithmic  delay  is  dependent   on  the  opBmizaBon  of  the  voice   processing  algorithms  
3.  PlaTorm  Diversity   •  Challenge   –  Maintain  same  user-­‐experience  across  different  plaTorms   •  PotenBal  SoluBons   –  Adapt  to  different  form-­‐factors   –  Auto  tuning  of  AcousBc  Echo  CancellaBon  (AEC)  across  the  devices.   AEC  characterisBcs  are  heavily  dependent  on  the  plaTorm   industrial  design  (mic  and  speaker  locaBon)   –  OpBmizaBon  of  the  DSP  algorithms  for  target  processor   –  OpBmizaBon/integraBon  with  the  plaTorm  audio  and  network   interfaces  
3.1  Example  Audio  CharacterisBcs     Mobile  Device     Flat  Delay   (msec)     ERL     Non-­‐Linearity     Jiaer  (msec)     Record  Event   Play  Event   Handset#1   330   -­‐3dB   No  non-­‐linearity   15   8   Handset#2   250   -­‐4dB   Non-­‐Linear   20   5   Handset#3   225   -­‐40dB   Highly  Non-­‐Linear   20   5   Legend:   •  Flat  Delay  -­‐    Delay  between  the  speech  sample  played  ay  the  APP  (sent  to  audio  interface  for  play  back)  to  the  Bme  the  echo  of  the  sample   is  received  by  the  APP  (data  capture  provided  by  the  audio  interface).       •  Non-­‐Linearity  -­‐  Because  of  the  post/pre-­‐processing  components  used  in  the  audio  path,  the  enBre  frequency  range  of  the  played  and   echoed  speech  is  not  modified  uniformly  and  this  nonlinear  distorBon  impacts  the  echo  cancellaBon       •  ERL  -­‐  Echo  return  loss  -­‐  specifies  the  loss/gain  encountered  by  the  signal  in  the  acousBc  environment       •   Ji]er  -­‐  variaBons  in  the  data  capture/play  out  intervals  
4.  Bandwidth  Variability   •  Challenge   –  Adapt  quickly  and  efficiently  to  varying  bandwidth  condiBons   •  PotenBal  SoluBons   –  Video  AdaptaBon  Parameters   •  Frames  Per  Second   •  Video  ResoluBon     •  Bit  Rates   –  Voice  AdaptaBon  Parameters   •  Data  Rate  AdaptaBon   •  Codec  Change   –  Diminishing  returns  for  bandwidth   •  Marginal  to  no  improvement  in  quality  beyond  a  certain  bandwidth  for  a  given  resoluBon  and  frame  rate   •  Don't  use  up  available  bandwidth  if  it  is  not  required.  A  judicious  choice  is  a  must  in  order  to  eliminate  blockiness   –  Bandwidth  control   •  Maintaining  a  given  bandwidth  and  the  right  split  of  codec  bitrate  and  FEC  in  the  presence  of  variable  loss   •  A  constant  bitrate  configuraBon  of  the  codec  is  essenBal.  Variable  bandwidth  can  play  havoc  with  pacing/  bandwidth   adaptaBon      
4.1  Example  Bandwidth  AdaptaBon   0   10000   20000   30000   40000   50000   60000   70000   80000   1   1110   2219   3328   4437   5546   6655   7764   8873   9982   11091   12200   13309   14418   15527   16636   17745   18854   19963   21072   22181   23290   24399   25508   26617   27726   28835   29944   31053   32162   33271   34380   35489   36598   37707   38816   39925   41034   42143   43252   44361   45470   46579   47688   48797   49906   51015   52124   53233   54342   55451   56560   57669   58778   59887   60996   Represents  Allo]ed  Bandwidth  in  bps  
6.  Packet  Losses   •  Challenge   –  MiBgate  packet  loss  to  provide  a  smooth  voice  and  video   experience   •  packet  loss  is  very  common  in  wireless  environments  parBcularly  on  Wi-­‐Fi   •  MiBgaBon  Approach   –  AdapBve  ji]er  buffer  and  packet  loss  concealment    to  miBgate   packet  losses   –  A  suitable  FEC  strategy  to  combat  loss  and  maintain  user   experience  with  minimal  latency   –  Balancing  FEC  protecBon,  quality,  latency  especially  at  low   bandwidths  
6.1  Example  Packet  Loss  &  MiBgaBon   •  Packet  loss  miBgaBon  algorithm   should  help  maintain  good   voice  call  quality  for  random   and  small  bursts   0   0.5   1   1.5   2   2.5   3   3.5   4   4.5   0%   3%   5%   10%   20%   30%   40%   50%   Voice  Quality  (PESQ)   Packet  Error  Rate  (PER)   App#1   App#2   App#3   Dropped  Call   Good  Voice  Quality  
7.  Network  Handoff   •  Challenge   –  Provide  seamless  call  handoff   •  MiBgate  dropped  calls  when  moving  from  one  access  network  to  the  other   •  PotenBal  SoluBons   –  Implement  make  before  break/call  anchoring  algorithms  that  can   seamlessly  handoff  when  moving  from  one  network  to  the  other   –  Some  of  the  3GPP  standards  support  VCC  (Voice  Call  ConBnuity)  for   seamless  handoff  between  circuit-­‐switched  networks  and  packet   networks  
7.1  Typical  Network  Scenario   Dropped  calls   when  changing   networks   Home   On-­‐the-­‐road     Enterprise   3G
8.  Interoperability   •  Challenge   – Interoperability  with  legacy  systems   •  PotenBal  SoluBons   – Because  of  the  open-­‐source  nature  of  WebRTC  it  is   possible  to  integrate  addiBonal  codecs  and  algorithms  to   WebRTC  codebase  for  interoperability   – Transcoding  might  be  required  in  the  infrastructure  to   support  some  legacy  systems  
Summary   •  PPA  (Performance,  Power,  AdaptaBon)  –  the  key  metrics  for  opBmal  voice/ video  experience  on  mobile  devices   •  Performance   –  For  improved  performance  all  the  parameters  discussed  have  to  be  tuned   carefully  for  device  and  networks   •  Power   –  OpBmize  for  target  apps  processor,  leverage  hardware  accelerators  wherever   available     –  Find  the  sweet  spot  -­‐  adopt  principle  of  good  enough  and  diminishing  returns  for   frame  rate         •  AdaptaBon   –  Adapt  quickly  and  efficiently  to  varying  bandwidth  condiBons,  network  packet  losses   and  form-­‐factors  
WebRTC  on  Mobile  -­‐     IntégraBon,  Challenges  &  SoluBons   Saraj  Mudigonda   ImaginaBon  Technologies   Saraj.mudigonda@imgtec.com  
Developing  Mobile  ApplicaBons  using   WebRTC   Thomas  Sheffler     Weemo      
Outline   •  A  ConstellaBon  of  OpBons   – browser  versus  naBve  Apps   •  Mobile  ConsideraBons   – WebRTC  is  hard.    Mobile  is  harder.   •  Case  study  across  plaTorms  using  an  SDK   – iOS,  Android,  Javascript  
Browser  vs  NaBve   •  HTML5+Javascript+WebRTC   – the  promise:  write  once,  run  everywhere!   – Chrome  leads,  Firefox  and  Opera  closely  follow   •  NaBve  App  on  iOS  and  Android   – deliver  a  naBve  app  experience   – what  does  WebRTC  interoperability  mean?  
WebRTC  PlaTorm  OpBons   •  Android   –  Browser:  Chrome,  Firefox,  Opera  (Mar  2014)   •  Note:  WebRTC  is  not  in  the  Android  WebView   –  NaBve:  SDKs   •  Android  device  market  is  incredibly  fragmented   •  iOS   –  Browser:  really  none   –  NaBve:  SDKs   •  iOS  plaTorm  is  relaBvely  well  managed   •  80%  iOS7  (ArsTechnica)   NaBve  App   WebView  
OpBons  Summary   Android   iOS   Browser   Chrome,  Firefox,  Opera   n/a   Webview   n/a   n/a   NaBve   Yes*   Yes*   *use  a  WebRTC  compaBble  SDK  
Browser  Interoperability   h]p://iswebrtcreadyyet.com/  
What  is  WebRTC?   •  A  Suite  of  Technologies  standardized  by  W3C   –  Codecs:  VP8,  Opus   –  Transport:  RTP,  encrypBon   –  SpecificaBons  for  describing  endpoints  (SDP)   –  Methods  for  endpoints  to  connect  (ICE,  STUN,  TURN)   –  Javascript  bindings  for   •  accessing  your  camera,  microphone   •  obtaining  SDP  entry  of  yourself   •  rendering  media  streams  described  by  an  SDP  entry  
What  WebRTC  is  not   •  A  way  to  idenBfy  users   –  authenBcaBon   •  A  way  to  locate  other  users   –  phonebook,  user  database   •  A  way  to  iniBate  calls  to  other  users   –  signaling   •  A  way  to  see  the  online  status  of  other  users   –  presence  
What  is  WebRTC-­‐compaBble?   •  NaBve  Apps  may  be  compaBble  if   –  they  can  interoperate  with  a  WebRTC  browser   •  NaBve  Interoperability   –  VP8,  Opus   –  RTP,  encrypBon   –  SDP,  ICE,  STUN,  TURN   •  May  or  may  not  present  bindings  similar  to  Javascript  
Compiling  WebRTC  yourself   •  Ref:   h]p://ninjaneBc.com/how-­‐to-­‐get-­‐started-­‐with-­‐webrtc-­‐ and-­‐ios-­‐without-­‐wasBng-­‐10-­‐hours-­‐of-­‐your-­‐life/   •  “The  current  HEAD  has  audio  and  video”  [May  2014]   •  Demo  App  uses  ICE  servers  apprtc.appspot.com   •  My  summary:  this  is  really  hard  
Back-­‐end  technolgies   •  Website  versus  Mobile  Backend   –  Website  development  across  plaTorms  (desktop,  iOS,  Android)  is   understood  (Rails,  Django,  Express,  etc)   –  Mobile  App  backend  development  may  be  unfamiliar   •  Parse   •  Why  do  you  need  one?   –  IdenBfy  users  (authenBcaBon,  roster)   –  Keep  your  API  keys  secret   –  Protect  access  to  your  realBme  resources   •  TURN  relays  cost  real  $$  
When  Worlds  Collide   •  If  you’re  coming  from  the  telephony  world   –  this  is  not  SIP,  it’s  not  a  so•phone   –  webservices  bring  their  own  problems   –  security,  cross-­‐site  request  forging,  Oauth2,  Javascript  code  is   “in  the  clear”,  mobile  authenBcaBon   •  If  you’re  coming  from  Webservices   –  RTC  is  not  request/response,  it’s  not  ReST   –  get  up  to  speed  on  Offer/Accept,  SDP,  RTP,  ICE  candidates  
IdenBfy  your  Use  Case   •  Is  two-­‐way  video  the  applicaBon?     •  Is  it  FaceBme?   •  Is  it  customer  assistance?   –  video  is  a  feature,  not  the  focus   –  does  it  get  out  of  the  way   –  one-­‐way  video?   •  Is  it  a  mulBway  call?  (meeBng)   •  The  way  screen  real  estate  and  compute  resources  is  used  is  VERY   different  from  the  desktop.  
These  are  very  different   • WVGA  or  HD   • 250  kbps  –  1  mbps   • Video  IS  the  focus   • 240x180  might  be  enough   • 100  kbps   • Video  floats  while  App  conBnues  
Is  Tablet  the  same  as  Mobile?   •  Tablets  do  not  roam  so  o•en:  more  tethered  to  a   basestaBon   •  May  not  have  a  cellular  network   •  Screensize?    Camera?   – Samsung  Galaxy  Tab  2  10”  front  camera  is  0.3MP   •  Ba]ery  life?  CPU?   •  Shaky  camera  effect  
ICE  in  One  Minute  
ICE  Candidates   RT   TURN   UDP  
ICE  Candidates   RT   STUN   UDP   UDP  
ICE  Candidates   RT   UDP  
Cellular  network  consideraBons   •  The  cellular  network  does  not  behave  like  a  consumer   home  network   –  do  not  expect  peer-­‐to-­‐peer  to  work  on  4G   •  Handoffs  between  cells  affects  IP  addresses   –  sudden  changes  in  network  connecBvity  [RFC5944:  mobilility]   •  QuesBon:  how  does  your  WebRTC  applicaBon  behave   in  the  context  of  mobile  networks?  
Handoffs  from  WiFi  to  Cellular   •  Mobile  apps  like  FaceBme  set  consumer   expectaBons   •  WebRTC  technologies  do  not  really  address   changing  network  topologies  
App  Development   •  How  is  an  App  different  from  a  Desktop?   –  Apps  are  sandboxed:  a  restricted  execuBon  environment   –  no  shared  memory  or  libraries   •  Screensharing   –  Mobile  environments  sandbox  apps  for  security   –  Apps  can  sample  their  own  pixels,  not  the  pixels  of  other  apps  
Power  State  TransiBons   •  Foreground/Background   – Can  a  user  depart  the  video  engagement  to  go  look  at   something  else?   – Can  the  App  or  Page  receive  noBficaBons  while  in  the   background?   •  Browser-­‐based  implementaBon   – background  window  is  not  acBve  
Use  WebRTC-­‐compaBble  SDK   •  Abstracts  WebRTC  endpoints.   •  AuthenBcates  users.    Presents  calls.   •  Built  on  WebRTC  technologies   – VP8,  Opus   – RTP,  encrypBon   – SDP,  ICE,  STUN,  TURN  
Examples  using  SDK   •  iOS,  Android,  Javascript   – iniBalize  package   – authenBcate   – make  and  receive  calls   – connect  to  UI  elements  
iOS  
IniBalize  the  SDK   ! static NSString *APP_ID = @"3b04d9add47c”;! ! @implementation AppDelegate! - (BOOL)application:(UIApplication *)application 
 didFinishLaunchingWithOptions:(NSDictionary *)launchOptions! {! ViewController* mainController = ! (ViewController*) self.window.rootViewController;! ! NSError *e;! Weemo *w = [Weemo WeemoWithAppID:APP_ID andDelegate:mainController error:&e];! return YES;! }! !
Mobile  AuthenBcaBon   Your   ApplicaBon   Auth   Controller   Web   Service   Cloud-­‐based   Video  Service   Web   Service   SDK   Video   Service   token   token   request  token   (proxy)   (uid,client,profile)   token   Auth   token  
AuthenBcate   static NSString *AUTH_HOST = @"http://192.168.1.233:8000";! ! - (IBAction)login:(id)sender {! ! dispatch_async(dispatch_get_main_queue(), ^{! self.auth = [[WeemoAuthSimpleClient alloc] init];! [self.auth authWithHost:AUTH_HOST uid:@"testuser1" success:^(NSString *token) {! ! [[Weemo instance] authenticateWithToken:token andType:USERTYPE_INTERNAL];! ! } failure:^(NSError *e) {! NSLog(@"Login failure:%@", e);! }];! });! }! !
Call,  Hangup   - (IBAction)call_button:(id)sender {! NSLog(@"call button");! [[Weemo instance] createCall:otherUsername];! }! ! - (IBAction)hangup_button:(id)sender {! NSLog(@"hangup button");! ! WeemoCall *call = [[Weemo instance] activeCall];! if (call) {! [call hangup];! }! }! !
Connect  call  to  UI   #pragma mark Weemo Delegate! ! - (void) weemoCallCreated:(WeemoCall *)call {! NSLog(@"weemoCallCreated:%@", call);! ! [call setViewVideoIn:self.v_videoIn];! [call setViewVideoOut:self.v_videoOut];! ! call.delegate = self;! ! }! UIView   v_videoIn   UIView   v_videoOut  
iOS  Summary   •  We  learned  how  to   – iniBalize  package   – authenBcate   – make  and  receive  calls   – connect  to  UI  elements  
Android   •  What  is  different?   – event  bus  instead  of  delegates   – decorators  to  register  event  listeners  
EventBus   •  Based  on  a  common  component  in  Google  Guava   •  Our  lightweight  implementaBon   •  Decouples  event  responsibility  from  class  hierarchy  
IniBalizaBon   public class FullscreenActivity extends Activity {! @Override! public void onStart() {! super.onStart();! Weemo.eventBus().register(this);! Weemo.initialize("APIKEY", this);! }! ! @WeemoEventListener! public void onConnected(final ConnectedEvent event) { ... }! ! @WeemoEventListener! public void onAuthenticated(final AuthenticatedEvent event) { ... }! !
Summary   •  WebRTC  in  Chrome  has  taken  many  man-­‐years   – there  are  more  things  to  think  about  for  mobile   •  Leverage  a  NaBve  SDK  for  your  Mobile  ApplicaBon  
Thank  You     Please  remember  to  complete  an  evaluaBon  of  today’s  sessions  
QuesBons  (hidden)  

More Related Content

PPTX
Echo in WebRTC; Why?
Muaz Khan
 
PPTX
WebRTC for Mobile - Challenges and Solutions
Amir Zmora
 
PDF
WebRTC and Mobile Integration
Giorgio Natili
 
PDF
Avb Module Datasheet
Antoine Rouget
 
PDF
Develop Smart Solutions with Raspberry Pi and EnableX Live Video API
Enablex io
 
PPT
Polycom - Overview
CXN-BE
 
PPT
Keyboard, Video And Mouse (KVM) Switch Solution
Premier Farnell
 
PPTX
Sommarkollo 2013 lync workshop
Tommy Clarke
 
Echo in WebRTC; Why?
Muaz Khan
 
WebRTC for Mobile - Challenges and Solutions
Amir Zmora
 
WebRTC and Mobile Integration
Giorgio Natili
 
Avb Module Datasheet
Antoine Rouget
 
Develop Smart Solutions with Raspberry Pi and EnableX Live Video API
Enablex io
 
Polycom - Overview
CXN-BE
 
Keyboard, Video And Mouse (KVM) Switch Solution
Premier Farnell
 
Sommarkollo 2013 lync workshop
Tommy Clarke
 

What's hot (20)

PPTX
Live, Low Delay, High Quality – How?
Bitmovin Inc
 
PPT
Acs ip-so c-10-tips-presentation
Design And Reuse
 
PPTX
Video Encoding and HTML5 Playback With Native DRM
Stefan Lederer
 
PPT
Lync Server 2013: Network Quality considerations in LAN, WAN and Wi-Fi
Ståle Hansen
 
PDF
Polycom Automation Framework - Polycom
qualisystems
 
PDF
UplinQ - enhance qualcomm® snapdragon™ audio using android audio ap_is
Satya Harish
 
PPTX
polycom , group series, setup, hardware
Mohamed El-Behery
 
PPTX
Polycom/Microsoft Solutions Q4 2011
Graham Walsh
 
PDF
WVIZ Desktop Videoconferencing 03
Videoguy
 
PPT
Track F- Designing the kiler soc - sonics
chiportal
 
PDF
Cloud Video Streaming without Plug-Ins
Bitmovin Inc
 
PDF
High Definition Fuzzing; Exploring HDMI vulnerabilities
E Hacking
 
PPTX
IbroadcastPro: multi format broadcasting
Wouter van der Ley
 
PDF
Biamp_Product_Brochure_TesiraFORTE
Amber James
 
PDF
SMPTE Toronto Presentation - Open-Source Software In Broadcasting: The Power ...
Brad Fortner
 
PDF
Cableworld
TELE-audiovision eng
 
PDF
SIP conferencing Application Server Enables Video ...
Videoguy
 
PDF
Yealink presentation(1)
Shaharyar Rao nethawk.com.pk
 
PPTX
Infocomm2012 basic building blocks of iptv mp v2
Will Bakewell
 
PDF
What the HEC? Security implications of HDMI Ethernet Channel and other relate...
44CON
 
Live, Low Delay, High Quality – How?
Bitmovin Inc
 
Acs ip-so c-10-tips-presentation
Design And Reuse
 
Video Encoding and HTML5 Playback With Native DRM
Stefan Lederer
 
Lync Server 2013: Network Quality considerations in LAN, WAN and Wi-Fi
Ståle Hansen
 
Polycom Automation Framework - Polycom
qualisystems
 
UplinQ - enhance qualcomm® snapdragon™ audio using android audio ap_is
Satya Harish
 
polycom , group series, setup, hardware
Mohamed El-Behery
 
Polycom/Microsoft Solutions Q4 2011
Graham Walsh
 
WVIZ Desktop Videoconferencing 03
Videoguy
 
Track F- Designing the kiler soc - sonics
chiportal
 
Cloud Video Streaming without Plug-Ins
Bitmovin Inc
 
High Definition Fuzzing; Exploring HDMI vulnerabilities
E Hacking
 
IbroadcastPro: multi format broadcasting
Wouter van der Ley
 
Biamp_Product_Brochure_TesiraFORTE
Amber James
 
SMPTE Toronto Presentation - Open-Source Software In Broadcasting: The Power ...
Brad Fortner
 
Cableworld
TELE-audiovision eng
 
SIP conferencing Application Server Enables Video ...
Videoguy
 
Yealink presentation(1)
Shaharyar Rao nethawk.com.pk
 
Infocomm2012 basic building blocks of iptv mp v2
Will Bakewell
 
What the HEC? Security implications of HDMI Ethernet Channel and other relate...
44CON
 
Ad

Viewers also liked (20)

PPTX
Retailers: Get Social During the Holidays
Adashmore Creative, LLC
 
PPTX
Sectoreseconmicos
Efren Beltran
 
PPTX
Maira
Efren Beltran
 
PPTX
Hangout draft
codycowan
 
PPTX
Presentación1
Efren Beltran
 
PPTX
E learning management system
chrt222
 
PPTX
Steer
Efren Beltran
 
PPT
如何做報告 (How to make a presentation?)
Frank Fang Kuo Yu
 
PPTX
David
Efren Beltran
 
PPTX
Samir
Efren Beltran
 
PPTX
Memes
Paul Re
 
PPTX
Sectores económicos
Efren Beltran
 
PPTX
Sectores económicos
Efren Beltran
 
PPTX
factores economicos
Efren Beltran
 
PPTX
Dibujos animados
Monica Reyes Chilicahua
 
PPTX
Sectores económicos
Efren Beltran
 
PPTX
Ix535 unit 1 wiki
nails26
 
PPTX
1x536 unit 2 how to boil water in the microwave
nails26
 
PPTX
educacion
Efren Beltran
 
PPTX
PIB
Efren Beltran
 
Retailers: Get Social During the Holidays
Adashmore Creative, LLC
 
Sectoreseconmicos
Efren Beltran
 
Maira
Efren Beltran
 
Hangout draft
codycowan
 
Presentación1
Efren Beltran
 
E learning management system
chrt222
 
Steer
Efren Beltran
 
如何做報告 (How to make a presentation?)
Frank Fang Kuo Yu
 
David
Efren Beltran
 
Samir
Efren Beltran
 
Memes
Paul Re
 
Sectores económicos
Efren Beltran
 
Sectores económicos
Efren Beltran
 
factores economicos
Efren Beltran
 
Dibujos animados
Monica Reyes Chilicahua
 
Sectores económicos
Efren Beltran
 
Ix535 unit 1 wiki
nails26
 
1x536 unit 2 how to boil water in the microwave
nails26
 
educacion
Efren Beltran
 
PIB
Efren Beltran
 
Ad

Similar to Delivering Great WebRTC on Mobile Devices (20)

PPT
VoLTE video telephony – a communication evolution
IMTC
 
PDF
HD Voice, telecom operators
John Gallagher
 
PDF
HD Voice: The Hurdles and how to overcome the codec war
John Gallagher
 
PPTX
VTU V SEM CNS Module 1 PPT 2018 Batch students
VENKATESHBHAT25
 
PPTX
Video Standards - Key Insights
Chetan Manchanda
 
PDF
Os Capouchtutorial
oscon2007
 
PDF
Considerations for Creating Streamed Video Content over 3G ...
Videoguy
 
PPTX
Voice oriented data communication
Ankit Anand
 
PDF
Videoconferencing Technical Considerations for IT Professionals
ClarkPowell
 
PDF
AN OPTIMIZED H.264-BASED VIDEO CONFERENCING SOFTWARE FOR ...
Videoguy
 
PPT
PowerPoint
Videoguy
 
PDF
A Practical Guide to WebRTC
vline
 
PDF
WebRTC Infrastructure the Hard Parts: Media
Dialogic Inc.
 
PPTX
Radvision webinar: Making Real Time Video Work Over The Internet
RADVISION Ltd.
 
PPT
Multimedia
BUDNET
 
PDF
SignalWire - Telecom of Tomorrow
Evan McGee
 
PDF
Dialogic
Videoguy
 
PDF
Multi media networking
cairo university
 
PDF
Quality Assurance for WebRTC Services
Tsahi Levent-levi
 
PDF
WebRTC Tutorial by Dean Bubley of Disruptive Analysis & Tim Panton of Westhaw...
Dean Bubley
 
VoLTE video telephony – a communication evolution
IMTC
 
HD Voice, telecom operators
John Gallagher
 
HD Voice: The Hurdles and how to overcome the codec war
John Gallagher
 
VTU V SEM CNS Module 1 PPT 2018 Batch students
VENKATESHBHAT25
 
Video Standards - Key Insights
Chetan Manchanda
 
Os Capouchtutorial
oscon2007
 
Considerations for Creating Streamed Video Content over 3G ...
Videoguy
 
Voice oriented data communication
Ankit Anand
 
Videoconferencing Technical Considerations for IT Professionals
ClarkPowell
 
AN OPTIMIZED H.264-BASED VIDEO CONFERENCING SOFTWARE FOR ...
Videoguy
 
PowerPoint
Videoguy
 
A Practical Guide to WebRTC
vline
 
WebRTC Infrastructure the Hard Parts: Media
Dialogic Inc.
 
Radvision webinar: Making Real Time Video Work Over The Internet
RADVISION Ltd.
 
Multimedia
BUDNET
 
SignalWire - Telecom of Tomorrow
Evan McGee
 
Dialogic
Videoguy
 
Multi media networking
cairo university
 
Quality Assurance for WebRTC Services
Tsahi Levent-levi
 
WebRTC Tutorial by Dean Bubley of Disruptive Analysis & Tim Panton of Westhaw...
Dean Bubley
 

Recently uploaded (20)

PPTX
A Presentation on Touch Screen Technology
memembruh336
 
PDF
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
PDF
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
PPTX
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
PDF
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
PPTX
A Presentation on Artificial Intelligence
memembruh336
 
PDF
Hindi spoken digit analysis for native and non-native speakers
IAESIJAI
 
PDF
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
PPTX
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
PDF
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
PDF
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
PPTX
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
PPTX
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
PDF
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
PPTX
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
PDF
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
PDF
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
PDF
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
PPTX
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
PDF
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 
A Presentation on Touch Screen Technology
memembruh336
 
Profit Center Accounting in SAP S/4HANA, S4F28 Col11
Libreria ERP
 
Building Integrated photovoltaic BIPV_UPV.pdf
SandeepSingh286037
 
Tartificialntelligence_presentation.pptx
ry7r52mzb4
 
Mushroom cultivation and it's methods.pdf
mailmeonrishi
 
A Presentation on Artificial Intelligence
memembruh336
 
Hindi spoken digit analysis for native and non-native speakers
IAESIJAI
 
ENT215_Completing-a-large-scale-migration-and-modernization-with-AWS.pdf
ssuser28425f
 
Group 1 Presentation -Planning and Decision Making .pptx
MatthewLewis227954
 
WOOl fibre morphology and structure.pdf for textiles
Rajendrakumar868651
 
Assigned Numbers - 2025 - Bluetooth® Document
Bloombase
 
TechTalks-8-2019-Service-Management-ITIL-Refresh-ITIL-4-Framework-Supports-Ou...
bonakiduza1024
 
cloud_computing_Infrastucture_as_cloud_p
mainakbhattacharya20
 
MIND Revenue Release Quarter 2 2025 Press Release
MIND CTI
 
TLE Review Electricity (Electricity).pptx
JayPolicarpio2
 
Unlocking AI with Model Context Protocol (MCP)
Brian McKeiver
 
Agricultural_Statistics_at_a_Glance_2022_0.pdf
SandeepSingh286037
 
DASA ADMISSION 2024_FirstRound_FirstRank_LastRank.pdf
rameswartudu05
 
OMC Textile Division Presentation 2021.pptx
JahangirAlam816069
 
Enhancing emotion recognition model for a student engagement use case through...
IAESIJAI
 

Delivering Great WebRTC on Mobile Devices

  • 2. Delivering  Great  WebRTC  on   Mobile  Devices   Amir  Zmora  
  • 3. Workshop  Presenters   •  Saraj  Mudigonda   – ImaginaBon  Technologies     •  Thomas  Sheffler     – Weemo    
  • 4. WebRTC  on  Mobile  -­‐     IntégraBon,  Challenges  &  SoluBons   Saraj  Mudigonda   ImaginaBon  Technologies   Saraj.mudigonda@imgtec.com  
  • 5. Agenda   •  Background   •  IntegraBon  OpBons   •  V.VoIP  Building  Blocks     •  V.VoIP  Mobile  ApplicaBons   •  Challenges  &  SoluBons   •  Summary  
  • 6. Background   Pre-WebRTC § Limited tools for developers § Real-time voice/video conferencing algorithms were limited to only few proprietary native and enterprise applications § Phone applications and web applications were two independent identities PC Modem Mobile IP Phone § Abundant Tools/APIs available for developers § Easy access to complex real-time voice/video algorithms § Enabling real-time communications on numerous platforms with different operating systems, sizes, functions and on browsers with no-plugins …. But it does not come free, it brings along multiple challenges Post-WebRTC
  • 7. IntegraBon  OpBons   Over-­‐The-­‐Top   ApplicaBons   Pre-­‐Loaded   ApplicaBons   Embedded   ApplicaBons   NATIVE  APPLICATIONS   Web  Browser-­‐ Based  ApplicaBons   •  Best-­‐effort  services  offered   by  non-­‐operator   •  Can  be  downloaded  from   Apple  Store/Google  Play   •  Bundled  with  the  plaTorm   •  Cannot  be  deleted  or   disabled   •  Operator  driven   •  Deeply  integrated   applicaBons   •  Part  of  the  NaBve  Dialer    
  • 8. V.VoIP  Building  Blocks   V.VoIP  Applica4on   WebRTC Media Engine Audio  Coding       Module   Video   Conferencing   Module   Voice/Video     QoS     Audio  Pre-­‐Processing  Module  –     AEC,  NC,  AGC     ICE STUN TURN Security TLS SRTP DTLS Audio/Video  Capture/Rendering   Signaling & User Interface Network  Interface  
  • 9. Mobile  V.VoIP  ApplicaBons   SIP  Signaling   Voice  +  Video  App   SIP  +  TradiBonal   Voice  +  Video  Codecs   Enterprise  Telephony    IR.92/IR.94  +  AMR     Video  +Voice  over  LTE   RCS  5.0/5.1   Rich  CommunicaBon  Suite  
  • 10. Challenges   I.  Device  Related   1.  Ba]ery  Life   2.  Latency  Variability   3.  PlaTorm  Diversity     II.  Network  Related   4.  Bandwidth  Variability   5.  Latency  Variability   6.  Packet  Losses     III.  Ecosystem  Related   7.  Network  Handoff   8.  Interoperability  
  • 11. 1.  Ba]ery  Life   •  Challenge   –  Improved  ba]ery  life  for  mobile  video/voice  conferencing     •  PotenBal  SoluBons   –  OpBmize  for  target  mobile  applicaBon  processors   •  MIPS  intensive  algorithms  such  as  echo  cancellaBon  and  codecs  should  be  opBmized  for  the  target   processors  thus  reducing  the  overall  mw/MHz  required   –  Leverage  hardware  accelerators  wherever  available   •  Not  all  plaTorms  expose  APIs  for  hardware  integraBon     –  Diminishing  returns  for  frame  rate   •  Significant  difference  in  perceivable  video  quality  between  10fps  and    15fps,  but  not  so  much  between   15fps  and  30fps  to  jusBfy  a  2x  increase  in  ba]ery  consumpBon   –  Principle  of  good  enough   •  ResoluBon  based  on  form  factor  of  the  devices.  No  need  for  HD  video  telephony  burning  up  CPU  and   your  hands  on  a  4  inch  screen.   •  If  user  really  wants  highest  possible  quality,  give  him  a  sehng  for  it  with  an  appropriate  warning  J  
  • 12. 0   10   20   30   40   50   60   70   80   90   100   Voice  NB  Call   Voice  WB  Call   Voice  +  SW  Video   Voice  +  HW  Video   C P U   U S A G E   %     1.1  Example  CPU  Usage  %   HD  Video   VGA  30fps  on  Single-­‐Core  CPU   Hardware  accelerators  not   only  offloads  CPU  and  reduces   power  consumpBon  but  also   enables  HD  resoluBon  Video    
  • 13. 2.  Latency  Variability   Propaga4on  +  Queuing  Delay  TX  Processing  Delay   RX  Processing  Delay   Network   Interface   Delay   Algorithmic   Delay  –   Encode/AEC   Audio/Video   Capture  Delay   Network   Interface   Delay   Algorithmic   Delay  –   Decode/AJB   Audio/Video   Rendering     Delay  
  • 14. 2.1  Latency  Variability   •  Challenge   –  Reduced  latency  for  superior  user  experience  and  to  avoid   talking  over  each  other   •  PotenBal  SoluBons   –  Reduced  algorithmic  delay  for  voice/video  encode/decode   and  DSP  algorithms  such  as  echo  cancellaBon   –  OpBmized  integraBon  with  audio,  video  and  network   interfaces   –  Dedicated  bearer  for  real-­‐Bme  applicaBons  -­‐  voice  and  video   conferencing  
  • 15. 2.2  Example  Latency  Measurements   0   50   100   150   200   250   300   350   400   1   2   3   Audio  Interface  Delay   Network  Interface  Delay   Algorithmic  Delay   Handset#   Handset#   Handset#   •  Audio  and  network  interface     delay  is  very  device  specific  and   contributes  to  major  chunk  of   the  latency  on  the  client   •  Algorithmic  delay  is  dependent   on  the  opBmizaBon  of  the  voice   processing  algorithms  
  • 16. 3.  PlaTorm  Diversity   •  Challenge   –  Maintain  same  user-­‐experience  across  different  plaTorms   •  PotenBal  SoluBons   –  Adapt  to  different  form-­‐factors   –  Auto  tuning  of  AcousBc  Echo  CancellaBon  (AEC)  across  the  devices.   AEC  characterisBcs  are  heavily  dependent  on  the  plaTorm   industrial  design  (mic  and  speaker  locaBon)   –  OpBmizaBon  of  the  DSP  algorithms  for  target  processor   –  OpBmizaBon/integraBon  with  the  plaTorm  audio  and  network   interfaces  
  • 17. 3.1  Example  Audio  CharacterisBcs     Mobile  Device     Flat  Delay   (msec)     ERL     Non-­‐Linearity     Jiaer  (msec)     Record  Event   Play  Event   Handset#1   330   -­‐3dB   No  non-­‐linearity   15   8   Handset#2   250   -­‐4dB   Non-­‐Linear   20   5   Handset#3   225   -­‐40dB   Highly  Non-­‐Linear   20   5   Legend:   •  Flat  Delay  -­‐    Delay  between  the  speech  sample  played  ay  the  APP  (sent  to  audio  interface  for  play  back)  to  the  Bme  the  echo  of  the  sample   is  received  by  the  APP  (data  capture  provided  by  the  audio  interface).       •  Non-­‐Linearity  -­‐  Because  of  the  post/pre-­‐processing  components  used  in  the  audio  path,  the  enBre  frequency  range  of  the  played  and   echoed  speech  is  not  modified  uniformly  and  this  nonlinear  distorBon  impacts  the  echo  cancellaBon       •  ERL  -­‐  Echo  return  loss  -­‐  specifies  the  loss/gain  encountered  by  the  signal  in  the  acousBc  environment       •   Ji]er  -­‐  variaBons  in  the  data  capture/play  out  intervals  
  • 18. 4.  Bandwidth  Variability   •  Challenge   –  Adapt  quickly  and  efficiently  to  varying  bandwidth  condiBons   •  PotenBal  SoluBons   –  Video  AdaptaBon  Parameters   •  Frames  Per  Second   •  Video  ResoluBon     •  Bit  Rates   –  Voice  AdaptaBon  Parameters   •  Data  Rate  AdaptaBon   •  Codec  Change   –  Diminishing  returns  for  bandwidth   •  Marginal  to  no  improvement  in  quality  beyond  a  certain  bandwidth  for  a  given  resoluBon  and  frame  rate   •  Don't  use  up  available  bandwidth  if  it  is  not  required.  A  judicious  choice  is  a  must  in  order  to  eliminate  blockiness   –  Bandwidth  control   •  Maintaining  a  given  bandwidth  and  the  right  split  of  codec  bitrate  and  FEC  in  the  presence  of  variable  loss   •  A  constant  bitrate  configuraBon  of  the  codec  is  essenBal.  Variable  bandwidth  can  play  havoc  with  pacing/  bandwidth   adaptaBon      
  • 19. 4.1  Example  Bandwidth  AdaptaBon   0   10000   20000   30000   40000   50000   60000   70000   80000   1   1110   2219   3328   4437   5546   6655   7764   8873   9982   11091   12200   13309   14418   15527   16636   17745   18854   19963   21072   22181   23290   24399   25508   26617   27726   28835   29944   31053   32162   33271   34380   35489   36598   37707   38816   39925   41034   42143   43252   44361   45470   46579   47688   48797   49906   51015   52124   53233   54342   55451   56560   57669   58778   59887   60996   Represents  Allo]ed  Bandwidth  in  bps  
  • 20. 6.  Packet  Losses   •  Challenge   –  MiBgate  packet  loss  to  provide  a  smooth  voice  and  video   experience   •  packet  loss  is  very  common  in  wireless  environments  parBcularly  on  Wi-­‐Fi   •  MiBgaBon  Approach   –  AdapBve  ji]er  buffer  and  packet  loss  concealment    to  miBgate   packet  losses   –  A  suitable  FEC  strategy  to  combat  loss  and  maintain  user   experience  with  minimal  latency   –  Balancing  FEC  protecBon,  quality,  latency  especially  at  low   bandwidths  
  • 21. 6.1  Example  Packet  Loss  &  MiBgaBon   •  Packet  loss  miBgaBon  algorithm   should  help  maintain  good   voice  call  quality  for  random   and  small  bursts   0   0.5   1   1.5   2   2.5   3   3.5   4   4.5   0%   3%   5%   10%   20%   30%   40%   50%   Voice  Quality  (PESQ)   Packet  Error  Rate  (PER)   App#1   App#2   App#3   Dropped  Call   Good  Voice  Quality  
  • 22. 7.  Network  Handoff   •  Challenge   –  Provide  seamless  call  handoff   •  MiBgate  dropped  calls  when  moving  from  one  access  network  to  the  other   •  PotenBal  SoluBons   –  Implement  make  before  break/call  anchoring  algorithms  that  can   seamlessly  handoff  when  moving  from  one  network  to  the  other   –  Some  of  the  3GPP  standards  support  VCC  (Voice  Call  ConBnuity)  for   seamless  handoff  between  circuit-­‐switched  networks  and  packet   networks  
  • 23. 7.1  Typical  Network  Scenario   Dropped  calls   when  changing   networks   Home   On-­‐the-­‐road     Enterprise   3G
  • 24. 8.  Interoperability   •  Challenge   – Interoperability  with  legacy  systems   •  PotenBal  SoluBons   – Because  of  the  open-­‐source  nature  of  WebRTC  it  is   possible  to  integrate  addiBonal  codecs  and  algorithms  to   WebRTC  codebase  for  interoperability   – Transcoding  might  be  required  in  the  infrastructure  to   support  some  legacy  systems  
  • 25. Summary   •  PPA  (Performance,  Power,  AdaptaBon)  –  the  key  metrics  for  opBmal  voice/ video  experience  on  mobile  devices   •  Performance   –  For  improved  performance  all  the  parameters  discussed  have  to  be  tuned   carefully  for  device  and  networks   •  Power   –  OpBmize  for  target  apps  processor,  leverage  hardware  accelerators  wherever   available     –  Find  the  sweet  spot  -­‐  adopt  principle  of  good  enough  and  diminishing  returns  for   frame  rate         •  AdaptaBon   –  Adapt  quickly  and  efficiently  to  varying  bandwidth  condiBons,  network  packet  losses   and  form-­‐factors  
  • 26. WebRTC  on  Mobile  -­‐     IntégraBon,  Challenges  &  SoluBons   Saraj  Mudigonda   ImaginaBon  Technologies   Saraj.mudigonda@imgtec.com  
  • 27. Developing  Mobile  ApplicaBons  using   WebRTC   Thomas  Sheffler     Weemo      
  • 28. Outline   •  A  ConstellaBon  of  OpBons   – browser  versus  naBve  Apps   •  Mobile  ConsideraBons   – WebRTC  is  hard.    Mobile  is  harder.   •  Case  study  across  plaTorms  using  an  SDK   – iOS,  Android,  Javascript  
  • 29. Browser  vs  NaBve   •  HTML5+Javascript+WebRTC   – the  promise:  write  once,  run  everywhere!   – Chrome  leads,  Firefox  and  Opera  closely  follow   •  NaBve  App  on  iOS  and  Android   – deliver  a  naBve  app  experience   – what  does  WebRTC  interoperability  mean?  
  • 30. WebRTC  PlaTorm  OpBons   •  Android   –  Browser:  Chrome,  Firefox,  Opera  (Mar  2014)   •  Note:  WebRTC  is  not  in  the  Android  WebView   –  NaBve:  SDKs   •  Android  device  market  is  incredibly  fragmented   •  iOS   –  Browser:  really  none   –  NaBve:  SDKs   •  iOS  plaTorm  is  relaBvely  well  managed   •  80%  iOS7  (ArsTechnica)   NaBve  App   WebView  
  • 31. OpBons  Summary   Android   iOS   Browser   Chrome,  Firefox,  Opera   n/a   Webview   n/a   n/a   NaBve   Yes*   Yes*   *use  a  WebRTC  compaBble  SDK  
  • 33. What  is  WebRTC?   •  A  Suite  of  Technologies  standardized  by  W3C   –  Codecs:  VP8,  Opus   –  Transport:  RTP,  encrypBon   –  SpecificaBons  for  describing  endpoints  (SDP)   –  Methods  for  endpoints  to  connect  (ICE,  STUN,  TURN)   –  Javascript  bindings  for   •  accessing  your  camera,  microphone   •  obtaining  SDP  entry  of  yourself   •  rendering  media  streams  described  by  an  SDP  entry  
  • 34. What  WebRTC  is  not   •  A  way  to  idenBfy  users   –  authenBcaBon   •  A  way  to  locate  other  users   –  phonebook,  user  database   •  A  way  to  iniBate  calls  to  other  users   –  signaling   •  A  way  to  see  the  online  status  of  other  users   –  presence  
  • 35. What  is  WebRTC-­‐compaBble?   •  NaBve  Apps  may  be  compaBble  if   –  they  can  interoperate  with  a  WebRTC  browser   •  NaBve  Interoperability   –  VP8,  Opus   –  RTP,  encrypBon   –  SDP,  ICE,  STUN,  TURN   •  May  or  may  not  present  bindings  similar  to  Javascript  
  • 36. Compiling  WebRTC  yourself   •  Ref:   h]p://ninjaneBc.com/how-­‐to-­‐get-­‐started-­‐with-­‐webrtc-­‐ and-­‐ios-­‐without-­‐wasBng-­‐10-­‐hours-­‐of-­‐your-­‐life/   •  “The  current  HEAD  has  audio  and  video”  [May  2014]   •  Demo  App  uses  ICE  servers  apprtc.appspot.com   •  My  summary:  this  is  really  hard  
  • 37. Back-­‐end  technolgies   •  Website  versus  Mobile  Backend   –  Website  development  across  plaTorms  (desktop,  iOS,  Android)  is   understood  (Rails,  Django,  Express,  etc)   –  Mobile  App  backend  development  may  be  unfamiliar   •  Parse   •  Why  do  you  need  one?   –  IdenBfy  users  (authenBcaBon,  roster)   –  Keep  your  API  keys  secret   –  Protect  access  to  your  realBme  resources   •  TURN  relays  cost  real  $$  
  • 38. When  Worlds  Collide   •  If  you’re  coming  from  the  telephony  world   –  this  is  not  SIP,  it’s  not  a  so•phone   –  webservices  bring  their  own  problems   –  security,  cross-­‐site  request  forging,  Oauth2,  Javascript  code  is   “in  the  clear”,  mobile  authenBcaBon   •  If  you’re  coming  from  Webservices   –  RTC  is  not  request/response,  it’s  not  ReST   –  get  up  to  speed  on  Offer/Accept,  SDP,  RTP,  ICE  candidates  
  • 39. IdenBfy  your  Use  Case   •  Is  two-­‐way  video  the  applicaBon?     •  Is  it  FaceBme?   •  Is  it  customer  assistance?   –  video  is  a  feature,  not  the  focus   –  does  it  get  out  of  the  way   –  one-­‐way  video?   •  Is  it  a  mulBway  call?  (meeBng)   •  The  way  screen  real  estate  and  compute  resources  is  used  is  VERY   different  from  the  desktop.  
  • 40. These  are  very  different   • WVGA  or  HD   • 250  kbps  –  1  mbps   • Video  IS  the  focus   • 240x180  might  be  enough   • 100  kbps   • Video  floats  while  App  conBnues  
  • 41. Is  Tablet  the  same  as  Mobile?   •  Tablets  do  not  roam  so  o•en:  more  tethered  to  a   basestaBon   •  May  not  have  a  cellular  network   •  Screensize?    Camera?   – Samsung  Galaxy  Tab  2  10”  front  camera  is  0.3MP   •  Ba]ery  life?  CPU?   •  Shaky  camera  effect  
  • 42. ICE  in  One  Minute  
  • 43. ICE  Candidates   RT   TURN   UDP  
  • 44. ICE  Candidates   RT   STUN   UDP   UDP  
  • 46. Cellular  network  consideraBons   •  The  cellular  network  does  not  behave  like  a  consumer   home  network   –  do  not  expect  peer-­‐to-­‐peer  to  work  on  4G   •  Handoffs  between  cells  affects  IP  addresses   –  sudden  changes  in  network  connecBvity  [RFC5944:  mobilility]   •  QuesBon:  how  does  your  WebRTC  applicaBon  behave   in  the  context  of  mobile  networks?  
  • 47. Handoffs  from  WiFi  to  Cellular   •  Mobile  apps  like  FaceBme  set  consumer   expectaBons   •  WebRTC  technologies  do  not  really  address   changing  network  topologies  
  • 48. App  Development   •  How  is  an  App  different  from  a  Desktop?   –  Apps  are  sandboxed:  a  restricted  execuBon  environment   –  no  shared  memory  or  libraries   •  Screensharing   –  Mobile  environments  sandbox  apps  for  security   –  Apps  can  sample  their  own  pixels,  not  the  pixels  of  other  apps  
  • 49. Power  State  TransiBons   •  Foreground/Background   – Can  a  user  depart  the  video  engagement  to  go  look  at   something  else?   – Can  the  App  or  Page  receive  noBficaBons  while  in  the   background?   •  Browser-­‐based  implementaBon   – background  window  is  not  acBve  
  • 50. Use  WebRTC-­‐compaBble  SDK   •  Abstracts  WebRTC  endpoints.   •  AuthenBcates  users.    Presents  calls.   •  Built  on  WebRTC  technologies   – VP8,  Opus   – RTP,  encrypBon   – SDP,  ICE,  STUN,  TURN  
  • 51. Examples  using  SDK   •  iOS,  Android,  Javascript   – iniBalize  package   – authenBcate   – make  and  receive  calls   – connect  to  UI  elements  
  • 53. IniBalize  the  SDK   ! static NSString *APP_ID = @"3b04d9add47c”;! ! @implementation AppDelegate! - (BOOL)application:(UIApplication *)application 
 didFinishLaunchingWithOptions:(NSDictionary *)launchOptions! {! ViewController* mainController = ! (ViewController*) self.window.rootViewController;! ! NSError *e;! Weemo *w = [Weemo WeemoWithAppID:APP_ID andDelegate:mainController error:&e];! return YES;! }! !
  • 54. Mobile  AuthenBcaBon   Your   ApplicaBon   Auth   Controller   Web   Service   Cloud-­‐based   Video  Service   Web   Service   SDK   Video   Service   token   token   request  token   (proxy)   (uid,client,profile)   token   Auth   token  
  • 55. AuthenBcate   static NSString *AUTH_HOST = @"http://192.168.1.233:8000";! ! - (IBAction)login:(id)sender {! ! dispatch_async(dispatch_get_main_queue(), ^{! self.auth = [[WeemoAuthSimpleClient alloc] init];! [self.auth authWithHost:AUTH_HOST uid:@"testuser1" success:^(NSString *token) {! ! [[Weemo instance] authenticateWithToken:token andType:USERTYPE_INTERNAL];! ! } failure:^(NSError *e) {! NSLog(@"Login failure:%@", e);! }];! });! }! !
  • 56. Call,  Hangup   - (IBAction)call_button:(id)sender {! NSLog(@"call button");! [[Weemo instance] createCall:otherUsername];! }! ! - (IBAction)hangup_button:(id)sender {! NSLog(@"hangup button");! ! WeemoCall *call = [[Weemo instance] activeCall];! if (call) {! [call hangup];! }! }! !
  • 57. Connect  call  to  UI   #pragma mark Weemo Delegate! ! - (void) weemoCallCreated:(WeemoCall *)call {! NSLog(@"weemoCallCreated:%@", call);! ! [call setViewVideoIn:self.v_videoIn];! [call setViewVideoOut:self.v_videoOut];! ! call.delegate = self;! ! }! UIView   v_videoIn   UIView   v_videoOut  
  • 58. iOS  Summary   •  We  learned  how  to   – iniBalize  package   – authenBcate   – make  and  receive  calls   – connect  to  UI  elements  
  • 59. Android   •  What  is  different?   – event  bus  instead  of  delegates   – decorators  to  register  event  listeners  
  • 60. EventBus   •  Based  on  a  common  component  in  Google  Guava   •  Our  lightweight  implementaBon   •  Decouples  event  responsibility  from  class  hierarchy  
  • 61. IniBalizaBon   public class FullscreenActivity extends Activity {! @Override! public void onStart() {! super.onStart();! Weemo.eventBus().register(this);! Weemo.initialize("APIKEY", this);! }! ! @WeemoEventListener! public void onConnected(final ConnectedEvent event) { ... }! ! @WeemoEventListener! public void onAuthenticated(final AuthenticatedEvent event) { ... }! !
  • 62. Summary   •  WebRTC  in  Chrome  has  taken  many  man-­‐years   – there  are  more  things  to  think  about  for  mobile   •  Leverage  a  NaBve  SDK  for  your  Mobile  ApplicaBon  
  • 63. Thank  You     Please  remember  to  complete  an  evaluaBon  of  today’s  sessions