CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine

Media Frameworks and
Swift:
This is Fine
Chris Adamson • @invalidname
CocoaConf Chicago, April 2017
Slides available at slideshare.net/invalidname
Code available at github.com/invalidstream

Who the what, now?
@invalidname

import Cocoa
import AVFoundation
import CoreMediaIO
if let devices = AVCaptureDevice.devices(),
let avDevices = devices.filter(
{$0 is AVCaptureDevice}) as? [AVCaptureDevice] {
for device in avDevices {
print("(device.description)")
}
}

<AVCaptureHALDevice: 0x100b16ab0 [Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E]>
<AVCaptureHALDevice: 0x100c1a7c0 [Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03]>
<AVCaptureHALDevice: 0x100c16910 [iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1]>
<AVCaptureHALDevice: 0x100d13900 [Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4]>
<AVCaptureHALDevice: 0x100a26850 [Soundflower (64ch)][SoundflowerEngine:1]>
<AVCaptureHALDevice: 0x100a26310 [HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3]>
<AVCaptureHALDevice: 0x100d13660 [Soundflower (2ch)][SoundflowerEngine:0]>
<AVCaptureDALDevice: 0x100a348f0 [iGlasses][iGlasses]>
<AVCaptureDALDevice: 0x100a28d00 [HD Pro Webcam C920][0x244000046d082d]>
Program ended with exit code: 0

CMIOObjectPropertyAddress prop =
{ kCMIOHardwarePropertyAllowScreenCaptureDevices,
kCMIOObjectPropertyScopeGlobal,
kCMIOObjectPropertyElementMaster };
UInt32 allow = 1;
CMIOObjectSetPropertyData( kCMIOObjectSystemObject,
&prop, 0, NULL, sizeof(allow), &allow );

var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))
var allow : UInt32 = 1
CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject),
&prop,
0,
nil,
UInt32(MemoryLayout<UInt32>.size),
&allow)

UInt32 allow = 1;
&prop,
0,
nil,
&allow)

UInt32 allow = 1;
&prop,
0,
nil,
&allow)
This
is
ﬁne

mScope: CMIOObjectPropertyScope(
kCMIOObjectPropertyScopeGlobal),

public typealias CMIOObjectPropertySelector = UInt32
public typealias CMIOObjectPropertyScope = UInt32
public typealias CMIOObjectPropertyElement = UInt32
public struct CMIOObjectPropertyAddress {
public var mSelector: CMIOObjectPropertySelector
public var mScope: CMIOObjectPropertyScope
public var mElement: CMIOObjectPropertyElement
public init()
public init(mSelector: CMIOObjectPropertySelector,
mScope: CMIOObjectPropertyScope,
mElement: CMIOObjectPropertyElement)
}

extension CMIOObjectPropertySelector {
static let allowScreenCaptureDevices = CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices)
}
extension CMIOObjectPropertyScope {
static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal)
}
extension CMIOObjectPropertyElement {
static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster)
}

mSelector: .allowScreenCaptureDevices,
mScope: .global,
mElement: .master)

Demo
http://github.com/invalidstream/audio-reverser

Reversing Audio
1. Decode the MP3/AAC to LPCM

Reversing Audio
2. Grab a buffer from the end

Reversing Audio
3. Reverse its samples in memory

Reversing Audio
4. Write it to the front of a new ﬁle

Reversing Audio
5. Repeat until fully baked

API Needs
• Convert from MP3/AAC to LPCM
• Write sequentially to audio ﬁle (.caf, .aif, .wav)
• Random-access read from audio ﬁle

Plan A (Swift)
• AV Foundation
• AVAssetReader/Writer can do format conversion
while reading/writing audio ﬁles
• Can’t (easily) read from arbitrary packet offsets;
meant to process everything forward

Plan B (C, Swift?)
• Audio Toolbox (part of Core Audio)
• ExtAudioFile can do format conversions while
reading/writing audio ﬁles
• AudioFile can read from arbitrary packet offset

// declare LPCM format we are converting to
AudioStreamBasicDescription format = {0};
format.mSampleRate = 44100.0;
format.mFormatID = kAudioFormatLinearPCM;
format.mFormatFlags = kAudioFormatFlagIsPacked |
kAudioFormatFlagIsSignedInteger;
format.mBitsPerChannel = 16;
format.mChannelsPerFrame = 2;
format.mBytesPerFrame = 4;
format.mFramesPerPacket = 1;
format.mBytesPerPacket = 4;

// declare LPCM format we are converting to
var format = AudioStreamBasicDescription(
mSampleRate: 44100.0,
mFormatID: kAudioFormatLinearPCM,
mFormatFlags: kAudioFormatFlagIsPacked +
kAudioFormatFlagIsSignedInteger,
mBytesPerPacket: 4,
mFramesPerPacket: 1,
mBytesPerFrame: 4,
mChannelsPerFrame: 2,
mBitsPerChannel: 16,
mReserved: 0)

// open AudioFile for output
AudioFileID forwardAudioFile;
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
kAudioFileFlags_EraseFile,
&forwardAudioFile);
IF_ERR_RETURN
#define IF_ERR_RETURN if (err != noErr) { return err; }

var forwardAudioFile: AudioFileID?
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }

kAudioFileCAFType,
&format,
&forwardAudioFile)
1. Uses a free function, rather than a method on AudioFile

kAudioFileCAFType,
&format,
&forwardAudioFile)
2. Errors are communicated via the return value, rather than
throws

kAudioFileCAFType,
&format,
&forwardAudioFile)
3. Some parameters are UInt32 constants, some are enums

kAudioFileCAFType,
&format,
&forwardAudioFile)
4. Audio format is passed as an
UnsafePointer<AudioStreamBasicDescription>

kAudioFileCAFType,
&format,
&forwardAudioFile)
5. Created object is returned via an in-out parameter

Pointer arithmetic!
// swap packets inside transfer buffer
for i in 0..<packetsToTransfer/2 {
let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket))
let swapDst = transferBuffer.advanced(by: transferBufferSize -
(Int(i+1) * Int(format.mBytesPerPacket)))
memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket))
memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket))
memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket))
}

extension AudioFileID {
init? (url: URL, fileType: UInt32,
format: AudioStreamBasicDescription, flags: AudioFileFlags) {
var fileId : AudioFileID?
var format = format
let err = AudioFileCreateWithURL(url as CFURL,
fileType,
&format,
flags,
&fileId)
guard err != noErr, let createdFile = fileId else { return nil }
self = createdFile
}
}

Been there, done that
• The Amazing Audio Engine !
• Novocaine (!?)
• EZAudio !
• AudioKit
• Superpowered
• etc…

/**
Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf
files for its forward and backward versions.
- parameter sourceURL: A file URL containing the source audio to be read from
- parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file
- parameter backwardURL: A file URL with the destination to write the backward file
*/
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReversingC.h
//
// Use this file to import your target's public headers that you would like to expose to Swift.
//
#import <CoreFoundation/CoreFoundation.h>
#import <AudioToolbox/AudioToolbox.h>
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReverser-Bridging-Header.h

if USE_SWIFT_CONVERTER {
err = convertAndReverseSwift(sourceURL: source as CFURL,
forwardURL: self.forwardURL as! CFURL,
backwardURL: self.backwardURL as! CFURL)
} else {
err = convertAndReverse(source as! CFURL,
self.forwardURL as! CFURL,
self.backwardURL as! CFURL)
}

C APIs on iOS/macOS
• Core Foundation
• Core Audio
• Core Media
• Video Toolbox
• Keychain
• IOKit
• OpenGL
• SQLite
• Accelerate
• OpenCV
• BSD, Mach
• etc…

Audio Units
• Discrete software objects for working with audio
• Generators, I/O, Filters/Effects, Mixers,
Converters
• Typically combined in a “graph” model
• Used by Garage Band, Logic, etc.

Demo
R(t) = C(t) x M(t)
https://github.com/invalidstream/ring-modulator-v3audiounit

Ring Modulator
• Multiplication of two signals
• One is usually a sine wave
• Originally implemented as a ring-shaped circuit

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2
-2.4
-1.6
-0.8
0.8
1.6
2.4
R(t) = C(t) x M(t)

Modulate! Modulate!
• Ring modulator best known
as the “Dalek” voice effect
on Doctor Who (circa
1963)
• Also used in early
electronic music

-(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription 
options:(AudioComponentInstantiationOptions)options 
error:(NSError **)outError {
self = [super initWithComponentDescription:componentDescription 
options:options
error:outError];
if (self == nil) {
return nil;
}
// ...
return self;
}
MyAudioUnit.m

https://github.com/apple/swift/blob/master/docs/
ABIStabilityManifesto.md

“Given the importance of getting the core ABI and the
related fundamentals correct, we are going to defer the
declaration of ABI stability out of Swift 4 while still
focusing the majority of effort to get to the point where
the ABI can be declared stable.”
—Ted Kremenek, Feb. 16, 2017 
“Swift 4, stage 2 starts now”
https://lists.swift.org/pipermail/swift-evolution/Week-of-
Mon-20170213/032116.html

// Block which subclassers must provide to implement rendering.
- (AUInternalRenderBlock)internalRenderBlock {
// Capture in locals to avoid Obj-C member lookups.
// If "self" is captured in render, we're doing it wrong. See sample code.
return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *timestamp,
AVAudioFrameCount frameCount,
NSInteger outputBusNumber,
AudioBufferList *outputData,
const AURenderEvent *realtimeEventListHead,
AURenderPullInputBlock pullInputBlock) {
// Do event handling and signal processing here.
return noErr;
};
}

Don’t do this
• An audio unit’s render block is called on a
realtime thread
• Therefore it cannot perform any action that could
block:
• I/O (ﬁle or network)
• Waiting on a mutex or semaphore

Also, don’t do this
• Call objc_msg_send()
• Capture any Objective-C or Swift object
• Allocate memory
Basically, if you touch anything in the block other than a pre-
allocated C struct or primitive, you’re asking for trouble.

AUValue *frequencyCapture = &frequency;
AudioStreamBasicDescription *asbdCapture = &asbd;
__block UInt64 *totalFramesCapture = &totalFrames;
AudioBufferList *renderABLCapture = &renderABL;
// BLOCK IMPLEMENTATION ON NEXT SLIDE
return noErr;
};

// BLOCK IMPLEMENTATION ON NEXT SLIDE
return noErr;
};
❌

// pull in samples to filter
pullInputBlock(actionFlags, timestamp, frameCount, 0, renderABLCapture);
// copy samples from ABL, apply filter, write to outputData
size_t sampleSize = sizeof(Float32);
for (int frame = 0; frame < frameCount; frame++) {
*totalFramesCapture += 1;
for (int renderBuf = 0; renderBuf < renderABLCapture->mNumberBuffers; renderBuf++) {
Float32 *sample = renderABLCapture->mBuffers[renderBuf].mData +
(frame * asbdCapture->mBytesPerFrame);
// apply modulation
Float32 time = totalFrames / asbdCapture->mSampleRate;
*sample = *sample * fabs(sinf(M_PI * 2 * time * *frequencyCapture));
memcpy(outputData->mBuffers[renderBuf].mData +
(frame * asbdCapture->mBytesPerFrame),
sample,
sampleSize);
}
}
return noErr;

Obj-C Instance Variables!
❌

https://github.com/apple/swift/blob/master/docs/
OwnershipManifesto.md

Certain kinds of low-level programming
require stricter performance guarantees.
Often these guarantees are less about
absolute performance than predictable
performance. For example, keeping up with
an audio stream is not a taxing job for a
modern processor, even with signiﬁcant per-
sample overheads, but any sort of
unexpected hiccup is immediately noticeable
by users.
—“Swift Ownership Manifesto”, 
February 2017

We believe that these problems can be addressed with an opt-in
set of features that we collectively call ownership. […]
Swift already has an ownership system, but it's “under the
covers”: it's an implementation detail that programmers have
little ability to inﬂuence. What we are proposing here is easy to
summarize:
• We should add a core rule to the ownership system,
called the Law of Exclusivity […]
• We should add features to give programmers more
control over the ownership system […]
• We should add features to allow programmers to express
types with unique ownership […]

“[Swift] is the ﬁrst industrial-quality systems
programming language that is as expressive and
enjoyable as a scripting language.”
https://developer.apple.com/library/content/documentation/
Swift/Conceptual/Swift_Programming_Language/

Waiting…
• ABI stability — will not be in Swift 4
• Ownership — unclear
• Are these traits sufﬁcient?

Strategies
• Use AV Foundation if you can
• Learn to balance C and Swift
• “Render undo C-sar what is C-sar’s…”
• The goal is to have idiomatic Swift, not Swift
that may work but looks like C

CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine

More Related Content

What's hot (20)

Similar to CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine (20)

More from Chris Adamson (20)

Recently uploaded (20)

CocoaConf Chicago 2017: Media Frameworks and Swift: This Is Fine