Media Frameworks Versus Swift (Swift by Northwest, October 2017)

Media Frameworks
Versus Swift
Chris Adamson • @invalidname
Swift by Northwest, October 2017
Slides available at slideshare.net/invalidname
Code available at github.com/invalidstream

Who the what, now?
@invalidname

import Cocoa
import AVFoundation
import CoreMediaIO
if let devices = AVCaptureDevice.devices(),
let avDevices = devices.filter(
{$0 is AVCaptureDevice}) as? [AVCaptureDevice] {
for device in avDevices {
print("(device.description)")
}
}

[Loopback Simulator][com.rogueamoeba.Loopback:E8577B20-0806-4472-A5E6-426CABCD6C8E]
[Loopback Line-In][com.rogueamoeba.Loopback:A00F38FD-C2B6-43FD-98B7-23BAA6FACB03]
[iMic USB audio system][AppleUSBAudioEngine:Griffin Technology, Inc:iMic USB audio system:220000:2,1]
[Loopback Keynote][com.rogueamoeba.Loopback:1936D2A3-6D0B-428E-899E-0ABE46628EA4]
[Soundflower (64ch)][SoundflowerEngine:1]
[HD Pro Webcam C920][AppleUSBAudioEngine:Unknown Manufacturer:HD Pro Webcam C920:1218B05F:3]
[Soundflower (2ch)][SoundflowerEngine:0]
[iGlasses][iGlasses]
[HD Pro Webcam C920][0x244000046d082d]
Program ended with exit code: 0

CMIOObjectPropertyAddress prop =
{ kCMIOHardwarePropertyAllowScreenCaptureDevices,
kCMIOObjectPropertyScopeGlobal,
kCMIOObjectPropertyElementMaster };
UInt32 allow = 1;
CMIOObjectSetPropertyData( kCMIOObjectSystemObject,
&prop, 0, NULL, sizeof(allow), &allow );

var prop = CMIOObjectPropertyAddress(
mSelector: CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices),
mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
mElement: CMIOObjectPropertyElement(
kCMIOObjectPropertyElementMaster))
var allow : UInt32 = 1
CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject),
&prop,
0,
nil,
UInt32(MemoryLayout<UInt32>.size),
&allow)

CMIOObjectPropertyAddress prop =
{ kCMIOHardwarePropertyAllowScreenCaptureDevices,
kCMIOObjectPropertyScopeGlobal,
kCMIOObjectPropertyElementMaster };
UInt32 allow = 1;
CMIOObjectSetPropertyData( kCMIOObjectSystemObject,
&prop, 0, NULL, sizeof(allow), &allow );
mScope: CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal),
var allow : UInt32 = 1
CMIOObjectSetPropertyData(CMIOObjectID(kCMIOObjectSystemObject),
&prop,
0,
nil,
UInt32(MemoryLayout<UInt32>.size),
&allow)
This
is
ﬁne

mScope: CMIOObjectPropertyScope(
kCMIOObjectPropertyScopeGlobal),

public typealias CMIOObjectPropertySelector = UInt32
public typealias CMIOObjectPropertyScope = UInt32
public typealias CMIOObjectPropertyElement = UInt32
public struct CMIOObjectPropertyAddress {
public var mSelector: CMIOObjectPropertySelector
public var mScope: CMIOObjectPropertyScope
public var mElement: CMIOObjectPropertyElement
public init()
public init(mSelector: CMIOObjectPropertySelector,
mScope: CMIOObjectPropertyScope,
mElement: CMIOObjectPropertyElement)
}

extension CMIOObjectPropertySelector {
static let allowScreenCaptureDevices = CMIOObjectPropertySelector(
kCMIOHardwarePropertyAllowScreenCaptureDevices)
}
extension CMIOObjectPropertyScope {
static let global = CMIOObjectPropertyScope(kCMIOObjectPropertyScopeGlobal)
}
extension CMIOObjectPropertyElement {
static let master = CMIOObjectPropertyElement(kCMIOObjectPropertyElementMaster)
}

mSelector: .allowScreenCaptureDevices,
mScope: .global,
mElement: .master)

Demo
http://github.com/invalidstream/audio-reverser

Reversing Audio
1. Decode the MP3/AAC to LPCM2. Grab a buffer from the end3. Reverse its samples in memory4. Write it to the front of a new ﬁle5. Repeat until fully baked

API Needs
• Convert from MP3/AAC to LPCM
• Write sequentially to audio ﬁle (.caf, .aif, .wav)
• Random-access read from audio ﬁle

Plan A (Swift)
• AV Foundation
• AVAssetReader/Writer can do format conversion
while reading/writing audio ﬁles
• Can’t (easily) read from arbitrary packet offsets;
meant to process everything forward

Plan B (C, Swift?)
• Audio Toolbox (part of Core Audio)
• ExtAudioFile can do format conversions while
reading/writing audio ﬁles
• AudioFile can read from arbitrary packet offset

// declare LPCM format we are converting to
AudioStreamBasicDescription format = {0};
format.mSampleRate = 44100.0;
format.mFormatID = kAudioFormatLinearPCM;
format.mFormatFlags = kAudioFormatFlagIsPacked |
kAudioFormatFlagIsSignedInteger;
format.mBitsPerChannel = 16;
format.mChannelsPerFrame = 2;
format.mBytesPerFrame = 4;
format.mFramesPerPacket = 1;
format.mBytesPerPacket = 4;

// declare LPCM format we are converting to
var format = AudioStreamBasicDescription(
mSampleRate: 44100.0,
mFormatID: kAudioFormatLinearPCM,
mFormatFlags: kAudioFormatFlagIsPacked +
kAudioFormatFlagIsSignedInteger,
mBytesPerPacket: 4,
mFramesPerPacket: 1,
mBytesPerFrame: 4,
mChannelsPerFrame: 2,
mBitsPerChannel: 16,
mReserved: 0)

// open AudioFile for output
AudioFileID forwardAudioFile;
err = AudioFileCreateWithURL(forwardURL,
kAudioFileCAFType,
&format,
kAudioFileFlags_EraseFile,
&forwardAudioFile);
IF_ERR_RETURN
#define IF_ERR_RETURN if (err != noErr) { return err; }

var forwardAudioFile: AudioFileID?
kAudioFileCAFType,
&format,
AudioFileFlags.eraseFile,
&forwardAudioFile)
if err != noErr { return err }

kAudioFileCAFType,
&format,
&forwardAudioFile)
1. Uses a free function, rather than a method on AudioFile

kAudioFileCAFType,
&format,
&forwardAudioFile)
2. Errors are communicated via the return value, rather than
throws

kAudioFileCAFType,
&format,
&forwardAudioFile)
3. Some parameters are UInt32 constants, some are enums

kAudioFileCAFType,
&format,
&forwardAudioFile)
4. Audio format is passed as an
UnsafePointer<AudioStreamBasicDescription>

kAudioFileCAFType,
&format,
&forwardAudioFile)
5. Created object is returned via an in-out parameter

Pointer arithmetic!
// swap packets inside transfer buffer
for i in 0..<packetsToTransfer/2 {
let swapSrc = transferBuffer.advanced(by: Int(i) * Int(format.mBytesPerPacket))
let swapDst = transferBuffer.advanced(by: transferBufferSize -
(Int(i+1) * Int(format.mBytesPerPacket)))
memcpy(swapBuffer, swapSrc, Int(format.mBytesPerPacket))
memcpy(swapSrc, swapDst, Int(format.mBytesPerPacket))
memcpy(swapDst, swapBuffer, Int(format.mBytesPerPacket))
}

extension AudioFileID {
init? (url: URL, fileType: UInt32,
format: AudioStreamBasicDescription, flags: AudioFileFlags) {
var fileId : AudioFileID?
var format = format
let err = AudioFileCreateWithURL(url as CFURL,
fileType,
&format,
flags,
&fileId)
guard err != noErr, let createdFile = fileId else { return nil }
self = createdFile
}
}

Been there, done that
• The Amazing Audio Engine 💀
• Novocaine (💀?)
• EZAudio 💀
• AudioKit
• Superpowered
• etc…

/**
Convert a source audio file (using any Core Audio-supported codec) and create LPCM .caf
files for its forward and backward versions.
- parameter sourceURL: A file URL containing the source audio to be read from
- parameter forwardURL: A file URL with the destination to write the decompressed (LPCM) forward file
- parameter backwardURL: A file URL with the destination to write the backward file
*/
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReversingC.h
//
// Use this file to import your target's public headers that you would like to expose to Swift.
//
#import <CoreFoundation/CoreFoundation.h>
#import <AudioToolbox/AudioToolbox.h>
OSStatus convertAndReverse(CFURLRef sourceURL, CFURLRef forwardURL, CFURLRef backwardURL);
AudioReverser-Bridging-Header.h

if USE_SWIFT_CONVERTER {
err = convertAndReverseSwift(sourceURL: source as CFURL,
forwardURL: self.forwardURL as! CFURL,
backwardURL: self.backwardURL as! CFURL)
} else {
err = convertAndReverse(source as! CFURL,
self.forwardURL as! CFURL,
self.backwardURL as! CFURL)
}

// open AVAudioFile for URL input
let sourceAudioFile = try AVAudioFile(forReading: sourceURL as URL,
commonFormat: .pcmFormatInt16,
interleaved: true)
var forwardAudioFile = try AVAudioFile(forWriting: forwardURL as URL,
settings: [:],
commonFormat: .pcmFormatInt16,
interleaved: true)
// convert to a flat file
let outputBufferSize: size_t = 0x8000 // 32 KB buffer
let framesPerBuffer: UInt32 = UInt32(outputBufferSize) / 4
let transferBuffer =
AVAudioPCMBuffer(pcmFormat: sourceAudioFile.processingFormat,
frameCapacity: framesPerBuffer)
var totalFrameCount: Int64 = 0

while(true) {
do {
try sourceAudioFile.read(into: transferBuffer)
} catch {
// FIXME: discern between empty read and genuine error.
// this could happen if last read(from:) exactly filled the
// buffer, and this one gets 0 frames.
// empty read should break, real error should throw
throw error
}
// this never happens, actually. earlier try fails instead
guard transferBuffer.frameLength > 0 else {
print ("done reading file")
break
}
// increment frame count
totalFrameCount += Int64(transferBuffer.frameLength)
// write to forwardFile
try forwardAudioFile.write(from: transferBuffer)
if transferBuffer.frameLength < transferBuffer.frameCapacity {
// didn't fill buffer; we're done
break
}
}

C APIs on iOS/macOS
• Core Foundation
• Core Audio
• Core Media
• Video Toolbox
• Keychain
• IOKit
• OpenGL
• SQLite
• Accelerate
• OpenCV
• BSD, Mach
• etc…

Audio Units
• Discrete software objects for working with audio
• Generators, I/O, Filters/Effects, Mixers,
Converters
• Typically combined in a “graph” model
• Used by Garage Band, Logic, etc.

Demo
R(t) = C(t) x M(t)
https://github.com/invalidstream/ring-modulator-v3audiounit

Ring Modulator
• Multiplication of two signals
• One is usually a long-period sine wave
• Originally implemented as a ring-shaped circuit

0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2
-2.4
-1.6
-0.8
0.8
1.6
2.4
0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2
-2.4
-1.6
-0.8
0.8
1.6
2.4
0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2
-2.4
-1.6
-0.8
0.8
1.6
2.4
0 0.8 1.6 2.4 3.2 4 4.8 5.6 6.4 7.2
-2.4
-1.6
-0.8
0.8
1.6
2.4
R(t) = C(t) x M(t)

Modulate! Modulate!
• Ring modulator best known
as the “Dalek” voice effect
on Doctor Who (circa
1963)
• Also used in early
electronic music

-(instancetype)initWithComponentDescription:(AudioComponentDescription)componentDescription 
options:(AudioComponentInstantiationOptions)options 
error:(NSError **)outError {
self = [super initWithComponentDescription:componentDescription 
options:options
error:outError];
if (self == nil) {
return nil;
}
// ...
return self;
}
MyAudioUnit.m

https://github.com/apple/swift/blob/master/docs/
ABIStabilityManifesto.md

“First, ABI stability is the center focus of Swift 5 — and
we will pivot much of our prioritization of efforts for
Swift 5 around it. With Swift 4, ABI stability was a
strong goal. In Swift 5, it is a *requirement* of the
release. Whatever ABI we have at the end of Swift 5 is
the ABI that we will have. ABI stability is an important
inﬂection point for the maturity of the language, and it
cannot be delayed any longer.”
—Ted Kremenek, Aug 8, 2017 
“Swift 5: start your engines”
https://lists.swift.org/pipermail/swift-evolution/Week-of-
Mon-20170807/038645.html

// Block which subclassers must provide to implement rendering.
- (AUInternalRenderBlock)internalRenderBlock {
// Capture in locals to avoid Obj-C member lookups.
// If "self" is captured in render, we're doing it wrong. See sample code.
return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *timestamp,
AVAudioFrameCount frameCount,
NSInteger outputBusNumber,
AudioBufferList *outputData,
const AURenderEvent *realtimeEventListHead,
AURenderPullInputBlock pullInputBlock) {
// Do event handling and signal processing here.
return noErr;
};
}

Don’t do this
• An audio unit’s render block is called on a
realtime thread
• Therefore it cannot perform any action that could
block:
• I/O (ﬁle or network)
• Waiting on a mutex or semaphore

Also, don’t do this
• Call objc_msg_send()
• Capture any Objective-C or Swift object
• Allocate memory
Basically, if you touch anything in the block other than a pre-
allocated C struct or numeric type (or a pointer to those types),
you are asking for trouble.

// Block which subclassers must provide to implement rendering.
- (AUInternalRenderBlock)internalRenderBlock {
// Capture in locals to avoid Obj-C member lookups.
// If "self" is captured in render, we're doing it wrong. See sample code.
AUValue *frequencyCapture = &frequency;
AudioStreamBasicDescription *asbdCapture = &asbd;
__block UInt64 *totalFramesCapture = &totalFrames;
AudioBufferList *renderABLCapture = &renderABL;
return ^AUAudioUnitStatus(AudioUnitRenderActionFlags *actionFlags,
const AudioTimeStamp *timestamp,
AVAudioFrameCount frameCount,
NSInteger outputBusNumber,
AudioBufferList *outputData,
const AURenderEvent *realtimeEventListHead,
AURenderPullInputBlock pullInputBlock) {
// Do event handling and signal processing here.
// BLOCK IMPLEMENTATION ON NEXT SLIDE
return noErr;
};
❌

// pull in samples to filter
pullInputBlock(actionFlags, timestamp, frameCount, 0, renderABLCapture);
// copy samples from ABL, apply filter, write to outputData
size_t sampleSize = sizeof(Float32);
for (int frame = 0; frame < frameCount; frame++) {
*totalFramesCapture += 1;
for (int renderBuf = 0; renderBuf < renderABLCapture->mNumberBuffers; renderBuf++) {
Float32 *sample = renderABLCapture->mBuffers[renderBuf].mData +
(frame * asbdCapture->mBytesPerFrame);
// apply modulation
Float32 time = totalFrames / asbdCapture->mSampleRate;
*sample = *sample * sinf(M_PI * 2 * time * *frequencyCapture);
memcpy(outputData->mBuffers[renderBuf].mData +
(frame * asbdCapture->mBytesPerFrame),
sample,
sampleSize);
}
}
return noErr;

Obj-C Instance Variables!
❌
AUValue *frequencyCapture = &frequency;
AudioStreamBasicDescription *asbdCapture = &asbd;
__block UInt64 *totalFramesCapture = &totalFrames;
AudioBufferList *renderABLCapture = &renderABL;

https://github.com/apple/swift/blob/master/docs/
OwnershipManifesto.md

Certain kinds of low-level programming
require stricter performance guarantees.
Often these guarantees are less about
absolute performance than predictable
performance. For example, keeping up with
an audio stream is not a taxing job for a
modern processor, even with signiﬁcant per-
sample overheads, but any sort of
unexpected hiccup is immediately noticeable
by users.
—“Swift Ownership Manifesto”, 
February 2017

We believe that these problems can be addressed with an opt-in
set of features that we collectively call ownership. […]
Swift already has an ownership system, but it's “under the
covers”: it's an implementation detail that programmers have
little ability to inﬂuence. What we are proposing here is easy to
summarize:
• We should add a core rule to the ownership system,
called the Law of Exclusivity […]
• We should add features to give programmers more
control over the ownership system […]
• We should add features to allow programmers to express
types with unique ownership […]

“[Swift] is the ﬁrst industrial-quality systems
programming language that is as expressive and
enjoyable as a scripting language.”
https://developer.apple.com/library/content/documentation/
Swift/Conceptual/Swift_Programming_Language/
Well, now that you mention it…

“systems programming language” removed

???Big Bang Heat Death of
the Universe

Waiting…
• ABI stability — Will be in Swift 5 (2018)
• Ownership — unclear
• Are these traits sufﬁcient?

Strategies
• Use AV Foundation if you can
• It does most of what anyone needs at this point
• Learn to balance C and Swift
• “Render undo C-sar what is C-sar’s…”
• The goal is to have idiomatic Swift, not Swift
that may work but looks like C

Media Frameworks
versus Swift
Chris Adamson • @invalidname
Swift by Northwest, October 2017
Slides available at slideshare.net/invalidname
Code available at github.com/invalidstream

Media Frameworks Versus Swift (Swift by Northwest, October 2017)

More Related Content

What's hot (20)

Similar to Media Frameworks Versus Swift (Swift by Northwest, October 2017) (20)

More from Chris Adamson (19)

Recently uploaded (20)

Media Frameworks Versus Swift (Swift by Northwest, October 2017)