Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

Core Image

The Most Fun API

You’re Not Using
Chris Adamson • @invalidname

CocoaConf Columbus, August 2014

“Core Image is an image processing and analysis
technology designed to provide near real-time
processing for still and video images.”

Agenda
• Images, Filters, and Contexts

• The Core Image Filter Gallery

• Neat Tricks with Built-In Filters

• Core Image on OS X

Core Image, Core Concepts
• Core Image is
of the time

• A chain of ﬁlters describes a “recipe” of processing
steps to be applied to one or more images

• “Stringly typed”

• You only get pixels when you render

Typical Workflow
• Start with a source CIImage

• Apply one or more filters

• Render resulting CIImage to a CIContext, or
convert CIImage out to another type

• A few filters take or produce types other than
CIImage (CIQRCodeGenerator)

CIImage
• An image provided to or produced by Core Image

• But no bitmap of pixel data!

• Immutable

• -imageByCroppingToRect,
-imageByApplyingTransform

• -extent — a CGRect of the image’s size

CIImage sources
• NSURL

• CGImageRef

• Bitmap or JPEG/PNG/TIFF in NSData

• OpenGL texture

• Core Video image/pixel buffer

CIContext
• Rendering destination for a CIImage (-
[drawImage:inRect:fromRect:])

• This is where you get pixels (also, this is the processor-
intenstive part)

• On iOS, must be created from an EAGLContext. On
Mac, can be created with CGContextRef

• Can also produce output as a CGImageRef, bitmap data,
or a CVPixelBuffer (iOS only)

CIFilter
• Performs an image processing operation

• Typically takes and produces a CIImage

• All parameters are provided via -[setValue:forKey:]

• Stringly-typed!

• Output is retrieved with -[valueForKey:]

–Core Image Cat
“I can has ﬁlterz?

Core Image Filter Reference
Filter Name
Parameters
Note the type & number to
provide
Categories
Watch for CICategoryBuiltIn
and CICategoryVideo
Example Figure
Availability
Watch for versioning and
OS X-only ﬁlters

Filter Categories
• Group ﬁlters by functionality: CICategoryBlur,
CICategoryGenerator,
CICategoryCompositeOperation, etc.

• Also group ﬁlters by availability and
appropriateness: CICategoryBuiltIn,
CICategoryVideo, CICategoryNonSquarePixels

CICategoryGenerator
• No input image, just produces an output

• CICategoryGradient is also output-only

• Example: CICheckerboardGenerator

CICategoryBlur
• Algorithmically spreads/blends pixels

• CICategorySharpen offers an opposite effect

• Example: CIGaussianBlur

CICategoryColorAdjustement
• Changes distribution of color throughout an image

• Example: CIColorControls (adjusts saturation,
brightness, contrast)

CICategoryColorEffect
• Color changes that affect the subjective nature of
the image

• Example: CIPhotoEffectNoir

CICategoryDistortionEffect
• Moves pixels to achieve an effect

• Example: CITorusLensDistortion

CICategoryStylize
• Various stylistic effects

• Example: CIPointillize

CICategoryGeometryAdjustment
• Moves pixels via cropping, afﬁne transforms, etc.

• Example: CICrop

CICategoryTileEffect
• Repeatedly copies all or part of an image

• Example: CIAfﬁneTile

CICategoryCompositeOperation
• Combines multiple images

• Example: CISourceOverCompositing

Creating CIColorControls Filter
_colorControlsFilter

Setting input values
[self
! ! ! ! ! ! ! ! ! ! ! !
[self
! ! ! ! ! ! ! ! ! ! ! !
[self
! ! ! ! ! ! ! ! ! ! ! !

Setting input image
CIImage
! ! ! ! ! ! ! ! ! !
[self
! ! ! ! ! ! ! !

Getting output image
ciImage = [
UIImage
self

Other output options
• Use a CIContext

• -[drawImage:inRect:fromRect:] draws pixels to
the EAGLContext (iOS) or CGContextRef (OS
X) that the CIContext was created from.

• CIContext can also render to a void* bitmap

• On iOS, can create a CVPixelBufferRef, typically
used for writing to a ﬁle with AVAssetWriter

Chaining ﬁlters
• Use the output of one ﬁlter as the input to the next

• This doesn’t cost anything, because the CIImages
just hold state, not pixels

Creating CIContext
if (!
! !
! ! ! ! ! !
! !
}!
! ! !
// make CIContext from GL context, clearing out default color space
self
! ! ! ! ! ! ! ! ! ! ! ! !
! ! ! !
! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !
!
Note: This is in a subclass of GLKView

Set up Sepia Tone ﬁlter
_sepiaToneFilter
[_sepiaToneFilter
! ! ! ! ! ! ! ! !

Set up Mask to Alpha ﬁlter
UIImage
_circleMaskFilter
CIImage
! ! ! ! ! ! ! ! ! ! ! ! circleImageUI.
[_circleMaskFilter
! ! ! ! ! ! !
_circleMask
circle-mask-100x100.png

Set up Blend with Mask ﬁlter
_constantColorGeneratorFilter
! ! ! ! ! ! ! ! ! !
[_constantColorGeneratorFilter
! ! ! [
! ! ! ! ! ! ! ! ! ! ! ! ! !
forKey
_backgroundAlphaFill
! ! ! ! ! ! ! ! ! ! !
!
_blendWithMaskFilter
[_blendWithMaskFilter
! ! ! ! ! ! ! !
! ! ! ! ! ! ! !

Apply ﬁlters
// Get CIImage from source image
CGImageRef
! ! ! ! ! ! ! !
loupeImage = [
!
// Apply sepia filter
[self
loupeImage = [
!
// Set sepia-filtered image as input to blend-with-mask
! ! ! ! ! !
loupeImage = [

Render in CIContext
if ([
! [
}!
! !
[self
!
// GL-on-Retina fix
CGRect
drawBoundsInPoints.
drawBoundsInPoints.
! !
// drawing to CIContext draws to the EAGLESContext it's based on
[self
! ! ! ! !! ! !
! ! ! ! ! ! ! !
!
// Refresh GLKView contents immediately
[self

Working with Video
• AVFoundation AVCaptureVideoDataOutput and
AVAssetReader deliver CMSampleBuffers

• CMSampleBuffers have timing information and
CVImageBuffers/CVPixelBuffers

• +[CIImage imageWithCVPixelBuffer:]

Chroma Key (“green screen”
recipe
• Use a CIColorCube to map green-ish colors to
transparent

• Use CISourceOverCompositing to draw this
alpha’ed image over another image

CIColorCube
Maps colors from one RGB “cube” to another
http://en.wikipedia.org/wiki/RGB_color_space

Using CIColorCube
CIColorCube maps green(-ish) colors to 0.0 alpha, all
other colors pass through

CISourceOverCompositing
inputBackgroundImage inputImage
outputImage

CIColorCube Dataconst unsigned int size = 64;!
size_t cubeDataSize = size * size * size * sizeof (float) * 4;!
float *keyCubeData = (float *)malloc (cubeDataSize);!
//! float *alphaMatteCubeData = (float *)malloc (cubeDataSize);!
//! float rgb[3], hsv[3], *keyC = keyCubeData, *alphaC = alphaMatteCubeData;!
float rgb[3], hsv[3], *keyC = keyCubeData;!
// Populate cube with a simple gradient going from 0 to 1!
for (int z = 0; z < size; z++){!
! rgb[2] = ((double)z)/(size-1); // Blue value!
! for (int y = 0; y < size; y++){!
! ! rgb[1] = ((double)y)/(size-1); // Green value!
! ! for (int x = 0; x < size; x ++){!
! ! ! rgb[0] = ((double)x)/(size-1); // Red value!
!
! ! ! // Convert RGB to HSV!
! ! ! // You can find publicly available rgbToHSV functions on the Internet!
!
! ! ! RGBtoHSV(rgb[0], rgb[1], rgb[2],!
! ! ! ! ! &hsv[0], &hsv[1], &hsv[2]);!
!
! ! ! // RGBtoHSV uses 0 to 360 for hue, while UIColor (used above) uses 0 to 1.!
! ! ! hsv[0] /= 360.0;!
! ! ! !
! ! ! // Use the hue value to determine which to make transparent!
! ! ! // The minimum and maximum hue angle depends on!
! ! ! // the color you want to remove!
! ! ! !
! ! ! bool keyed = (hsv[0] > minHueAngle && hsv[0] < maxHueAngle) &&!
! ! ! (hsv[1] > minSaturation && hsv[1] < maxSaturation) &&!
! ! ! (hsv[2] > minBrightness && hsv[2] < maxBrightness);!
! ! ! !
! ! ! float alpha = keyed ? 0.0f : 1.0f;!
! ! ! !
! ! ! // re-calculate c pointer!
! ! ! keyC = (((z * size * size) + (y * size) + x) * sizeof(float)) + keyCubeData;!
! ! ! !
! ! ! // Calculate premultiplied alpha values for the cube!
! ! ! keyC[0] = rgb[0] * alpha;!
! ! ! keyC[1] = rgb[1] * alpha;!
! ! ! keyC[2] = rgb[2] * alpha;!
! ! ! keyC[3] = alpha;!
! ! ! ! ! ! ! !
! ! }!
! }!
}!
See “Chroma Key Filter Recipe” in Core Image Programming Guide

Create CIColorCube from
mapping data
// build the color cube filter and set its data to above!
self.colorCubeFilter = [CIFilter filterWithName:@"CIColorCube"];!
[self.colorCubeFilter setValue:[NSNumber numberWithInt:size]!
! !! ! ! ! ! ! ! ! forKey:@"inputCubeDimension"];!
NSData *data = [NSData dataWithBytesNoCopy:keyCubeData!
! !! ! ! ! ! ! ! length:cubeDataSize!
! !! ! ! ! ! freeWhenDone:YES];!
[self.colorCubeFilter setValue:data forKey:@"inputCubeData"];!

Create CISourceOverCompositing
// source over filter!
self.backgroundImage = [UIImage imageNamed:!
! !! ! ! ! ! ! ! ! ! @"img_washington_small_02.jpg"]; !
self.backgroundCIImage = [CIImage imageWithCGImage:!
! !! ! ! ! ! ! ! ! ! ! self.backgroundImage.CGImage];!
self.sourceOverFilter = [CIFilter filterWithName:!
! !! ! ! ! ! ! ! ! ! @"CISourceOverCompositing"];!
[self.sourceOverFilter setValue:self.backgroundCIImage !
! !! ! ! ! ! ! ! forKeyPath:@"inputBackgroundImage"];!

Apply Filters in Capture Callback
CIImage *bufferCIImage = [CIImage imageWithCVPixelBuffer:cvBuffer];!
!
[self.colorCubeFilter setValue:bufferCIImage !
! !! ! ! ! ! ! ! ! ! forKey:kCIInputImageKey];!
CIImage *keyedCameraImage = [self.colorCubeFilter valueForKey:!
! !! ! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];!
!
[self.sourceOverFilter setValue:keyedCameraImage !
! !! ! ! ! ! ! ! ! forKeyPath:kCIInputImageKey];!
!
CIImage *compositedImage = [self.sourceOverFilter valueForKeyPath:!
! !! ! ! ! ! ! ! ! ! ! kCIOutputImageKey];
Then draw compositedImage to CIContext as before

Other Points of Interest
• CIQRCodeGenerator filter — Converts data (e.g., a string) to
a QR Code

• CILenticularHaloGenerator filter — aka, lens flare

• CIDetector — Class (not a filter) to find features in images.
Currently only supports face finding (returned as an array of
CIFeatures). Optionally detects smiles and eye blinks within
faces.

• CIImage has a red-eye enhancement that takes the array of
face CIFeatures to tell it where to apply the effect

Core Image on OS X
• Core Image is part of QuartzCore (or Image Kit), so
you don’t @import CoreImage

• Many more ﬁlters are available

• Can create your own ﬁlter with OpenGL Shading
Language (plus some CI extensions). See CIKernel.

• Also available in iOS 8

• Filters can be set on CALayers

CALayer Filters on OS X
• Views must be layer-backed (obviously)

• Must also call -[NSView
setLayerUsesCoreImageFilters:] on 10.9+

• CALayer has properties: filters, compositingFilter,
backgroundFilters, minificationFilter,
magnificationFilter

• These exist on iOS, but do nothing

Adding CIPixellate to layer’s
ﬁlters
self
! !! ! ! ! ! ! ! !
self
[self
! !! ! ! ! ! [
! !! ! ! ! ! ! ! !
!
[self
! !! ! !
! !! ! ! ! ! ! ! ! !
self
! !! ! !

Updating a layer’s ﬁlters
-(void
! [
! !! ! ! !
! !! ! ! ! ! ! ! ! ! !
! !! ! ! !
}

Wrap Up: Stuff to Remember
• Get psyched about ﬁlters, but remember to check
that they’re on your targeted platform/version.

• Drawing to a CIContext on iOS must be GL-
backed (e.g., with a GLKView)

Q&A
Slides and code will be posted to:

http://www.slideshare.net/invalidname/
!
@invalidname

http://subfurther.com/blog

Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)

More Related Content

Viewers also liked (16)

Similar to Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014) (20)

More from Chris Adamson (14)

Recently uploaded (20)

Core Image: The Most Fun API You're Not Using (CocoaConf Columbus 2014)