[JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial Detection
1 like1,570 views
The document describes a KittyCam project that uses a Raspberry Pi, camera, and PIR motion sensor to detect when a cat is in view of the camera and take a photo. It uses Node.js and several frameworks like Johnny-Five and KittyDar. When motion is detected, it takes a photo using Raspistill, detects if a cat face is present using KittyDar, and if so uploads the photo to Cloudinary and sends a text with the image link. It provides details on the hardware components, software stack including Node.js, and implementation of the various functions.
![girlie_mac@
Sketch
● Language for Arduino
● Loosely based on C
● Wut, C?
● HALP!!1!!!
#include <Servo.h>
#include <Wire.h>
#include <Firmata.h>
#define I2C_WRITE B00000000
#define I2C_READ B00001000
#define I2C_READ_CONTINUOUSLY B00010000
#define I2C_STOP_READING B00011000
#define I2C_READ_WRITE_MODE_MASK B00011000
#define I2C_10BIT_ADDRESS_MODE_MASK B00100000
#define I2C_MAX_QUERIES 8
#define I2C_REGISTER_NOT_SPECIFIED -1
#define MINIMUM_SAMPLING_INTERVAL 1
int analogInputsToReport = 0;
byte previousPINs[TOTAL_PORTS];
byte pinConfig[TOTAL_PINS];
byte portConfigInputs[TOTAL_PORTS];
int pinState[TOTAL_PINS];
unsigned long currentMillis;](https://image.slidesharecdn.com/jskongresskittycam-161128230754/85/JS-Kongress-2016-KittyCam-js-Raspberry-Pi-Camera-w-Cat-Facial-Detection-6-320.jpg)
![girlie_mac@
PIR Sensor > Run Camera
const child_process = require('child_process');
board.on('ready', () => {
const motion = new five.Motion('P1-7');
motion.on('motionstart', () => { // Motion detected
let filename = 'photo/image_'+i+'.jpg';
let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1'];
let spawn = child_process.spawn('raspistill', args);
spawn.on('exit', function() {
console.log('A photo is saved as '+filename);
...
motion
detected!
Take a photo!
Spawns a new process w/ a given shell command](https://image.slidesharecdn.com/jskongresskittycam-161128230754/85/JS-Kongress-2016-KittyCam-js-Raspberry-Pi-Camera-w-Cat-Facial-Detection-30-320.jpg)
![girlie_mac@
Processing Image
spawn.on('exit', () => {
let imgPath = __dirname + '/' + filename;
// Child process: read the file and detect cats with KittyDar
let args = [imgPath];
let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js');
fork.send(args);
// the child process is completed
fork.on('message', (base64) => {
if(base64) {
uploadToCloud(base64); // Send to cloud storage
}
}); ...
Create another worker
by running a new
instance of V8 engine.](https://image.slidesharecdn.com/jskongresskittycam-161128230754/85/JS-Kongress-2016-KittyCam-js-Raspberry-Pi-Camera-w-Cat-Facial-Detection-31-320.jpg)
![girlie_mac@
detectCatsFromPhoto.js
const kittydar = require('kittydar');
const Canvas = require('canvas');
process.on('message', (m) => {
fs.readFile(m[0], (err, data) => {
...
let canvas = new Canvas(w, h);
let ctx = canvas.getContext('2d');
ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h);
let cats = kittydar.detectCats(canvas);
console.log('There are', cats.length, 'cats in this photo');
if(cats.length > 0) {base64Img = canvas.toDataURL();}
process.send(base64Img);
process.exit(0);
});
Running in an another process
a cat detected!
KittyDar](https://image.slidesharecdn.com/jskongresskittycam-161128230754/85/JS-Kongress-2016-KittyCam-js-Raspberry-Pi-Camera-w-Cat-Facial-Detection-32-320.jpg)
[JS Kongress 2016] KittyCam.js - Raspberry Pi Camera w/ Cat Facial Detection
- 1. girlie_mac@ KittyCam.js Smile, you’re on camera!Smile, you’re on camera! Tomomi Imura
- 2. girlie_mac@ Hello, I’m Tomomi & I’m a: ● San Francisco Dweller ● Front-End Engineer ● Open Web + Tech Advocate ● N00b Hardware Hacker ● Sr. Developer Advocate at Nexmo ● Cat lady of InterWeb
- 3. girlie_mac@
- 5. girlie_mac@ Arduino ● MCU-based kit ● Open-Source hardware & software (IDE) ● The first developer-friendly boards
- 6. girlie_mac@ Sketch ● Language for Arduino ● Loosely based on C ● Wut, C? ● HALP!!1!!! #include <Servo.h> #include <Wire.h> #include <Firmata.h> #define I2C_WRITE B00000000 #define I2C_READ B00001000 #define I2C_READ_CONTINUOUSLY B00010000 #define I2C_STOP_READING B00011000 #define I2C_READ_WRITE_MODE_MASK B00011000 #define I2C_10BIT_ADDRESS_MODE_MASK B00100000 #define I2C_MAX_QUERIES 8 #define I2C_REGISTER_NOT_SPECIFIED -1 #define MINIMUM_SAMPLING_INTERVAL 1 int analogInputsToReport = 0; byte previousPINs[TOTAL_PORTS]; byte pinConfig[TOTAL_PINS]; byte portConfigInputs[TOTAL_PORTS]; int pinState[TOTAL_PINS]; unsigned long currentMillis;
- 7. girlie_mac@ Raspberry Pi ● $35 single-board computer ● Broadcom chip with ARM-compatible CPU & GPU ● Runs Linux ● More language choices: C, C++, Python...
- 8. girlie_mac@ But… I want to code in
- 9. girlie_mac@ Johnny-Five ● JavaScript robotics framework ● Works with Arduino-compatible Boards ● IO plugins for more platform supports ● http://johnny-five.io/
- 10. girlie_mac@ Awww, JavaScript all the way!
- 12. girlie_mac@ KittyCam Raspberry Pi camera with cat facial detection! ● Hardware: Raspberry Pi, camera, and PIR sensor ● Software: Node.js + J5 + More open-source goodies
- 14. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!
- 15. girlie_mac@
- 16. girlie_mac@ Raspberry Pi 3 PIR Motion Sensor Camera Module F/F Jumper Wires (x3)
- 17. girlie_mac@
- 18. girlie_mac@
- 19. girlie_mac@ 1. Raspberry Pi 3 2. 5MP Camera Board Module 3. Pyroelectric Infrared (PIR) motion sensor 4. Female/Female wires
- 20. girlie_mac@ Programming Raspberry Pi Pre-installed on Raspbian OS: C / C++
- 21. girlie_mac@ Programming RPi with Node.js
- 22. girlie_mac@
- 23. girlie_mac@ Installing Node ARM $ wget https://nodejs.org/dist/v4.4.5/node-v4.4.5-l inux-armv7l.tar.xz $ tar -xvf node-v4.4.5-linux-armv7l.tar.xz $ cd node-v4.4.5-linux-armv7l $ sudo cp -R * /usr/local/
- 24. girlie_mac@ kittyCam.js 1. Detect motion 2. Take a photo 3. Cat facial detection 4. Store the photo in cloud 5. Real-time view on web 6. SMS the photo link HELL, YEAH!
- 25. girlie_mac@ kittyCam.js Tech Stack 1. Detect motion w/ Johnny-Five IR.Motion obj 2. Take a photo w/ Raspistill, command line tool 3. Cat facial detection w/ KittyDar 4. Store the photo in Cloudinary 5. Publish & subscribe the url to display on web via PubNub 6. Send a text message via Nexmo
- 26. girlie_mac@ $ Raspistill canvas catDetect.js app.js kittyDar Motion detected take a photo Store the photo if cats are detected display photos on web browsers real-time anywhere Johnny-Five w/ raspi-io child process Returns url Notify with SMS via
- 27. girlie_mac@ Johnny-Five w/ Raspi-io const five = require('johnny-five'); const raspi = require('raspi-io'); let board = new five.Board({io: new raspi()}); board.on('ready', () => { console.log('board is ready'); ... });
- 28. girlie_mac@ Motion const five = require('johnny-five'); const raspi = require('raspi-io'); const board = new five.Board({io: new raspi()}); board.on('ready', function() { // Create a new `motion` hardware instance const motion = new five.Motion('P1-7'); ... }); a PIR is wired on pin 7 (GPIO 4) VCC Ground Data
- 29. girlie_mac@ Raspistill Command Line Tool $ raspistill -o myPhoto.jpg
- 30. girlie_mac@ PIR Sensor > Run Camera const child_process = require('child_process'); board.on('ready', () => { const motion = new five.Motion('P1-7'); motion.on('motionstart', () => { // Motion detected let filename = 'photo/image_'+i+'.jpg'; let args = ['-w', '320', '-h', '240', '-o', filename, '-t', '1']; let spawn = child_process.spawn('raspistill', args); spawn.on('exit', function() { console.log('A photo is saved as '+filename); ... motion detected! Take a photo! Spawns a new process w/ a given shell command
- 31. girlie_mac@ Processing Image spawn.on('exit', () => { let imgPath = __dirname + '/' + filename; // Child process: read the file and detect cats with KittyDar let args = [imgPath]; let fork = child_process.fork(__dirname+'/detectCatsFromPhoto.js'); fork.send(args); // the child process is completed fork.on('message', (base64) => { if(base64) { uploadToCloud(base64); // Send to cloud storage } }); ... Create another worker by running a new instance of V8 engine.
- 32. girlie_mac@ detectCatsFromPhoto.js const kittydar = require('kittydar'); const Canvas = require('canvas'); process.on('message', (m) => { fs.readFile(m[0], (err, data) => { ... let canvas = new Canvas(w, h); let ctx = canvas.getContext('2d'); ctx.drawImage(img, 0, 0, w, h, 0, 0, w, h); let cats = kittydar.detectCats(canvas); console.log('There are', cats.length, 'cats in this photo'); if(cats.length > 0) {base64Img = canvas.toDataURL();} process.send(base64Img); process.exit(0); }); Running in an another process a cat detected! KittyDar
- 33. girlie_mac@ KittyDar ● Open-source JavaScript cat facial detection written by Heather Arthur ● Takes a canvas obj & calculate the locations of cats in the image let cats = kittydar.detectCats(canvas);
- 34. girlie_mac@ KittyDar: Behind the Scene 1. Chops up the image up into many “windows” 2. Extracts data by measuring a set of gradients, from light & dark in order to find edges 3. Compares the direction of these edges to the edges found in known cat images Neural network (JSON w/ vector data) is pre-trained w/ thousands pics of cats & non-cats
- 37. girlie_mac@ Cat Facial Detection The annotation data sequence: (Number of points, default is 9), (Left Eye), (Right Eye), (Mouth), (Left Ear-1), (Left Ear-2), (Left Ear-3), (Right Ear-1), (Right Ear-2), and (Right Ear-3)
- 38. girlie_mac@ Cat Facial Data 9 247 94 294 92 273 127 213 81 207 29 247 53 286 50 320 20 322 74 00000561_012.jpg.cat00000561_012.jpg
- 39. girlie_mac@ Cat Facial Data Collection
- 41. girlie_mac@ Send the Pic w/ Kitty to Cloud const cloudinary = require('cloudinary'); // the child process is completed fork.on('message', (base64) => { if(base64) { cloudinary.uploader.upload(base64, (result) => { // Done! - Get the URL and do more stuff }); } else deletePhoto(imgPath); });
- 42. girlie_mac@ View Photos Real-time via Socket const pubnub = require(pubnub); Publish Subscribe
- 44. girlie_mac@ Send SMS via Nexmo const Nexmo = require(nexmo); let nexmo = new Nexmo({//config with API keys}); nexmo.message.sendSms( FROM_NUMBER, TO_NUMBER, ' '+ url, options, (err, responseData) => { if (err) console.log(err); else console.dir(responseData); });
- 45. girlie_mac@
- 46. girlie_mac@ QA Team Lead QA: Jamie Ginger Basil Alice Yugi Venom @kittenVenom Ozzy
- 48. girlie_mac@ Next Steps ● Upgrade Hardware ○ Raspberry Pi 3 ○ NoIR Night Vision Camera ● Upgrade Node (was 0.12) & all dependencies ● ES5 to ES6 ● More features (Maybe) ○ Cat Identification w/ RFID ○ Photo Booth w/ Filter effects & props
- 49. girlie_mac@ More Cat Projects w/ Node.js
- 50. girlie_mac@ https://http.cat Thanks Rogério Vicente for the .cat domain & the API! Previously: HTTP Status Cats
- 51. girlie_mac@ HTTP Status Cats Slack Bot Install it at: http://girliemac.com/ slack-httpstatuscats/ Tutorial on Medium.com: https://goo.gl/saOUl0
- 52. @girlie_mac Next Project? ● Selfie bot (à la Mannie the Selfie Cat) Mannie the Selfie Cat by @yoremahm on Instagram https://www.instagram.com/yoremahm/
- 54. girlie_mac@ Attribution: Emoji by Emoji-One (CC-BY 4.0)