Building a Thought Controlled Drone
35 likes5,433 views
Jim McKeeth presented on the development of a thought-controlled drone at the InterDrone Conference in 2015, discussing technical details about various drones and brain-computer interface (BCI) headsets. He outlined the capabilities of the Parrot AR.Drone 2.0 and Bebop drones, as well as EEG headset options from different manufacturers for measuring brain activity. The presentation emphasized the integration of BCI technology in controlling drones and the potential future applications in autonomous systems and user interfaces.
![Command Strings
• Connect via WiFi access point to 192.168.1.1 UDP port 5556
• Command string format: AT*[cmd]=[seq],[arg array]
• Sequence number must not repeat
• Argument array is comma separated typically
• Floating point numbers must be encoded as decimals (more later)
• Send command every 30 ms (>2 seconds = session dropped)](https://image.slidesharecdn.com/qf4hoygbsloulpegt4yk-signature-e989c4db3e68839b4d905cea027c8e5514a9354dc2f9eaa4b1d8e72239d31887-poli-150311142644-conversion-gate01/85/Building-a-Thought-Controlled-Drone-30-320.jpg)
![Command String Examples
• FlatTrim: AT*FTROM=[seq]
• Movement: AT*PCMD=[seq],1,Roll, Pitch, Vert, Yaw
• Values of 1.0 to -1.0
• Use 0 in all values to hover
• Takeoff: AT*REF=[seq],290718208
• Land: AT*REF=[seq],290717696
• Emergency: AT*REF=[seq],290717952](https://image.slidesharecdn.com/qf4hoygbsloulpegt4yk-signature-e989c4db3e68839b4d905cea027c8e5514a9354dc2f9eaa4b1d8e72239d31887-poli-150311142644-conversion-gate01/85/Building-a-Thought-Controlled-Drone-31-320.jpg)
Building a Thought Controlled Drone
- 1. Building a Thought-Controlled Drone Jim McKeeth, Embarcadero Technologies jim.mckeeth@embarcadero.com @JimMcKeeth Presented at InterDrone Conference Sept 9th, 2015
- 2. About Jim McKeeth • Lead World Wide Developer Evangelist • Longtime developer • Object Pascal, Java, C#, JavaScript, Objective-C, etc. • Invented and patented swipe to unlock in 2000 • US Patent # 8352745 & 6766456, etc. • Improvisational performer with ComedySportz Boise • I am not a neuroscientist, but I find this all fascinating!
- 3. • Founded 1993 • Provides Architect, Database and Developer Tools • 3.2 Million Customers including 97% of Fortune 2000 • 35+ Product Awards – Consistent Innovation • 500+ Employees in 29 Countries • www.embarcadero.com
- 4. Agenda • Drone technical details • Headset technical details • Drone API • Headset SDK • Concerns, Challenges and the Future
- 5. More Information • My email: jim.mckeeth@embarcadero.com • My blog: delphi.org • This Slide Deck http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone • Overview Slide Deck http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input • The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC
- 6. “In five years, science fiction will be real!” — Claus Torp Jensen (@ClausTorpJensen) Senior Technical Staff Member and Chief Architect API Economy at IBM (@ibmapimgt) What will the world be like in five years?
- 7. The Pieces Emotiv EPOC Brain-Computer Interface Parrot.AR Drone 2 Simi-Autonomous Connected App Platform
- 9. The Drone: Parrot AR.Drone 2.0 • Price: $299.99 • Runs Linux internally, but not user accessible • Contains it’s own WiFi access point • Connect via UDP & TCP • Send control strings • Maintain transmission rate to keep alive • Can also receive status and video back
- 10. Completely Rebuildable HD Battery $60 Gears+Shafts $15 Tools $20 Cross bars $30 Screws $7 Propellers $12 4x Engines $200 Bottom structure $60 Main board $110 Nav board $70 Charger $40 Outdoor haul $25 ———————— Total $650
- 11. New contender: Parrot Bebop • Price: $499.99 • Lots of upgrades, especially camera • Also contains own WiFi access point • Whole new SDK • iOS, Android & Linux • Published Nov 2014 • Incompatible with AR.Drone 2.0
- 12. SDK vs. API • Parrot AR.Drone 2.0 provides both an API and SDK (Bebop has just an SDK at this point.) • An SDK is a software library that does additional work locally - makes it easier to handle video streaming, etc. • An API is a specification of how to connect and control. • SDK works with specific tools and platforms. • Typically SDK uses API. • My examples use the API so it works across platforms.
- 13. Brain Computer Interface (BCI) Headset Options
- 14. Electroencephalography (EEG) • Measures electrical activity along the scalp. • Detects voltage fluctuations resulting from ionic current flows within neurons of brain. Image source: https://commons.wikimedia.org/wiki/File:EEG_cap.jpg
- 15. Neural Oscillation / Brain WavesBeta 15-30Hz Waking Consciousness and Reasoning Alpha 7-14Hz Deep Relaxation Delta 4Hz Deep Sleep Theta 4-7Hz Light Meditation And Sleeping Mu 8-13Hz Voluntary Movement Gamma 30-100Hz Conscious Perception Waveforms by Hugo Gamboa
- 16. NeuroSky’s MindWave (also in XWave headsets) • EEG with 1 dry electrode • Measures “attention,” “meditation” and eye-blinks • Bluetooth communication • SDK for iOS, Android, PC & Mac • $129.99 to $79.95 retail neurosky.com • Hacking: http://makezine.com/2011/05/10/behind- the-brain-blinker/
- 17. InteraXon’s Muse • 7 dry EEG sensors via Bluetooth • Focuses on “brain fitness” • 4 hour battery • Android, Mac & Windows SDK now, iOS coming soon • Works with Windows, OS X, iOS, Android, etc. • $299 at choosemuse.com
- 18. OpenBCI • Open source hardware & software • Bluetooth, Arduino, EEG • 8 wet Electrodes per board (daisy chainable) • Full open EEG & BCI platform • SDK provides EEG data • www.openbci.com starting at $449 • No iOS or Android yet
- 19. Emotiv EPOC • 16 wet electrodes • 14 EEG electrodes + 2 reference electrodes • Gyroscope to read head movements • 4 mental states, 13 conscious thoughts & facial expressions – 4 Processing suites • Wireless (encrypted) for Windows, Linux or Mac • $399 for Desktop • $499 for Bluetooth Smart (EPOC+) • www.emotiv.com
- 20. Emotiv Insight • Expected eventually • Starting at $299 (estimate) • 5 dry sensors + 2 reference • Bluetooth 4.0 LE (Smart) • Battery Life: 4+ hours • SDK: Android, iOS, Mac, Linux and Windows Platforms • www.emotiv.com • Comparison http://bit.ly/1cfliTn
- 21. Emotiv EPOC SDK Overview
- 22. Expressiv Suite • Detect facial expressions • Eyelid & eyebrow positions • Wink, blink, furrow • Horizontal eye movement • Smile, laugh, clenching & smirking • EEG sensors picking up signals to muscles (not brain waves) • Very fast (10ms) Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
- 23. Uses for Facial Detection • Anything requiring fast input • Facial gesture detection • Basic eye tracking • Fast input & responses Image Source: https://commons.wikimedia.org/wiki/File:Duchenne-FacialExpressions.jpg
- 24. Affectiv Suite • Detect emotions / mental states • Excitement • Engagement or Boredom • Meditation • Frustration • Passive detection • Variable levels of each • Short & long term tracking Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
- 25. Uses for Emotion Tracking • Realtime media evaluation • Stress analysis and management • Mood monitoring • For more accuracy combine with other data sources • Eye tracking, heart rate, etc. Image Source: https://commons.wikimedia.org/wiki/File:Drama-icon.svg
- 26. Cognitiv Suite • Detect conscious thoughts • Requires training • Detect 13 thought patterns with intensity level (4 at a time) • Left, right, up, down, forward, pull, CW, CCW, left, right, sway-back, sway-forward, Disappear • Not as fast as Expresiv Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
- 27. Uses for Cognitive Thought Identification • Whole new input method • Variable intensity for fine control • Observe subconscious responses • More training improves accuracy • Improves the model • Trains your brain to focus Image Source: https://commons.wikimedia.org/wiki/File:ThinkingMan_Rodin.jpg
- 28. Brain-Computer Interface Internet of Things Autonomous Vehicles expectations Speech-to-Speech Translation time Plateau of Productivity Slope of EnlightenmentTrough of Disillusionment Peak of Inflated Expectations Innovation Trigger Plateau will be reached in: more than 10 years5 to 10 years2 to 5 yearsless than 2 years obsolete before plateau Gartner's 2014 Hype Cycle for Emerging Technologies August 11, 2014 Smart Advisors Data Science Prescriptive Analytics Neurobusiness Biochips Affective Computing Smart Robots 3D Bioprinting Systems Volumetric and Holographic Displays Software-Defined Anything Quantum Computing Human Augmentation Connected Home Quantified Self Smart WorkspaceVirtual Personal Assistants Digital Security Bioacoustic Sensing Natural-Language Question Answering Consumer 3D Printing Cryptocurrencies Complex-Event Processing Big Data In-Memory Database Management Content Analytics Hybrid Cloud Computing Gamification Augmented Reality Machine-to-Machine Communication Services Mobile Health Monitoring Cloud Computing NFC Virtual Reality Gesture Control In-Memory Analytics Activity Streams Enterprise 3D Printing 3D Scanners Consumer Telematics Speech Recognition Wearable User Interfaces
- 29. Using the AR.Drone API
- 30. Command Strings • Connect via WiFi access point to 192.168.1.1 UDP port 5556 • Command string format: AT*[cmd]=[seq],[arg array] • Sequence number must not repeat • Argument array is comma separated typically • Floating point numbers must be encoded as decimals (more later) • Send command every 30 ms (>2 seconds = session dropped)
- 31. Command String Examples • FlatTrim: AT*FTROM=[seq] • Movement: AT*PCMD=[seq],1,Roll, Pitch, Vert, Yaw • Values of 1.0 to -1.0 • Use 0 in all values to hover • Takeoff: AT*REF=[seq],290718208 • Land: AT*REF=[seq],290717696 • Emergency: AT*REF=[seq],290717952
- 32. Float as Integer • Single precision floating point number is 32 bits • Integer is 32 bits • Map IEEE-754 bit value from float to integer • -0.8 = 0xBF4CCCCD = 10000001011001100110011001 = 1085485875
- 33. Drone Concerns • Manual override - keyboard override BCI, just in case • Send emergency command to clear error state • Start with FlatTrims to level drone • Use state machine to send command 30 times a second 1. Keyboard override 2. BCI Input 3. Hover
- 34. ToDo with Drone • Process NavData (sent back on UDP port 5554) • Get battery data to know remaining flight time, errors, etc. • Process Video stream (sent back on UDP port 5555) • Display on Google Glass for HUD • Find use for drone “Animations” like flips • Maybe use LED animations when receiving BCI input
- 35. Using Emotiv EDK
- 36. SDK Format • On Windows a Visual C++ DLL provides the interface • Examples in VC++, C#.NET, Java, Python & Matlab • All my code is in Object Pascal • On OS X a dylib provides the interface • Examples in Qt & Xcode
- 37. Emotiv EDK Workflow • Call EE_EngineConnect to connect • Create event handler EE_EmoEngineEventCreate • Create state structure EE_EmoStateCreate • Define actions to listen for and then train those actions (more later) • Every 10 ms call EE_EngineGetNextEvent (more later)
- 38. Handle Cognitive Actions • If state is EDK_OK then call EE_EmoEngineEventGetType • If EventType is not EE_EmulatorError then call EE_EmoEngineEventGetUserId • If EventType is EE_CognitivEvent then it was a training event (next slide) • If EventType is EE_EmoStateUpdated then get EE_EmoEngineEventGetEmoState • Read ES_CognitivGetCurrentAction and ES_CognitivGetCurrentActionPower
- 39. Listen & Train • Listening: • Call EE_CognitivSetActiveActions with mask of Actions • Call EE_CognitivSetTrainingAction with the ID of the action to train • Call EE_CognitivSetTrainingControl with COG_START • On EE_CognitivEventGetType of EE_CognitivTrainingSucceeded call EE_CognitivSetTrainingControl with COG_ACCEPT
- 40. New Emotiv Xavier SDK
- 41. New Emotiv Xavier SDK • EPOC+ and Insight are Bluetooth Smart • Connect to mobile devices - iOS & Android • Allows to additional user definable cognitive actions • Still only listens for 4 cognitive actions at a time
- 43. Thoughts to Movement • Moves while thought active, otherwise hover in position • Speed of movement = intensity of thought • How to allow more than 4 directions of movement? • Use Expressiv facial gesture to switch between sets • Future consideration: Use strobing grid of controls
- 44. Typing via BCI Jonathan Wolpaw Wadsworth Center Albany, NY ~15 seconds / key . . . . . . . . . A B C D E F . . G H I J K L . . M N O P Q R . . S T U V W X . . Y Z 1 2 3 4 . . 5 6 7 8 9 0 . . . . . . . . .
- 46. Purpose of Feedback • Using a visual feedback gives the wearer something to focus on • Otherwise they often change their thought pattern when the drone is moving • Also gives them immediate awareness of what is picked up
- 47. Moverio BT-200 Product Overview Integrated Sensors Accelerometer Compass Gyro Head Set Camera Earbuds with Microphone +x +y +z
- 48. Epson Moverio BT-200 Specs Epson BT-200 Retail Price $699 Display Type Transparent Resolution 960×540 QHD Android v4.0.4 API Level 15 - Ice Cream Sandwich Camera VGA, 640x480 WLAN 802.11b/g/n 2.4GHz Bluetooth 3.0 micro USB v2.0 (On-The-Go) GPS P-GPS CPU TI OMAP 4460 (ARM Cortex A9 Dual Core 1.2GHz) RAM 1GB Flash 8GB microSD Max 32GB
- 49. Laptop EDK My App Data Flow Electrical Signals Wireless Signal AT*REF=1,290718208 Provides WiFi Hotspot Feedback
- 50. Q & A
- 51. More Information • My email: jim.mckeeth@embarcadero.com • My blog: delphi.org • This Slide Deck http://www.slideshare.net/jimmckeeth/build-brain-controlled-drone • Overview Slide Deck http://www.slideshare.net/jimmckeeth/jim-mc-keeth-wearable-thought-input • The Code: https://github.com/jimmckeeth/Delphi-Emotiv-EPOC