Cold front - bridging the web and the physical world
0 likes131 views
The document discusses the integration of smart devices with web technologies, emphasizing low friction connectivity through progressive web apps, Bluetooth, and USB. It highlights the advantages and limitations of existing technologies like Bluetooth Low Energy and NFC for device interaction and data exchange, while addressing the need for user-friendly experiences. The author encourages building the future of web interactivity with these advancements, focusing on usability and accessibility.
![Requesting device
navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] })
.then(device => { /* ... */ })
.catch(error => { console.log(error); });
navigator.bluetooth.requestDevice({
filters: [{ services: [0x1234, 0x12345678, '99999999-0000-1000-8000-00805f9b34fb'] }]
})
.then(device => { /* ... */ })
.catch(error => { console.log(error); });
You may provide either the full Bluetooth UUID or a short 16- or 32-bit form, if not a standardized service
Standardized services are easy to connect to](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-52-320.jpg)
![Requesting device
navigator.bluetooth.requestDevice({
filters: [{ name: "Kenneth's robot" }],
optionalServices: ['battery_service']
})
.then(device => { /* ... */ })
.catch(error => { console.log(error); });
You can also search by advertized name. Services can be optional, but needs to be listed if you want to access them when
available](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-53-320.jpg)
![Connect to a device
navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] })
.then(device => {
// Human-readable name of the device.
console.log(device.name);
// Attempts to connect to remote GATT Server.
return device.gatt.connect();
})
.then(server => { /* ... */ })
.catch(error => { console.log(error); });
When you have a device, you can connect to the GATT (Generic Attributes) server from where you can access
Characteristics](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-56-320.jpg)
![API example
let device;
navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] })
.then(selectedDevice => {
device = selectedDevice;
return device.open(); // Begin a session.
})
.then(() => device.selectConfiguration(1)) // Select configuration #1 for the device.
.then(() => device.claimInterface(2)) // Request exclusive control over interface #2.
.then(() => device.controlTransferOut({
requestType: 'class',
recipient: 'interface',
request: 0x22,
value: 0x01,
index: 0x02
})) // Ready to receive data
.then(() => device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5.
.then(result => {
let decoder = new TextDecoder();
console.log('Received: ' + decoder.decode(result.data));
})
.catch(error => { console.log(error); });
In many ways similar in spirit to Web Bluetooth, but of course USB specific
try {
let device = await navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] })
await device.open(); // Begin a session.
await device.selectConfiguration(1)) // Select configuration #1 for the device.
await device.claimInterface(2)) // Request exclusive control over interface #2.
await device.controlTransferOut({
requestType: 'class',
recipient: 'interface',
request: 0x22,
value: 0x01,
index: 0x02
});
// Ready to receive data
let result = device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5.
let decoder = new TextDecoder();
console.log('Received: ' + decoder.decode(result.data));
}
catch(error) {
console.log(error);
}](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-81-320.jpg)
![NFC reader/writer
The Web NFC API covers reading from and writing to NFC tags: https://w3c.github.io/web-nfc/
try {
await navigator.nfc.push({
data: [{ recordType: "url", data: "https://w3c.github.io/web-nfc/" }]
});
console.log("Message pushed.");
} catch(err) {
console.log("Push failed :-( try again.");
}
Behind a flag
In Chrome](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-91-320.jpg)
![NFC reader/writer
navigator.nfc.watch(message => {
if (message.data[0].recordType == 'empty') {
navigator.nfc.push({
data: [{
recordType: "json",
mediaType: "application/json",
data: { firstName: 'Kenneth' }
}]
});
} else {
console.log(`Read msg written by ${message.url}`;
processMessage(message);
}
});
Behind a flag
In Chrome
function processMessage(message) {
for (let d of message.data) {
if (d.recordType == "json") {
console.log(`Hello ${record.data.firstName}!`);
}
if (d.recordType == "opaque" && d.mediaType == 'image/png') {
const blob = new Blob(d.data, d.mediaType);
const img = document.createElement("img");
img.src = URL.createObjectURL(blob);
img.onload = _ => URL.revokeObjectURL(img.src);
document.querySelector('images').appendChild(img);
}
}
}
}](https://image.slidesharecdn.com/coldfront-bridgingthewebandthephysicalworld2-180313132606/85/Cold-front-bridging-the-web-and-the-physical-world-92-320.jpg)
Cold front - bridging the web and the physical world
- 1. Bridging the web and the physical world Kenneth Rohde Christiansen Web Platform Architect
- 2. Hi! I am Kenneth
- 3. Danish - happiest people in the world second most
- 8. What makes a thing "smart"?
- 9. Devices which are complicated or slow to use, are they really smart?
- 10. ➫ Solve a real issue ➫ Be there when you need them ➫ Ease to use, with low friction
- 12. I don't want to download your app, I just want to do stuff
- 13. Now! Now! Nowww!
- 14. With many devices around usapps don't scale well
- 15. Too many icons! Lack of overview Lots of updates pending for stuff I never use!
- 16. Too many icons! ➫ Lack of overview ➫ Updates pending for stuff I never use!
- 17. Let's look at usages
- 18. There are different common usages - Everyday use turn on the light! - Occasionally change temperature of my fridge - Once paying for parking when travelling
- 19. Better model?
- 20. We want low friction and ease of use
- 23. Focus on speed HTTP 2.0 push, Service Workers, PRPL pattern, streams, scrolling, CSS Houdini, Compositing …
- 24. Fast with Service Worker caching
- 26. The web is up to task
- 29. But that can't be web?! No address bar - super fast - icon on homescreen
- 30. We call that experience for Progressive Web Apps https://developers.google.com/web/progressive-web-apps/
- 31. Great for one-time use The low friction of the web
- 32. Always handy if I want it to be Add to homescreen
- 33. No clutter Looks and acts like an app
- 34. And safe
- 35. But... Copyright Club Skirts Dinah Shore Weekend https://commons.wikimedia.org/wiki/File:White_Diamonds_Party.jpg
- 36. Devices live outside the sandbox
- 37. And yes, the web can do that too!
- 38. Keep in mind Users and companies don't care about how stuff works, it should just work
- 40. But first, it might make sense to listen You can actually win stuff!
- 41. Lars and Coldfront has joined up for a quiz! More on that later! WebLight
- 42. Advantages Widely used, Low energy, Wireless
- 43. Advantages Cheap, Easy to use, Beacon functionality
- 44. Disadvantages Security (just radio waves), Not powered, Stability, Speed
- 47. ➫ Based on Bluetooth 4+ (Low Energy) GATT protocol ➫ Easy to use, modern web API ➫ Uses promises and typed arrays (ie. Uint8Array etc) ➫ No Classic mode support ➫ Only Central, not Peripheral Chrome only for now
- 49. Drawing by Noa Lee Shaked
- 50. An example HTTPS only User Gesture required to request device
- 52. Requesting device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); navigator.bluetooth.requestDevice({ filters: [{ services: [0x1234, 0x12345678, '99999999-0000-1000-8000-00805f9b34fb'] }] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You may provide either the full Bluetooth UUID or a short 16- or 32-bit form, if not a standardized service Standardized services are easy to connect to
- 53. Requesting device navigator.bluetooth.requestDevice({ filters: [{ name: "Kenneth's robot" }], optionalServices: ['battery_service'] }) .then(device => { /* ... */ }) .catch(error => { console.log(error); }); You can also search by advertized name. Services can be optional, but needs to be listed if you want to access them when available
- 54. BLE Services and Characteristics GATT (Generic Attribute profile) defines how two BLE devices transfer data back and forth using concepts known as Services and Characteristics ➫ Services are logical entities, and contain specific chunks of data, known as Characteristics ➫ Characteristics encapsulate a single data point Services and Characterics are identified by 16- or 128 bit UUID's Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
- 55. BLE Services and Characteristics In JavaScript you can think of a Service being something like an object - ie. a collection of properties Characteristics are similar to properties on an object Characteristic Characteristic Characteristic Characteristic Characteristic Service Service Profile
- 56. Connect to a device navigator.bluetooth.requestDevice({ filters: [{ services: ['battery_service'] }] }) .then(device => { // Human-readable name of the device. console.log(device.name); // Attempts to connect to remote GATT Server. return device.gatt.connect(); }) .then(server => { /* ... */ }) .catch(error => { console.log(error); }); When you have a device, you can connect to the GATT (Generic Attributes) server from where you can access Characteristics
- 57. Deal with Characteristics .then(service => { // Getting Battery Level Characteristic... return service.getCharacteristic('battery_level'); }) .then(characteristic => { // Reading Battery Level... return characteristic.readValue(); }) .then(value => { console.log('Battery percentage is ' + value.getUint8(0)); }) readValue() returns a DataView (part of the Typed Array family), so it is easy to extract values from.
- 58. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_control_point')) .then(characteristic => { // Writing 1 is the signal to reset energy expended. const resetEnergyExpended = Uint8Array.of(1); return characteristic.writeValue(resetEnergyExpended); }) .then(_ => { console.log('Energy expended has been reset.'); }) Writing is easy too!
- 59. Deal with Characteristics .then(service => service.getCharacteristic('heart_rate_measurement')) .then(characteristic => characteristic.startNotifications()) .then(characteristic => { characteristic.addEventListener('characteristicvaluechanged', handleCharacteristicValueChanged); console.log('Notifications have been started.'); }) .catch(error => { console.log(error); }); function handleCharacteristicValueChanged(event) { var value = event.target.value; console.log('Received ' + value); // TODO: Parse Heart Rate Measurement value. // See https://github.com/WebBluetoothCG/demos/blob/gh-pages/heart-rate-sensor/heartRateSensor.js } GATT notifications - subscribe to Characteristics changes
- 60. Bootstrapping - aka. how do people find my site from my device
- 61. Bluetooth devices can advertize web sites around them! That is Eddystone
- 62. Shows as notifications Click, and they open
- 63. Shows as notifications Click, and they open Can be regular sites No additional Bluetooth functionality needed
- 64. In near future Clicking on a Physical Web notification*, will allow the opened page to auto connect to device *Eddystone URL
- 65. Get started today! (or tomorrow) https://developers.google.com/web/updates/2015/07/ interact-with-ble-devices-on-the-web
- 66. Advantages Widely in use, Cheap, Cabled, More Secure, Fast, Stable, Can power devices
- 68. Safe and secure USB has no radio interference USB connector can be locked behind a hatch
- 69. Recall Users and companies don't care about how stuff works - it should just work Bluetooth and USB coexists just fine!
- 71. Nice example showing ➫ A device capable of running JS! ➫ Bootstrapping and auto connect ➫ Device being powered over USB
- 72. Three different transfer types Bulk, Interrupt, Isochronous
- 73. Bulk Bulk transfers are the fastest* but have no guaranteed timing Printers and most CDC** (e.g serial) use bulk transfers. * Bulk transfers will use spare un-allocated bandwidth on the bus after all other transactions have been allocated. If the bus is busy with isochronous and/or interrupt then bulk data may slowly trickle over the bus. ** Communications Device Class
- 74. Interrupt Interrupt transfers have guaranteed maximum latency, or time between transaction attempts Mice and keyboards use interrupt transfers.
- 75. Isochronous Isochronous transfers have guaranteed timing but no error correcting Streaming audio and video use isochronous transfers
- 76. Web USB specific headers Creates further security and restrictions
- 77. Web USB specific headers ➫ Optional ➫ Required for bootstrapping popup ➫ Restrict which sites can access the peripheral
- 79. Issues The Linux modemmanager (enabled by default on most distros) hijacks (claims for 16sec) CDC devices unless their VID/PID are blacklisted (like the Arduino Leonardo) Note: Only affects CDC devices
- 80. Issues Windows autoloads drivers for a range of classes, e.g. HID, MSD and CDC (Win 10) Other Web USB devices additionally requires MS OS descriptors in order to autoload (or INF file, see below) Earlier versions (< 10) of Windows, requires a signed "driver", which is basically a signed INF text file, binding the VIP/PID to usbser.sys (the default USB serial driver) More info: https://medium.com/@larsgk/web-enabling-legacy-devices-dc3ecb9400ed#.fj7bd1ba3
- 81. API example let device; navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] }) .then(selectedDevice => { device = selectedDevice; return device.open(); // Begin a session. }) .then(() => device.selectConfiguration(1)) // Select configuration #1 for the device. .then(() => device.claimInterface(2)) // Request exclusive control over interface #2. .then(() => device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 })) // Ready to receive data .then(() => device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. .then(result => { let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); }) .catch(error => { console.log(error); }); In many ways similar in spirit to Web Bluetooth, but of course USB specific try { let device = await navigator.usb.requestDevice({ filters: [{ vendorId: 0x2341 }] }) await device.open(); // Begin a session. await device.selectConfiguration(1)) // Select configuration #1 for the device. await device.claimInterface(2)) // Request exclusive control over interface #2. await device.controlTransferOut({ requestType: 'class', recipient: 'interface', request: 0x22, value: 0x01, index: 0x02 }); // Ready to receive data let result = device.transferIn(5, 64)) // Waiting for 64 bytes of data from endpoint #5. let decoder = new TextDecoder(); console.log('Received: ' + decoder.decode(result.data)); } catch(error) { console.log(error); }
- 82. Availability Works in Chrome today on Linux, Mac and Windows
- 84. Wow! But there is more! Photo by Andrés Þór
- 85. Many companies often need to track stuff
- 86. Bar codes and similar tech ➫ Quickly read and track things and products ➫ Get information about things around you ➫ Quick actions
- 87. Bluetooth Beacons might be the best solution
- 88. Or even NFC tags!
- 89. Or Quick Response codes and barcodes + they are cheap and printable
- 90. Barcode reader The Shape Detection API covering barcodes, faces and more: https://wicg.github.io/shape-detection-api/ try { let barcodeDetector = new BarcodeDetector(); // Assuming |theImage| is e.g. a <img> content, or a Blob. let detectedCodes = await barcodeDetector.detect(theImage); for (const barcode of detectedCodes) { const box = barcode.boundingBox; console.log(`Barcode ${barcode.rawValue} @(${box.x}, ${box.y}) with size ${box.width}x${box.height}`); } } catch(err) { console.error(`Barcode Detection failed, boo: ${err}`); } Behind a flag In Chrome
- 91. NFC reader/writer The Web NFC API covers reading from and writing to NFC tags: https://w3c.github.io/web-nfc/ try { await navigator.nfc.push({ data: [{ recordType: "url", data: "https://w3c.github.io/web-nfc/" }] }); console.log("Message pushed."); } catch(err) { console.log("Push failed :-( try again."); } Behind a flag In Chrome
- 92. NFC reader/writer navigator.nfc.watch(message => { if (message.data[0].recordType == 'empty') { navigator.nfc.push({ data: [{ recordType: "json", mediaType: "application/json", data: { firstName: 'Kenneth' } }] }); } else { console.log(`Read msg written by ${message.url}`; processMessage(message); } }); Behind a flag In Chrome function processMessage(message) { for (let d of message.data) { if (d.recordType == "json") { console.log(`Hello ${record.data.firstName}!`); } if (d.recordType == "opaque" && d.mediaType == 'image/png') { const blob = new Blob(d.data, d.mediaType); const img = document.createElement("img"); img.src = URL.createObjectURL(blob); img.onload = _ => URL.revokeObjectURL(img.src); document.querySelector('images').appendChild(img); } } } }
- 93. Pretty cool, even Sir Tim Berners-Lee is excited
- 95. But what about Connecting to other devices to control or receive data
- 97. But my already has sensors! Gimme gimme access pretty pretty please with sugar on top!
- 98. Could an API work for Local and Remote sensors? Web Platform, Node, IoT?
- 100. Find sensors hard? Read my motion sensors explainer https://w3c.github.io/motion-sensors/ Gravity Sensor Absolute Orientation Sensor Linear Acceleration Sensor Geomagnetic Orientation Sensor Low pass filter Sensor fusion
- 101. Very easy to use API Low-level, so you can do your own filters and sensor fusion const accel = new Accelerometer({ frequency: 50 }); const gyro = new Gyroscope({ frequency: 50 }); gyro.onreading = _ => { … } accel.start(); gyro.start(); Behind a flag In Chrome
- 102. let timestamp = null; gyro.onreading = _ => { let norm = Math.sqrt(accel.x ** 2 + accel.y ** 2) + accel.z ** 2); let xAccel = (accel.x / norm) * 90; let yAccel = (accel.y / norm) * -90; let zAccel = (accel.z / norm); let xGyro = gyro.x * 180 / Math.PI; let yGyro = gyro.y * 180 / Math.PI; let zGyro = gyro.z * 180 / Math.PI; let dt = timestamp ? ((gyro.timestamp - this.timestamp) / 1000) : 0; timestamp = gyro.timestamp; // Complementary filter // E.g.: new angle = 98% * (current angle + gyro rotation rate) + (2% * Accelerometer angle) this.alpha = weight * (this.alpha + zGyro * dt) + (1.0 - weight) * zAccel; this.beta = weight * (this.beta + xGyro * dt) + (1.0 - weight) * xAccel; this.gamma = weight * (this.gamma + yGyro * dt) + (1.0 - weight) * yAccel; console.log(`(${this.alpha}, ${this.beta}, ${this.gamma})`); };
- 103. Let's look at a video! ➫ Daydream controller exposed via Web Bluetooth ➫ Exposed as Generic Sensors ➫ Try it https://sensor-compass.appspot.com/
- 107. That is quite tiny
- 112. Summary The web is ready for nice app like experiences ⬤ ⬤ ⬤ ⬤
- 113. Summary You can effortless connect to devices over Bluetooth and USB Chrome for now, but expect other browser vendors to follow suit ⬤ ⬤ ⬤ ⬤
- 114. Summary NFC support and barcode readers are coming too! ⬤ ⬤ ⬤ ⬤
- 115. Summary Generic Sensors exposes sensors to the web Plus the API is easy to replace with custom implementation, say, via Web Bluetooth ⬤ ⬤ ⬤ ⬤
- 116. The web is ready, let's build the future
- 117. Follow me on Twitter, where I will post these slides today @kennethrohde