What wearable data means for clinical researchers

By: Dr. Sam Volchenboum

Dr. Sam Volchenboum is the director of the Center for Research Informatics at the University of Chicago, a board-certified pediatric hematologist and oncologist, and the cofounder of Litmus Health, a data science platform for early-stage clinical trials.

. . .

Earlier this year, FitBit released findings from 6 billion hours worth of sleep data. It will eventually be shared in its raw form with scientific institutions and in science journals.

The release comes a year and a half after Fitbit faced a class-action lawsuit regarding the accuracy of their heart-rate data, which have been shown to be inaccurate by a margin of up to 20 beats per minute by researchers at California State Polytechnic University, Pomona.

At its release, we were asked about the lawsuit nearly every day for more than a month. It risked sending many back to the drawing board who had been optimistically experimenting with biotelemetry.

As tools for collecting research data, there have always been concerns about the utility of the data that wearables, sensors and monitors provide. Here we seek to present a balanced view of the state of these and other related issues, and, ultimately, chart a viable path forward.

The challenge when the lawsuit emerged, and still now, is that we must properly plan for and accommodate variance in remotely collected data, which we argue is a standard and well-known problem for any research data, whether it is collected in the hospital, clinic or at home.

Seeking answers

Many investigators and their sponsors consider it obvious that the information needed to create new therapeutic solutions for recalcitrant clinical problems is everywhere around us.

“We’ve got to stop leaving so much of our patients’ data on the table,” Dr. Sam Blackman of Juno Therapeutics has argued. “Quantifying patients’ health outside the clinic is essential to faster, more efficient pipelines.”

While we couldn’t agree more, few of these hardware devices are FDA-approved, and despite their proven widespread appeal with consumers, scientific validation is often difficult, expensive and often intractable.

The admissibility of these data is similarly in question, with data provenance front-and-center. For example, simple, consistent timestamp and time series protocols do not currently exist in ways that would suffice for a Title 21 CFR Part 11 audit, much less formal claims-making to the FDA. And that’s just the beginning.

A call for standards

Market education for consumer-device manufacturers remains a priority. In a recent conversation, we were astonished to find how low-resolution their basic understanding is of the requirements for the collection of data in clinical trials.

We don’t fault this particular company, or any other. As we want better devices, we must do a better job of telling manufacturers what kinds of measurements and outputs we need. It is critical that the research community develop, promote and use standards for data collection, storage and reporting that can be easily understood and adopted by everyone, including non-experts.

The Institute of Electrical and Electronics Engineers (IEEE) is with us on this point. Standards are essential for these new data sources to be viable in the long term. Importantly, the FDA has mandated all electronic data be submitted using standards, such as those mandated by CDISC. This is the administration’s official guidance, and the looming deadline is causing a gratifying flurry of activity.

Standards and data provenance aren’t sexy, but they are absolutely essential to any compelling future vision of clinical research.

We expect pharma to invest primarily in later-stage data transformations instead of applying these standards throughout their data collection and storage pipelines — which will be big business for companies like Capsenta, which can make quick wine out of dirty water.

But over time, there will be a trickle-down effect, and we’ll start to see standards as a way of life, used throughout every pipeline, at every opportunity. It will become easier and easier to credibly innovate using novel data.

CDISC, if you’re not familiar, is a global nonprofit dedicated to the creation and support of standards for clinical research data. While the big-picture electronic data reporting standards are very mature and becoming more widely adopted, the mobile device components remain under development.

We are under pressure to get it correct right out of the gate. Anything less will lead to a hampering of innovation and continued dependence on old, outdated methods of collection, storage and transmission.

As we see it, the world will be a better place if we never see another patient in a clinic lobby furiously trying to remember several weeks’ worth of information while they scribble their data onto a clipboard form.

Next-generation devices

A better breed of purpose-built, less-variant, yet thoroughly modern devices are coming to market.

The Google X’s Baseline Study project and Alphabet’s Art Levinson-led longevity play, Calico, offer motivation enough for the internet giant to build a better remote monitor. Surely the Google team surveyed what’s available today off-the-shelf and found their options markedly wanting. We’re not at all surprised that Google decided to take the DIY approach.

But Google’s clinical wearable is, to be clear, just a means to a greater end. Without a way to collect data at the point of experience, any large-scale study of disease is confined to outcomes that patients themselves remember weeks or months after the fact, placed alongside traditional clinician-observed realities in-clinic. Those two sources of information no longer suffice — not just for Google, but for all of us.

Very soon we won’t have to choose between usability and accuracy.

We believe there is a confluence of factors — increasing attention to quality-of-life, the demand for personalized therapies, the growing need for lean clinical trials — that the group or company that becomes facile at measuring, interpreting and ultimately creating quality-of-life metrics will reap enormous financial benefits. There is no doubt that this motive is at least partially driving Google’s business decisions in this area.

We’ll continue to see new clinically focused entrants from usual and unusual suspects alike.

Right now there is a huge technical and experience gap between modern, alluring consumer-centric devices and sensors and their hardened brethren on the traditional medical side, where user experience is at best an afterthought. This delta will steadily close. Very soon we won’t have to choose between usability and accuracy.

But until such a time as the devices themselves have sufficiently evolved, and even with established standards in hand, we’re still left with the near-term reality that the accuracy and precision of these devices are sufficiently low, so as to render them ineffective for clinical decision-making.

Modeling of errors

Moving data from a proprietary device into an analytics environment remains a difficult problem — more difficult than one might imagine. This issue is being slowly solved on a device-by-device basis.

The most pressing need lies in the modeling of errors, or what we affectionately call MOE. This is not something that any single device maker alone can accomplish.

In fact, MOE is the most natural area for a data commons and the associated open-sourcing of software. Getting reliable device data into an electronic data capture (EDC) instrument is not an area of specialization likely to help any single health IT startup or large conglomerate. MOE is one of those classic areas where a rising tide floats all boats.

Let us expand on that a bit. The great thing about wearable and other tracking devices — the new ones in particular — is the volume, velocity and variety of the data they create. While it is a clichéd term, “big data” is an important concept here, and collecting such torrents of information facilitates the application of machine learning and other modeling methods to build sophisticated models for each data type.

By leveraging the information on normal subjects being collected by Google as part of their Google Baseline Study, the variance of each data type can be modeled and used to understand data collected as part of a research study. This is essentially what MOE is, and its public availability to clinicians and researchers will be critical to the use of wearables and other devices.

Even just a rote comparison of Fitbit data to iPhone and Android accelerometer and GPS data is an opportunity to triangulate truth and quantify the margin of error. As such, heterogeneous sources are enormously preferable to homogenous stacks, which is why we recommend against hardware-specific solutions.

Let’s not let the pursuit of the perfect device stand in the way of progress. In fact, any expectation of high accuracy and precision is unfair.

As researchers, perfection isn’t the goal. Rather, our pressing need is to understand precisely how imprecise these devices are.

We don’t need to eliminate variance; rather, we need to predict it. Effective modeling of device error will allow researchers to normalize data across different devices and, moreover, different participants.

In closing

We are converging on a time when the whole world could become a big clinical trial, with patients contributing their own data in a way that respects their privacy and allows highly granular control of access rights and permissions.

Over time, the devices themselves will get better and better. Standards and modeling of errors, however, offer nearer-term help.

Regardless, remember it’s not the data that matters as much as what we do with it. Patients tell us every day how they’re doing by their actions. Are we really listening?

. . .

Dr. Sam Volchenboum is the director of the Center for Research Informatics at the University of Chicago, a board-certified pediatric hematologist and oncologist, and the cofounder of Litmus Health, a data science platform for early-stage clinical trials.