Earlier this year, BBC TV technology presenter Lara Lewington decided to find out whether wearable activity trackers gave people an accurate indication of the exercise they do. She spent a week wearing four of the latest trackers and found that they each gave a different record of how many calories she’d burnt off and the steps she’d taken: the range was 2,649 calories, with a 23 per cent difference in the recorded number of steps. Not long afterwards, researchers at the Perelman School of Medicine and the Center for Health Incentives and Behavioral Economics at the University of Pennsylvania, US, claimed that smartphone apps are just as capable of accurately tracking physical activity as fitness trackers and other wearables.

Participants in this study walked on a treadmill wearing a pedometer, two accelerometers, three wearables and two smartphones running four apps. The researchers counted people walking 500 steps and 1,500 steps and then checked to see what the apps and wearables had recorded. They found that wearables logged between 22.7 per cent and 1.5 per cent less than the observed step count, while apps recorded between 6.7 per cent less and 6.2 per cent more.

The researchers concluded that, if everyone has a smartphone and can download free apps, why – if there’s not much difference in accuracy – would you pay for a wearable device?

In January 2015 there was more bad news for wearables. Iowa State University announced that seven of the most popular wearable devices provided inaccurate readings. The Iowa researchers gave 50 participants seven wearable devices each, and recorded how many calories the devices said each individual burned off.

As a control, they simultaneously tested each participant with a system that determines calorie burn by analysing oxygen levels in exhaled breath – widely recognised as the most accurate way to measure calorie consumption. The Iowa team found that all seven devices were at least 15 per cent inaccurate, with the worst offender out by 40 per cent.

Acceptable margins of error
Newspaper headlines made much of the inaccuracies and discrepancies. However, Greg Welk, an Iowa State University professor of kinesiology, says a device that is 10–15 per cent out is actually providing a reasonably accurate estimate of calories burnt. Mitesh Patel, from the University of Pennsylvania, also said the devices his team tested were “all pretty accurate”.

“You can’t expect a perfect measurement,” comments Yang Bai, the lead Iowa State University researcher. “Ten to 15 per cent out is good – 5 per cent is difficult to achieve.”

Bai explains that the actual number of calories a person burns during exercise varies depending on the person and what they’re doing during the workout. She adds that tracking technology in fitness devices doesn’t take the amount of resistance into account – say, for instance, if you’re running up a hill or lifting a heavier weight. “When the activity changes, or you change speeds, this will affect how many calories you burn off,” she adds.

Overstating calories?
The challenge of accurately measuring calories isn’t limited to trackers, as Liz Dickinson, CEO of wearable company MioGlobal, explains, pointing out that counting calories based on steps taken gives only an indication of what an average person of average fitness might burn off during the exercise. “It’s imprecise,” she says. “Calories actually burn off depending on metabolic expenditure, which varies with age, fitness level and gender.

“If you input all this information it gives a picture of who you are, but you still only get a crude reading.”

So if the reading is only crude when all this data is inputted, what happens when people using fitness equipment in a gym just press ‘quick start’, or at most input their age and weight? Dave Wright, CEO of heart rate system MYZONE, believes 99 per cent of people using a piece of fitness equipment don’t provide enough information to get any sort of accurate indication of the calories they burn.

Wright also believes some equipment manufacturers overstate the number of calories burned. “It makes members feel better on their piece of equipment, which helps the supplier sell more of its brand to health clubs,” he says. “But if you want accuracy, the more data points you collect, the more accurate the equation and the more precise the calorie count.”

Everyday activity vs exercise
Returning specifically to the trackers vs apps debate, Doug McClure, executive VP of product marketing at FitLinxx – which has recently launched its AmpStrip device – doesn’t accept that smartphone apps do as effective a job as wearables in tracking fitness. “People might carry their phones with them all day, but the phone isn’t always on their person – it’s sitting on a desk, in a bag, or a drawer,” he points out. “If you’re running on a treadmill for an hour, then a phone app will do a similar job, but in the real world phone apps fall down.”

Dickinson says another drawback to smartphone apps is that – at least for now – they tend to measure steps, not heart rate. Meanwhile, new wearables such as the Apple Watch and MioGlobal’s latest general purpose tracker, MioFuse, use more advanced technology to measure heart rate. In the case of MioFuse, the technology measures the volume of blood under the skin, with an algorithm calculating the heartbeat during the workout. “This technology doesn’t calculate heart rate from an electrical beat of the heart, though, which would be more accurate still,” comments Wright.

“It also relies on the person’s arms staying still and the device being tight enough not to let light in from the outside. The algorithm predicts the movement of the wrist, which is OK for regular movement – running or walking. But if the movement is irregular, it can’t measure heart rate in real time and will be inaccurate during vigorous exercise.”

Wright believes these devices are good for the 23 hours of the day when someone isn’t exercising, but says they will need an accurate device, such as a chest strap, to more accurately measure what goes on during vigorous activity.

Even then there are challenges to obtaining accurate data, because the analogue chips in cardio equipment rely on the heart rate belts that come with this chip, says Wright. “The issue with analogue chest straps is that, with multiple people exercising across numerous pieces of cardio equipment, you get cross-talk – your heart rate appears on someone else’s machine. You can’t guarantee the readings are yours.”

Progress or accuracy?
When it comes to accuracy of calculating calories, it seems the main issue is not whether the device is wearable, an app or a piece of gym equipment. It’s more that, with current tech – or at least tech that’s appropriate for everyday use – it’s very hard to calculate calories accurately.

Mick Rice, operations director at Pulse, confirms: “There’s lots of data we don’t know and can’t get at – for instance, a person’s BMI. Anything other than an oxygen mask to some degree gives you an average.” McClure agrees: “The question is: how close can we get to that gold standard in a consumer-friendly environment?”

But if existing technology can’t measure calorie count and heart rate accurately, how – short of people running around with oxygen masks on – can technology measure fitness progress? “If you can’t get perfect data, don’t worry about perfect data,” suggests Jeff Bartree, networked fitness product manager at Precor US. “Instead, measure relevant data.

“If you walk 10,000 steps or do 45 minutes on a cardio machine and you’re told you’ve burnt 450 calories, you also know you’ll burn even more if you increase the intensity. You may not know exactly how many calories you burn off, but you’ll get in better shape, and that’s the real aim.”

Rice agrees. “All this is just an index that encourages people to keep up levels of activity and track trends,” he says. “Time, distance travelled or intensity provide more accurate indicators. Calories burned gives an indicator of sorts, so too heart rate and step count. So why not put them all into a wearable device? People are motivated by different things. We want users to hold on to some method of keeping up an active, healthy lifestyle, so we should offer them as many indicators as possible.”

Bartree agrees, adding: “It’s also useful if technology allows the user to record what they do across a number of different types of exercise.”

Watch this space
Eventually, technological advances will enable trackers to be more accurate. “We’ll get better data recording, better algorithms and better ways of communicating this to users in meaningful and useful ways,” says McClure. “Today’s technology is much more accurate than early tracking tech and I don’t see any reason why this trend wouldn’t continue.”

Bartree believes people, and headline writers, need to be more patient: “This technology is a relatively new thing for mass market fitness. It might not be 100 per cent accurate at the moment, but now it’s caught on there will be lots of clever people looking at it and improving it.”

Currently, the general consensus is that tracking relative progress and offering people different measures to motivate them – to be used as an index against which to measure progress – should be more important than absolute accuracy.