From fitness watches to industrial temperature sensors, Internet-connected devices are the way of the future. But many businesses are unprepared for the giant leap in complexity that comes with developing IoT devices. Learn how the most sophisticated embedded development teams establish a benchmark for success.
Every product launch comes with challenges—and nowhere is this more true than when launching Internet of Things (IoT) devices.
Transitioning from traditional hardware to IoT devices requires a giant leap in complexity. Product and engineering teams must factor in the integration of radio stacks, networking protocols, and security measures on top of considerations like chip space and low-power modes for battery operation. Not only are the hardware, software, and ecosystems infinitely more complex, but it’s a rapidly evolving industry with surprisingly few established best practices.
“A lot of businesses are still figuring out the journey,” said Andie Cockerill, Vice President of Customer Experience at Memfault. Andie works with big and small IoT companies across a variety of industries, ranging from consumer electronics to industrial machinery. She hears their challenges—and successes—firsthand.
“Managing devices in a controlled environment is one thing, but putting them in consumers’ hands is another. Most development teams grapple with the unpredictability of real-world deployments.” —Andie Cockerill, Vice President of Customer Experience, Memfault
Lessons from the Field
So how are the most sophisticated development teams working to address these issues? What should IoT developers and engineering leaders keep in mind for a seamless transition from pre-launch to post-launch?
Memfault brought together IoT experts from some of the industry’s top companies to discuss how to master IoT device quality—not only on Day 1, but six months down the road. Here are their recommendations for getting the data you need to deliver products that perform in the real world.
Meet the Experts
- Tyler Hoffman is the co-founder and head of developer experience at Memfault. He has nearly a decade of embedded engineering expertise.
- Shiva Rajagopal is a Staff Embedded Software Engineer at Google as part of the Fitbit team.
- Simon Ford is the founder of Blecon, a company enabling low-cost cloud connectivity. He has 20+ years of IoT, mobile, and microelectronics experience.
- Andie Cockerill is the Vice President of Customer Experience at Memfault. She works with innovative IoT teams to understand their goals and design solutions.
10 IoT Development Best Practices
1: Think about monitoring while you’re developing—not after.
Every release manager and support manager knows the high level of stress on launch day. (Not to mention, the lack of sleep in the weeks leading up to it.) It’s not surprising that post-launch monitoring and testing often gets pushed to the back burner.
Shiva Rajagopal, Staff Embedded Software Engineer at Google, explained: “Development teams are often so hyper-focused on developing and testing on their debugger that they forget to factor in monitoring until they get to production. It's a very easy trap to fall into.
Shiva’s #1 tip for developers is to shift their mindset toward monitoring earlier in the process.
“You really need to be thinking about monitoring while you're developing. At the end of the day, your customers or other users are not going to have a debugger. And so you need to make sure that you can monitor and test that stuff while it's in progress.” —Shiva Rajagopal, Staff Embedded Software Engineer, Google
2: Establish early access to the data you’ll need later.
Businesses are making huge investments throughout the development process, but data accessibility often gets overlooked for resourcing. Other times, the decision to invest in an embedded observability tool comes much later, when deploying it earlier could have accelerated the process.
Simon Ford, an IoT veteran and the founder of Blecon, said: “Imagine your system is deployed. What are you going to be looking for? What tools do you need? Move that to the start. You’ll be solving different problems on the bench and quickly iterating to try and get out bugs later on. But if you can distill that down to the same set of tools or parameters that you can use in different ways each time, that’s really powerful.”
A lot of people ship in beta, but they don’t yet have a way to retrieve their data. They haven’t yet built their hub, set up their app, or created a connectivity path. This can be a key moment to establish visibility and practice using embedded observability tools to accelerate your development.
Tyler Hoffman, Co-Founder and Head of Developer Experience at Memfault, saw the value of this firsthand while working as an embedded software engineer at Fitbit.
“You’ll need to test your connectivity and find a way to get a data path, but you won’t yet have the hub and everything built. It can be challenging—you're flying the plane while you're building it. Implement tools early in the process that will enable you to extract and access the data you’ll need.” —Tyler Hoffman, Co-Founder and Head of Developer Experience, Memfault
3: Identify “go and no-go” metrics.
Engineering teams are strapped for resources, and managers need to decide where to deploy their team’s time. It’s not feasible to just have people sitting and monitoring all day long.
The most sophisticated teams establish key metrics ahead of time to provide a top-level view of performance.
What do you want to measure to give you a pulse on the health of your product? You need to be able to have a quick indication of good versus bad. Set the bar so you can be data-driven and release ready.
The experts recommend focusing on three key areas: stability, connectivity, and battery life. These will play a vital role in the success or failure of every IoT device.
See a new framework to measure embedded device quality in the field
4: Keep it simple.
What you monitor doesn’t have to be complex. Think about the one clear metric that is meaningful to your business. What signal will tell you at a glance if your product is able to fulfill its purpose or if something is malfunctioning to interfere with that?
Simon calls this metric the “smell test.” He shared an example of a giant online retailer measuring cash going through their register. If that number dropped at any point, they knew something was wrong.
“Ask yourself, what’s the smell test that makes everyone look and act quickly? It’s fundamental to identify this one thing for your business.” —Simon Ford, Founder of Blecon
Andie said, “We often recommend going to your support teams. They will know what your top ‘customer-impacting moment’ is. For example, Memfault works with a lot of audio companies. The key thing for them is audio dropouts. That's very, very noticeable for their customers. If it’s something that you can pick up and give a signal on, that's something your executives often want to look at as well.”
5: Get executive buy-in.
When a report lands on a CTO or CEO’s desk, they need to be able to quickly understand it, even if they’re not deeply involved in the development process.
Simon explained, “Often it's an engineer who is shouting that something is important, but no one pays attention because no one has agreed what important looks like."
"If everyone knows what’s important and why, then the organization can respond. It’s often not the technical engineering of a debugging problem that makes the difference. It’s the agility of the organization.” —Simon Ford, Founder of Blecon
Make sure your whole team agrees on what metrics are most important, all the way up to senior management. This will make it a lot easier to respond and make quick decisions about where to allocate resources.
6: Use dashboards to make analysis faster.
Having the right data in the right place at the right time to move efficiently is a huge problem for companies. Data often lives in pockets that you have to query and treasure hunt to find the right info. It’s spread across data lakes, BI tools, excel spreadsheets, and a variety of other places.
And unless you have a very sophisticated, well-resourced BI team and system, it’s often up to your development team to figure out how to make all the data you’re collecting easy to understand. Even if you do have a BI team, data scientists aren’t familiar with the type of data coming from these devices. They may have a hard time figuring out the right way to frame the data to give you the answers you need.
A lot of product managers and engineers try to solve this on their own. But it can be extremely time-consuming, piling more work onto an already full plate. Instead of hacking together spreadsheets, SQL queries, and graphs to make sense of the huge amounts of data you’re collecting, look for tools that put crucial data at your fingertips.
Explore the new dashboards in Memfault's embedded observability tool
7: Don’t jump to fix squeaky wheels.
Every engineering manager faces the daily challenge of prioritization. They’re strapped for resources and need to make hard decisions about where to invest precious time. Oftentimes, the default answer is to assign resources to the latest customer complaint. But there are several problems with this.
First of all, how do you know if it’s a standalone noisy customer or something that’s truly impacting a wide swath of your customer base? Secondly, a very small percent of customers who experience a problem will actually call in. And that means there could be significant unreported issues that are silently killing your brand.
“There might be one person that saw a bug that's going to kill the entire watch in two months. And you have to make sure to take that just as seriously as everyone who's saying, 'Hey, I started an exercise, and the device crashed after five minutes.' Both of them are very important in completely different ways, and one's way more visible than the other. But you have to make sure that each issue gets the attention it needs.” —Shiva Rajagopal, Staff Embedded Software Engineer, Google
Simon added, “The joy of having metrics and data is that you can start making much better decisions. You can discover what I call the hidden problems. These are the crashes that no one reports, or the times when it’s taking multiple times to connect. They’re the things that are hurting your brand and causing your customers to lose trust, but are not reported.”
8: Steal from software teams.
For years, software teams have had solid processes and tools to help them work efficiently and effectively. And while firmware engineering is a unique beast, there are proven best practices you can leverage.
For example, Shiva advocates for working in agile as much as possible, even in pre-production. Although certain parts of development will inevitably require a waterfall approach, you can front-load those in the schedule and making sure you're able to test them. Once that hardware comes in or once that module has landed, agile has a lot of great benefits.
Shiva explained, “Agile just philosophically focuses more on—how do we get this thing to a workable state and then iterate on it? I have seen agile work quite well during my time at Fitbit, and now at Google."
"It helps to think about how you're developing in terms of: How do we get something working and how do we start measuring that and iterating on it?” —Shiva Rajagopal, Staff Embedded Software Engineer, Google
Another popular approach is dogfooding, which all the experts agreed can be one of the best ways to get visibility before you launch a product. Have internal users and a beta cohort actually use your product the same way your customers will.
9: Analyze costs at a wide scale.
Many engineering teams are hyper aware of the cost of a chip and the need to save code space on a device. But overlooking the importance of collecting diagnostic data due to concerns about storage costs and space limitations can be detrimental.
Simon commented, “Consider the cost of an engineering team’s time, or the cost of deploying a product and facing a recall. What’s the cost to the brand if something goes wrong? When you look at your cost analysis at a wider scale, things become very clear.”
“The trade-off of implementing observability from the beginning versus dealing with performance issues at scale is significant.” —Andie Cockerill, Vice President of Customer Experience, Memfault
The experts agreed that cost analysis should extend beyond hardware expenses to encompass broader considerations like deployment costs, risk mitigation, and brand reputation. It’s always possible to distill data down to small amounts, even on tiny devices like those in the Bluetooth space, where diagnostics and product analytics can be crucial.
By adopting a more comprehensive approach to cost analysis, engineering teams can make informed decisions that prioritize the long-term success of their IoT projects.
10: Keep your team’s mental health in check
Businesses invest millions of dollars and years of time to develop IoT products. Engineering teams face an extraordinary amount of pressure as launch day approaches.
Shiva noted, “When this product that you’ve been working on for months and months is about to go out the door, and all you're hearing about is the negatives, the bugs, and the potential stability issues—it's really disheartening. You want to make sure that your team is staying motivated and that they are able to function the best they can.”
Set your IoT devices up for success.
Is your team considering how to monitor device quality, health, and performance in the field?
Watch our keynote with Memfault CEO François Baldassari, in which he walked through a data-driven framework for device quality and reliability. This new framework represents a first for the IoT industry and is based on discussions with IoT companies of all sizes, across multiple industries.
Watch Now: How to Measure Embedded Device Quality in the Field