Estimate project
← All posts

Building transportation mode recognition software end to end – a success story

This success story shows how motivation, know-how, and commitment pay off. Countless projects exist, each with its own problems that call for inventive approaches and technologies.

8 min read
Machine LearningCase Study

Introduction

This success story shows how motivation, know-how, and commitment pay off. Countless projects exist, each with its own problems that call for inventive approaches and technologies. Many different skill sets can also help get the job done within the expected timeframe. We have plenty of software engineering experience behind us, we share knowledge well within the team, and, frankly, it usually takes a lot to surprise us.

Every now and then, though, a truly unusual task lands on our desk. You probably know the feeling: you go through the starting assumptions of a project, talk to the client, and right then you realize an adventure is about to begin.

Nothing drives a project to success like a motivated team, and nothing fuels motivation like a thrilling, challenging engineering problem. This project gave us all of that and more.

This post describes how we went from the idea of a sophisticated transport mode recognition solution powered by machine learning to a solid proof-of-concept (PoC), and then began getting ready for a minimum viable product (MVP).

Make yourself a coffee. This story is not a short one 🙂

Task

Any good IT project starts with a clearly defined problem. The whole project architecture is built around it, but it also gets the team moving and keeps it driven and focused. Grasping the problem and what it entails may well be the single most important part of a project.

Our client delivered on that front. They came to us with the challenge of recognizing modes of transportation from mobile phone data. We were to build an end-to-end solution that uses human activity recognition to compute avoided CO2 emissions.

The client’s main goal is to reward people for traveling in a sustainable way. The client’s app detects the transportation mode automatically, computes how much CO2 was avoided relative to a car with a combustion engine, and rewards users based on that – a worthy idea and a product that makes sense.

Technically, the task is a tough one. It means handling a constant stream of mobile sensor data and calls for very precise machine learning models. On top of that, grasping the spatial context is essential, because users move around cities. This puts the task where science meets engineering, which makes it a perfect fit for our company.

Team

As someone wise once put it, you must gather your party before venturing forth. Our machine learning team is made up of engineers who specialize in machine learning. A deep neural network sits at the heart of the solution, but it can’t work on its own. A mobile application has to be built both to access the sensor data and to communicate with the user. Since several microservices consume the model’s inference results and cooperate to compute the CO2 avoidance emission score, the system also needs a platform that can orchestrate its other components.

For this reason, we joined forces with other companies: a software house with excellent Scala developers experienced in big data and blockchain solutions, and a partner company specialising in building management and user-facing applications.

That is how the project kicked off: with an engineering team made up of Machine Learning engineers, Scala software developers, iOS developers, DevOps engineers, Quality Assurance Engineers, and Geographic Information System experts. It goes without saying that this was a fascinating combination. In parallel, other team members coordinated our work, acting as Product Owners / Managers, designing the User experience / Interface, and running Blockchain workshops. Setting up coordination across the work of all three companies took roughly two weeks.

Crucially, the client’s team took part in the engineering effort from the very first day. They were always on hand to respond to the countless questions we threw at them every day. The collaboration between the client and the engineers proved to be essential. To the client’s team: thank you for your attitude and your support!

Planning before coding

We had a great deal of freedom and could design the architecture of the solution from the ground up. We were eager to start coding, but before that we needed to be sure we understood what the client expected, so that we could plan well. The deadline for the task was fixed. We wanted the client to be able to show their solution and share their idea with the world as early as possible.

Our aim was to be able to follow the progress and phases of the PoC at any moment. We wanted the client to stay in control of what was going on in their project even though development moved fast. This also mattered a lot from the machine learning perspective, since it requires large amounts of high-quality data, which has to be gathered and processed correctly. For us, this meant delivering the first version of the whole platform within two months of starting work. Getting the timing right was key to meeting the constraints set by the field research (data collection).

To make sure every part was ready on time, we chose to run a series of workshops with the client. Together we produced a detailed outline of the whole system, covering machine learning, the requirements for the cloud platforms, the mobile application, and, last but not least, blockchain.

The first round of workshops produced the transport mode detection (TMD) research document, which laid out the steps required to build a flexible machine learning solution. This in turn led to the design of the platform architecture and of a research (annotation) mode for the mobile application.

Research mode activated

Machine learning projects require huge amounts of data – and not data of any kind. How accurate a model is depends directly on how good the gathered data is. In new kinds of projects, the data set a problem needs hardly ever exists, so it has to be built.

You can’t simply head out into the field and grab the data. First, we had to define a set of rules to follow while collecting it. What ensures that this step is done carefully is approaching the whole project as scientific research. This approach has never let us down, as it suits machine learning projects perfectly.

The rules took the form of a mobile application and an annotation manual. Next, we put together a team of volunteers willing to help us gather data. Every participant received an iPhone and an Apple Watch to record measurements while traveling between home and work.

During the research we gathered over 150 million records describing how people move.
map

An example of the data gathered during the research, in one of the research areas.

It is worth stressing that by the time the first team member headed out to gather data, we already had a functioning cloud platform that processed the data coming from a mobile phone and prepared it for machine learning. This was at the end of the project’s second month. The pace of the project was remarkable.

Feedback-driven development

The system would never have come to life without constant communication between us. A well-thought-out architecture helps define the framework the engineering team works within. Still, success hinges on how effectively we can share information about the work on the mobile application, the cloud platform, or machine learning. Slack and Google Meet were a big help here.

Have I mentioned that the team was remote and everybody worked from home?

Wrapping it all up

Six months into development, the system PoC was ready for user acceptance tests (UAT). The cloud platform was complete and able to handle requests coming from the mobile application. The machine learning transport mode recognition service was up and ready to run inference on the collected sensor data.

There is no better way to test a solution driven by machine learning than to use it in real-life conditions. So our research team, supported by QA and our clients, set off on their trips. We went through various scenarios, from walking to commuting by public transport, and wrote down plenty of notes to fine-tune the system.

The outcome exceeded our expectations. Every part of the platform fit together, and the solution works.

Our client is active and takes part in many conferences in Europe and the US. Large-scale trials are also in preparation, and a beta version is open for registration.

Our work culminated in an extensive report for our client, in which we documented every part of the platform in detail. A step like this matters for preserving knowledge about the project. It makes it possible to launch the next stages quickly and to grow the team when needed. Solid documentation also matters for the freedom to operate (FTO) verification process, or even for obtaining a patent.

Summary

It was a wonderful adventure. Within roughly six months, we designed and built an advanced proof-of-concept. The solution can produce and use a deep learning model that recognizes transport modes. To give the client the functionality they wanted, we brought together many skill sets: machine learning, backend development, mobile development, blockchain, GIS, and scientific research.

This success would not have been possible without the broad experience of the team members and the trust our clients put in us.

The hardest part of the project was assembling all the tasks in time to begin designing and training machine learning models early on. We wanted to leave as much time as we could for experimenting with various neural network architectures and data processing methods. This demanded a great deal of discipline from everyone on the team, as well as an understanding that every piece of work on the cloud platform or the mobile application has a direct impact on the machine learning results.

So what comes next?

We are now heading toward an MVP. It is time to plan the upcoming tasks and get the machine running faster. More news soon. Every commute counts 🌿