Stepping into on-edge Machine Learning
How I built a DIY project that brings deep learning onto an Arduino Pro board.
How I built a DIY project that brings deep learning onto an Arduino Pro board.
Well, it has happened at last: I have fallen for machine learning on embedded systems for good. The symptoms of this new infatuation showed up gradually and shyly at first. Initially, I was captivated by what could be done with a Raspberry Pi. Pairing Linux with Python meant I could get results quickly, and for a machine learning engineer it felt like moving the same work over from a larger machine with hardly any changes. On top of that, building something physical and bringing it to life with my own code was a refreshing break from writing software meant purely for the backend.
Next came the Nvidia Jetson. It is a capable piece of hardware, designed with deep learning and computer vision in mind right from the first boot. Setting it up and working with the environmental shield was a joy. Even so, small as it is, I kept feeling that it was too bulky to fit every scenario I had in mind. More precisely, my Mavic could not lift it.
That is how I ended up with the Arduino Nano. I must say it both impressed and disappointed me. The platform has a lot to offer: plenty of room for expansion, components that are easy to get, and the ability to run compact machine learning models. Any accessory that works with the popular Arduino Uno can be used with the Nano as well. Working on ML models, however, is not much fun, because the memory limits are very tight.
Still, I did not give up – my faith in Arduino is boundless. Meanwhile, on-edge machine learning kept getting more exciting, and I kept coming up with new ideas for applications. What I needed was a stronger device of a similar size. This is how I found the Arduino Portenta.
In this post, I walk through my first DIY project that runs deep learning on an Arduino Pro board. Make yourself a coffee – it is going to be a lengthy read 🙂
Problem
First, we need a sensible problem to tackle. Environmental sciences are my passion, so whenever I look for side projects, I lean toward Earth Sciences, remote sensing, and environmental protection. Winter here is not especially severe, yet even I am not keen on leaving my cozy apartment, so coming up with something fun was not easy.
Luckily, the answer was close at hand. It came from my children, who are too young to grasp why waste should be sorted and recycled. As a result, rubbish ends up in whichever bin is nearest, and there ought to be a way to stop that.
The plan is straightforward. Build and program waste bins so that they help children learn proper recycling habits.
On top of that, building it has to be enjoyable. In my case, that means going completely overboard with technology 😅. So let us use computer vision, deep learning, embedded ML, and a bit of the Internet of Things.
Searching for an interesting project made me reflect more on embedded ML in industry, agriculture, urban management, environmental monitoring, and other fields. The potential is huge, but so are the drawbacks and constraints. It deserves an in-depth discussion, and I sense I should learn from outside sources. If you can recommend a good conference on IoT and ML, I would really appreciate hearing about it.
Unboxing
To my pleasant surprise, the key components were delivered just one day after I ordered them. For this project, I chose the Arduino Portenta H7 and extended it with a vision shield. Its 320x320 pixels camera sensor is sufficient for classifying basic kinds of waste. The shield also supports LoRa, which will come in useful for later projects. An Ethernet variant is available too.
Arduino Portenta H7 + Vision Shield
Preparing the board
Attaching the shield to the board is simple. The assembly seems robust enough to be used with some extra protection, but I did not want to damage it while prototyping. Hobbyists have not widely adopted this board yet, so there are no ready-made cases. Thankfully, the module is small, and 3d printing a case for it is no trouble. I often rely on an excellent project called “customizable everything box.” It has a huge number of remixes, so finding one that suits a DIY build is easy.
Arduino Portenta H7 + Vision Shield
Here comes the tricky bit. Before we could use the board, it had to be flashed. I intend to use the Portenta in two scenarios.
The first scenario covers collecting data and preparing the model, for which I plan to use Edge Impulse. Happily, they provide an excellent tutorial on hooking the board up to their platform. Be sure to connect to the device through your browser – it makes everything go smoothly.
The second scenario combines running inference with an ML model and driving some extra hardware. Since I like MicroPython and CircuitPython, I went with OpenMV, which is installed differently. Once you download the IDE and plug in the board, you will be prompted that the device can be flashed.
Keep in mind that the two scenarios cannot be used interchangeably.
If your development machine cannot connect to the device, there is one more hurdle to overcome. Removing modemmanager and adding a few udev rules should solve it. More details are in the Porenta setup troubleshooting guide.
Assemble
The first item on my to-do list was getting the colorful wastebaskets ready. There are three bins in the setup, and each color stands for a different kind of waste: blue (paper), yellow (plastic and metal), and green (glass). The bin with the controller sits in the middle. I cut a rectangular opening in its top and mounted the Arduino board with the camera there. Two LED strips are attached to the inner walls of every bin. Everything runs off a small power bank. Naturally, the wiring inside resembles spaghetti 😅.
IoT trash bin preparations
Data preparation
Data can be prepared in several ways: I could either use an existing dataset or build one myself. Even though there are public labeled datasets I could have adapted to this problem, I chose to collect a small sample set of my own. The reasons were the particular characteristics of the Arduino camera sensor and my wish to test what the Edge Impulse platform can do. It turned out to be the right call. The data acquisition feature in Edge Impulse is easy to use and let me get through this step quickly. Linking the platform directly to the device was an excellent idea.
Data samples. Nowhere near what you would call trash – this is the Internet, and I wanted the post to look nice.
Preparing the model
I stuck with the Edge Impulse workflow and trained the machine learning model with the TensorflowLite framework. To begin, I set up an Impulse, which serves as a blueprint for the training pipeline. After that, I handled data preprocessing. Lastly, I picked a predefined model architecture and started training. Thanks to the available transfer learning option, the model converged quickly and gave good results. I still wonder how far Edge Impulse can be customized, but I will find out soon on a more demanding use case.
Deployment
When it comes to deployment, there are many options to choose from. The recommended one is installing the model as the device firmware. Since I wanted to add some custom behavior, I ended up deploying the model as an OpenMV package. This produces a package with the model, a text file with the labels, and a template MicroPython inference script. The only thing left to do was to edit that script and add the lines that drive the LEDs.
Results
With everything assembled, it was time for user acceptance testing. The kids had fun for the first 15 minutes, but it was still worth the effort.
Draft results of Inference with customized logic.
Naturally, I did not get the result I wanted right away. Iterating on machine learning models is normal. I repeated the whole cycle – from collecting data through training to customizing the model – a number of times before the solution became usable.
Summary
All in all, the time was well invested. I learned plenty and really enjoyed building and programming the solution. I firmly believe that machine learning on the edge will matter a lot going forward, as there are many fields where this approach fits. At the same time, I feel a lot of work remains. Luckily, Edge Impulse and OpenMV make life much easier.
If you would like to dig deeper into automated waste classification, take a look at an article written by our Senior Machine Learning Engineer.
A Senior Machine Learning Engineer from our machine learning team adds:
I have long been curious which fields would gain from processing data on the edge instead of the conventional setup, where lightweight nodes forward data to a central server with more computing power. When you have to maintain a network of devices, it is simpler to build very basic devices that can run on batteries for as long as several years than to build more sophisticated, energy-hungry devices with advanced processing (here, a machine learning model) that drain their batteries quickly or even have to be plugged into the grid. In my experience with IoT systems, the most typical use case is collecting data, with processing done centrally. I wonder whether embedded ML will turn out to have the edge over that approach.