Estimate project
← All posts

DevOps in Embedded Development: Connecting Software and Hardware

This article went through a lot on DevOps in Embedded Development. We've built a foundation by showing how tools such as Docker, QEMU, GitHub Actions and Terraform can bring structure and automation to the workflow.

9 min read
DevOpsDockerEmbedded DevelopmentGitHub Actions

Embedded systems have long been viewed as a world apart from mainstream software development, one typically marked by manual processes, complex hardware dependencies and long release cycles. Yet as embedded systems grow ever more complex and connected, DevOps principles, with their focus on automation, collaboration and continuous delivery, turn out to be not merely helpful but indispensable.

This article builds on the ideas from my online talk at the Embedded Israel meetup.

Why DevOps for embedded?

What DevOps fundamentally promises – quicker feedback loops, automated pipelines and greater reliability – matters even more in embedded development. In contrast to web or mobile applications, embedded systems talk directly to physical hardware, which brings its own specific complexities. Flashing firmware by hand, plugging cables in and out, and postponed testing on physical boards can slow development down considerably.

Adopting DevOps lets us automate a lot of these traditionally manual steps. This means:

  • Quicker iteration: Build once, test frequently and get rapid feedback, not only on code changes but also on how the device actually behaves.
  • Spotting problems early: Find issues early in the development cycle, cutting the cost and effort of fixing them down the line.
  • Greater reliability and scalability: Automated testing and deployment result in more stable products and let teams take on bigger, more complex projects.

Dealing with the specific challenges

The advantages are obvious, but embedded DevOps also brings a distinct set of challenges that need tackling:

  • Hardware-software integration: The code has to work flawlessly with particular hardware, which makes simulation and automated testing harder.
  • Scarce resources: Microcontrollers are frequently heavily limited in CPU, RAM and storage, which calls for optimized code and efficient toolchains.
  • Real-time requirements: Plenty of embedded systems are subject to strict timing constraints, and continuous integration and deployment (CI/CD) setups need to help verify them.
  • Safety and reliability: In critical sectors such as automotive or medical, thorough testing and certification come first.
  • Varied hardware platforms: Teams frequently deal with several hardware platforms, each with its own toolchains and debugging procedures.

Laying the groundwork: managing the toolchain and environment

Deciding how to manage the build environment is among the first key choices in an embedded workflow.

Native vs. virtualized builds

For small projects, native builds are simple, but environment drift can cause “it works on my machine” issues. Virtualized builds, Docker-based ones in particular, give a consistent, reproducible environment by pinning compiler versions, dependencies and build scripts. Such stability is essential for CI/CD and for developing together.

Cross-compilation and static linking

Because embedded devices frequently don’t have the resources to compile code themselves, cross-compilation is essential. Code is compiled on a stronger host machine for the target embedded architecture. Numerous vendors supply official toolchains, for instance Espressif’s ESP-IDF for ESP32 or ARM compilers for STM32 through Buildroot or Yocto, which makes the process smoother.

Static linking is also widespread: every required library is linked straight into the final binary. This makes deployment easier because missing-library problems disappear, though binaries may end up bigger.

Emulation with QEMU

QEMU works as a virtualizer that lets developers emulate embedded systems (such as a Raspberry Pi or STM32) on their host machine. This is extremely useful for:

  • Early development: Trying out software before the physical hardware exists.
  • CI/CD integration: Executing virtual test runs inside the CI pipeline.

QEMU does have limitations, though, particularly when it comes to peripheral support (e.g., SPI, I2C sensors, custom GPIO). Even so, it is still a powerful way to speed up early development and CI.

Docker: A game-changer for embedded development

And now, my favourite bit: these days Docker transforms every kind of development, largely because it fixes some of the messiest issues we deal with – inconsistent toolchains, clashing dependencies and painful onboarding.

Here are three ways it makes things easier:

  • Reproducible toolchain environment: Docker images bundle the precise compiler, linker and SDK versions, keeping every development environment consistent.
  • Isolated cross-compilation targets: Teams can build separate Docker images for different boards or architectures (e.g., STM32, ESP32), which prevents toolchain clashes.
  • CI/CD integration & quicker onboarding: Docker images plug smoothly into CI/CD platforms such as GitHub Actions, and newcomers can start working fast thanks to a ready-made environment.

CI/CD with GitHub Actions

Thanks to its smooth integration with Git repositories and its flexibility, GitHub Actions is a great fit for embedded projects. Workflows, written as YAML files in the .github/workflows directory, make automated builds, cross-compilation and testing possible. They describe a sequence of automated steps, for example building, testing or deploying code, that are triggered by events such as pushes, pull requests or schedules.

Features that matter most for embedded workflows are:

  • buildx: A Docker CLI plugin that lets you build Docker images for several embedded architectures right inside the pipeline.
  • QEMU runners: Let GitHub Actions emulate target CPUs, making substantial testing possible without physical hardware.

GitHub runners: Public vs. self-hosted

Workflows don’t run on their own; they need runners, i.e. virtual or physical machines on which the jobs are actually executed.

Runners come in two kinds: GitHub hosted runners and self hosted runners.

When testing has to interact with physical hardware, self-hosted runners are essential. Scaling them means setting up several instances to process actions concurrently.

Scaling embedded CI: Labels & matrix builds

To make CI/CD workflows more flexible and scalable, GitHub Actions offers advanced configuration options that let teams adapt their automation to complex hardware and software requirements. Two particularly useful ones are:

  • Runner labels: Self-hosted runners can be given labels (e.g., arm64, lcd-test, usb-flash). Workflows can then pick specific runners by these labels, sending jobs to the right hardware.
name: Flash Firmware

on:
  workflow_dispatch:

jobs:
  flash:
    runs-on: [self-hosted, arm64, usb-flash]
    steps:
      - uses: actions/checkout@v3

      - name: Flash firmware to device
        run: |
          ./scripts/flash.sh
  • Matrix builds: This powerful feature runs several build variants in parallel. A matrix can, for instance, be set up to build firmware for various boards (e.g., STM32, ESP32) and modes (e.g., debug, release) at the same time, shortening testing cycles.
name: Build Firmware

on:
  push:
    branches: [main]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        board: [stm32, esp32]
        mode: [debug, release]
    steps:
      - uses: actions/checkout@v3

      - name: Build ${{ matrix.board }} firmware in ${{ matrix.mode }} mode
        run: |
          make BOARD=${{ matrix.board }} MODE=${{ matrix.mode }}

Optimizing your embedded CI pipeline

A well-optimized embedded CI pipeline builds more efficiently and shortens feedback time, which allows quicker iteration and better code quality. Concentrate on cutting out unneeded steps, making use of caching and running tasks in parallel wherever you can.

On top of that, make sure sensitive data such as credentials and API keys is stored securely through environment variables or secrets management tools. Audit your pipeline for vulnerabilities on a regular basis and follow security best practices to avoid possible breaches.

Caching

Embedded toolchains tend to be big. Caching toolchains and build outputs can cut build times considerably by avoiding repeated downloads and compilation.

Below are a few examples of caching with the action cache.

jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Cache ARM GCC Toolchain
        uses: actions/cache@v4
        with:
          path: ~/.toolchains/gcc-arm-none-eabi
          key: ${{ runner.os }}-arm-gcc-${{ hashFiles('toolchain-install-script.sh') }}
          restore-keys: |
            ${{ runner.os }}-arm-gcc-

      - name: Install Toolchain
        run: ./toolchain-install-script.sh

Here, the script installs the ARM toolchain only when it isn’t already in the cache. Thanks to hashFiles, the cache is invalidated whenever the script changes.

Dependency cleanup

Check pipelines for dependencies you don’t need and pass flags such as no-install-recommends during installation, so containers stay smaller and builds run faster.

Secure pipelines

Handle sensitive data (signing keys, tokens, Wi-Fi passwords) as secrets. Rely on GitHub Actions secrets and permissions rather than hardcoding credentials. Add code scanning actions to detect hidden keys, outdated libraries and vulnerable GitHub Actions versions.

Here’s how a secret key can be used in your GitHub Action:

name: Secure Firmware Upload

on:
  push:
    branches: [main]

permissions:
  contents: read
  id-token: write

jobs:
  upload:
    runs-on: [self-hosted, usb-flash]

    steps:
      - uses: actions/checkout@v3

      - name: Decrypt firmware signing key
        run: |
          echo "${{ secrets.SIGNING_KEY }}" | gpg --batch --import
[...]

Lastly, on the security front, it’s a good idea to add GitHub Actions that scan your code for hidden keys, outdated libraries and even GitHub Actions versions that are out of date and need updating.

The example below shows how to analyze your code with the Codeql action.

security-scan:
    name: Code & Dependency Scan
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      # GitHub's built-in code scanning with CodeQL
      - name: Initialize CodeQL
        uses: github/codeql-action/init@v3
        with:
          languages: cpp

      - name: Perform CodeQL Analysis
        uses: github/codeql-action/analyze@v3

Testing strategies: From unit to hardware

Testing embedded systems goes further than just compiling code. We usually split testing into three levels: unit, integration and hardware-in-the-loop (HIL) tests.

Unit tests:

They run on the host system and concentrate on whether the logic is correct. They are fast and very easy to automate.

Integration tests:

They bring different modules together and can be cross-compiled to run either on emulators such as QEMU or straight on target boards.

Hardware-in-the-Loop (HIL) tests:

At this level, code runs on real hardware. HIL testing needs a more elaborate setup, but DevOps principles can automate essential parts of it, such as flashing devices, running tests and collecting results. It frequently requires a physical link between the target board and a self-hosted CI runner, usually over USB, UART or GPIO. Depending on what the tests require, extra components such as relays, measuring instruments or even cameras (to test display output) may be necessary.

So, even though DevOps tooling makes the testing workflow smoother, the underlying infrastructure and cabling always have to be built and looked after. Once that’s done, this foundation allows powerful, repeatable testing on exactly the hardware your customers will be using.

Infrastructure as Code (IaC)

Describing infrastructure (e.g., cloud resources, CI/CD environments) as code with tools such as Terraform or Pulumi brings many benefits.

Declarative tools guarantee that infrastructure is always created the same way, removing the human error that comes with creating cloud resources by hand. This also makes setup simpler: rather than working out how to configure cloud resources, you just declare them and supply the required parameters, such as names, address ranges or sizes.

Describing your target infrastructure in Terraform and setting up the first environment can take a while, but every following environment will be created far more quickly. What’s more, rolling out a change to all environments described with Infrastructure as Code (IaC) will require just one module modification and applying the Terraform code.

Summary

This article went through a lot on DevOps in Embedded Development, and still we’ve barely scratched the surface. The topic is deep, spanning software, hardware, infrastructure, testing, deployment and security. We’ve built a foundation by showing how tools such as Docker, QEMU, GitHub Actions and Terraform can bring structure and automation to the workflow. The point is not only to speed up development, but to improve repeatability, scalability and reliability, even with real hardware in the loop.

Putting these practices in place takes time and discipline, but in the long run they give your team, your product quality and your customers a major advantage. We’ve seen what DevOps can do; now it’s time to act. There’s no need to rebuild your whole process at once. Begin with small steps – for example, put your toolchain in Docker or create a basic GitHub workflow. Focus on repeatability and feedback, and bring hardware into your pipeline from day one instead of treating it as an afterthought.

Above all, don’t go down this path by yourself. Share your setups, write internal documentation and contribute to tooling. The easier we make embedded DevOps to adopt, the stronger our teams and products will grow. It is no longer a nice-to-have; it’s a competitive necessity.