Pytorch, TensorFlow, JAX, and Flax compared by an ML Engineer
With this blog post and its accompanying examples, grasping particular caveats and weaknesses of the various frameworks becomes straightforward.
Which Deep Learning framework comes out on top?
Stepping into the ML world, you may feel overwhelmed by the number of libraries to pick from, with camps as divided as political parties or religions (nearly on par with front-end frameworks). Out of the many options, a handful stand out as the most popular: Pytorch, Tensorflow (+ Keras), Pytorch Lightning, and, more recently, JAX (together with its NN framework - Flax). Each of them can be used at a different level of abstraction: Pytorch Lightning and Keras sit higher up, Tensorflow, PyTorch, and Flax offer more control, and JAX works at the lowest level.
In this blog post, I want to present qualitative examples that help compare the strengths and weaknesses of these frameworks, so that their pros and cons become easier to grasp.
Tensorflow + Keras
TensorFlow was created by Google Brain and released in 2015. The framework covers a broad range of machine learning tasks, from basic linear regression to sophisticated deep learning architectures, and gives low-level control over the model architecture.
Higher-level control, by contrast, is available through Keras, which became a core component of Tensorflow starting with version 2). Keras 3.0, however, is once more stepping away from being Tensorflow-only and supports several backends: JAX, Tensorflow, and Pytorch.
Tensorflow is also known for the wide range of platforms it targets: mobile devices (Tensorflow Lite), large-scale distributed systems (Tensorflow Serving), production-grade deployment (TFX - Tensorflow Extended), and web deployment (Tensorflow.js for running models in Node.js).
Pytorch

How popular the frameworks are in new Machine Learning papers, source: visio.ai
Pytorch was released in 2016 by Facebook’s AI research Lab (FAIR) and rapidly won over researchers and developers. This was mostly thanks to its dynamic computation graph and ease of use, which knocked Tensorflow off the stable position it held at the time.
In our in-depth analysis of the pneumonia model, we trained and evaluated the model with PyTorch.
Its Dynamic Computation Graph (the graph being constructed on the fly) was revolutionary back then, making network debugging and runtime changes much easier. This architecture and API fit Pythonic-style development better than other frameworks did (e.g. Caffe). Another major advantage was its robust support for automatic gradient computation, which considerably simplified implementing machine learning models. PyTorch also lets you structure code in a flexible, modular way, which made it hugely popular for R&D.
As time passed and its popularity in the developer community grew, the Pytorch ecosystem expanded considerably, with excellent documentation, specialised tools and libraries (covering various domains – audio, computer vision or text/nlp – as well as deployment, e.g. edge deployment), and also more niche ones: the medical computer vision library Monai, the GenAI diffusion library Diffusers, or, for 3d point cloud data, Pytorch Geometrics and Pytorch3D. On top of that, libraries such as fast.ai make it possible to get started quickly with building meaningful, capable solutions, giving easy access to the newest ideas from the research world.
Pytorch Lightning
Pytorch Lightning is a high-level framework layered on Pytorch that makes developing and training deep learning models simpler. It hides complex operations that previously had to be written by hand (e.g., training loops and validation steps), letting researchers concentrate on the essential parts of model development.
The framework’s key advantages are:
- straightforward scaling across multiple GPUs and nodes,
- automatic handling of routine tasks such as checkpointing, logging or early stopping.
- an even more modular structure, which makes code easier to read and maintain.
JAX +Flax
JAX is an open-source library for numerical computation created by Google that pairs Python’s flexibility with hardware acceleration. Its key advantages are:
- automatic differentiation that works natively with Python and NumPy.
- Just-In-Time Compilation (JIT), which speeds up Python functions by turning them into optimised machine code that runs on CPU, GPU and TPU.
- broad vectorisation and parallelism, enabling code to run efficiently on various hardware platforms.
Flax, in turn, is a high-level neural network library built on JAX, also developed by Google. Its main goal is a modular design that is easy to customise. It manages model parameters, state, and randomness in a structured way, which lets it keep model logic apart from training logic and makes code easier to debug.
The JAX and Flax combination is the go-to tool for both Deepmind and Google AI.
Analysis
We have gone over the general background of and motivation for each framework, but we also want some qualitative comparisons, such as popularity, along with UX and performance measures like execution time, ease of writing, and so on. The next section gives a clearer picture of which framework suits which task best.
Popularity
The view from GitHub stars
Below is a comparison of how popular the selected frameworks are on GitHub.
GitHub popularity
Pytorch clearly leads (81k stars), with Tensorflow (74k stars) close behind, followed by its tightly linked framework, Keras (61k). JAX (29k stars) and Pytorch Lightning (28k stars) are newer libraries, while Flax (6k stars) is the least popular, being the newest of all.
The research view
Furthermore, if we look at which framework new papers use, Pytorch clearly dominates in the research community (over 75% of new deep learning papers rely on Pytorch). This ultimately shows that people are more inclined to use new models in Pytorch; that said, aggregators such as Hugging Face’s Transformer frequently make it possible to stay model agnostic without converting models by hand.
Simple computer vision example
Experimental setup
For this solution, we concentrate on a simple example that uses:
- a Neural Network made of 5 Conv layers with Batch Normalization (since it makes the training and testing phases behave differently)
- the Imagenette dataset, comprising the 10 most accessible classes of ImageNet, which gives 9472 examples.
- the Adam optimizer with a plain learning rate of 0.001.
Performance
This section looks at two main implementation styles in Machine Learning: one suited to large datasets that works in a streaming manner, and another best for smaller datasets that fit entirely in GPU memory.
Out-of-the-box implementation (stream-based)
| Pytorch | Pytorch lightning | Tensorflow + Keras | JAX |
| 232s (180s using the native loader) | 180s | 64s | 84s |
Here, data is loaded in a streaming manner, meaning every image is fetched through a data loader only when it is needed. Each setup could be tuned further in many ways, but our goal is a simple, out-of-the-box implementation. The results show Tesorflow and JAX to be faster and Pytroch and Pytorch lightning to be slower. The explanation is simple: Pytorch and Pytorch lightning load images with PIL, whereas Tensorflow and FLAX rely on the native TF implementation. By experimenting with a custom Dataloader for Pytorch (using torchvision.io.read_image()), we managed to cut the processing time by 22% (to 180s). That is still considerably slower than TF processing (which internally uses the highly optimised tf.io.read_file()).
All in-memory implementation (memory-based)
Unlike the previous setup, here we used a subset of 3/10 classes because of memory constraints) and put the entire dataset on the GPU (reducing the impact of slower DataLoaders). With this setup, Pytorch appears considerably faster than the alternatives, PyTorch Lightning and JAX achieve comparable times, and Tensorflow is the slowest. This may suggest that Pytorch Lightning and Keras carry some additional overhead because of their extra features.
Both examples (stream-based and memory-based) are simple experiments on the Imagenette dataset. Rather than proving which framework is best, they nicely highlight certain caveats hidden behind the basic API. Some of these caveats, already touched upon (such as data loading), will be explained, and others will be covered in the next section.
Coding UX and analysis
Every example uses a comparable default setup, loading a subset of Imagenet called Imagenette (the 10 most accessible classes of ImageNet) with 9472 examples. Below is a comparison of lines of code (LOCs), excluding imports and configuration that has to be set.
LOCs compared across implementations in different frameworks
Pytorch and Tensorflow need the most lines, while JAX is the most compact (though it may require some system variables to be set). What should stand out here are the characteristics of each framework. Pytorch enables strong modularisation of the codebase (with an OOP approach to building models), and Pytorch Lightning goes a step further by applying a similar approach to the data plane. The result is much cleaner code that is easier to understand.
Tensorflow began with a more functional style but later switched to a subclassed API (inspired by Pytorch’s success), which enables this as well. Datasets and dataloaders in that framework, however, are still more often written in a functional style.
The philosophy of Flax and JAX is to keep the model architecture and the training characteristics independent of each other. Accordingly, you are expected to create a TrainingState that has access to the model’s parameters, the selected optimizer, and the learning rate, without touching the exact architecture of the model.
It is worth noting that Tensorflow has built up some technical debt over the years, visible in breaking changes (and, in effect, poor support and compatibility between Keras 3.0 and TF2)
Interesting differences between frameworks
Different initialisers
Another notable difference among the frameworks lies in how they initialise weights by default:
- Tensorflow initialises weights with Xavier Uniform,
- Pytorch initialises weights with Kaiming Uniform,
- Flax initialises weights with LeCun Normal.
Distribution of initial weights compared for the Xavier, Kaming and LeCun initialisations
Kaming considers the number of incoming units of a layer (for a convolution layer: the number of input channels times the kernel size), whereas Xavier initialisation also considers the outgoing units (fan_out). LeCun initialisation (a variant of Xavier initialisation) works better for layers with tanh activation.
They may all seem alike, but different initializiers can strongly affect how models begin training and which direction they take, so paying attention to such details can make a big difference when porting code to a new framework.
Different processing backends
As mentioned in the performance section, there are some differences in how images – and, consequently, data augmentation – are handled. Pytorch relies on PIL, whereas Tensorflow and JAX use more native solutions. This cuts both ways: PIL-based image augmentations are easier to perform, but they may be less optimised than native solutions and end up lengthening preprocessing (as happened in our stream-based example).
Different GPU memory allocation algorithms
Pytorch and Pytorch Lightning allocate memory incrementally, taking more as the need arises. Tensorflow and JAX, by contrast, allocate greedily, which can lead to odd errors when they are used within the same scope. This well-known issue is described on the JAX website and can be fixed with a few lines of code.
These differing strategies can affect processing time, but they can also grab more memory whenever needed, preventing other components that may require the GPU from using it.
Support for existing models
The analysis as a whole shows that JAX is consistently a solid and readable choice. Because it is the newest and not yet widely adopted, though, more models with trained weights are still needed. This can be worked around by using HuggingFace and its transformer library, but that may not be ideal when you want to keep dependencies to a minimum.
Conclusions
With this blog post and its accompanying examples, grasping particular caveats and weaknesses of the various frameworks becomes straightforward. Some are mature (such as Pytorch), some carry technical debt from earlier solutions (such as Tensorflow), and some have yet to prove their considerable potential while their community keeps growing every day, e.g., JAX + Flax.