A 2023 Guide to Human Pose Estimation
This post walks through a broad range of topics on human pose estimation – from the basic definitions, through example use cases, to metrics and datasets.
What is Human Pose Estimation?
Human pose estimation is a well-known Computer Vision task whose history spans over 20 years. The field is about locating the joints of the human body (for instance knees and wrists), referred to as key points, in pictures or video footage.
There are three distinct ways of representing the whole human body:
- a skeleton-based model (also known as a kinematic model)
- a contour-based model (also known as a planar model)
- a volume-based model (also known as a volumetric model)
a) skeleton-based, b) contour-based c) volume-based | Source: ” A comprehensive survey on human pose estimation”
The goal of human pose estimation is to build a skeleton-based representation first and then to process it in whatever way the target application requires. From this point on, we will talk about skeleton-based datasets, metrics and solutions. We concentrate on 2D human pose estimation, since 3-D approaches frequently start from 2D algorithms and only afterwards regress the results into three-dimensional space.
Why is it hard?
The key difficulties in Human Pose Estimation revolve around:
- occlusions,
- atypical human poses (see the image below),
- key points that are not visible,
- varying lighting and clothing.
To deal with these issues, practitioners either design new methods or build demanding datasets, so that the resulting models can withstand such problems. Later on we will look at several solutions that attack these problems through more sophisticated architectures.
Right arm and left leg key points predicted wrongly because of an unusual pose, Source: TU Delft
Use Cases
AI-powered personal trainers
Thanks to recent progress in Computer Vision, combined with gyms being shut for months, apps that support people exercising at home became much more popular. Under the hood, every one of these apps relies on a Human Pose Estimation model to follow the user’s movements, count repetitions and propose corrections to their technique.
The Kaia health mobile app: kaiahealth.de
Augmented Reality & CGI
Gathering data for Augmented Reality and CGI is costly and slow (for example when motion capture technology is used), so human pose estimation models are frequently applied to obtain cheaper but still precise avatars for films or Augmented Reality apps. Disney Research is investigating how to generate avatars from Human Pose Estimation to make AR applications feel closer to real life.
Source: AR Poser
Interactive Gaming Experience
Several body-tracking cameras have been introduced to make gaming more immersive. Microsoft Kinect, for example, relies internally on 3d human pose estimation to follow the player’s movements and translate them into the actions of the in-game hero. The picture below shows a little toddler playing a game about flying like a bird.
A toddler playing a bird flying game, Source: Towards Data Science
Cashier-less shops
Cashier-less shops such as Amazon GO owe their growth to progress in Human Pose Estimation. Analysing and tracking shoppers makes it possible to determine which products, and how many of them, a customer put into their basket. As a result, people can walk out of the shop without queueing for a long time to pay.
Amazon Go in Seattle – a prototype cashier-less shop. Source: Wikipedia
Monitoring aggressive behaviors
Human Pose Estimation makes it possible to spot violent and dangerous behaviour in cities. Karlsruhe Institute of Technology and Mannheim Police Headquarters are working together, and their solution is going to be tested in real-life conditions in 2023.
Detecting street fights with Human Pose Estimation. Source: “Where are we with Human Pose Estimation in Real-World Surveillance?”
Datasets
MPII
The MPII dataset contains 40k person instances, each annotated with 16 joints. The training and validation sets hold 22k and 3k person instances respectively.
COCO
The COCO dataset contains 200k images with 250k person instances annotated with 17 key points.
Below you can see how the body joint annotations of MPII and COCO differ.
Source: TU Delft
OCHuman
This dataset concentrates on difficult, occluded examples. It was presented in 2019 together with the paper “Pose2Seg: Detection Free Human Instance Segmentation”. It contains 5081 images with 10375 person instances, every one of which is heavily occluded (MaxIOU >0.5).
Metrics
PCK - Percentage of Correct Keypoints
The MPII dataset uses this metric: a detected joint counts as correct if the distance between its predicted and true locations falls below a given threshold. To tie the threshold to body size, it is commonly expressed as a fraction of the head segment length. The main metric for the MPII dataset is PCK@0.5, which means that only joints lying within 0.5 ∗ ℎ𝑒𝑎𝑑_𝑏𝑜𝑛𝑒_𝑙𝑒𝑛𝑔ℎ𝑡 of the true position are treated as correctly detected.
<math xmlns="http://www.w3.org/1998/Math/MathML" display="block">
<mi>O</mi>
<mi>K</mi>
<mi>S</mi>
<mo>=</mo>
<mfrac>
<mrow>
<munder>
<mo data-mjx-texclass="OP">∑</mo>
<mi>i</mi>
</munder>
<mi>e</mi>
<mi>x</mi>
<mi>p</mi>
<mo stretchy="false">(</mo>
<mo>−</mo>
<mfrac>
<msubsup>
<mi>d</mi>
<mi>i</mi>
<mn>2</mn>
</msubsup>
<mrow>
<mn>2</mn>
<msup>
<mi>s</mi>
<mn>2</mn>
</msup>
<mo>∗</mo>
<msubsup>
<mi>k</mi>
<mi>i</mi>
<mn>2</mn>
</msubsup>
</mrow>
</mfrac>
<mo stretchy="false">)</mo>
<mo>∗</mo>
<mi>δ</mi>
<mo stretchy="false">(</mo>
<msub>
<mi>v</mi>
<mi>i</mi>
</msub>
<mo>></mo>
<mn>0</mn>
<mo stretchy="false">)</mo>
</mrow>
<mrow>
<munder>
<mo data-mjx-texclass="OP">∑</mo>
<mi>i</mi>
</munder>
<mi>δ</mi>
<mo stretchy="false">(</mo>
<msub>
<mi>v</mi>
<mi>i</mi>
</msub>
<mo>></mo>
<mn>0</mn>
<mo stretchy="false">)</mo>
</mrow>
</mfrac>
</math>
OKS - Object Keypoint Similarity
OKS is the primary metric of the COCO dataset. Here d is the distance between the predicted and the true position of the key point i is the visibility flag of the key point i, s is the scale of the object and is a per key point constant controlling the falloff (computed by the authors of the COCO dataset).
Put simply, OKS plays the role of a similarity measure, much like IOU does in Object Detection for Image Segmentation. This metric is usually reported through Average Precision (AP@50, AP@75, and the mean over 10 steps between @50 and @95) and average Recall (using the same steps as AP).
Deep Learning models
Over the history of Human Pose Estimation, many solutions were built on classical Computer Vision, concentrating on body parts and on variations in colour and contrast. In recent years, however, deep learning solutions have taken over the field, so the next part concentrates on them.
There are two families of deep learning solutions:
- top-down: first detect the person, then regress key points inside the selected bounding box.
- bottom-up: find key points without assigning identities, then group them into individual persons.
Top-down approaches
As mentioned earlier, the top-down family includes many examples with good results. All of them learn from heatmaps of key point locations rather than from the key point positions themselves. This approach turned out to give noticeably better and more robust results. Here we restrict ourselves to the 3 pose estimation architectures that had the greatest influence and shaped this family of approaches.
Overview of the Hourglass structure – a chain of identically built Hourglass modules that repeatedly lower (with strided convolutions) and then raise (with nearest neighbor upsampling) the resolution. From: original paper.
The hourglass approach in fact stacks several modules that share exactly the same structure. Every module performs downsampling and upsampling (which makes it look like an hourglass). This design captures both local context (for example, where the wrist is) and global context (for example, how the body is oriented). To improve learning, intermediate supervision is applied after every module, comparing the predicted heatmaps against the true positions.
The HRNet architecture: an arrow denotes convolution, a downward arrow strided convolution, and an upward arrow upsampling. Source: original paper
Earlier solutions moved from high to low and back to high resolution, whereas this architecture attempts to keep high resolution during the entire process, as the figure above shows. It begins with high resolution, and at every depth step it adds more parallel scales, each receiving information from the higher, equal or lower resolutions of the preceding steps. Because high-resolution features are available at every step, HRNet has remained at the top of most HPE leaderboards.
(a) The ViTPose framework. (b) The transformer block. (c) The classic decoder. (d) The simple decoder. (e) The decoders for several datasets, Source: original paper
Once vision transformers appeared and gained traction in Computer Vision, it was only a question of time until a “Transformer for Pose Estimation” would be proposed. This solution is built from a set of TransformerBlocks (each combining Layer Normalization, Multi Headed Self Attention and a Feed Forward Network) plus a decoder module. Once the encoder has extracted the features, a fairly simple decoder is applied (twice: a Deconvolution Layer followed by Batch Normalization and Relu, and then a linear predictor layer). The network scales easily and doesn’t need convolutional layers to be carefully assembled with a calculated number of parameters, yet it still delivers very strong results.
It also appears to perform well on multi-person pose estimation with heavy occlusions (it currently leads the OCHuman dataset).
Bottom-up approaches
As noted earlier, bottom-up approaches generate several skeletons in one pass, which tends to make them faster and better suited to real-time solutions, and they also do better in crowded scenes for multi person pose estimation.
How OpenPose produces human skeletons for a given image – the pipeline, Source: original paper
OpenPose is likely the best-known bottom up approach model, since it was published as an open-source library in 2018. Part of its popularity also comes from being among the first real-time solutions offering reliable, broadly applicable human pose estimation.
This is how the architecture works:
- features are first extracted by the initial few layers
- Next come two branches of convolutional layers: the first produces 18 confidence maps, each standing for a specific part of the human pose skeleton, and the second produces 38 Part Affinity Fields (PAFs), which express how strongly parts are associated (a bipartite graph of keypoint to keypoint connections) .
- Keypoint to keypoint connections with low confidence are pruned: PAFs make it possible to drop links between keypoints that are unlikely to belong to the same person instance.
- Once pruning is done, several human poses are regressed.
These properties make the solution a good fit for real time multi person pose estimation.
Omnipose paper
The Omnipose architecture, Source: original paper
Omnipose is at the moment the top-performing bottom up approach architecture, and its structure is fairly simple. It opens with two 3x3 convolutions and a ResnetBottleneck block. These are followed by 3 HRNet blocks, each enriched with Gaussian heat map modulation (introduced in the Dark paper). This enhancement changes how the exact keypoint location is determined: instead of searching for the maximum value (as earlier solutions did), it assumes the heatmap of a given keypoint follows a Gaussian distribution and tries to locate that distribution’s centre.
The key contribution of the Omnipose architecture is the Waterfall Atrous Spatial Pyramid module (WASPv2), shown below.

The Waterfall Atrous Spatial Pyramid module, Source: original paper
The module draws partly on the WASPv1 module (first introduced in the Unipose paper), but adds the idea of merging backbone features with low-level features. With all of these enhancements, the architecture currently ranks among the best-performing ones.
MoveNet paper

The MoveNet architecture, Source: Tensorflow blog
MoveNet is a widely used bottom up approach model that relies on heatmaps to pinpoint human key points precisely. Its architecture is made up of:
- a feature extractor - basically MobileNetV2 combined with a feature pyramid network (FPN)
- a set of prediction heads - based on a CenterNet-inspired scheme, slightly modified to boost speed and accuracy.
The 4 prediction heads are:
- a person center heatmap,
- a keypoint regression field that predicts the keypoints of a person,
- a person key point heatmap that predicts every keypoint no matter which person instance it belongs to,
- a 2D per-keypoint offset field that predicts the offset from every pixel of the output feature map to the location of each keypoint.
Sample images from the Activity dataset. Source: Tensorflow blog
The architecture was trained on CocoDataset extended with Activity, a private Google dataset focused on demanding fitness and yoga poses. The new architecture together with the robust training data make this solution fairly resilient to occlusions and unfamiliar poses. The Google team published it as a ready-to-use solution with tf-js. The model comes in two variants: “lightning”, optimised for speed, and “thunder”, targeting better accuracy (while still keeping 30+FPS). A live demo is available on their website.
Our ML team’s model suggestions
Clearly, there is a range of solutions to pick from. Our recommendations are as follows:
- With plenty of data and the option to work offline, VIT, Omnipose or HRNet should be your first choice.
- For real-time or crowd applications, look at MoveNet or OpenPose.
- When data is scarce, smaller models will likely do better (e.g. MoveNet “lightning”, HRNet-32 or OmniPose-Lite), or larger variants combined with careful and extensive augmentations.
Summary
Human Pose Estimation is a demanding yet fascinating area, commonly applied in sports and gaming. This post covered a broad range of topics: the basic definitions and challenges, several use cases, the metrics and datasets used to evaluate models, and the most influential top-down and bottom-up approaches, including those that are currently SOTA.
