LDPose: Towards Inclusive Human Pose Estimation for Limb-Deficient Individuals in the Wild

Abstract

Limb-Deficient Pose Estimation Task

Task Definition

25-Keypoint Annotation Schema

Figure: Demonstration about the 25 Keypoints used in the LDPose. The MSCOCO 17 standard body keypoints are shown in green, and the newly introduced 8 residual limb keypoints are highlighted in blue. Real-world examples of residual limbs, both with and without prosthetics, are also presented.

Standard Body Keypoints (17)

Nose, Eyes (×2), Ears (×2), Shoulders (×2), Elbows (×2), Wrists (×2), Hips (×2), Knees (×2), Ankles (×2)

Residual Limb Endpoints (8)

Above Left/Right Elbow Residual Limb End (×2), Below Left/Right Elbow Residual Limb End (×2), Above Left/Right Knee Residual Limb End (×2), Below Left/Right Knee Residual Limb End (×2)

Impact and Applications

Limb-Deficient Pose (LDPose) Dataset

Dataset Features

• Comprehensive Coverage: Various major limb deficiency types including upper limb, lower limb, and bilateral deficiencies
• Diverse Scenarios: Images collected from Paralympic competitions, training sessions, and daily life activities
• Expert Annotation: Guided by internationally accredited para-athletics classifiers to ensure medical accuracy
• Inclusive Representation: Diverse ethnic backgrounds and age groups to promote inclusivity
• High Quality Standards: Rigorous quality control with multiple annotation rounds and validation

Data Examples

Original Image

Pose Annotation

Upper Limb Deficiency: Athletes in competitive sports

Original Image

Pose Annotation

Lower Limb Deficiency: Daily life activities and mobility

Original Image

Pose Annotation

Bilateral Deficiency: Paralympic training sessions

Data Format

{
  "info": {
    "description": "LDPose Dataset",
    "version": "1.0",
    "year": 2025,
    "contributor": "LDPose Team"
  },
  "images": [
    {
      "id": 1,
      "file_name": "000000001.jpg",
      "width": 640,
      "height": 480,
      "date_captured": "2024-01-01 12:00:00",
      "url": "http://example.com/images/000000001.jpg"
    }
  ],
  "annotations": [
    {
      "id": 1,
      "image_id": 1,
      "category_id": 1,
      "bbox": [100, 50, 200, 350],
      "area": 70000,
      "iscrowd": 0,
      "keypoints": [
        320, 140, 2, 310, 135, 2, 330, 135, 2, 305, 140, 2, 335, 140, 2,
        290, 180, 2, 350, 180, 2, 280, 240, 2, 0, 0, 0, 270, 290, 0,
        0, 0, 0, 300, 280, 2, 340, 280, 2, 295, 380, 2, 345, 380, 2,
        290, 470, 2, 350, 470, 2, 0, 0, 0, 360, 200, 2, 0, 0, 0,
        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0
      ],
      "num_keypoints": 14
    }
  ],
  "categories": [
    {
      "id": 1,
      "name": "LDHuman",
      "supercategory": "",
      "keypoints": [
        "nose", "left_eye", "right_eye", "left_ear", "right_ear",
        "left_shoulder", "right_shoulder", "left_elbow", "right_elbow",
        "left_wrist", "right_wrist", "left_hip", "right_hip",
        "left_knee", "right_knee", "left_ankle", "right_ankle",
        "left_above_elbow_res", "right_above_elbow_res",
        "left_below_elbow_res", "right_below_elbow_res",
        "left_above_knee_res", "right_above_knee_res",
        "left_below_knee_res", "right_below_knee_res"
      ],
      "skeleton": [
        [0,1],[1,3],[0,2],[2,4],[3,5],[4,6],[5,7],[6,8],[7,9],[8,10],[5,11],[6,12],[12,14],[11,13],[13,15],[14,16],
        [5, 17], [7, 19], [6, 18], [7, 20],[11, 21], [12, 22], [13, 23], [14, 24]
      ]
    }
  ]
}

Limb-Deficient Loss (LDLoss)

Our proposed LDLoss (also called Limb Visibility Loss or paraLoss) is designed to better distinguish residual limb keypoints by contrasting them with intact limb keypoints. The loss function operates on limb pairs, processing visibility predictions for both endpoints of each limb through a cross-entropy loss mechanism.

Algorithm: Limb-Deficient Loss (LDLoss)

Algorithm 1: Limb-Deficient Loss (LDLoss)

Require: kpt_vis_preds ∈ ℝ^(B×K×C), vis_targets ∈ {0,1}^(B×K), 
         LimbPairs (P pairs), loss_weight

1:  # Extract prediction vectors for both endpoints of each limb (expanded by pair)
2:  PredPairs ← []
3:  for each (i, j) in LimbPairs do
4:      # Select visibility predictions for keypoint i and j, both with dimension B×1×C
5:      Pi ← kpt_vis_preds[:, i, :]           # B×C
6:      Pj ← kpt_vis_preds[:, j, :]           # B×C
7:      # Concatenate both endpoints of the same limb as a "two-sample batch" for CE
8:      PredPairs.append( concat(Pi, Pj, dim=0) )  # 2B×C
9:  end for
10: PredPairwiseConfidence ← stack(PredPairs, dim=0)      # P×(2B)×C
11: PredPairwiseConfidence ← reshape(PredPairwiseConfidence, (P*2B, C))

12: # Generate corresponding target label sequence (consistent with concatenation order)
13: VisPairs ← []
14: for each (i, j) in LimbPairs do
15:     Ti ← vis_targets[:, i]   # B
16:     Tj ← vis_targets[:, j]   # B
17:     VisPairs.append( concat(Ti, Tj, dim=0) )  # 2B
18: end for
19: Visibilities ← stack(VisPairs, dim=0)                 # P×(2B)
20: Visibilities ← reshape(Visibilities, (P*2B))          # P*2B

21: # Compute cross-entropy loss on the paired "sample batch"
22: Loss_CE ← CrossEntropyLoss(PredPairwiseConfidence, Visibilities)

23: return loss_weight × Loss_CE

Benchmark

Experimental Environment

Limb-Deficient Metrics (LD Metrics)

COCO Metrics evaluate only the visible keypoints in the ground truth. This works well for individuals with complete bodies. However, for individuals with limb deficiencies, some keypoints do not exist. For example, if a person has lost their left upper limb, the left elbow and wrist are set to (0,0) with zero visibility. If a model predicts these points with high confidence, COCO Metrics ignore the error and give a misleading high score.

To address this limitation, we propose LD Metrics, which extends COCO Metrics by incorporating adaptive keypoint selection based on the realistic distribution of related keypoints. We introduce an adaptive weight, γ_i, into the OKS calculation to enforce consistency among associated keypoints. The new OKS_LDPose is defined as follows:

Here, d_i, κ_i, s, and δ(v_i > 0) are defined as in the original COCO evaluation. The weight γ_i is set to 0 when the prediction of multiple related keypoints violates realistic anatomical configurations, even if the predicted keypoints are precise and marked as visible in annotation.

For instance, if keypoint 21 (left above-knee residual limb endpoint) is correctly predicted but any of the associated keypoints (i.e., keypoint 13 [left knee], keypoint 23 [left below-knee residual], or keypoint 15 [left ankle]) are simultaneously predicted in a manner that contradicts the expected limb structure (such as predicting both a residual limb and a natural joint), then γ₂₁ is set to 0. Otherwise, γ_i is 1.

Baseline Methods

We benchmarked our dataset using state-of-the-art human pose estimation methods:

Experimental Results

Quantitative Results

We benchmark our LDPose dataset using state-of-the-art pose estimation methods. The table shows COCO metrics (AP, AP⁵⁰, AP⁷⁵, AR) and our proposed LD Metrics (AP, AP⁵⁰, AP⁷⁵, AR) for evaluating residual limb keypoint detection.

Case Study Visualization

Figure: ViTPose with ViT-l backbone case study. First image column: LDPose ground truth. Second image column: Pretrained ViTPose inference. Third image column: LDPose fine-tuned ViTpose. Fourth image column: LDPose fine-tuned ViTpose with LDLoss.

Resource Downloads

@inproceedings{ying2023ldpose,
  title={LDPose: Towards Inclusive Human Pose Estimation for Limb-Deficient Individuals in the Wild},
  author={Ying, Jiaying and Du, Heming and Zhang, Kaihao and Li, Lincheng and Yu, Xin},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={},
  year={2025}
}