Clarification on `load_mediapipe_directory` expected input format

[GerrySant]

I’ve been trying to convert MediaPipe Holistic pose estimations into the .pose format using pose-format, but I’m having some trouble understanding what exact JSON structure load_mediapipe_directory expects.

---

🧩 Context

I extracted the pose skeleton from a video using the following MediaPipe Holistic code:

import cv2
import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision
import os
import json

signal = "/home/gsantm/store/data/aligned_yolo_cropped_how2sign/test/clips/-fZc293MpJk_0-1-rgb_front.mp4"
output_root = "/home/gsantm/store/pose_estimators/mediapipe_holistic/output"
model_path = os.path.expanduser("~/store/pose_estimators/mediapipe_holistic/pose_landmarker_lite.task")

sample_name = os.path.splitext(os.path.basename(signal))[0]
output_dir = os.path.join(output_root, sample_name)
os.makedirs(output_dir, exist_ok=True)

BaseOptions = mp.tasks.BaseOptions
PoseLandmarker = mp.tasks.vision.PoseLandmarker
PoseLandmarkerOptions = mp.tasks.vision.PoseLandmarkerOptions
VisionRunningMode = mp.tasks.vision.RunningMode

options = PoseLandmarkerOptions(
    base_options=BaseOptions(model_asset_path=model_path),
    running_mode=VisionRunningMode.IMAGE
)

with PoseLandmarker.create_from_options(options) as landmarker:
    cap = cv2.VideoCapture(signal)
    if not cap.isOpened():
        raise RuntimeError(f"Error opening video file: {signal}")

    frame_idx = 0
    while True:
        ret, frame = cap.read()
        if not ret:
            break

        frame_rgb = cv2.cvtColor(frame, cv2.COLOR_BGR2RGB)
        mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=frame_rgb)

        result = landmarker.detect(mp_image)

        if result.pose_landmarks:
            landmarks = [
                {"x": lm.x, "y": lm.y, "z": lm.z, "visibility": lm.visibility}
                for lm in result.pose_landmarks[0]
            ]
        else:
            landmarks = []

        frame_json_path = os.path.join(output_dir, f"frame_{frame_idx}.json")
        with open(frame_json_path, "w") as f:
            json.dump({
                "frame": frame_idx,
                "landmarks": landmarks
            }, f, indent=2)

        frame_idx += 1

    cap.release()

This creates a directory like this:

/home/.../-fZc293MpJk_0-1-rgb_front/
├── frame_0.json
├── frame_1.json
├── frame_2.json
└── ...

Each JSON looks like this:

{
  "frame": 0,
  "landmarks": [
    {
      "x": 0.4936003088951111,
      "y": 0.19043749570846558,
      "z": -1.181049108505249,
      "visibility": 0.9997064471244812
    },
    {
      "x": 0.5202051401138306,
      "y": 0.15045350790023804,
      "z": -1.1277518272399902,
      "visibility": 0.9993149042129517
    },
(...)
    {
      "x": 0.4366815686225891,
      "y": 1.9366867542266846,
      "z": 0.12023007124662399,
      "visibility": 0.04526037722826004
    }
  ]
}

---

🧠 What I tried

Then I tried converting them into .pose format using:

from pose_format.utils.holistic import load_mediapipe_directory

signal = "/home/.../-fZc293MpJk_0-1-rgb_front"
pose = load_mediapipe_directory(signal, fps=50, width=674, height=588)

with open(f"{signal}/pose.pose", "wb") as f:
    pose.write(f)

But I get this error:

Traceback (most recent call last):
  File "...", line 16, in <module>
    pose = load_mediapipe_directory(output_path, fps=fps, width=width, height=height)
  File ".../pose_format/utils/holistic.py", line 397, in load_mediapipe_directory
    num_pose_points = first_frame["pose_landmarks"]["num_landmarks"]
KeyError: 'num_landmarks'

This makes sense because my JSON files do not have a num_landmarks key.
I couldn’t find where this value comes from, since it doesn’t seem to appear in the MediaPipe result object (after result = landmarker.detect(mp_image)).

---

🔍 What I’ve checked

Reading the code here, it seems num_landmarks and related metadata are expected to already exist in the JSON files — and it looks like in your implementation you assign these values internally.

---

❓Questions

1. Is there any example or script that shows how you extracted and formatted the MediaPipe Holistic outputs to make them compatible with load_mediapipe_directory?
2. How can I know how many landmarks should be included for:
   • face_landmarks
   • pose_landmarks
   • left_hand_landmarks
   • right_hand_landmarks

I just want to add that, while I understand you expect people to use your internal code for pose estimation (which makes sense and has never encountered any issue using your code), my goal is simply to understand how to correctly map MediaPipe Holistic outputs to your .pose format — and potentially generalize this process for other pose estimators in the future.

[bricksdont]

Can you clarify why you are using .mediapipe.tasks.vision.PoseLandmarker.detect() instead of mediapipe.solutions.holistic.Holistic.process()?

Isn't the landmarker a more coarse-grained tool with a lot fewer keypoints per frame? How many landmarks are there per frame after running your code?

This library is compatible only with mediapipe holistic, not with other related mediapipe tools. I suspect that is the key answer, but not 100% sure

a different explanation: mediapipe has changed and the code in this library has not been updated yet

[bricksdont]

After a quick test, mediapipe.solutions.holistic.Holistic.process() also does not lead to a result object that has num_landmarks (example code). @AmitMY do you know more?

[bricksdont]

You can get the number of landmarks for each component like this:

>>> num_landmarks = len(results.pose_landmarks.landmark)
>>> print(f'Total number of landmarks: {num_landmarks}')
Total number of landmarks: 33

[AmitMY]

Why are you not using load_holistic or the CLI tool? (video_to_pose -i video.mp4 -o video.pose --format mediapipe)

[bricksdont]

@AmitMY Sant would like to understand how a format based on JSON can be converted to .pose, and thought that would give some hints.

But regardless, I think it's a legitimate question to ask what load_mediapipe_directory is for. I see this was added by @J22Melody: 825f075

@J22Melody what's the use case for which you added this function?

[J22Melody]

IIRC, I added the load_mediapipe_directory function to load JSON-format poses into pose-format, likely from some old data for the shared task. This should serve the same purpose that @GerrySant wants.

I did not have any impression on num_landmarks, I suspect this was added later and then broke the load_mediapipe_directory function. Can you check?

[cleong110]

If it helps, I have some examples of poses with the "standard" number of landmarks here, and "refined" landmarks (10 extra points) here.

I gave it a go, writing out the data to the JSON folder format and trying to read it back, I think the Header stuff might be wrong.

https://colab.research.google.com/drive/1cCIVu_Kszpv-2Dq63bBtIrUoxknaNUZ2?usp=sharing

[bricksdont]

thank you Colin for chipping in! @GerrySant please have a look at Colin's code

also, @J22Melody I checked now, num_landmarks is part of the legacy JSON format we used for WMT-SLT22, data produced by Surrey. Example JSON for one frame:

{
    "face_landmarks": {
        "num_landmarks": 478,
        "fields": [
            "x",
            "y",
            "z"
        ],
        "landmarks": []
    },
    "pose_landmarks": {
        "num_landmarks": 33,
        "fields": [
            "x",
            "y",
            "z",
            "visibility"
        ],
        "landmarks": []
    },
    "pose_world_landmarks": {
        "num_landmarks": 33,
        "fields": [
            "x",
            "y",
            "z",
            "visibility"
        ],
        "landmarks": []
    },
    "left_hand_landmarks": {
        "num_landmarks": 21,
        "fields": [
            "x",
            "y",
            "z"
        ],
        "landmarks": []
    },
    "right_hand_landmarks": {
        "num_landmarks": 21,
        "fields": [
            "x",
            "y",
            "z"
        ],
        "landmarks": []
    }
}

to make load_mediapipe_directory more useful it should not fail if num_landmarks is not in the JSON

finally, @GerrySant maybe also look at how Openpose JSONs are loaded, that should be more straightforward