Skip to content

Pose

The pose submodule covers the full lifecycle of pose landmarks: extracting them from video with MediaPipe, describing their skeleton connectivity, transforming/augmenting them, and plotting them.

Across the submodule, a pose sequence is represented as a numpy array of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark (usually 3: x, y, z).

Extracting landmarks with MediaPipe

sign_language_tools.pose.mediapipe.extraction wraps MediaPipe's holistic landmarker to extract pose, hands and face landmarks from a video file frame by frame.

from sign_language_tools.pose.mediapipe.extraction import (
    load_holistic_landmarker,
    extract_poses_from_video_file,
)

landmarker = load_holistic_landmarker("holistic_landmarker.task")
poses = extract_poses_from_video_file("video.mp4", landmarker, show_progress=True)

poses["pose"].shape        # (T, 33, 3)
poses["left_hand"].shape   # (T, 21, 3)
poses["right_hand"].shape  # (T, 21, 3)
poses["face"].shape        # (T, 478, 3)

Extracted landmarks (skeleton only, skeleton overlaid on the frame, and the original frame)

Frames where a body part wasn't detected are filled with NaN; see InterpolateMissing to fill them in.

Skeleton connectivity

sign_language_tools.pose.mediapipe.edges and sign_language_tools.pose.mediapipe.vertices provide the connectivity (edges) and landmark groupings (vertices) for the MediaPipe skeletons, and sign_language_tools.pose.openpose.edges provides the equivalent for OpenPose. These are typically used to draw skeletons (see Visualization and the Video Player) or to build graph structures over the landmarks.

from sign_language_tools.pose.mediapipe.edges import POSE_EDGES, HAND_EDGES, FACE_EDGES
from sign_language_tools.pose.mediapipe.vertices import LIPS_VERTICES

Transforms

sign_language_tools.pose.transforms provides data-augmentation and preprocessing transforms for pose sequences, compatible with PyTorch/TensorFlow data pipelines. They can be composed with Compose from the common submodule.

import numpy as np
from sign_language_tools.pose.transforms import (
    Concatenate, TemporalRandomCrop, HorizontalFlip, Split, InterpolateMissing,
)

landmarks = {
    "pose": np.load("examples/ressources/pose.npy"),
    "left_hand": np.load("examples/ressources/left_hand.npy"),
    "right_hand": np.load("examples/ressources/right_hand.npy"),
}

pipeline = [
    InterpolateMissing(),
    Concatenate(["pose", "right_hand", "left_hand"]),
    TemporalRandomCrop(size=60),
    HorizontalFlip(),
    Split({"pose": 33, "right_hand": 21, "left_hand": 21}),
]

x = landmarks
for transform in pipeline:
    x = transform(x)

Transforms operating on a single array (e.g. TemporalRandomCrop, HorizontalFlip, InterpolateMissing) take and return a (T, L, C) array. Concatenate and Split bridge between a dict[str, np.ndarray] of landmark groups (e.g. {"pose": ..., "left_hand": ...}) and a single concatenated (T, L, C) array, so that group-agnostic transforms can be applied uniformly.

Visualization

sign_language_tools.pose.visualization provides matplotlib plotting for landmarks, one frame or a whole sequence at a time. For interactive playback synced to a video, see the Video Player instead.

from sign_language_tools.pose.visualization import plot_landmarks, plot_landmarks_sequence
from sign_language_tools.pose.mediapipe.edges import POSE_EDGES, HAND_EDGES

# A single frame
plot_landmarks(landmarks["pose"][0], connections=POSE_EDGES)

# A short sequence, one subplot per frame
fig = plot_landmarks_sequence(
    {"pose": landmarks["pose"][:5], "right_hand": landmarks["right_hand"][:5]},
    {"pose": POSE_EDGES, "right_hand": HAND_EDGES},
)

Reference

Extraction

sign_language_tools.pose.mediapipe.extraction.load_holistic_landmarker

load_holistic_landmarker(
    model_path: str,
    use_gpu: bool = False,
    options: HolisticLandmarkerOptions | None = None,
)

Load a MediaPipe holistic landmarker for video inference.

Parameters:

Name Type Description Default
model_path str

Path to the holistic landmarker .task model file.

required
use_gpu bool

Whether to run inference on GPU instead of CPU. Defaults to False.

False
options HolisticLandmarkerOptions | None

Custom landmarker options. If None, default options are used with the model running in VIDEO mode. Defaults to None.

None

Returns:

Name Type Description
HolisticLandmarker

The initialized holistic landmarker, ready to process video frames.

sign_language_tools.pose.mediapipe.extraction.extract_poses_from_video_file

extract_poses_from_video_file(
    video_path: str,
    holistic_landmarker: HolisticLandmarker,
    show_progress=False,
) -> dict[str, np.ndarray]

Extract holistic pose landmarks from every frame of a video file.

Parameters:

Name Type Description Default
video_path str

Path to the video file to process.

required
holistic_landmarker HolisticLandmarker

Landmarker used to run detection on each frame, e.g. as returned by load_holistic_landmarker.

required
show_progress bool

Whether to display a progress bar while iterating over the video frames. Defaults to False.

False

Returns:

Type Description
dict[str, ndarray]

dict[str, np.ndarray]: Mapping from landmark group ("pose", "left_hand", "right_hand", "face") to an array of shape (T, L, C), with T the number of frames, L the number of landmarks in the group and C=3 (x, y, z).

Edges and vertices

sign_language_tools.pose.mediapipe.edges

Edge (connectivity) definitions for MediaPipe hand, pose and face landmarks.

Each constant is a tuple of (start_index, end_index) pairs, where the indices refer to positions in the corresponding landmark array (hand: (21, C), pose: (33, C), face: (478, C)). These are typically used to draw skeletons or build adjacency structures (e.g. graph-based models) over the landmarks.

"Upper"/"lower" pose edges split the body pose skeleton into the upper body (arms, shoulders, face landmarks 0-10) and the lower body (hips and legs).

sign_language_tools.pose.mediapipe.vertices

Vertex groups for the MediaPipe face mesh, derived from its edge topology.

Each constant below is a tuple of landmark indices into the (478, C) array of face landmarks produced by MediaPipe (see results.face_landmarks in extract_poses_from_video_file).

The vertex tuples are computed from the corresponding FACEMESH_* edge definitions in sign_language_tools.pose.mediapipe.facemesh rather than listed by hand, so they always stay in sync with the edges: any vertex that appears in an edge is included, and duplicates are removed.

"Left"/"right" follow MediaPipe's convention, i.e. the subject's own left and right, not the viewer's.

sign_language_tools.pose.openpose.edges

Transforms

sign_language_tools.pose.transforms.center.CenterOnLandmarks

CenterOnLandmarks(landmark_idx: int | tuple[int, ...])

Bases: Transform

Centers a pose sequence on one or more reference landmarks.

For every frame, the mean position of the given reference landmark(s) is subtracted from all landmarks, so that the reference point becomes the new origin.

Parameters:

Name Type Description Default
landmark_idx int | tuple[int, ...]

Index, or tuple of indices, of the landmark(s) used as the reference point. If several indices are given, their mean position is used as the origin.

required
Example

import numpy as np from sign_language_tools.pose.transforms import CenterOnLandmarks pose_sequence = np.random.rand(10, 33, 3) # (T, L, C) transform = CenterOnLandmarks(landmark_idx=0) centered = transform(pose_sequence) centered.shape (10, 33, 3)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Centers the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The centered pose sequence, with the same shape as pose_sequence.

sign_language_tools.pose.transforms.clip.Clip

Clip(min_value: float = 0.0, max_value: float = 1.0)

Bases: Transform

Clips the coordinates of a pose sequence to a fixed value range.

Any value below min_value is set to min_value, and any value above max_value is set to max_value.

Parameters:

Name Type Description Default
min_value float

Lower bound of the clipping range.

0.0
max_value float

Upper bound of the clipping range.

1.0
Example

import numpy as np from sign_language_tools.pose.transform import Clip pose_sequence = np.array([[[-0.2, 0.5, 1.3]]]) # (T, L, C) transform = Clip(min_value=0.0, max_value=1.0) transform(pose_sequence) array([[[0. , 0.5, 1. ]]])

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Clips the pose sequence coordinates.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The clipped pose sequence, with the same shape as pose_sequence.

sign_language_tools.pose.transforms.concatenate.Concatenate

Concatenate(
    landmark_sets: Sequence[str] = None,
    landmark_axis: int = 1,
)

Bases: Transform

Concatenates multiple pose sequences along the landmark axis.

Accepts either a dict mapping body part names to pose sequences (e.g. {"upper_body": ..., "left_hand": ..., "right_hand": ...}) or a list of pose sequences. All pose sequences must have compatible shapes along every axis except the concatenation axis.

Parameters:

Name Type Description Default
landmark_sets Sequence[str]

Ordered list of keys to select and order entries from a dict input. Ignored when the input is a list. If None, keys are sorted alphabetically.

None
landmark_axis int

Axis along which to concatenate. Defaults to 1 (the L dimension for pose sequences of shape (T, L, C)).

1
Example

import numpy as np from sign_language_tools.pose.transform import Concatenate upper_body = np.random.rand(10, 8, 2) # (T, L, C) left_hand = np.random.rand(10, 21, 2) # (T, L, C) transform = Concatenate(landmark_sets=["upper_body", "left_hand"]) pose_sequence = transform({"upper_body": upper_body, "left_hand": left_hand}) pose_sequence.shape (10, 29, 2)

__call__

__call__(
    pose_sequences: Union[
        dict[str, ndarray], list[ndarray]
    ],
) -> np.ndarray

Concatenates the pose sequences.

Parameters:

Name Type Description Default
pose_sequences Union[dict[str, ndarray], list[ndarray]]

Either a dict mapping body part names to pose sequences, or a list of pose sequences, each of shape (T, L, C) (with a possibly different L per entry).

required

Returns:

Type Description
ndarray

The concatenated pose sequence, of shape (T, sum(L), C).

sign_language_tools.pose.transforms.drop_coordinates.DropCoordinates

DropCoordinates(coord: int | str)

Bases: Transform

Drops one of the x, y, z coordinates from a pose sequence.

Parameters:

Name Type Description Default
coord int | str

Coordinate to drop, either as an index (0, 1, 2) or as a name ("x", "y", "z").

required
Example

import numpy as np from sign_language_tools.pose.transforms import DropCoordinates pose_sequence = np.random.rand(10, 33, 3) # (T, L, C) transform = DropCoordinates(coord="z") transform(pose_sequence).shape (10, 33, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Drops the configured coordinate from the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, 3), where T is the number of frames and L the number of landmarks.

required

Returns:

Type Description
ndarray

The pose sequence without the dropped coordinate, of shape

ndarray

(T, L, 2).

sign_language_tools.pose.transforms.drop_frames.DropRandomFrames

DropRandomFrames(
    max_n: Optional[int] = None,
    max_p: Optional[float] = None,
    fill_value: float = 0.0,
)

Bases: Transform

sign_language_tools.pose.transforms.edge_normalize.NormalizeByReferenceEdge

NormalizeByReferenceEdge(
    ref_edge: tuple[int, int],
    per_frame: bool = False,
    eps: float = 1e-08,
)

Bases: Transform

Scale a pose sequence using a reference edge as the scale unit.

Parameters:

Name Type Description Default
ref_edge tuple[int, int]

Indices (p1, p2) of the two landmarks defining the reference edge.

required
per_frame bool

If True, compute and apply a scaling factor independently for each frame, forcing the reference edge to length 1 in every frame. If False (default), compute a single scaling factor from the mean edge length over the whole sequence, which removes the sequence's overall scale while preserving relative scale changes across frames.

False
eps float

Small value added to the edge length to avoid division by zero.

1e-08

sign_language_tools.pose.transforms.filter.FilterEmpty

FilterEmpty()

Bases: Transform

Removes frames where every landmark is missing.

A frame is considered empty when all of its landmark coordinates are NaN (e.g. a frame where pose estimation failed on every landmark), and such frames are dropped from the sequence.

Example

import numpy as np from sign_language_tools.pose.transform import FilterEmpty pose_sequence = np.random.rand(3, 5, 2) # (T, L, C) pose_sequence[1] = np.nan # frame 1 is entirely missing transform = FilterEmpty() transform(pose_sequence).shape (2, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Removes empty frames from the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The pose sequence without its empty frames, of shape

ndarray

(T', L, C) with T' <= T.

sign_language_tools.pose.transforms.filter.FilterLandmarks

FilterLandmarks(filtered_indices: set[int] | list[int])

Bases: Transform

Removes a fixed set of landmarks from a pose sequence.

Parameters:

Name Type Description Default
filtered_indices set[int] | list[int]

Indices of the landmarks to remove.

required
Example

import numpy as np from sign_language_tools.pose.transform import FilterLandmarks pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = FilterLandmarks(filtered_indices=[0, 2]) transform(pose_sequence).shape (10, 3, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Removes the configured landmarks from the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The pose sequence without the filtered landmarks, of shape

ndarray

(T, L - len(filtered_indices), C).

sign_language_tools.pose.transforms.flatten.Flatten

Flatten(mode: str = 'landmarks')

Bases: Transform

Flattens the last two dimensions of a pose sequence.

Parameters:

Name Type Description Default
mode str

Either "landmarks" (default), which merges the landmark and coordinate axes into one per frame ((T, L, C) -> (T, L*C)), or "features", which merges the frame and coordinate axes into one per landmark ((T, L, C) -> (L, T*C)).

'landmarks'
Example

import numpy as np from sign_language_tools.pose.transform import Flatten pose_sequence = np.random.rand(10, 5, 3) # (T, L, C) Flatten(mode="landmarks")(pose_sequence).shape (10, 15) Flatten(mode="features")(pose_sequence).shape (5, 30)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Flattens the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The flattened pose sequence, of shape (T, L*C) if

ndarray

mode="landmarks", or (L, T*C) if mode="features".

sign_language_tools.pose.transforms.flatten.Unflatten

Unflatten(
    n_landmarks: int,
    n_channels: int,
    mode: str = "landmarks",
)

Bases: Transform

Reshapes a flattened pose sequence back to (T, L, C).

Inverts a Flatten operation performed with the same mode, n_landmarks, and n_channels.

Parameters:

Name Type Description Default
n_landmarks int

Number of landmarks L.

required
n_channels int

Number of coordinates per landmark C, typically 3 for (x, y, z).

required
mode str

Must match the mode used by Flatten. If "features", a transpose is applied to restore the original (T, L, C) layout.

'landmarks'
Example

import numpy as np from sign_language_tools.pose.transform import Flatten, Unflatten pose_sequence = np.random.rand(10, 5, 3) # (T, L, C) flat = Flatten(mode="landmarks")(pose_sequence) Unflatten(n_landmarks=5, n_channels=3)(flat).shape (10, 5, 3)

__call__

__call__(flat_pose_sequence: ndarray) -> np.ndarray

Unflattens the pose sequence.

Parameters:

Name Type Description Default
flat_pose_sequence ndarray

Flattened pose sequence, of shape (T, L*C) if mode="landmarks", or (L, T*C) if mode="features".

required

Returns:

Type Description
ndarray

The pose sequence reshaped back to (T, L, C).

sign_language_tools.pose.transforms.flip.HorizontalFlip

HorizontalFlip(x_origin: float = 0.5)

Bases: Transform

sign_language_tools.pose.transforms.interpolate.InterpolateMissing

InterpolateMissing(method: str = 'linear')

Bases: Transform

Fills missing landmarks (NaN coordinates) in a pose sequence by interpolation.

Builds an interpolation function over the frame axis (see get_landmark_interpolation_function) to fill in NaN landmarks from observed ones, then fills any values still missing after that (e.g. leading/trailing frames) using nearest-neighbor extrapolation. If the whole pose sequence is NaN, it is returned unchanged.

Parameters:

Name Type Description Default
method str

Interpolation method used for observed landmarks. One of "linear", "nearest", "previous", "next".

'linear'
Example

import numpy as np from sign_language_tools.pose.transform import InterpolateMissing pose_sequence = np.array([[[0.0]], [[np.nan]], [[2.0]]]) # (T, L, C) transform = InterpolateMissing(method="linear") transform(pose_sequence).flatten() array([0., 1., 2.])

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Fills missing landmarks in the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C) containing observed and missing (NaN) landmarks.

required

Returns:

Type Description
ndarray

The pose sequence with missing landmarks replaced by

ndarray

interpolated values, with the same shape as pose_sequence

ndarray

(or unchanged if it is entirely missing).

sign_language_tools.pose.transforms.noise.GaussianNoise

GaussianNoise(scale: float)

Bases: Transform

Adds Gaussian noise to every coordinate of a pose sequence.

Parameters:

Name Type Description Default
scale float

Standard deviation of the Gaussian noise added to each coordinate.

required
Example

import numpy as np np.random.seed(0) from sign_language_tools.pose.transform import GaussianNoise pose_sequence = np.zeros((10, 5, 2)) # (T, L, C) transform = GaussianNoise(scale=0.01) transform(pose_sequence).shape (10, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Adds Gaussian noise to the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The pose sequence with added noise, with the same shape as

ndarray

pose_sequence.

sign_language_tools.pose.transforms.normalize.MinMaxNormalization

MinMaxNormalization()

Bases: Transform

Rescales a pose sequence to the [0, 1] range per coordinate.

For each coordinate channel (e.g. x, y, z), computes the min and max across all frames and landmarks (ignoring NaNs), and rescales the pose sequence so that channel falls within [0, 1]. Channels with zero range are left at zero instead of dividing by zero.

Example

import numpy as np from sign_language_tools.pose.transform import MinMaxNormalization pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = MinMaxNormalization() normalized = transform(pose_sequence) bool(normalized.min() >= 0 and normalized.max() <= 1) True

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Rescales the pose sequence to [0, 1].

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The rescaled pose sequence, with the same shape as

ndarray

pose_sequence.

sign_language_tools.pose.transforms.normalize.Standardization

Standardization()

Bases: Transform

Standardizes a pose sequence to zero mean and unit variance per coordinate.

For each coordinate channel (e.g. x, y, z), computes the mean and standard deviation across all frames and landmarks (ignoring NaNs), and rescales the pose sequence accordingly. Channels with zero variance are left at zero instead of dividing by zero.

Example

import numpy as np from sign_language_tools.pose.transform import Standardization pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Standardization() transform(pose_sequence).shape (10, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Standardizes the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The standardized pose sequence, with the same shape as

ndarray

pose_sequence.

sign_language_tools.pose.transforms.normalize.FixedResolutionNormalization

FixedResolutionNormalization(width: int, height: int)

Bases: Transform

Normalizes a pose sequence expressed in pixel coordinates to [-1, 1].

Divides x and y coordinates by a fixed (width, height) resolution, then rescales the result from [0, 1] to [-1, 1]. Useful when landmarks were extracted from images/videos of a known, fixed size.

Parameters:

Name Type Description Default
width int

Width (in pixels) used to normalize the x-coordinate.

required
height int

Height (in pixels) used to normalize the y-coordinate.

required
Example

import numpy as np from sign_language_tools.pose.transform import FixedResolutionNormalization pose_sequence = np.array([[[0.0, 0.0], [640.0, 480.0]]]) # (T, L, C) transform = FixedResolutionNormalization(width=640, height=480) transform(pose_sequence).tolist() [[[-1.0, -1.0], [1.0, 1.0]]]

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Normalizes the pose sequence to [-1, 1].

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, 2), where T is the number of frames, L the number of landmarks, and the last dimension holds (x, y) pixel coordinates.

required

Returns:

Type Description
ndarray

The normalized pose sequence, with the same shape as

ndarray

pose_sequence.

sign_language_tools.pose.transforms.optical_flow.ToOpticalFlow

ToOpticalFlow(fps: float | None = None)

Bases: Transform

Computes the per-landmark displacement between consecutive frames.

For each frame and each landmark, computes the Euclidean distance between its position in the current and previous frame (the first frame is compared against the origin). This yields a per-landmark optical-flow-like signal that is high when a landmark moves quickly.

Parameters:

Name Type Description Default
fps float | None

If given, multiplies the displacement by this frame rate to convert it from a per-frame value into a velocity (per second).

None
Example

import numpy as np from sign_language_tools.pose.transform import ToOpticalFlow pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = ToOpticalFlow() transform(pose_sequence).shape (10, 5)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Computes the optical flow of the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

Array of shape (T, L) containing, for each frame and

ndarray

landmark, the displacement (or velocity if fps is set) since

ndarray

the previous frame.

sign_language_tools.pose.transforms.padding.Padding

Padding(
    min_length: int,
    location="right",
    mode: str = "constant",
    constant_value: float = 0.0,
    return_mask: bool = False,
)

Bases: Transform

sign_language_tools.pose.transforms.resample.Resample

Resample(new_length: int, method: str = 'linear')

Bases: Transform

Resamples a pose sequence to a fixed number of frames.

Uses interpolation over the frame axis to compute new, evenly spaced frames. See get_landmark_interpolation_function.

Parameters:

Name Type Description Default
new_length int

Number of frames in the resampled pose sequence.

required
method str

Interpolation method used to compute new frames. One of "linear", "nearest", "previous", "next".

'linear'
Example

import numpy as np from sign_language_tools.pose.transform import Resample pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Resample(new_length=20) transform(pose_sequence).shape (20, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Resamples the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The resampled pose sequence, of shape (new_length, L, C).

sign_language_tools.pose.transforms.resample.RandomResample

RandomResample(
    min_length: int, max_length: int, method: str = "linear"
)

Bases: Resample

Resamples a pose sequence to a randomly chosen number of frames.

Each time the transform is called, the target length is drawn uniformly at random between min_length (inclusive) and max_length (exclusive).

Parameters:

Name Type Description Default
min_length int

Lower bound (inclusive) for the random target length.

required
max_length int

Upper bound (exclusive) for the random target length.

required
method str

Interpolation method used to compute new frames. One of "linear", "nearest", "previous", "next".

'linear'
Example

import numpy as np from sign_language_tools.pose.transform import RandomResample pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomResample(min_length=5, max_length=15) new_pose_sequence = transform(pose_sequence) new_pose_sequence.shape[1:] (5, 2) 5 <= new_pose_sequence.shape[0] < 15 True

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Resamples the pose sequence to a randomly chosen length.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The resampled pose sequence, of shape (new_length, L, C),

ndarray

where new_length is randomly drawn between min_length and

ndarray

max_length.

sign_language_tools.pose.transforms.rotate_2d.Rotation2D

Rotation2D(angle: float, center=(0.5, 0.5))

Bases: Transform

sign_language_tools.pose.transforms.rotate_2d.RandomRotation2D

RandomRotation2D(
    angle_range=(-pi / 12, pi / 12), center=(0.5, 0.5)
)

Bases: Rotation2D

sign_language_tools.pose.transforms.rotate_2d.MakeReferenceEdgeHorizontal

MakeReferenceEdgeHorizontal(ref_idx: tuple[int, int])

Bases: Transform

Rotates the pose in each frame so the reference edge is horizontal. The rotation aligns the reference vector with the positive x-axis.

This transform only works with pose sequences of shape (T, L, 2) where T is the number of frames and L the number of landmarks.

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Applies the rotation to a pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

A numpy array of shape (T, L, 2).

required

Returns:

Type Description
ndarray

np.ndarray: The rotated pose sequence of shape (T, L, 2).

sign_language_tools.pose.transforms.scale.Scale

Scale(scaling_factor: float, center=(0.5, 0.5))

Bases: Transform

Scales a pose sequence around a fixed center point.

Multiplies every coordinate by scaling_factor, then re-centers the result so that center stays fixed.

Parameters:

Name Type Description Default
scaling_factor float

Factor applied to every coordinate.

required
center

(x, y) reference point that stays fixed by the scaling.

(0.5, 0.5)
Example

import numpy as np from sign_language_tools.pose.transform import Scale pose_sequence = np.array([[[0.5, 0.5], [1.0, 1.0]]]) # (T, L, C) transform = Scale(scaling_factor=2.0) result = transform(pose_sequence) result[0, 0] # the center point is left unchanged array([0.5, 0.5]) result[0, 1] # points further from the center move further away array([1.5, 1.5])

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Scales the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The scaled pose sequence.

sign_language_tools.pose.transforms.scale.RandomScale

RandomScale(
    min_scale: float, max_scale: float, center=(0.5, 0.5)
)

Bases: Scale

Scales a pose sequence by a randomly chosen factor around a fixed center.

Each time the transform is called, the scaling factor is drawn uniformly at random between min_scale and max_scale.

Parameters:

Name Type Description Default
min_scale float

Lower bound for the random scaling factor.

required
max_scale float

Upper bound for the random scaling factor.

required
center

(x, y) reference point that stays fixed by the scaling.

(0.5, 0.5)
Example

import numpy as np from sign_language_tools.pose.transform import RandomScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomScale(min_scale=0.8, max_scale=1.2) transform(pose_sequence).shape (10, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Scales the pose sequence by a randomly chosen factor.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The scaled pose sequence.

sign_language_tools.pose.transforms.smoothing.SavitzkyGolayFiltering

SavitzkyGolayFiltering(
    window_length: int, polynom_order: int
)

Bases: Transform

Smooth a pose sequence with a Savitzky-Golay filter, applied along the time axis.

If the sequence is shorter than the filter window, it is returned unchanged.

Parameters:

Name Type Description Default
window_length int

Length of the filter window.

required
polynom_order int

Order of the polynomial used to fit each window.

required

sign_language_tools.pose.transforms.split.Split

Split(groups: dict[str, tuple[int, int]])

Bases: Transform

Splits a pose sequence into named groups of landmarks.

Parameters:

Name Type Description Default
groups dict[str, tuple[int, int]]

Mapping from a group name to a (start, end) range of landmark indices (end exclusive) to include in that group.

required
Example

import numpy as np from sign_language_tools.pose.transform import Split pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Split(groups={"left": (0, 2), "right": (2, 5)}) result = transform(pose_sequence) result["left"].shape (10, 2, 2) result["right"].shape (10, 3, 2)

__call__

__call__(pose_sequence: ndarray) -> dict[str, np.ndarray]

Splits the pose sequence into the configured groups.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
dict[str, ndarray]

A dict mapping each group name to its corresponding sub-sequence

dict[str, ndarray]

of landmarks, of shape (T, end - start, C).

sign_language_tools.pose.transforms.temporal_crop.TemporalCrop

TemporalCrop(size: int, location: str = 'start')

Bases: Transform

Crops a pose sequence to a fixed number of frames.

If the pose sequence is already shorter than or equal to size, it is returned unchanged.

Parameters:

Name Type Description Default
size int

Number of frames to keep.

required
location str

Where to crop from: "start", "center", or "end".

'start'
Example

import numpy as np from sign_language_tools.pose.transform import TemporalCrop pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalCrop(size=4, location="center") transform(pose_sequence).shape (4, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Crops the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The cropped pose sequence, of shape (size, L, C) (or

ndarray

unchanged if T <= size).

sign_language_tools.pose.transforms.temporal_crop.TemporalRandomCrop

TemporalRandomCrop(size: int)

Bases: Transform

Crops a pose sequence to a fixed number of frames at a random offset.

If the pose sequence is already shorter than or equal to size, it is returned unchanged. Otherwise, a random start position is chosen so that the whole crop fits within the sequence.

Parameters:

Name Type Description Default
size int

Number of frames to keep.

required
Example

import numpy as np from sign_language_tools.pose.transform import TemporalRandomCrop pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalRandomCrop(size=4) transform(pose_sequence).shape (4, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Crops the pose sequence at a random offset.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The cropped pose sequence, of shape (size, L, C) (or

ndarray

unchanged if T <= size).

sign_language_tools.pose.transforms.temporal_scale.TemporalScale

TemporalScale(scale: float, time_axis=0)

Bases: Resample

Resamples a pose sequence to a length scaled by a fixed factor.

The target length is computed as ceil(T * scale), where T is the size of the pose sequence along time_axis. The actual resampling is delegated to Resample.

Parameters:

Name Type Description Default
scale float

Factor applied to the sequence length. Values above 1.0 stretch the sequence (more frames), values below 1.0 compress it (fewer frames).

required
time_axis

Axis of the pose sequence along which the length is measured and scaled.

0
Example

import numpy as np from sign_language_tools.pose.transform import TemporalScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalScale(scale=1.5) transform(pose_sequence).shape (15, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Resamples the pose sequence to its scaled length.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The resampled pose sequence, of shape (ceil(T * scale), L, C).

sign_language_tools.pose.transforms.temporal_scale.RandomTemporalScale

RandomTemporalScale(
    min_scale: float, max_scale: float, time_axis=0
)

Bases: Resample

Resamples a pose sequence to a length scaled by a randomly chosen factor.

Each time the transform is called, the scaling factor is drawn uniformly at random between min_scale and max_scale, and the target length is computed as ceil(T * scale).

Parameters:

Name Type Description Default
min_scale float

Lower bound for the random scaling factor.

required
max_scale float

Upper bound for the random scaling factor.

required
time_axis

Axis of the pose sequence along which the length is measured and scaled.

0
Example

import numpy as np from sign_language_tools.pose.transform import RandomTemporalScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomTemporalScale(min_scale=0.5, max_scale=1.5) transform(pose_sequence).shape[1:] (5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Resamples the pose sequence to a randomly scaled length.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The resampled pose sequence, of shape (ceil(T * scale), L, C)

ndarray

where scale is randomly drawn between min_scale and

ndarray

max_scale.

sign_language_tools.pose.transforms.to_img.ToRGBImage

ToRGBImage(
    normalize: bool = True, fill_z_with_zero: bool = True
)

Bases: Transform

sign_language_tools.pose.transforms.translation.Translation

Translation(dx: float, dy: float)

Bases: Transform

Translates a pose sequence by a fixed offset.

Adds dx to the x-coordinate and dy to the y-coordinate of every landmark.

Parameters:

Name Type Description Default
dx float

Offset added to the x-coordinate.

required
dy float

Offset added to the y-coordinate.

required
Example

import numpy as np from sign_language_tools.pose.transform import Translation pose_sequence = np.zeros((1, 1, 2)) # (T, L, C) transform = Translation(dx=0.1, dy=-0.2) result = transform(pose_sequence) tuple(result[0, 0]) (0.1, -0.2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Translates the pose sequence.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The translated pose sequence.

sign_language_tools.pose.transforms.translation.RandomTranslation

RandomTranslation(
    dx_range=(-0.2, 0.2), dy_range=(-0.2, 0.2)
)

Bases: Transform

Translates a pose sequence by a randomly chosen offset.

Each time the transform is called, the x and y offsets are drawn uniformly at random from dx_range and dy_range respectively.

Parameters:

Name Type Description Default
dx_range

(min, max) range for the random x offset.

(-0.2, 0.2)
dy_range

(min, max) range for the random y offset.

(-0.2, 0.2)
Example

import numpy as np from sign_language_tools.pose.transform import RandomTranslation pose_sequence = np.zeros((10, 5, 2)) # (T, L, C) transform = RandomTranslation(dx_range=(-0.2, 0.2), dy_range=(-0.2, 0.2)) transform(pose_sequence).shape (10, 5, 2)

__call__

__call__(pose_sequence: ndarray) -> np.ndarray

Translates the pose sequence by a randomly chosen offset.

Parameters:

Name Type Description Default
pose_sequence ndarray

Pose sequence of shape (T, L, C), where T is the number of frames, L the number of landmarks, and C the number of coordinates per landmark.

required

Returns:

Type Description
ndarray

The translated pose sequence.

Visualization

sign_language_tools.pose.visualization.landmarks.plot_landmarks

plot_landmarks(
    landmarks: ndarray,
    connections: Optional[Tuple[Edge, ...]] = None,
    *,
    ax=None,
    vertex_size: float = 0.01,
    vertex_color: str = "lime",
    edge_color: str = "white",
    background_color: str = "black",
    text_color: str = "red",
    aspect_ratio: float = 1,
    show_axis: bool = False,
    show_indices: bool = False,
    x_lim: Optional[Tuple[float, float]] = None,
    y_lim: Optional[Tuple[float, float]] = None,
    refocus: bool = False,
    focus_pad: float = 0.02
)

Plots landmarks on a matplotlib axis.

Parameters:

Name Type Description Default
landmarks ndarray

Landmarks to plot, of shape (L, 2) or (L, 3), where L is the number of landmarks. Each landmark is drawn as a vertex.

required
connections Optional[Tuple[Edge, ...]]

Edges to draw between vertices, as (start, end) index pairs. If None, no edges are drawn.

None
ax

Matplotlib axes to draw on. Defaults to the current axes.

None
vertex_size float

Diameter of each vertex marker.

0.01
vertex_color str

Color of the vertex markers.

'lime'
edge_color str

Color of the edges.

'white'
background_color str

Color of the axes background.

'black'
text_color str

Color of the landmark indices, if shown.

'red'
aspect_ratio float

Aspect ratio (width / height) used to display the landmarks.

1
show_axis bool

Whether to draw the x- and y-axis.

False
show_indices bool

Whether to draw the index of each landmark next to its vertex.

False
x_lim Optional[Tuple[float, float]]

Override for the computed x-axis limits.

None
y_lim Optional[Tuple[float, float]]

Override for the computed y-axis limits.

None
refocus bool

If True, compute x_lim/y_lim to fit tightly around the landmarks instead of using the default [0, 1] unit square.

False
focus_pad float

Padding added around the landmarks when refocus is True. Ignored otherwise.

0.02

Returns:

Type Description

The matplotlib axes the landmarks were drawn on.

Example

import numpy as np from sign_language_tools.pose.visualization import plot_landmarks landmarks = np.random.rand(21, 2) ax = plot_landmarks(landmarks)

sign_language_tools.pose.visualization.landmarks.plot_landmarks_sequence

plot_landmarks_sequence(
    landmarks: dict[Any, ndarray],
    connections: Optional[
        dict[Any, Tuple[Edge, ...]]
    ] = None,
    **kwargs
)

Plots a temporal sequence of landmark groups, one subplot per frame.

Parameters:

Name Type Description Default
landmarks dict[Any, ndarray]

Mapping from a group name (e.g. "left_hand") to its landmark sequence, of shape (T, L, 2) or (T, L, 3). All groups must share the same sequence length T.

required
connections Optional[dict[Any, Tuple[Edge, ...]]]

Mapping from a group name to the edges drawn for that group, as in plot_landmarks. Groups without an entry are drawn without edges.

None
**kwargs

Additional keyword arguments forwarded to plot_landmarks for every vertex group and every frame.

{}

Returns:

Type Description

The matplotlib figure containing one subplot per frame.

Example

import numpy as np from sign_language_tools.pose.visualization import plot_landmarks_sequence landmarks = {"right_hand": np.random.rand(10, 21, 2)} fig = plot_landmarks_sequence(landmarks)