Pose¶
The pose submodule covers the full lifecycle of pose landmarks: extracting them from video with
MediaPipe, describing their skeleton connectivity, transforming/augmenting them, and plotting them.
Across the submodule, a pose sequence is represented as a numpy array of shape (T, L, C), where
T is the number of frames, L the number of landmarks, and C the number of coordinates per
landmark (usually 3: x, y, z).
Extracting landmarks with MediaPipe¶
sign_language_tools.pose.mediapipe.extraction wraps MediaPipe's holistic landmarker to extract
pose, hands and face landmarks from a video file frame by frame.
from sign_language_tools.pose.mediapipe.extraction import (
load_holistic_landmarker,
extract_poses_from_video_file,
)
landmarker = load_holistic_landmarker("holistic_landmarker.task")
poses = extract_poses_from_video_file("video.mp4", landmarker, show_progress=True)
poses["pose"].shape # (T, 33, 3)
poses["left_hand"].shape # (T, 21, 3)
poses["right_hand"].shape # (T, 21, 3)
poses["face"].shape # (T, 478, 3)

Frames where a body part wasn't detected are filled with NaN; see
InterpolateMissing to fill
them in.
Skeleton connectivity¶
sign_language_tools.pose.mediapipe.edges and sign_language_tools.pose.mediapipe.vertices provide
the connectivity (edges) and landmark groupings (vertices) for the MediaPipe skeletons, and
sign_language_tools.pose.openpose.edges provides the equivalent for OpenPose. These are typically
used to draw skeletons (see Visualization and the Video Player) or to
build graph structures over the landmarks.
from sign_language_tools.pose.mediapipe.edges import POSE_EDGES, HAND_EDGES, FACE_EDGES
from sign_language_tools.pose.mediapipe.vertices import LIPS_VERTICES
Transforms¶
sign_language_tools.pose.transforms provides data-augmentation and preprocessing transforms for
pose sequences, compatible with PyTorch/TensorFlow data pipelines. They can be composed with
Compose from the common
submodule.
import numpy as np
from sign_language_tools.pose.transforms import (
Concatenate, TemporalRandomCrop, HorizontalFlip, Split, InterpolateMissing,
)
landmarks = {
"pose": np.load("examples/ressources/pose.npy"),
"left_hand": np.load("examples/ressources/left_hand.npy"),
"right_hand": np.load("examples/ressources/right_hand.npy"),
}
pipeline = [
InterpolateMissing(),
Concatenate(["pose", "right_hand", "left_hand"]),
TemporalRandomCrop(size=60),
HorizontalFlip(),
Split({"pose": 33, "right_hand": 21, "left_hand": 21}),
]
x = landmarks
for transform in pipeline:
x = transform(x)
Transforms operating on a single array (e.g. TemporalRandomCrop, HorizontalFlip,
InterpolateMissing) take and return a (T, L, C) array. Concatenate and Split bridge between
a dict[str, np.ndarray] of landmark groups (e.g. {"pose": ..., "left_hand": ...}) and a single
concatenated (T, L, C) array, so that group-agnostic transforms can be applied uniformly.
Visualization¶
sign_language_tools.pose.visualization provides matplotlib plotting for landmarks, one frame or a
whole sequence at a time. For interactive playback synced to a video, see the Video Player
instead.
from sign_language_tools.pose.visualization import plot_landmarks, plot_landmarks_sequence
from sign_language_tools.pose.mediapipe.edges import POSE_EDGES, HAND_EDGES
# A single frame
plot_landmarks(landmarks["pose"][0], connections=POSE_EDGES)
# A short sequence, one subplot per frame
fig = plot_landmarks_sequence(
{"pose": landmarks["pose"][:5], "right_hand": landmarks["right_hand"][:5]},
{"pose": POSE_EDGES, "right_hand": HAND_EDGES},
)
Reference¶
Extraction¶
sign_language_tools.pose.mediapipe.extraction.load_holistic_landmarker ¶
load_holistic_landmarker(
model_path: str,
use_gpu: bool = False,
options: HolisticLandmarkerOptions | None = None,
)
Load a MediaPipe holistic landmarker for video inference.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
model_path
|
str
|
Path to the holistic landmarker |
required |
use_gpu
|
bool
|
Whether to run inference on GPU instead of CPU. Defaults to |
False
|
options
|
HolisticLandmarkerOptions | None
|
Custom landmarker options. If |
None
|
Returns:
| Name | Type | Description |
|---|---|---|
HolisticLandmarker |
The initialized holistic landmarker, ready to process video frames. |
sign_language_tools.pose.mediapipe.extraction.extract_poses_from_video_file ¶
extract_poses_from_video_file(
video_path: str,
holistic_landmarker: HolisticLandmarker,
show_progress=False,
) -> dict[str, np.ndarray]
Extract holistic pose landmarks from every frame of a video file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
video_path
|
str
|
Path to the video file to process. |
required |
holistic_landmarker
|
HolisticLandmarker
|
Landmarker used to run detection on each
frame, e.g. as returned by |
required |
show_progress
|
bool
|
Whether to display a progress bar while iterating over the
video frames. Defaults to |
False
|
Returns:
| Type | Description |
|---|---|
dict[str, ndarray]
|
dict[str, np.ndarray]: Mapping from landmark group ( |
Edges and vertices¶
sign_language_tools.pose.mediapipe.edges ¶
Edge (connectivity) definitions for MediaPipe hand, pose and face landmarks.
Each constant is a tuple of (start_index, end_index) pairs, where the indices
refer to positions in the corresponding landmark array (hand: (21, C), pose:
(33, C), face: (478, C)). These are typically used to draw skeletons or
build adjacency structures (e.g. graph-based models) over the landmarks.
"Upper"/"lower" pose edges split the body pose skeleton into the upper body (arms, shoulders, face landmarks 0-10) and the lower body (hips and legs).
sign_language_tools.pose.mediapipe.vertices ¶
Vertex groups for the MediaPipe face mesh, derived from its edge topology.
Each constant below is a tuple of landmark indices into the (478, C) array of
face landmarks produced by MediaPipe (see results.face_landmarks in
extract_poses_from_video_file).
The vertex tuples are computed from the corresponding FACEMESH_* edge
definitions in sign_language_tools.pose.mediapipe.facemesh rather than
listed by hand, so they always stay in sync with the edges: any vertex that
appears in an edge is included, and duplicates are removed.
"Left"/"right" follow MediaPipe's convention, i.e. the subject's own left and right, not the viewer's.
sign_language_tools.pose.openpose.edges ¶
Transforms¶
sign_language_tools.pose.transforms.center.CenterOnLandmarks ¶
CenterOnLandmarks(landmark_idx: int | tuple[int, ...])
Bases: Transform
Centers a pose sequence on one or more reference landmarks.
For every frame, the mean position of the given reference landmark(s) is subtracted from all landmarks, so that the reference point becomes the new origin.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
landmark_idx
|
int | tuple[int, ...]
|
Index, or tuple of indices, of the landmark(s) used as the reference point. If several indices are given, their mean position is used as the origin. |
required |
Example
import numpy as np from sign_language_tools.pose.transforms import CenterOnLandmarks pose_sequence = np.random.rand(10, 33, 3) # (T, L, C) transform = CenterOnLandmarks(landmark_idx=0) centered = transform(pose_sequence) centered.shape (10, 33, 3)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Centers the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The centered pose sequence, with the same shape as |
sign_language_tools.pose.transforms.clip.Clip ¶
Clip(min_value: float = 0.0, max_value: float = 1.0)
Bases: Transform
Clips the coordinates of a pose sequence to a fixed value range.
Any value below min_value is set to min_value, and any value above
max_value is set to max_value.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_value
|
float
|
Lower bound of the clipping range. |
0.0
|
max_value
|
float
|
Upper bound of the clipping range. |
1.0
|
Example
import numpy as np from sign_language_tools.pose.transform import Clip pose_sequence = np.array([[[-0.2, 0.5, 1.3]]]) # (T, L, C) transform = Clip(min_value=0.0, max_value=1.0) transform(pose_sequence) array([[[0. , 0.5, 1. ]]])
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Clips the pose sequence coordinates.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The clipped pose sequence, with the same shape as |
sign_language_tools.pose.transforms.concatenate.Concatenate ¶
Concatenate(
landmark_sets: Sequence[str] = None,
landmark_axis: int = 1,
)
Bases: Transform
Concatenates multiple pose sequences along the landmark axis.
Accepts either a dict mapping body part names to pose sequences
(e.g. {"upper_body": ..., "left_hand": ..., "right_hand": ...}) or a
list of pose sequences. All pose sequences must have compatible shapes
along every axis except the concatenation axis.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
landmark_sets
|
Sequence[str]
|
Ordered list of keys to select and order entries from a dict input. Ignored when the input is a list. If None, keys are sorted alphabetically. |
None
|
landmark_axis
|
int
|
Axis along which to concatenate. Defaults to |
1
|
Example
import numpy as np from sign_language_tools.pose.transform import Concatenate upper_body = np.random.rand(10, 8, 2) # (T, L, C) left_hand = np.random.rand(10, 21, 2) # (T, L, C) transform = Concatenate(landmark_sets=["upper_body", "left_hand"]) pose_sequence = transform({"upper_body": upper_body, "left_hand": left_hand}) pose_sequence.shape (10, 29, 2)
__call__ ¶
__call__(
pose_sequences: Union[
dict[str, ndarray], list[ndarray]
],
) -> np.ndarray
Concatenates the pose sequences.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequences
|
Union[dict[str, ndarray], list[ndarray]]
|
Either a dict mapping body part names to pose
sequences, or a list of pose sequences, each of shape
|
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The concatenated pose sequence, of shape |
sign_language_tools.pose.transforms.drop_coordinates.DropCoordinates ¶
DropCoordinates(coord: int | str)
Bases: Transform
Drops one of the x, y, z coordinates from a pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
coord
|
int | str
|
Coordinate to drop, either as an index ( |
required |
Example
import numpy as np from sign_language_tools.pose.transforms import DropCoordinates pose_sequence = np.random.rand(10, 33, 3) # (T, L, C) transform = DropCoordinates(coord="z") transform(pose_sequence).shape (10, 33, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Drops the configured coordinate from the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence without the dropped coordinate, of shape |
ndarray
|
|
sign_language_tools.pose.transforms.drop_frames.DropRandomFrames ¶
DropRandomFrames(
max_n: Optional[int] = None,
max_p: Optional[float] = None,
fill_value: float = 0.0,
)
Bases: Transform
sign_language_tools.pose.transforms.edge_normalize.NormalizeByReferenceEdge ¶
NormalizeByReferenceEdge(
ref_edge: tuple[int, int],
per_frame: bool = False,
eps: float = 1e-08,
)
Bases: Transform
Scale a pose sequence using a reference edge as the scale unit.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ref_edge
|
tuple[int, int]
|
Indices |
required |
per_frame
|
bool
|
If |
False
|
eps
|
float
|
Small value added to the edge length to avoid division by zero. |
1e-08
|
sign_language_tools.pose.transforms.filter.FilterEmpty ¶
FilterEmpty()
Bases: Transform
Removes frames where every landmark is missing.
A frame is considered empty when all of its landmark coordinates are NaN (e.g. a frame where pose estimation failed on every landmark), and such frames are dropped from the sequence.
Example
import numpy as np from sign_language_tools.pose.transform import FilterEmpty pose_sequence = np.random.rand(3, 5, 2) # (T, L, C) pose_sequence[1] = np.nan # frame 1 is entirely missing transform = FilterEmpty() transform(pose_sequence).shape (2, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Removes empty frames from the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence without its empty frames, of shape |
ndarray
|
|
sign_language_tools.pose.transforms.filter.FilterLandmarks ¶
FilterLandmarks(filtered_indices: set[int] | list[int])
Bases: Transform
Removes a fixed set of landmarks from a pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
filtered_indices
|
set[int] | list[int]
|
Indices of the landmarks to remove. |
required |
Example
import numpy as np from sign_language_tools.pose.transform import FilterLandmarks pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = FilterLandmarks(filtered_indices=[0, 2]) transform(pose_sequence).shape (10, 3, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Removes the configured landmarks from the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence without the filtered landmarks, of shape |
ndarray
|
|
sign_language_tools.pose.transforms.flatten.Flatten ¶
Flatten(mode: str = 'landmarks')
Bases: Transform
Flattens the last two dimensions of a pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
mode
|
str
|
Either |
'landmarks'
|
Example
import numpy as np from sign_language_tools.pose.transform import Flatten pose_sequence = np.random.rand(10, 5, 3) # (T, L, C) Flatten(mode="landmarks")(pose_sequence).shape (10, 15) Flatten(mode="features")(pose_sequence).shape (5, 30)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Flattens the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The flattened pose sequence, of shape |
ndarray
|
|
sign_language_tools.pose.transforms.flatten.Unflatten ¶
Unflatten(
n_landmarks: int,
n_channels: int,
mode: str = "landmarks",
)
Bases: Transform
Reshapes a flattened pose sequence back to (T, L, C).
Inverts a Flatten operation performed with the same mode,
n_landmarks, and n_channels.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
n_landmarks
|
int
|
Number of landmarks |
required |
n_channels
|
int
|
Number of coordinates per landmark |
required |
mode
|
str
|
Must match the mode used by |
'landmarks'
|
Example
import numpy as np from sign_language_tools.pose.transform import Flatten, Unflatten pose_sequence = np.random.rand(10, 5, 3) # (T, L, C) flat = Flatten(mode="landmarks")(pose_sequence) Unflatten(n_landmarks=5, n_channels=3)(flat).shape (10, 5, 3)
__call__ ¶
__call__(flat_pose_sequence: ndarray) -> np.ndarray
Unflattens the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
flat_pose_sequence
|
ndarray
|
Flattened pose sequence, of shape
|
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence reshaped back to |
sign_language_tools.pose.transforms.flip.HorizontalFlip ¶
HorizontalFlip(x_origin: float = 0.5)
Bases: Transform
sign_language_tools.pose.transforms.interpolate.InterpolateMissing ¶
InterpolateMissing(method: str = 'linear')
Bases: Transform
Fills missing landmarks (NaN coordinates) in a pose sequence by interpolation.
Builds an interpolation function over the frame axis (see
get_landmark_interpolation_function) to fill in NaN landmarks from
observed ones, then fills any values still missing after that (e.g.
leading/trailing frames) using nearest-neighbor extrapolation. If the
whole pose sequence is NaN, it is returned unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
method
|
str
|
Interpolation method used for observed landmarks. One of
|
'linear'
|
Example
import numpy as np from sign_language_tools.pose.transform import InterpolateMissing pose_sequence = np.array([[[0.0]], [[np.nan]], [[2.0]]]) # (T, L, C) transform = InterpolateMissing(method="linear") transform(pose_sequence).flatten() array([0., 1., 2.])
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Fills missing landmarks in the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence with missing landmarks replaced by |
ndarray
|
interpolated values, with the same shape as |
ndarray
|
(or unchanged if it is entirely missing). |
sign_language_tools.pose.transforms.noise.GaussianNoise ¶
GaussianNoise(scale: float)
Bases: Transform
Adds Gaussian noise to every coordinate of a pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scale
|
float
|
Standard deviation of the Gaussian noise added to each coordinate. |
required |
Example
import numpy as np np.random.seed(0) from sign_language_tools.pose.transform import GaussianNoise pose_sequence = np.zeros((10, 5, 2)) # (T, L, C) transform = GaussianNoise(scale=0.01) transform(pose_sequence).shape (10, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Adds Gaussian noise to the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The pose sequence with added noise, with the same shape as |
ndarray
|
|
sign_language_tools.pose.transforms.normalize.MinMaxNormalization ¶
MinMaxNormalization()
Bases: Transform
Rescales a pose sequence to the [0, 1] range per coordinate.
For each coordinate channel (e.g. x, y, z), computes the min and max
across all frames and landmarks (ignoring NaNs), and rescales the pose
sequence so that channel falls within [0, 1]. Channels with zero
range are left at zero instead of dividing by zero.
Example
import numpy as np from sign_language_tools.pose.transform import MinMaxNormalization pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = MinMaxNormalization() normalized = transform(pose_sequence) bool(normalized.min() >= 0 and normalized.max() <= 1) True
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Rescales the pose sequence to [0, 1].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The rescaled pose sequence, with the same shape as |
ndarray
|
|
sign_language_tools.pose.transforms.normalize.Standardization ¶
Standardization()
Bases: Transform
Standardizes a pose sequence to zero mean and unit variance per coordinate.
For each coordinate channel (e.g. x, y, z), computes the mean and standard deviation across all frames and landmarks (ignoring NaNs), and rescales the pose sequence accordingly. Channels with zero variance are left at zero instead of dividing by zero.
Example
import numpy as np from sign_language_tools.pose.transform import Standardization pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Standardization() transform(pose_sequence).shape (10, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Standardizes the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The standardized pose sequence, with the same shape as |
ndarray
|
|
sign_language_tools.pose.transforms.normalize.FixedResolutionNormalization ¶
FixedResolutionNormalization(width: int, height: int)
Bases: Transform
Normalizes a pose sequence expressed in pixel coordinates to [-1, 1].
Divides x and y coordinates by a fixed (width, height) resolution,
then rescales the result from [0, 1] to [-1, 1]. Useful when
landmarks were extracted from images/videos of a known, fixed size.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
width
|
int
|
Width (in pixels) used to normalize the x-coordinate. |
required |
height
|
int
|
Height (in pixels) used to normalize the y-coordinate. |
required |
Example
import numpy as np from sign_language_tools.pose.transform import FixedResolutionNormalization pose_sequence = np.array([[[0.0, 0.0], [640.0, 480.0]]]) # (T, L, C) transform = FixedResolutionNormalization(width=640, height=480) transform(pose_sequence).tolist() [[[-1.0, -1.0], [1.0, 1.0]]]
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Normalizes the pose sequence to [-1, 1].
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The normalized pose sequence, with the same shape as |
ndarray
|
|
sign_language_tools.pose.transforms.optical_flow.ToOpticalFlow ¶
ToOpticalFlow(fps: float | None = None)
Bases: Transform
Computes the per-landmark displacement between consecutive frames.
For each frame and each landmark, computes the Euclidean distance between its position in the current and previous frame (the first frame is compared against the origin). This yields a per-landmark optical-flow-like signal that is high when a landmark moves quickly.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
fps
|
float | None
|
If given, multiplies the displacement by this frame rate to convert it from a per-frame value into a velocity (per second). |
None
|
Example
import numpy as np from sign_language_tools.pose.transform import ToOpticalFlow pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = ToOpticalFlow() transform(pose_sequence).shape (10, 5)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Computes the optical flow of the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
Array of shape |
ndarray
|
landmark, the displacement (or velocity if |
ndarray
|
the previous frame. |
sign_language_tools.pose.transforms.padding.Padding ¶
Padding(
min_length: int,
location="right",
mode: str = "constant",
constant_value: float = 0.0,
return_mask: bool = False,
)
Bases: Transform
sign_language_tools.pose.transforms.resample.Resample ¶
Resample(new_length: int, method: str = 'linear')
Bases: Transform
Resamples a pose sequence to a fixed number of frames.
Uses interpolation over the frame axis to compute new, evenly spaced
frames. See get_landmark_interpolation_function.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
new_length
|
int
|
Number of frames in the resampled pose sequence. |
required |
method
|
str
|
Interpolation method used to compute new frames. One of
|
'linear'
|
Example
import numpy as np from sign_language_tools.pose.transform import Resample pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Resample(new_length=20) transform(pose_sequence).shape (20, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Resamples the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The resampled pose sequence, of shape |
sign_language_tools.pose.transforms.resample.RandomResample ¶
RandomResample(
min_length: int, max_length: int, method: str = "linear"
)
Bases: Resample
Resamples a pose sequence to a randomly chosen number of frames.
Each time the transform is called, the target length is drawn
uniformly at random between min_length (inclusive) and max_length
(exclusive).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_length
|
int
|
Lower bound (inclusive) for the random target length. |
required |
max_length
|
int
|
Upper bound (exclusive) for the random target length. |
required |
method
|
str
|
Interpolation method used to compute new frames. One of
|
'linear'
|
Example
import numpy as np from sign_language_tools.pose.transform import RandomResample pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomResample(min_length=5, max_length=15) new_pose_sequence = transform(pose_sequence) new_pose_sequence.shape[1:] (5, 2) 5 <= new_pose_sequence.shape[0] < 15 True
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Resamples the pose sequence to a randomly chosen length.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The resampled pose sequence, of shape |
ndarray
|
where |
ndarray
|
|
sign_language_tools.pose.transforms.rotate_2d.Rotation2D ¶
Rotation2D(angle: float, center=(0.5, 0.5))
Bases: Transform
sign_language_tools.pose.transforms.rotate_2d.RandomRotation2D ¶
RandomRotation2D(
angle_range=(-pi / 12, pi / 12), center=(0.5, 0.5)
)
Bases: Rotation2D
sign_language_tools.pose.transforms.rotate_2d.MakeReferenceEdgeHorizontal ¶
MakeReferenceEdgeHorizontal(ref_idx: tuple[int, int])
Bases: Transform
Rotates the pose in each frame so the reference edge is horizontal. The rotation aligns the reference vector with the positive x-axis.
This transform only works with pose sequences of shape (T, L, 2) where T is the number of frames and L the number of landmarks.
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Applies the rotation to a pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
A numpy array of shape (T, L, 2). |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
np.ndarray: The rotated pose sequence of shape (T, L, 2). |
sign_language_tools.pose.transforms.scale.Scale ¶
Scale(scaling_factor: float, center=(0.5, 0.5))
Bases: Transform
Scales a pose sequence around a fixed center point.
Multiplies every coordinate by scaling_factor, then re-centers the
result so that center stays fixed.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scaling_factor
|
float
|
Factor applied to every coordinate. |
required |
center
|
|
(0.5, 0.5)
|
Example
import numpy as np from sign_language_tools.pose.transform import Scale pose_sequence = np.array([[[0.5, 0.5], [1.0, 1.0]]]) # (T, L, C) transform = Scale(scaling_factor=2.0) result = transform(pose_sequence) result[0, 0] # the center point is left unchanged array([0.5, 0.5]) result[0, 1] # points further from the center move further away array([1.5, 1.5])
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Scales the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The scaled pose sequence. |
sign_language_tools.pose.transforms.scale.RandomScale ¶
RandomScale(
min_scale: float, max_scale: float, center=(0.5, 0.5)
)
Bases: Scale
Scales a pose sequence by a randomly chosen factor around a fixed center.
Each time the transform is called, the scaling factor is drawn
uniformly at random between min_scale and max_scale.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_scale
|
float
|
Lower bound for the random scaling factor. |
required |
max_scale
|
float
|
Upper bound for the random scaling factor. |
required |
center
|
|
(0.5, 0.5)
|
Example
import numpy as np from sign_language_tools.pose.transform import RandomScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomScale(min_scale=0.8, max_scale=1.2) transform(pose_sequence).shape (10, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Scales the pose sequence by a randomly chosen factor.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The scaled pose sequence. |
sign_language_tools.pose.transforms.smoothing.SavitzkyGolayFiltering ¶
SavitzkyGolayFiltering(
window_length: int, polynom_order: int
)
Bases: Transform
Smooth a pose sequence with a Savitzky-Golay filter, applied along the time axis.
If the sequence is shorter than the filter window, it is returned unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
window_length
|
int
|
Length of the filter window. |
required |
polynom_order
|
int
|
Order of the polynomial used to fit each window. |
required |
sign_language_tools.pose.transforms.split.Split ¶
Split(groups: dict[str, tuple[int, int]])
Bases: Transform
Splits a pose sequence into named groups of landmarks.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
groups
|
dict[str, tuple[int, int]]
|
Mapping from a group name to a |
required |
Example
import numpy as np from sign_language_tools.pose.transform import Split pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = Split(groups={"left": (0, 2), "right": (2, 5)}) result = transform(pose_sequence) result["left"].shape (10, 2, 2) result["right"].shape (10, 3, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> dict[str, np.ndarray]
Splits the pose sequence into the configured groups.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
dict[str, ndarray]
|
A dict mapping each group name to its corresponding sub-sequence |
dict[str, ndarray]
|
of landmarks, of shape |
sign_language_tools.pose.transforms.temporal_crop.TemporalCrop ¶
TemporalCrop(size: int, location: str = 'start')
Bases: Transform
Crops a pose sequence to a fixed number of frames.
If the pose sequence is already shorter than or equal to size, it is
returned unchanged.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
size
|
int
|
Number of frames to keep. |
required |
location
|
str
|
Where to crop from: |
'start'
|
Example
import numpy as np from sign_language_tools.pose.transform import TemporalCrop pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalCrop(size=4, location="center") transform(pose_sequence).shape (4, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Crops the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The cropped pose sequence, of shape |
ndarray
|
unchanged if |
sign_language_tools.pose.transforms.temporal_crop.TemporalRandomCrop ¶
TemporalRandomCrop(size: int)
Bases: Transform
Crops a pose sequence to a fixed number of frames at a random offset.
If the pose sequence is already shorter than or equal to size, it is
returned unchanged. Otherwise, a random start position is chosen so
that the whole crop fits within the sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
size
|
int
|
Number of frames to keep. |
required |
Example
import numpy as np from sign_language_tools.pose.transform import TemporalRandomCrop pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalRandomCrop(size=4) transform(pose_sequence).shape (4, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Crops the pose sequence at a random offset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The cropped pose sequence, of shape |
ndarray
|
unchanged if |
sign_language_tools.pose.transforms.temporal_scale.TemporalScale ¶
TemporalScale(scale: float, time_axis=0)
Bases: Resample
Resamples a pose sequence to a length scaled by a fixed factor.
The target length is computed as ceil(T * scale), where T is the
size of the pose sequence along time_axis. The actual resampling is
delegated to Resample.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
scale
|
float
|
Factor applied to the sequence length. Values above |
required |
time_axis
|
Axis of the pose sequence along which the length is measured and scaled. |
0
|
Example
import numpy as np from sign_language_tools.pose.transform import TemporalScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = TemporalScale(scale=1.5) transform(pose_sequence).shape (15, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Resamples the pose sequence to its scaled length.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The resampled pose sequence, of shape |
sign_language_tools.pose.transforms.temporal_scale.RandomTemporalScale ¶
RandomTemporalScale(
min_scale: float, max_scale: float, time_axis=0
)
Bases: Resample
Resamples a pose sequence to a length scaled by a randomly chosen factor.
Each time the transform is called, the scaling factor is drawn
uniformly at random between min_scale and max_scale, and the target
length is computed as ceil(T * scale).
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
min_scale
|
float
|
Lower bound for the random scaling factor. |
required |
max_scale
|
float
|
Upper bound for the random scaling factor. |
required |
time_axis
|
Axis of the pose sequence along which the length is measured and scaled. |
0
|
Example
import numpy as np from sign_language_tools.pose.transform import RandomTemporalScale pose_sequence = np.random.rand(10, 5, 2) # (T, L, C) transform = RandomTemporalScale(min_scale=0.5, max_scale=1.5) transform(pose_sequence).shape[1:] (5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Resamples the pose sequence to a randomly scaled length.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The resampled pose sequence, of shape |
ndarray
|
where |
ndarray
|
|
sign_language_tools.pose.transforms.to_img.ToRGBImage ¶
ToRGBImage(
normalize: bool = True, fill_z_with_zero: bool = True
)
Bases: Transform
sign_language_tools.pose.transforms.translation.Translation ¶
Translation(dx: float, dy: float)
Bases: Transform
Translates a pose sequence by a fixed offset.
Adds dx to the x-coordinate and dy to the y-coordinate of every
landmark.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dx
|
float
|
Offset added to the x-coordinate. |
required |
dy
|
float
|
Offset added to the y-coordinate. |
required |
Example
import numpy as np from sign_language_tools.pose.transform import Translation pose_sequence = np.zeros((1, 1, 2)) # (T, L, C) transform = Translation(dx=0.1, dy=-0.2) result = transform(pose_sequence) tuple(result[0, 0]) (0.1, -0.2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Translates the pose sequence.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The translated pose sequence. |
sign_language_tools.pose.transforms.translation.RandomTranslation ¶
RandomTranslation(
dx_range=(-0.2, 0.2), dy_range=(-0.2, 0.2)
)
Bases: Transform
Translates a pose sequence by a randomly chosen offset.
Each time the transform is called, the x and y offsets are drawn
uniformly at random from dx_range and dy_range respectively.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
dx_range
|
|
(-0.2, 0.2)
|
|
dy_range
|
|
(-0.2, 0.2)
|
Example
import numpy as np from sign_language_tools.pose.transform import RandomTranslation pose_sequence = np.zeros((10, 5, 2)) # (T, L, C) transform = RandomTranslation(dx_range=(-0.2, 0.2), dy_range=(-0.2, 0.2)) transform(pose_sequence).shape (10, 5, 2)
__call__ ¶
__call__(pose_sequence: ndarray) -> np.ndarray
Translates the pose sequence by a randomly chosen offset.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
pose_sequence
|
ndarray
|
Pose sequence of shape |
required |
Returns:
| Type | Description |
|---|---|
ndarray
|
The translated pose sequence. |
Visualization¶
sign_language_tools.pose.visualization.landmarks.plot_landmarks ¶
plot_landmarks(
landmarks: ndarray,
connections: Optional[Tuple[Edge, ...]] = None,
*,
ax=None,
vertex_size: float = 0.01,
vertex_color: str = "lime",
edge_color: str = "white",
background_color: str = "black",
text_color: str = "red",
aspect_ratio: float = 1,
show_axis: bool = False,
show_indices: bool = False,
x_lim: Optional[Tuple[float, float]] = None,
y_lim: Optional[Tuple[float, float]] = None,
refocus: bool = False,
focus_pad: float = 0.02
)
Plots landmarks on a matplotlib axis.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
landmarks
|
ndarray
|
Landmarks to plot, of shape |
required |
connections
|
Optional[Tuple[Edge, ...]]
|
Edges to draw between vertices, as |
None
|
ax
|
Matplotlib axes to draw on. Defaults to the current axes. |
None
|
|
vertex_size
|
float
|
Diameter of each vertex marker. |
0.01
|
vertex_color
|
str
|
Color of the vertex markers. |
'lime'
|
edge_color
|
str
|
Color of the edges. |
'white'
|
background_color
|
str
|
Color of the axes background. |
'black'
|
text_color
|
str
|
Color of the landmark indices, if shown. |
'red'
|
aspect_ratio
|
float
|
Aspect ratio (width / height) used to display the landmarks. |
1
|
show_axis
|
bool
|
Whether to draw the x- and y-axis. |
False
|
show_indices
|
bool
|
Whether to draw the index of each landmark next to its vertex. |
False
|
x_lim
|
Optional[Tuple[float, float]]
|
Override for the computed x-axis limits. |
None
|
y_lim
|
Optional[Tuple[float, float]]
|
Override for the computed y-axis limits. |
None
|
refocus
|
bool
|
If |
False
|
focus_pad
|
float
|
Padding added around the landmarks when |
0.02
|
Returns:
| Type | Description |
|---|---|
|
The matplotlib axes the landmarks were drawn on. |
Example
import numpy as np from sign_language_tools.pose.visualization import plot_landmarks landmarks = np.random.rand(21, 2) ax = plot_landmarks(landmarks)
sign_language_tools.pose.visualization.landmarks.plot_landmarks_sequence ¶
plot_landmarks_sequence(
landmarks: dict[Any, ndarray],
connections: Optional[
dict[Any, Tuple[Edge, ...]]
] = None,
**kwargs
)
Plots a temporal sequence of landmark groups, one subplot per frame.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
landmarks
|
dict[Any, ndarray]
|
Mapping from a group name (e.g. |
required |
connections
|
Optional[dict[Any, Tuple[Edge, ...]]]
|
Mapping from a group name to the edges drawn for
that group, as in |
None
|
**kwargs
|
Additional keyword arguments forwarded to
|
{}
|
Returns:
| Type | Description |
|---|---|
|
The matplotlib figure containing one subplot per frame. |
Example
import numpy as np from sign_language_tools.pose.visualization import plot_landmarks_sequence landmarks = {"right_hand": np.random.rand(10, 21, 2)} fig = plot_landmarks_sequence(landmarks)