Annotations¶

The annotations submodule manipulates segment-based annotations: gloss/lemma boundaries, sign activity, or any other labeling of a video's timeline.

A set of segments is represented as a numpy array of shape (M, 2) ([start, end]) or (M, 3) ([start, end, label]), with an inclusive end: [start, end] covers frames start, ..., end. This is the same convention used by VideoPlayer.attach_segments.

Transforms¶

sign_language_tools.annotations.transforms converts and manipulates segments: merging adjacent or overlapping spans, rendering them as a dense per-frame label vector (and back), scaling/moving them for data augmentation, or building BIO tags for sequence labeling.

import numpy as np
from sign_language_tools.annotations.transforms import (
    MergeSegments, SegmentsToFrameLabels, FrameLabelsToSegments,
)

segments = np.array([
    [2, 5, 1],
    [6, 12, 2],
    [14, 17, 0],
    [18, 25, 1],
])  # (M, 3): [start, end, label]

segments = MergeSegments()(segments)

frame_labels = SegmentsToFrameLabels()(segments)   # (T,) dense label vector
segments_back = FrameLabelsToSegments()(frame_labels)

Utilities¶

sign_language_tools.annotations.utils provides temporal IoU (intersection-over-union) and non-maximum suppression, useful to evaluate or post-process segment predictions.

from sign_language_tools.annotations.utils.iou import pairwise_temporal_intersection_over_union
from sign_language_tools.annotations.utils.nms import non_maximum_suppression

tiou_map = pairwise_temporal_intersection_over_union(target_segments, predicted_segments)
kept = non_maximum_suppression(predicted_segments, iou_threshold=0.5)

Visualization¶

sign_language_tools.annotations.visualization plots segments as colored spans on a matplotlib timeline.

import numpy as np
from sign_language_tools.annotations.visualization import plot_segments_on_timeline

segments = np.array([[0, 10], [15, 25]])
plot_segments_on_timeline(segments, labels=["bonjour", "merci"])

Reference¶

Transforms¶

sign_language_tools.annotations.transforms.scale.ScaleSegments ¶

ScaleSegments(
    factor: float = 0.8,
    location: Literal["center", "start", "end"] = "center",
    min_length: float = 1.0,
    max_length: float = 1000.0,
)

Bases: Transform

Scales the length of segments by a fixed factor.

Each segment's length is multiplied by factor and clipped to [min_length, max_length]. The resized segment is then repositioned according to location, so that either its center, its start, or its end stays fixed.

Parameters:

Name	Type	Description	Default
`factor`	`float`	Multiplicative factor applied to each segment's length. Values below 1.0 shrink segments, values above 1.0 grow them.	`0.8`
`location`	`Literal['center', 'start', 'end']`	Which point of the segment stays fixed while resizing: `"center"` keeps the midpoint fixed, `"start"` keeps the start fixed, and `"end"` keeps the end fixed.	`'center'`
`min_length`	`float`	Lower bound applied to the new segment length after scaling.	`1.0`
`max_length`	`float`	Upper bound applied to the new segment length after scaling.	`1000.0`

Note

If segments has an integer dtype, the repositioned start/end values are truncated (not rounded) to that dtype. Pass a float array if fractional precision matters.

Example

import numpy as np from sign_language_tools.annotations.transforms import ScaleSegments segments = np.array([[10., 20.], [30., 60.]]) # (M, 2) transform = ScaleSegments(factor=0.5, location="center") transform(segments) array([[12.5, 17.5], [37.5, 52.5]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Scales the segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` containing the start and end of `M` segments.	required

Returns:

Type	Description
`ndarray`	The scaled segments, with the same shape as `segments`.

sign_language_tools.annotations.transforms.scale.RandomRelativeScaleSegments ¶

RandomRelativeScaleSegments(
    scale_std: float = 0.5,
    location: Literal["center", "start", "end"] = "center",
    min_length: float = 1.0,
    max_length: float = 1000.0,
)

Bases: ScaleSegments

Scales the length of segments by a random factor drawn per segment.

Each segment's length is multiplied by an independent factor sampled as 1 + scale_std * N(0, 1), then clipped to [min_length, max_length] and repositioned according to location, exactly like ScaleSegments.

Parameters:

Name	Type	Description	Default
`scale_std`	`float`	Standard deviation of the random scaling factor around 1.0.	`0.5`
`location`	`Literal['center', 'start', 'end']`	Which point of the segment stays fixed while resizing: `"center"` keeps the midpoint fixed, `"start"` keeps the start fixed, and `"end"` keeps the end fixed.	`'center'`
`min_length`	`float`	Lower bound applied to the new segment length after scaling.	`1.0`
`max_length`	`float`	Upper bound applied to the new segment length after scaling.	`1000.0`

Example

import numpy as np from sign_language_tools.annotations.transforms import RandomRelativeScaleSegments segments = np.array([[10., 20.], [30., 60.]]) # (M, 2) transform = RandomRelativeScaleSegments(scale_std=0.3) scaled = transform(segments) scaled.shape (2, 2)

sign_language_tools.annotations.transforms.boundaries.SegmentsToBoundaries ¶

SegmentsToBoundaries(
    width: int | None = None,
    relative_width: float | None = None,
    min_width: int = 1,
    exclude_start: bool = False,
    exclude_end: bool = False,
    min_start: int | None = 0,
    max_end: int | None = None,
    boundary_labels: tuple[int, int] = (1, 1),
)

Bases: Transform

Transform segments into boundary regions centered on their transitions.

A segment is a [start, end] interval (both inclusive, in frames) marking an event in a sequence — for example, a sign in a sign-language video. A boundary is a short interval centered on a transition (the point where a segment starts or ends) that can be used as a training target for models that detect segment onsets/offsets rather than segment content.

For each input segment, this transform can emit up to two boundaries: * a start-boundary centered on the segment's first frame, and * an end-boundary centered on the frame immediately after the segment's last frame (i.e. on end + 1).

Centering the end-boundary on end + 1 rather than on end means the boundary marks the transition out of the segment (between the last in-segment frame and the first out-of-segment frame), which is symmetric with the start-boundary marking the transition into the segment.

Boundary placement uses integer arithmetic when width is given as an integer: a boundary of width w centered on transition t covers [t - w // 2, t - w // 2 + w - 1]. For even widths this is asymmetric by one frame (more on the right of the transition than the left), which keeps everything on the frame grid without rounding artifacts.

The emitted boundaries are [b_start, b_end, label] triples. The output is sorted by b_start and may contain overlapping or duplicate boundaries when segments are close together — downstream code should handle this if needed. Boundaries that fall entirely outside [min_start, max_end] after clamping are dropped.

Parameters:

Name	Type	Description	Default
`width`	`int \| None`	Fixed boundary width in frames. Exactly one of `width` or `relative_width` must be provided.	`None`
`relative_width`	`float \| None`	Boundary width as a fraction of each segment's length, rounded to the nearest integer. Exactly one of `width` or `relative_width` must be provided.	`None`
`min_width`	`int`	Lower bound on the boundary width, applied after `relative_width` is resolved. Defaults to 1 so very short segments still produce a visible boundary.	`1`
`exclude_start`	`bool`	If True, do not emit start-boundaries.	`False`
`exclude_end`	`bool`	If True, do not emit end-boundaries.	`False`
`min_start`	`int \| None`	If not None, boundary starts are clamped to at least this value. Typically 0 to keep boundaries inside the timeline.	`0`
`max_end`	`int \| None`	If not None, boundary ends are clamped to at most this value. Set this to the sequence length minus one to keep boundaries inside the timeline.	`None`
`boundary_labels`	`tuple[int, int]`	Labels assigned to (start-boundaries, end-boundaries) respectively. Using distinct values (e.g. `(1, 2)`) lets downstream code tell the two boundary types apart.	`(1, 1)`

Example

import numpy as np from sign_language_tools.annotations.transforms import SegmentsToBoundaries segments = np.array([[0, 6], [7, 9], [13, 16]]) # (M, 2) transform = SegmentsToBoundaries(width=4, boundary_labels=(1, 2)) transform(segments) array([[ 0, 1, 1], [ 5, 8, 1], [ 5, 8, 2], [ 8, 11, 2], [11, 14, 1], [15, 18, 2]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Transforms segments into their transition boundaries.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, K)` with `K > 2` containing at least the start and end of `M` segments; extra columns are ignored.	required

Returns:

Type	Description
`ndarray`	Array of shape `(M', 3)` where each row is
`ndarray`	`[boundary_start, boundary_end, label]`, sorted by
`ndarray`	`boundary_start`. The dtype matches the input dtype.

sign_language_tools.annotations.transforms.bio_tags.BioTags ¶

BioTags(
    width: int | None = None,
    relative_width: float | None = None,
    b_label: int = 1,
    i_label: int = 2,
)

Bases: Transform

Split each segment into a Beginning tag followed by an Inside tag.

Segments use inclusive-end convention: [start, end] covers frames start, ..., end. Each segment of length L produces: * a B-tag covering [start, start + w - 1] with label b_label, and * an I-tag covering [start + w, end] with label i_label, where w is the B-tag width (either width or round(relative_width * L), minimum 1). If w >= L the I-tag is omitted, so single-frame segments produce a B-tag only.

Parameters:

Name	Type	Description	Default
`width`	`int \| None`	Fixed B-tag width, in frames. Exactly one of `width` or `relative_width` must be specified.	`None`
`relative_width`	`float \| None`	B-tag width as a fraction of each segment's length, rounded to the nearest integer (minimum 1). Exactly one of `width` or `relative_width` must be specified.	`None`
`b_label`	`int`	Label assigned to the B-tag (beginning) of each segment.	`1`
`i_label`	`int`	Label assigned to the I-tag (inside) of each segment.	`2`

Example

import numpy as np from sign_language_tools.annotations.transforms import BioTags segments = np.array([[0, 6], [8, 13]]) # (M, 2) transform = BioTags(width=2) transform(segments) array([[ 0, 1, 1], [ 2, 6, 2], [ 8, 9, 1], [10, 13, 2]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Splits each segment into B/I tags.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` containing the start and end of `M` segments.	required

Returns:

Type	Description
`ndarray`	Array of shape `(M', 3)` containing the start, end, and label
`ndarray`	of the resulting B/I tags, sorted by start.

sign_language_tools.annotations.transforms.offset.SegmentsToBoundaryOffsets ¶

SegmentsToBoundaryOffsets(
    sequence_length: int | None = None,
    background_value: float = -1.0,
    dtype: dtype = np.float32,
)

Bases: Transform

Render segments as a per-frame (start_offset, end_offset) series.

Segments use inclusive-end convention: [start, end] covers frames start, ..., end. For each frame t inside a segment: * start_offset[t] = t - start (frames since segment began) * end_offset[t] = end - t (frames until segment ends) Both offsets are 0 at the boundary frames and positive in between. Frames not covered by any segment get background_value.

When segments overlap, later segments overwrite earlier ones for the overlapping frames (sorted by start). To avoid surprises, run RemoveOverlapping upstream.

Parameters:

Name	Type	Description	Default
`sequence_length`	`int \| None`	Length of the output series. If None, inferred from the segments as `max(end) + 1`. Pass explicitly when frames may extend past the last annotated segment.	`None`
`background_value`	`float`	Value for frames outside any segment, written to both columns.	`-1.0`
`dtype`	`dtype`	Output dtype. Float because offsets are commonly used as regression targets, and the background sentinel may be non-integer.	`float32`

Example

import numpy as np from sign_language_tools.annotations.transforms import SegmentsToBoundaryOffsets segments = np.array([[2, 5]]) # (M, 2) transform = SegmentsToBoundaryOffsets(sequence_length=8) transform(segments) array([[-1., -1.], [-1., -1.], [ 0., 3.], [ 1., 2.], [ 2., 1.], [ 3., 0.], [-1., -1.], [-1., -1.]], dtype=float32)

call ¶

__call__(
    segments: ndarray, sequence_length: int | None = None
) -> np.ndarray

Renders the segments as a per-frame offset series.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` containing the start and end of `M` segments.	required
`sequence_length`	`int \| None`	Overrides the `sequence_length` passed to the constructor for this call only.	`None`

Returns:

Type	Description
`ndarray`	Array of shape `(sequence_length, 2)` containing the
`ndarray`	`(start_offset, end_offset)` pair for each frame.

sign_language_tools.annotations.transforms.silence.CloseShortSilences ¶

CloseShortSilences(max_silence: int)

Bases: Transform

Close short silences between consecutive segments by extending them.

Segments use inclusive-end convention: [start, end] covers frames start, ..., end. The silence between two segments [a, b] and [c, d] (with c > b) is c - b - 1 empty frames.

For each adjacent pair whose silence is at most max_silence frames, the boundary is placed near the midpoint of the gap. When the silence has odd length, the extra frame is absorbed into the earlier segment.

Like RemoveOverlapping, this is a single-pass transform: cascading effects (where closing one gap might shift a boundary into another nearby gap) are not handled. In practice this only matters when max_silence is large relative to segment lengths.

Parameters:

Name	Type	Description	Default
`max_silence`	`int`	Maximum number of empty frames between two segments for the gap to be closed. Must be non-negative.	required

Example

import numpy as np from sign_language_tools.annotations.transforms import CloseShortSilences segments = np.array([[0, 6], [8, 13], [25, 27], [30, 41]]) # (M, 2) transform = CloseShortSilences(max_silence=3) transform(segments) array([[ 0, 7], [ 8, 13], [25, 28], [29, 41]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Closes short silences between consecutive segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` containing the start and end of `M` segments.	required

Returns:

Type	Description
`ndarray`	The segments with short silences closed, sorted by start.

sign_language_tools.annotations.transforms.frame_labels.SegmentsToFrameLabels ¶

SegmentsToFrameLabels(
    vector_size: int | None = None,
    background_label: int = 0,
    fill_label: int = 1,
    dtype: dtype = np.int64,
)

Bases: Transform

Renders segments as a dense per-frame label vector.

Segments use inclusive-end convention: [start, end] covers frames start, ..., end. For (M, 2) segments, all frames within a segment get fill_label. For (M, 3) segments, the third column is used as the per-segment label. Frames not covered by any segment get background_label. When segments overlap, later segments (in array order) overwrite earlier ones for the overlapping frames.

Parameters:

Name	Type	Description	Default
`vector_size`	`int \| None`	Length of the output label vector. If None, inferred from the segments as `max(end) + 1`. Pass explicitly when frames may extend past the last annotated segment.	`None`
`background_label`	`int`	Label assigned to frames outside any segment.	`0`
`fill_label`	`int`	Label assigned to frames inside a segment when `segments` has no label column (shape `(M, 2)`).	`1`
`dtype`	`dtype`	Output dtype of the label vector.	`int64`

Example

import numpy as np from sign_language_tools.annotations.transforms import SegmentsToFrameLabels segments = np.array([[2, 4, 1], [6, 8, 2]]) # (M, 3) transform = SegmentsToFrameLabels() transform(segments) array([0, 0, 1, 1, 1, 0, 2, 2, 2], dtype=int64)

call ¶

__call__(
    segments: ndarray, vector_size: int | None = None
) -> np.ndarray

Renders the segments as a label vector.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, 3)` containing the start, end, and optionally the label of `M` segments.	required
`vector_size`	`int \| None`	Overrides the `vector_size` passed to the constructor for this call only.	`None`

Returns:

Type	Description
`ndarray`	Label vector of shape `(vector_size,)`.

sign_language_tools.annotations.transforms.frame_labels.FrameLabelsToSegments ¶

FrameLabelsToSegments(
    background_classes: tuple[int, ...] = (0,),
    include_labels: bool = True,
)

Bases: Transform

Inverse of SegmentsToFrameLabels: groups a label vector into segments.

Contiguous runs of identical labels become [start, end, label] segments, using the same inclusive-end convention as the rest of the module ([start, end] covers frames start, ..., end). Runs whose label is in background_classes are excluded from the output.

Parameters:

Name	Type	Description	Default
`background_classes`	`tuple[int, ...]`	Labels considered background; segments with one of these labels are dropped from the output.	`(0,)`
`include_labels`	`bool`	If True, the output segments include the label as a third column. If False, only `[start, end]` is returned.	`True`

Example

import numpy as np from sign_language_tools.annotations.transforms import FrameLabelsToSegments labels = np.array([0, 0, 1, 1, 1, 0, 2, 2, 2]) transform = FrameLabelsToSegments() transform(labels) array([[2, 4, 1], [6, 8, 2]])

call ¶

__call__(labels: ndarray) -> np.ndarray

Groups the label vector into segments.

Parameters:

Name	Type	Description	Default
`labels`	`ndarray`	Label vector of shape `(T,)`, where `T` is the number of frames.	required

Returns:

Type	Description
`ndarray`	Array of shape `(M, 3)`, or `(M, 2)` if `include_labels` is
`ndarray`	False, containing the `M` non-background segments.

sign_language_tools.annotations.transforms.merge.MergeSegmentsOnTransition ¶

MergeSegmentsOnTransition(
    transitions: list[tuple[int, int]], new_value: int = 1
)

Bases: Transform

Merges adjacent segment pairs whose labels match a given transition.

Segments are sorted by start. For each pair of adjacent segments (the end of one immediately followed by the start of the next, with no gap), if their (label_a, label_b) pair is listed in transitions, the pair is merged into a single segment spanning both, labeled new_value.

Note

This is a single-pass, pairwise transform: a chain of 3 or more consecutive segments that all match a transition is merged as overlapping pairs rather than collapsed into one segment. In practice this only matters when the same transition can repeat back-to-back.

Parameters:

Name	Type	Description	Default
`transitions`	`list[tuple[int, int]]`	List of `(label_a, label_b)` pairs. A pair of adjacent segments is merged when their labels match one of these pairs, in order.	required
`new_value`	`int`	Label assigned to the merged segments.	`1`

Example

import numpy as np from sign_language_tools.annotations.transforms import MergeSegmentsOnTransition segments = np.array([[2, 5, 1], [6, 10, 2], [12, 15, 3]]) # (M, 3) transform = MergeSegmentsOnTransition(transitions=[(1, 2)], new_value=9) transform(segments) array([[ 2, 10, 9], [12, 15, 3]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Merges matching adjacent segment pairs.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 3)` containing the start, end, and label of `M` segments.	required

Returns:

Type	Description
`ndarray`	The segments after merging, sorted by start.

sign_language_tools.annotations.transforms.merge.MergeSegments ¶

MergeSegments()

Bases: Transform

Merges overlapping or touching segments into single segments.

Segments are sorted by start, then greedily merged: a segment is folded into the current run whenever its start falls at or before current_end + 1 (i.e. it overlaps or touches the run with no gap). If segments has a label column, the merged segment keeps the label of the first segment in each run.

Example

import numpy as np from sign_language_tools.annotations.transforms import MergeSegments segments = np.array([ ... [2, 5, 1], ... [6, 12, 2], ... [14, 17, 0], ... [18, 25, 1], ... [67, 78, 3], ... ]) # (M, 3) transform = MergeSegments() transform(segments) array([[ 2, 12, 1], [14, 25, 0], [67, 78, 3]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Merges the overlapping or touching segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, 3)` containing the start and end (and optionally a label) of `M` segments.	required

Returns:

Type	Description
`ndarray`	The merged segments, sorted by start.

sign_language_tools.annotations.transforms.move.MoveSegments ¶

MoveSegments(dx: float = 0.0)

Bases: Transform

Shifts segments in time by a fixed offset.

Only the start and end columns are shifted; any extra column (e.g. a per-segment label) is left untouched.

Parameters:

Name	Type	Description	Default
`dx`	`float`	Offset added to the start and end of every segment. Negative values shift segments earlier.	`0.0`

Example

import numpy as np from sign_language_tools.annotations.transforms import MoveSegments segments = np.array([[10, 20, 1], [30, 50, 2]]) # (M, 3) transform = MoveSegments(dx=5) transform(segments) array([[15, 25, 1], [35, 55, 2]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Shifts the segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, 3)` containing the start and end (and optionally a label) of `M` segments.	required

Returns:

Type	Description
`ndarray`	The shifted segments, with the same shape as `segments`.

sign_language_tools.annotations.transforms.move.RandomRelativeMoveSegments ¶

RandomRelativeMoveSegments(dx_std: float = 0.4)

Bases: Transform

Shifts segments in time by a random offset proportional to their length.

Each segment is shifted independently by dx_std * N(0, 1) * length, where length is that segment's duration. Only the start and end columns are shifted; any extra column (e.g. a per-segment label) is left untouched.

Parameters:

Name	Type	Description	Default
`dx_std`	`float`	Standard deviation of the random shift, expressed as a fraction of each segment's length.	`0.4`

Example

import numpy as np from sign_language_tools.annotations.transforms import RandomRelativeMoveSegments segments = np.array([[10, 20, 1], [30, 50, 2]]) # (M, 3) transform = RandomRelativeMoveSegments(dx_std=0.4) moved = transform(segments) moved.shape (2, 3)

call ¶

__call__(segments: ndarray) -> np.ndarray

Shifts the segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, 3)` containing the start and end (and optionally a label) of `M` segments.	required

Returns:

Type	Description
`ndarray`	The shifted segments, with the same shape and dtype as
`ndarray`	`segments`.

sign_language_tools.annotations.transforms.overlapping.RemoveOverlapping ¶

RemoveOverlapping(min_gap: int = 0)

Bases: Transform

Remove overlap between segments and enforce a minimum frame gap.

Segments use inclusive-end convention: [start, end] covers frames start, start+1, ..., end, so length = end - start + 1.

Two segments A, B (sorted by start) are considered too close when B.start - A.end <= min_gap. The boundary is placed near the midpoint of the overlap/touch region; when min_gap is odd, the extra empty frame falls on B's side.

Parameters:

Name	Type	Description	Default
`min_gap`	`int`	Minimum number of empty frames required between two consecutive segments. Must be non-negative.	`0`

Example

import numpy as np from sign_language_tools.annotations.transforms import RemoveOverlapping segments = np.array([[2, 5], [6, 14], [12, 17]]) # (M, 2) transform = RemoveOverlapping(min_gap=1) transform(segments) array([[ 2, 5], [ 7, 13], [15, 17]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Removes overlap between the segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, K)` with `K > 2`; only the first two columns (start, end) are modified, extra columns pass through unchanged.	required

Returns:

Type	Description
`ndarray`	Segments with overlaps resolved, sorted by start.

sign_language_tools.annotations.transforms.fill_between.FillBetween ¶

FillBetween(
    start_value: int = 1,
    end_value: int = 1,
    fill_value: int = 1,
    max_width: int | None = None,
    bidirectional: bool = False,
)

Bases: Transform

Fills the gap between consecutive segments that form a transition.

Segments are sorted by start, then each adjacent pair is checked: if the first segment's label equals start_value and the second's label equals end_value (or the reverse, when bidirectional is True), a new segment is created to cover the gap between them, labeled fill_value.

Parameters:

Name	Type	Description	Default
`start_value`	`int`	The label of the segment where the filling starts.	`1`
`end_value`	`int`	The label of the segment where the filling ends.	`1`
`fill_value`	`int`	The label to use for the newly created segments that fill the gaps.	`1`
`max_width`	`int \| None`	The maximum width (duration) of a gap to be filled. If None, all identified gaps will be filled.	`None`
`bidirectional`	`bool`	If True, also considers gaps between an `end_value` segment followed by a `start_value` segment, not only the reverse.	`False`

Example

import numpy as np from sign_language_tools.annotations.transforms import FillBetween segments = np.array([[2, 5, 1], [14, 17, 1], [30, 34, 1]]) # (M, 3) transform = FillBetween(start_value=1, end_value=1, fill_value=1) transform(segments) array([[ 2, 5, 1], [ 6, 13, 1], [14, 17, 1], [18, 29, 1], [30, 34, 1]])

call ¶

__call__(segments: ndarray) -> np.ndarray

Fills the gaps between matching consecutive segments.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 3)` containing the start, end, and label of `M` segments.	required

Returns:

Type	Description
`ndarray`	The segments with filling segments inserted, sorted by start.

Utilities¶

sign_language_tools.annotations.utils.iou.temporal_intersection_over_union ¶

temporal_intersection_over_union(
    target_segment: Union[tuple[float, float], ndarray],
    segments: ndarray,
) -> np.ndarray

Computes the temporal intersection-over-union (tIoU) between a target segment and K segments. The IoU is also called the Jaccard index or the Jaccard similarity coefficient.

The tIoU between a segment A and a segment B is: $\frac{|A \cup B|}{|A \cap B|}$.

Parameters:

Name	Type	Description	Default
`target_segment`	`Union[tuple[float, float], ndarray]`	A target segment which is a tuple (start, end) or a Numpy array of shape (2).	required
`segments`	`ndarray`	A Numpy array of shape (K, 2) containing K segments (start, end).	required

Returns:

Name	Type	Description
`iou_scores`	`ndarray`	The IoU score between the interval and each of the K segments. This is a Numpy array of shape (K).

sign_language_tools.annotations.utils.iou.pairwise_temporal_intersection_over_union ¶

pairwise_temporal_intersection_over_union(
    target_segments: ndarray, candidate_segments: ndarray
) -> np.ndarray

Computes the temporal intersection-over-union (tIoU) between all pairs of target and candidate segments. The IoU is also called the Jaccard index or the Jaccard similarity coefficient.

The tIoU between a segment A and a segment B is: $\frac{|A \cup B|}{|A \cap B|}$. Args: target_segments: A Numpy array of shape (N, 2) containing N target segments (start, end). candidate_segments: A Numpy array of shape (M, 2) containing M candidate segments (start, end).

Returns:

Name	Type	Description
`iou_map`	`ndarray`	A Numpy array of size (N, M) containing the IoU score for all pairs of target and candidate segments.

sign_language_tools.annotations.utils.iou.get_segment_positive_negative_count ¶

get_segment_positive_negative_count(
    tiou_map: ndarray, tiou_threshold: float
)

sign_language_tools.annotations.utils.iou.calculate_segment_prediction_metrics ¶

calculate_segment_prediction_metrics(
    tiou_map: ndarray, tiou_threshold: float = 0.5
)

sign_language_tools.annotations.utils.nms.non_maximum_suppression ¶

non_maximum_suppression(
    proposals: ndarray, iou_threshold: float = 0.8
) -> np.ndarray

Filter out a set of segment proposals using the Non-Maximum Suppression (NMS) algorithm. A high scoring proposal is selected, then only proposals with a IoU score less than a given threshold are kept. This operation removes proposals that overlap too much.

Parameters:

Name	Type	Description	Default
`proposals`	`ndarray`	A Numpy array of shape (K, 3) containing K proposals with scores associated to them. Each proposal is in the form (start, end, score).	required
`iou_threshold`	`float`	The IoU threshold used to filter out proposals that overlap too much.	`0.8`

Returns:

Type	Description
`ndarray`	A numpy array of shape (L, 3) that is a subset of the K initial proposals.

sign_language_tools.annotations.utils.nms.soft_nms ¶

soft_nms(
    proposals: ndarray,
    alpha: float = 0.4,
    t1: float = 0.5,
    t2: float = 0.9,
    n_proposals: int = 100,
) -> np.ndarray

Filter out a set of segment proposals using the Soft Non-Maximum Suppression (soft NMS) algorithm with a gaussian decay.

Parameters:

Name	Type	Description	Default
`proposals`	`ndarray`	A Numpy array of shape (K, 3) containing K proposals with scores associated to them. Each proposal is in the form (start, end, score).	required
`alpha`	`float`	Alpha value of Gaussian decaying function. Default = 0.4.	`0.4`
`t1`	`float`	Minimum thresholds for soft NMS. Default = 0.5.	`0.5`
`t2`	`float`	Maximum thresholds for soft NMS. Default = 0.9.	`0.9`
`n_proposals`	`int`	Maximum number of retained proposals. Default = 100.	`100`

Returns:

Type	Description
`ndarray`	A numpy array of shape (L, 3) that is a subset of the K initial proposals.

Visualization¶

sign_language_tools.annotations.visualization.timeline.plot_segments_on_timeline ¶

plot_segments_on_timeline(
    segments: ndarray,
    labels: list[str] = None,
    y_lim: tuple[float, float] = (0, 0.5),
    ax: Axes | None = None,
    alpha: float = 0.5,
    colors: Any = None,
    cmap: str = "tab20",
    **kwargs: Any
)

Plot segments as colored spans on a timeline.

Each segment is drawn as a vertical span between its start and end frame. If labels is given, the label of each segment is drawn centered in its span. Colors can be assigned automatically (one color per unique label), given as a single color applied to every segment, or given explicitly per segment.

Parameters:

Name	Type	Description	Default
`segments`	`ndarray`	Array of shape `(M, 2)` or `(M, 3)`, where `M` is the number of segments and columns are `[start, end]` or `[start, end, label]`, with an inclusive end.	required
`labels`	`list[str]`	Optional list of `M` labels, one per segment. Segments with a `None` label are drawn without text.	`None`
`y_lim`	`tuple[float, float]`	Tuple `(y_min, y_max)` giving the vertical extent of each span, as axis-relative coordinates in `[0, 1]`.	`(0, 0.5)`
`ax`	`Axes \| None`	Matplotlib axes to draw on. Defaults to the current axes.	`None`
`alpha`	`float`	Opacity of the segment spans.	`0.5`
`colors`	`Any`	Colors to use for the segments. Can be: `None` (default): a color per unique label is picked automatically from `cmap` if `labels` is given, otherwise matplotlib's default color is used for every segment. A single color, applied to every segment. A list of `M` colors, one per segment.	`None`
`cmap`	`str`	Name of the matplotlib colormap used for automatic per-label coloring. Ignored if `colors` is not `None`.	`'tab20'`
`**kwargs`	`Any`	Additional keyword arguments forwarded to `Axes.axvspan`.	`{}`

Example

import numpy as np segments = np.array([[0, 10], [15, 25]]) plot_segments_on_timeline(segments, labels=["a", "b"])

Annotations¶

Transforms¶

Utilities¶

Visualization¶

Reference¶

Transforms¶

sign_language_tools.annotations.transforms.scale.ScaleSegments ¶

__call__ ¶

sign_language_tools.annotations.transforms.scale.RandomRelativeScaleSegments ¶

sign_language_tools.annotations.transforms.boundaries.SegmentsToBoundaries ¶

__call__ ¶

sign_language_tools.annotations.transforms.bio_tags.BioTags ¶

__call__ ¶

sign_language_tools.annotations.transforms.offset.SegmentsToBoundaryOffsets ¶

__call__ ¶

sign_language_tools.annotations.transforms.silence.CloseShortSilences ¶

__call__ ¶

sign_language_tools.annotations.transforms.frame_labels.SegmentsToFrameLabels ¶

__call__ ¶

sign_language_tools.annotations.transforms.frame_labels.FrameLabelsToSegments ¶

__call__ ¶

sign_language_tools.annotations.transforms.merge.MergeSegmentsOnTransition ¶

__call__ ¶

sign_language_tools.annotations.transforms.merge.MergeSegments ¶

__call__ ¶

sign_language_tools.annotations.transforms.move.MoveSegments ¶

__call__ ¶

sign_language_tools.annotations.transforms.move.RandomRelativeMoveSegments ¶

__call__ ¶

sign_language_tools.annotations.transforms.overlapping.RemoveOverlapping ¶

__call__ ¶

sign_language_tools.annotations.transforms.fill_between.FillBetween ¶

__call__ ¶

Utilities¶

sign_language_tools.annotations.utils.iou.temporal_intersection_over_union ¶

sign_language_tools.annotations.utils.iou.pairwise_temporal_intersection_over_union ¶

sign_language_tools.annotations.utils.iou.get_segment_positive_negative_count ¶

sign_language_tools.annotations.utils.iou.calculate_segment_prediction_metrics ¶

sign_language_tools.annotations.utils.nms.non_maximum_suppression ¶

sign_language_tools.annotations.utils.nms.soft_nms ¶

Visualization¶

sign_language_tools.annotations.visualization.timeline.plot_segments_on_timeline ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶

call ¶