SAM 2 — Segment Anything Model 2

Overview#

SAM 2 is the second-generation Segment Anything model from Meta FAIR. The first SAM (Kirillov et al., arXiv:2304.02643, April 2023) introduced the idea of a promptable foundation model for image segmentation — a single model that, given a sparse prompt (a point, a box, a coarse mask, or a text embedding), would return a high-quality mask for any object in any image. It was trained on SA-1B (11M images, 1.1B masks) and instantly became the default segmentation engine for annotation tooling, medical-imaging labelling, and many CV preprocessing pipelines.

SAM 2, released on 29 July 2024, generalises the same promptable paradigm to video. The core insight is that a video is just a sequence of images that share object identity over time. SAM 2 adds a memory bank that stores feature representations from previously prompted and predicted frames, and a memory-attention block that lets the per-frame decoder condition on that history. Prompt the object once on one frame and SAM 2 tracks the corresponding masklet through the rest of the clip; correct it mid-clip with a positive or negative click and the corrected state propagates forward.

On Yobitel infrastructure SAM 2 powers two production surfaces. Inside MediQuery — Yobitel's clinical-imaging AI application — SAM 2 is the segmentation engine clinicians use to annotate radiology series for downstream analytics and model training. On Yobibyte, SAM 2 is exposed as a managed image- and video-segmentation endpoint that customers consume from a workspace without touching CUDA or Triton. This entry helps you stand up SAM 2 in production — picking the right variant, flags and SKU for interactive annotation, offline batch masking or live video tracking — whether you are running raw upstream on your own cluster or routing through Yobibyte's managed endpoint.

Quick start#

The example below installs SAM 2, loads the Large variant, and demonstrates two flows: a single-image segmentation given a point click, and a video segmentation given a click on frame 0 with propagation through the rest of the clip. Both flows run on a single H100, L40S or L4 with appropriate VRAM headroom.

# 1. Install (from Meta's official package)
# pip install "sam2 @ git+https://github.com/facebookresearch/sam2"

# 2. Image segmentation
from sam2.sam2_image_predictor import SAM2ImagePredictor
from sam2.build_sam import build_sam2
from PIL import Image
import numpy as np

sam2 = build_sam2(
    config_file="configs/sam2_hiera_l.yaml",
    ckpt_path="checkpoints/sam2_hiera_large.pt",
    device="cuda",
)
predictor = SAM2ImagePredictor(sam2)

image = np.array(Image.open("xray.png").convert("RGB"))
predictor.set_image(image)
masks, scores, logits = predictor.predict(
    point_coords=np.array([[512, 384]]),  # click on the object
    point_labels=np.array([1]),            # 1 = foreground
    multimask_output=True,
)

# 3. Video segmentation with cross-frame propagation
from sam2.build_sam import build_sam2_video_predictor

video_predictor = build_sam2_video_predictor(
    config_file="configs/sam2_hiera_l.yaml",
    ckpt_path="checkpoints/sam2_hiera_large.pt",
)

state = video_predictor.init_state(video_path="clip.mp4")

# Click on the object in frame 0
_, _, _ = video_predictor.add_new_points_or_box(
    inference_state=state,
    frame_idx=0,
    obj_id=1,
    points=np.array([[320, 240]]),
    labels=np.array([1]),
)

# Propagate the masklet through the rest of the video
for frame_idx, obj_ids, mask_logits in video_predictor.propagate_in_video(state):
    pass  # write per-frame masks to disk / queue

How it works#

SAM 2 is a five-component model wrapped in a per-frame inference loop with cross-frame memory. The image encoder (Hiera, a Hierarchical Vision Transformer pre-trained with Masked Autoencoder) processes each input frame into multi-scale feature maps. A memory-attention block lets the current frame cross-attend to features stored from earlier prompted and predicted frames. The prompt encoder turns user input (points, boxes, mask hints) into prompt embeddings, and a lightweight transformer mask decoder produces three candidate masks per object plus IoU confidence scores.

The memory bank is the architectural piece that makes video work. It is a FIFO buffer of recent frame features augmented with the features of prompted frames — small, typically a handful of frames worth — but enough to preserve object identity across motion, occlusion and brief out-of-view periods. When a user adds a corrective click mid-clip, the corrected frame enters the memory bank and the propagation continues from that corrected state. The result is interactive: every refinement is immediately reflected forward through the rest of the video.

For images, SAM 2 short-circuits the video machinery — the memory bank is empty, the memory-attention block is bypassed, and the model behaves like a faster, higher-quality version of original SAM. This is why the same model can serve both `SAM2ImagePredictor` and `SAM2VideoPredictor` from one weights file. On Yobibyte's managed endpoint, the platform picks the predictor based on the input type (single image vs MP4 / RTSP feed) so workspace customers do not need to think about it.

• Image encoder — Hiera ViT (T / S / B+ / L) with MAE pretraining. Produces multi-scale per-frame features.
• Memory attention — cross-attends current-frame features with the memory bank's stored features.
• Prompt encoder — sparse (points, boxes) and dense (mask) prompt encoders, inherited from SAM.
• Mask decoder — lightweight transformer decoder emitting three candidate masks plus IoU scores.
• Memory bank — FIFO buffer of recent frames plus prompted-frame features; small but sufficient for identity tracking.
• Memory encoder — distils mask + image features into compact memory tokens before storage.

Reference and specifications#

The table below is the canonical reference for the four SAM 2 variants and the most-used predictor flags. Parameter counts and FPS are from Meta's published numbers and Yobitel lab measurements on TensorRT-exported engines; treat throughput as planning anchors, not contractual.

Workload patterns#

Three deployment shapes cover the bulk of production SAM 2 use: interactive annotation behind a UI, offline batch masking across a video archive, and live tracking on streamed video. Each pattern targets a different latency budget, batch size and quality bar. These are also the three shapes Yobibyte automates for managed customers — the flags below are what a team running raw upstream signs up to hand-tune; on Yobibyte the workspace SLO derives them.

Pattern A — interactive annotation. Single user, sub-200 ms response to each click. Use SAM 2 Base+ on L40S or SAM 2 Large on H100 with `multimask_output=True`. Pattern B — batch masking across a video archive. Throughput-optimised; per-clip latency irrelevant; pre-extract frames to local SSD; run SAM 2 Large with `propagate_in_video` in long sweeps. Pattern C — live RTSP tracking. Single-stream propagation behind a lightweight gateway; use SAM 2 Small or Base+ for sustained 25-30 FPS on L4 / L40S.

# Pattern A — interactive annotation flow (sub-200 ms per click)
from sam2.sam2_image_predictor import SAM2ImagePredictor
from sam2.build_sam import build_sam2

sam2 = build_sam2("configs/sam2_hiera_b+.yaml",
                  "checkpoints/sam2_hiera_base_plus.pt",
                  device="cuda")
predictor = SAM2ImagePredictor(sam2)
predictor.set_image(image)  # encode once per image
# every subsequent click reuses the cached image embedding
masks, scores, _ = predictor.predict(
    point_coords=clicks,
    point_labels=labels,
    multimask_output=True,
)

# Pattern B — offline batch masking across a video archive
import os
from sam2.build_sam import build_sam2_video_predictor
predictor = build_sam2_video_predictor(
    "configs/sam2_hiera_l.yaml",
    "checkpoints/sam2_hiera_large.pt",
)
for clip in os.listdir("archive/"):
    state = predictor.init_state(video_path=f"archive/{clip}",
                                 offload_video_to_cpu=True)
    # seed object from a pre-computed bounding box per clip
    predictor.add_new_points_or_box(state, frame_idx=0,
                                    obj_id=1, box=seed_box[clip])
    for fi, oid, ml in predictor.propagate_in_video(state):
        save_mask(clip, fi, ml)

# Pattern C — live RTSP tracking with a fixed memory window
state = predictor.init_state(video_path="rtsp://camera/01",
                             async_loading_frames=True,
                             offload_state_to_cpu=False)

Sizing and capacity planning#

SAM 2 sizing is governed by three quantities: encoder cost per frame, memory-bank size per active object, and concurrent stream / user count. The table below assumes FP16 inference on Yobitel-deployable accelerators at 1024x1024 image resolution; FP8 on Hopper roughly doubles per-stream throughput. Mask decode is cheap; the encoder is the bottleneck.

VRAM budgets are dominated by activations and the memory bank for video. SAM 2 Large with a 64-frame memory window holds roughly 4-6 GB of active state per video stream; SAM 2 Small holds roughly 1 GB. Multi-object tracking multiplies this by the number of tracked objects per frame. For annotation tooling on H100, SAM 2 Large supports 50+ concurrent annotators per GPU. For live RTSP tracking, plan one accelerator per 4-8 streams at SAM 2 Base+ depending on resolution.

Limits and quotas#

Raw SAM 2 imposes no hard limits beyond available VRAM. The values below are operational ceilings observed in production and the corresponding defaults enforced on Yobibyte managed SAM 2 endpoints.

Observability#

Production SAM 2 monitoring focuses on encode time, memory-bank size and mask-quality scores rather than raw model latency. The minimum useful Prometheus surface for a self-managed deployment:

• Per-frame encode latency — histogram. Spikes indicate decoder contention or VRAM pressure.
• Per-click predict latency — histogram for interactive flows; target p95 below 200 ms.
• Memory bank size (frames) and (bytes) — per active video session.
• Mask IoU score distribution — sliding-window histogram; leftward drift signals model degradation or domain shift.
• Active sessions — gauge per endpoint; drives autoscaling decisions.
• Standard GPU telemetry — DCGM exporter for SM occupancy, memory pressure, NVLink utilisation.

Cost and FinOps#

Most SAM 2 cost comes from accelerator-hours; the model weights and the SA-V dataset are free. The ranges below are Yobitel NeoCloud on-demand reference prices for common SAM 2 deployment SKUs; reserved pricing is materially lower at 12+ month commitment. All figures USD; treat as planning anchors.

Security and compliance#

SAM 2's permissive licence makes the legal posture simple — the operational concerns are data residency and privacy. Inputs to SAM 2 are typically images or video that may contain identifiable people, sensitive medical content, or operational footage.

• Licence — SAM 2 weights and code are Apache 2.0. No copyleft concerns, suitable for closed-source SaaS.
• Data residency — for NCSC OFFICIAL workloads, pin the endpoint to Yobitel UK Sovereign region; for HIPAA / patient imaging, use MediQuery's dedicated SAM 2 deployment which routes through HIPAA-aligned Yobitel infrastructure.
• Input redaction — pre-blur or pre-redact faces before feeding video to SAM 2 if downstream consumers do not need identifiable subjects.
• Audit trail — Yobibyte logs per-prompt input hashes, predicted mask hashes, and predictor variant; sufficient evidence for most internal audit requirements.
• Model supply chain — pin a specific commit of the `sam2` package in production and verify checkpoint SHA against Meta's published values.

Migration and alternatives#

SAM 2 has clearly displaced the original SAM for general use; the meaningful alternatives in 2026 are domain-specific or different problem framings.

Troubleshooting#

The failure modes below cover roughly 80 percent of SAM 2 production tickets observed in Yobitel Managed Operations runbooks.

Where it fits in the Yobitel stack#

SAM 2 sits inside two Yobitel surfaces. In MediQuery, SAM 2 is the segmentation engine clinicians use to annotate radiology series — CT slices, MR sequences, pathology slides — for downstream analytics and model training. The MediQuery deployment runs SAM 2 Base+ on L40S behind a HIPAA-aligned audit log and the clinician-facing UI; it is not a generic SAM 2 endpoint but a clinical product built on top of one. On Yobibyte, SAM 2 is exposed as a managed image- and video-segmentation endpoint: customers submit images or video to a workspace and Yobibyte picks the variant, the placement and the concurrency to meet the workspace SLO.

Yobitel NeoCloud provides the underlying L4 / L40S / H100 capacity for both surfaces and for any self-managed SAM 2 deployment customers prefer to run themselves. InferenceBench tracks SAM 2 throughput-per-dollar across Yobitel and peer providers for the four variants and three workload shapes — the same data informs Yobibyte's internal placement decisions via Omniscient Compute.

• [MediQuery](/products/ai-applications/mediquery) — clinical SAM 2 deployment for radiology annotation.
• [Yobibyte](/products/yobibyte) — managed SAM 2 image and video segmentation endpoint.
• [Yobitel NeoCloud](/services/neocloud) — L4 / L40S / H100 capacity for self-managed deployments.
• [InferenceBench](/products/inferencebench) — public SAM 2 throughput-per-dollar tracking.