Skip to content

Dense Prediction and Vision Tasks

mlxcel predict is the offline file interface for checkpoints that return depth, normals, segmentation, point maps, keypoints, per-instance masks, meshes, or Gaussian splats. These outputs have shapes and lifetimes that do not fit the token stream from generate or the box rows from detect, so they have their own subcommand and a shared file contract.

mlxcel predict

mlxcel predict --model models/dinov2-small --image input.png --output outputs/features --task features
mlxcel predict --model models/video-depth --video clip.mp4 --output outputs/depth
mlxcel predict --model models/sapiens2-pose --image person.png --output outputs/pose \
  --opts '{"boxes":[[140,90,920,700]],"flip_test":true}'
Flag Default Description
-m, --model <PATH> required Dense-prediction checkpoint directory.
--image <PATH> - Input image. Conflicts with --video; exactly one is required.
--video <PATH> - Input video.
-o, --output <DIR> required Output directory, created when absent.
--task <NAME> family default Model task. Required when the family supports more than one.
--prompt <TEXT> - Text prompt. Repeatable. Families without prompt support reject it.
--boxes <X1,Y1,X2,Y2[;...]> - Pixel-spanning xyxy boxes. Repeatable, and several boxes may share one flag separated by semicolons.
--mask <PATH> - Object mask for SAM 3D Objects; RGBA alpha is used when omitted.
--pointmap <PATH> - Safetensors file with an HWC XYZ points tensor for SAM 3D Objects.
--max-frames <N> 32 Maximum sampled video frames.
--target-fps <FPS> - Time-based uniform video sampling instead of a consecutive prefix.
--opts <JSON> - Model-specific options as a JSON object.

The command adds the selected task under task, the prompt strings under prompts, and the parsed numeric boxes under boxes before calling the predictor, so a model reads every option from one map. A prompt or box on a family that does not accept it is an error.

Video Depth Anything and SAM 3 / 3.1 track keep at most --max-frames consecutive frames from the start of the clip when --target-fps is unset. Every other family samples uniformly at 2 fps, capped at --max-frames. Setting --target-fps switches the first group to uniform sampling in time too. A sampled set whose decoded RGBA worst-case size would exceed 512 MiB is rejected before ffmpeg decodes it. Existing files with the standard names are replaced and unrelated files in the output directory are left alone.

Families

Family Detection Tasks Notable output files
DINOv2 dinov2, dinov2_with_registers features result.safetensors, meta.json
Sapiens2 sapiens2 pose, seg, normal, pointmap, matting, backbone keypoints.json, segmentation.png, normal.png
SAM 3 sam3, sam3_video detect, segment, track instances.json, masks/000.png, masks/f000012_o003.png
SAM 3.1 sam3.1, sam3.1_video detect, segment, track as SAM 3
Video Depth Anything video_depth_anything depth depth.png, depth/000000.png, ...
MoGe3 moge3 geometry depth.png, normal.png, mask.png
SAM 3D Objects sam3d_objects mesh mesh.obj or gaussian.ply
SAM 3D Body sam3d_body body_mesh mesh.obj with vertices, keypoints_3d, joint_coords, camera

PP-DocLayoutV3 and RF-DETR Base and Seg Small run under detect, not predict, because they return boxes. PP-DocLayoutV3 emits boxes in reading order, which mlxcel generate --layout-detections consumes for Falcon-OCR. A segmentation checkpoint adds --masks-dir <DIR>, which writes one source-resolution 8-bit mask PNG per detection in printed order.

Output Contract

Every successful run writes:

Path Contents
result.safetensors Every output tensor converted to contiguous f32, except masks, which stays u8. Instances include boxes (N, 4) and scores (N,); masks is (M, H, W) at source resolution. Video maps stack on axis 0.
meta.json Family, task, source size, inference time after MLX evaluation, finite min and max for every tensor, model metadata, and flattened intrinsics when present.
keypoints.json Present when people were detected. Each entry holds bbox, (x, y, score) points in source pixels, and optional point names.
instances.json Scored instances with source-pixel bbox, label, mask_index, optional video object_id and frame, and a relative mask PNG path. Empty for outputs without instances.
masks/000.png Image instance mask, 8-bit single-channel, values 0 and 255. The number is the masks tensor row.
masks/f000012_o003.png Video mask for frame 12 and stable object 3.
depth.png 16-bit grayscale. The finite minimum and maximum become 0 and 65535; NaN and infinity become 0.
normal.png 8-bit RGB from (normal + 1) / 2, clamped to [0, 1].
mask.png 8-bit grayscale, 0 at or below 0.5 and 255 above it.
segmentation.png 8-bit class IDs.

Sapiens2 publishes a fixed 1024x768 input (height by width). Segmentation and matting stretch to that canvas; normal and pointmap estimation preserve the aspect ratio with symmetric padding and remove it before resizing back to source resolution. The loader accepts both the earlier nested backbone schema and the current flat Transformers schema, for the 0.4B, 0.8B, 1B, and 5B architectures. Pose accepts up to 16 boxes and processes them one at a time to bound peak memory; it does not run a person detector, so omit boxes to run on the full image.

Video Depth Anything preserves the source aspect ratio, resizes the lower bound to 518 with 14-pixel multiples, and runs the upstream 32-frame window with 10-frame overlap, keyframe conditioning, and an eight-frame linear blend. Metric checkpoints keep their absolute scale and report units: "metres"; relative checkpoints report units: "relative_disparity".

MoGe3 estimates a camera-space point map, depth, a validity mask, surface normals, and normalized camera intrinsics from one image. Its --opts fields are resolution_level (0 to 9), num_tokens, refine_steps (0 to 16, default 3), force_projection, apply_mask, fov_x, and return_per_step.