Skip to content

Distributed Inference

This section covers mlxcel's distributed inference capabilities, which allow models to be spread across multiple GPUs or nodes to handle models that are too large for a single device or to increase throughput.

Contents

Document Description
Overview Concepts, strategy comparison, infrastructure, and cluster configuration
Tensor Parallelism Splitting weight matrices across GPUs within a single model layer
Pipeline Parallelism Assigning model layers to different GPUs in a pipeline
Disaggregated Inference Separating prompt processing from token generation onto different nodes

Choosing a Strategy

flowchart TD
    A{"Model fits on\none GPU?"} -->|Yes| B["Single-node serving\n(no distributed mode needed)"]
    A -->|No| C{"How to split?"}
    C --> D{"Multiple GPUs\non same host?"}
    C --> E{"Multiple\nhosts?"}
    C --> F{"Separate prefill\nfrom decode?"}
    D --> G["**Tensor Parallelism**\nLow-latency\nNVLink/PCIe required"]
    E --> H["**Pipeline Parallelism**\nHigher latency per request\nGood for large models"]
    F --> I["**Disaggregated Inference**\nBest for mixed workloads\nLong prompts + variable decode"]

See Overview for a detailed comparison table.