Skip to content

Guides · Workspace authoring

Profiling, runtime images, and invocation patches

Profiling is prepared on a server and requested on a selected Eval or Bench; it is not a separate workload kind. Enable capture-target preparation on the server, a server case, or an invocation patch. The three capture deadlines are optional; omitted ones use built-in defaults that dry-run renders, so declare them only to size a particular capture:

[servers.example]
profiling = true
capture_arm_deadline_seconds = 120
capture_control_deadline_seconds = 120
capture_finalization_deadline_seconds = 1800

Select the capture mechanism on the server, a case, or an invocation patch (--set server.profiler.mechanism=engine_trace); omission resolves to managed collection:

[servers.example.profiler]
mechanism = "engine_trace"

Managed collection wraps each captured rank process tree with Nsight Systems. Engine trace instead lets the framework’s own profiler write per-rank traces into a record-owned directory that InferLab assigns at planning. Integrations that cannot honor engine_trace reject it while planning; the backend support matrix lists which do. Profiling owns the capture lifecycle, placement limits, and coverage rules.

An engine-trace server declares no profiler.nsys escape inputs. Combining mechanism = "engine_trace" with nsys escapes in one server or role fails workspace load; a combination assembled across layers, such as a case or --set selecting engine_trace for a server with escapes, fails resolution. A server case may declare only profiler.mechanism; nsys escapes belong to the server and its roles.

Declaring profiler.mechanism or nsys escape inputs on a server whose profiling resolves off (no profiling = true and no requested --capture) fails resolution with a typed error naming the declaration and how to enable profiling; profiler declarations are never silently dropped.

capture_arm_deadline_seconds is one budget for preparing and arming every selected rank target. capture_control_deadline_seconds bounds each framework range action that opens a window, and each managed-collection close. capture_finalization_deadline_seconds is one budget for session inspection, any required collection stop, asynchronous report completion, report coverage, and every engine-trace window close; its default depends on the resolved mechanism. Profiling describes how these budgets combine during a capture.

A server may replace the dedicated Nsight Systems fields or add launch and start options and environment; role declarations merge after the server layer:

[servers.example.profiler.nsys]
executable = "/path/to/nsight-systems/bin/nsys"
trace = ["cuda", "nvtx", "osrt"]
launch_options = []
start_options = []
sampling = "none"
context_switch = "none"
[servers.example.profiler.nsys.env]
PATH = "/path/to/nsight-systems/bin:/usr/bin"

executable, trace, sampling, and context_switch replace the server values. Role option lists append after server lists, and role environment entries win by key. The effective environment applies to launch, collection start, session inspection, and collection stop; only the launch invocation passes it onward to the wrapped server. InferLab rejects options that attempt to replace managed session, report, range, export, overwrite, or launch-wait facts.

Select OS runtime tracing (osrt in trace) only when the experiment needs it, and size the lifecycle deadlines for its added capture cost; removing it is not a general cure for startup or finalization timing.

Request a capture as described in Profiling.

A runtime image definition selects one stack, a base image that InferLab resolves to an immutable per-platform digest, one or more platforms, an optional subset of the stack’s source_paths, and recipe-referenced validations:

[images.vllm-runtime]
stack = "vllm"
base_image = "example.com/micromamba:1.0"
platforms = ["linux/amd64"]
packages = ["upstream/vllm"]
[[images.vllm-runtime.validations]]
recipe = "smoke"
server_case = "tp1"

Omitting packages selects every stack source path. Each selected path builds one wheel for every project the stack’s Pixi environment installs from inside it, at the directory the Pixi manifest declares. A submodule whose Python project lives below its root (for example SGLang’s python/) is selected by its root path, while the manifest points the dependency at the project directory. A selected path the environment installs nothing from fails image resolution, so when a stack also lists source paths consumed only while building another package, declare packages as the subset the environment installs. Every locked registry package must carry a hash: a locked PyPI package without one fails image resolution naming the package, because the image could not pin it; take such packages from conda instead and relock. A validation names only a recipe and optional server case; it does not restate model, placement, server, or measurement facts. Builds require a clean workspace. The only builder kind in local bindings is local-docker:

[builders.local]
kind = "local-docker"

Build workflow covers building, including the toolchain runtime that package builds need. Portable contexts, image metadata, and built packages exclude model locators, builder hosts, workspace paths, placements, and other machine-private facts.

Per-machine container bindings apply to every server container on that machine, including image validations, but not to inferlab run. They may pass environment values by name, grant absolute device paths, lift the memlock limit, and add only IPC_LOCK, SYS_NICE, or SYS_PTRACE; InferLab never requests privileged mode. pass_env may not name variables InferLab manages in the container (HOME, USER, LOGNAME, CUDA_VISIBLE_DEVICES, CUDA_DEVICE_ORDER, CONDA_PREFIX):

[machines.local.container]
pass_env = ["HF_TOKEN"]
devices = ["/dev/infiniband"]
memlock_unlimited = true
capabilities = ["IPC_LOCK", "SYS_NICE"]

A workspace may also declare a digest-pinned image it did not build:

[external_images.official]
reference = "example.com/vllm@sha256:<64-hex-digest>"
integration = "vllm"

The integration claim must equal the integration of every server stack the image is selected for. An image supplies only the engine: for external and built images alike, InferLab lowers commands inside the image with the workspace’s own integration packages, mounted read-only ahead of the image’s site-packages, never adapter packages baked into the image, so an image keeps working across adapter protocol changes. (A local launch without an image lowers in the stack’s own Pixi environment.) The workspace must commit and install a framework-free Pixi environment named adapter that contains inferlab-integration-<integration>; its InferLab dependencies, such as the adapter SDK and a shared Gateway package, are mounted with it. The image provides the Python interpreter and third-party packages such as Pydantic.

By default InferLab replaces an external image’s entrypoint with the rendered command, because an entrypoint may itself be a fixed serving command. When the image’s own entrypoint is the supported way to run commands in it, for example an entrypoint that prepares the image’s CUDA or NCCL runtime before executing its arguments, declare entrypoint = "image": serving processes and inferlab run commands then keep that entrypoint and receive the command as its arguments. The recorded server command shows which path ran.

Variables that settings or the integration set for a containerized process, such as extra_env, reach the container verbatim and apply only inside it; the host-side container client keeps its launch machine’s own environment.

Built and external image selection and ad-hoc probes cover how a launch or an inferlab run command selects an image.

Use repeatable --set for temporary typed changes. Values use TOML syntax and later assignments win.

Terminal window
inferlab serve start example \
--set server.readiness_timeout_seconds=1800 \
--set server.settings.max_model_len=32768 \
--set server.devices=4 \
--dry-run
inferlab recipe run qualify \
--set evals.gsm8k.limit=100 \
--set evals.gsm8k.trials=5 \
--set evals.gsm8k.concurrency=8 \
--set 'benches.random-8k1k.concurrency=[1, 8]' \
--dry-run

Recipe measurement patches may name only Eval and Bench definitions selected by that recipe’s workload suite. They cannot change identities, kinds, suite membership, the gate, or the selected server.

The evals.gsm8k.trials patch above repeats one lm-eval task; read Repeated trials for which tasks accept it and how trials are seeded.