Guides · Workspace authoring
Profiling, runtime images, and invocation patches
Workload profiling
Section titled “Workload profiling”Profiling is prepared on a server and requested on a selected Eval or Bench; it is not a separate workload kind. Enable capture-target preparation on the server, a server case, or an invocation patch. The three capture deadlines are optional; omitted ones use built-in defaults that dry-run renders, so declare them only to size a particular capture:
[servers.example]profiling = truecapture_arm_deadline_seconds = 120capture_control_deadline_seconds = 120capture_finalization_deadline_seconds = 1800Select the capture mechanism on the server, a case, or an invocation patch
(--set server.profiler.mechanism=engine_trace); omission resolves to
managed collection:
[servers.example.profiler]mechanism = "engine_trace"Managed collection wraps each captured rank process tree with Nsight Systems.
Engine trace instead lets the framework’s own profiler write per-rank traces
into a record-owned directory that InferLab assigns at planning. Integrations
that cannot honor engine_trace reject it while planning; the
backend support matrix
lists which do. Profiling owns the capture
lifecycle, placement limits, and coverage rules.
An engine-trace server declares no profiler.nsys escape inputs. Combining
mechanism = "engine_trace" with nsys escapes in one server or role fails
workspace load; a combination assembled across layers, such as a case or
--set selecting engine_trace for a server with escapes, fails resolution.
A server case may declare only profiler.mechanism; nsys escapes belong to
the server and its roles.
Declaring profiler.mechanism or nsys escape inputs on a server whose
profiling resolves off (no profiling = true and no requested --capture)
fails resolution with a typed error naming the declaration and how to enable
profiling; profiler declarations are never silently dropped.
capture_arm_deadline_seconds is one budget for preparing and arming every
selected rank target. capture_control_deadline_seconds bounds each framework
range action that opens a window, and each managed-collection close.
capture_finalization_deadline_seconds is one budget for session inspection,
any required collection stop, asynchronous report completion, report coverage,
and every engine-trace window close; its default depends on the resolved
mechanism. Profiling describes how these
budgets combine during a capture.
A server may replace the dedicated Nsight Systems fields or add launch and start options and environment; role declarations merge after the server layer:
[servers.example.profiler.nsys]executable = "/path/to/nsight-systems/bin/nsys"trace = ["cuda", "nvtx", "osrt"]launch_options = []start_options = []sampling = "none"context_switch = "none"
[servers.example.profiler.nsys.env]PATH = "/path/to/nsight-systems/bin:/usr/bin"executable, trace, sampling, and context_switch replace the server
values. Role option lists append after server lists, and role environment
entries win by key. The effective environment applies to launch, collection
start, session inspection, and collection stop; only the launch invocation
passes it onward to the wrapped server. InferLab rejects options that attempt
to replace managed session, report, range, export, overwrite, or launch-wait
facts.
Select OS runtime tracing (osrt in trace) only when the experiment needs
it, and size the lifecycle deadlines for its added capture cost; removing it
is not a general cure for startup or finalization timing.
Request a capture as described in Profiling.
Runtime images and ad-hoc execution
Section titled “Runtime images and ad-hoc execution”A runtime image definition selects one stack, a base image that InferLab
resolves to an immutable per-platform digest, one or more platforms, an
optional subset of the stack’s source_paths, and recipe-referenced
validations:
[images.vllm-runtime]stack = "vllm"base_image = "example.com/micromamba:1.0"platforms = ["linux/amd64"]packages = ["upstream/vllm"]
[[images.vllm-runtime.validations]]recipe = "smoke"server_case = "tp1"Omitting packages selects every stack source path. Each selected path builds
one wheel for every project the stack’s Pixi environment installs from inside
it, at the directory the Pixi manifest declares. A submodule whose Python
project lives below its root (for example SGLang’s python/) is selected by
its root path, while the manifest points the dependency at the project
directory. A selected path the environment installs nothing from fails image
resolution, so when a stack also lists source paths consumed only while
building another package, declare packages as the subset the environment
installs. Every locked registry package must carry a hash: a locked PyPI
package without one fails image resolution naming the package, because the
image could not pin it; take such packages from conda instead and relock. A
validation names only a recipe and optional server case; it does not restate
model, placement, server, or measurement facts. Builds require a clean
workspace. The only builder kind in local bindings is local-docker:
[builders.local]kind = "local-docker"Build workflow covers building, including the toolchain runtime that package builds need. Portable contexts, image metadata, and built packages exclude model locators, builder hosts, workspace paths, placements, and other machine-private facts.
Per-machine container bindings apply to every server container on that
machine, including image validations, but not to inferlab run. They may pass environment values
by name, grant absolute device paths, lift the memlock limit, and add only
IPC_LOCK, SYS_NICE, or SYS_PTRACE; InferLab never requests privileged
mode. pass_env may not name variables InferLab manages in the container
(HOME, USER, LOGNAME, CUDA_VISIBLE_DEVICES, CUDA_DEVICE_ORDER,
CONDA_PREFIX):
[machines.local.container]pass_env = ["HF_TOKEN"]devices = ["/dev/infiniband"]memlock_unlimited = truecapabilities = ["IPC_LOCK", "SYS_NICE"]A workspace may also declare a digest-pinned image it did not build:
[external_images.official]reference = "example.com/vllm@sha256:<64-hex-digest>"integration = "vllm"The integration claim must equal the integration of every server stack the
image is selected for. An image supplies only the engine: for external and
built images alike, InferLab lowers commands inside the image with the
workspace’s own integration packages, mounted read-only ahead of the image’s
site-packages, never adapter packages baked into the image, so an image keeps
working across adapter protocol changes. (A local launch without an image
lowers in the stack’s own Pixi environment.) The workspace must commit and
install a framework-free Pixi environment named adapter that contains
inferlab-integration-<integration>; its InferLab dependencies, such as the
adapter SDK and a shared Gateway package, are mounted with it. The image
provides the Python interpreter and third-party packages such as Pydantic.
By default InferLab replaces an external image’s entrypoint with the rendered
command, because an entrypoint may itself be a fixed serving command. When the
image’s own entrypoint is the supported way to run commands in it, for example
an entrypoint that prepares the image’s CUDA or NCCL runtime before executing
its arguments, declare entrypoint = "image": serving processes and
inferlab run commands then keep that entrypoint and receive the command as
its arguments. The recorded server command shows which path ran.
Variables that settings or the integration set for a containerized process,
such as extra_env, reach the container verbatim and apply only inside it;
the host-side container client keeps its launch machine’s own environment.
Built and external image selection
and ad-hoc probes cover how a launch or an
inferlab run command selects an image.
Invocation patches
Section titled “Invocation patches”Use repeatable --set for temporary typed changes. Values use TOML syntax and
later assignments win.
inferlab serve start example \ --set server.readiness_timeout_seconds=1800 \ --set server.settings.max_model_len=32768 \ --set server.devices=4 \ --dry-run
inferlab recipe run qualify \ --set evals.gsm8k.limit=100 \ --set evals.gsm8k.trials=5 \ --set evals.gsm8k.concurrency=8 \ --set 'benches.random-8k1k.concurrency=[1, 8]' \ --dry-runRecipe measurement patches may name only Eval and Bench definitions selected by that recipe’s workload suite. They cannot change identities, kinds, suite membership, the gate, or the selected server.
The evals.gsm8k.trials patch above repeats one lm-eval task; read
Repeated trials for which tasks accept it
and how trials are seeded.