Infer-forge：围绕 SGLang 的 Harness、Loop 与 Graph 工程

14 阅读 - 阅读 来源: LMSYS
LMSYS SGLang 推理优化 工程系统 任务图 MonoRepo
Infer-forge：围绕 SGLang 的 Harness、Loop 与 Graph 工程

1. 引言

推理优化看似是代码层面的局部修改，但其有效性却是全局性的。一个内核、通信路径或调度变更，只有在由模型、工作负载、SLO、服务拓扑、运行时版本和加速器平台共同定义的特定部署点上才有意义。同一补丁可能在一个部署点带来提升，却在另一个点导致回退。

因此首要需求是可靠执行。重现部署点不仅需要模型能力，还需工具、环境、上下文、内存、验证和安全边界保持稳定。Harness Engineering 将这些周边条件转化为可复现、可检查的执行系统，形成抽象 Agent = Model + Harness 的基础。

The inference deployment space stacks five layers of static configuration. Model shows Ling, Qwen, DeepSeek, Kimi, GLM, and MiniMax. Serving Scenario runs Modality (text, image, video) into Traffic Shape (input and output length, media count, resolution, QPS, concurrency, cache reuse) into SLO (TTFT, TPOT, throughput, E2E latency). Serving Topology separates Deployment Architecture—Colocated PD, PD Disaggregation, and EPD Disaggregation, each listing the node roles it is built from — Prefill and Decode together, then Prefill and Decode as separate roles, then Encoder alongside them — from Parallelism—TP, PP, DP, and EP—because the two are chosen independently. Versioned Runtime Profiles is a stack of tabbed cards labelled Service A rev. 12, Service B rev. 7, and Service C rev. 21, plus a fourth paler card behind them all, blank and showing only its top edge, whose narrow tab carries an ellipsis for the profiles not drawn, the front card holding an Engine Configuration and a Container Image whose digest is pinned alongside its Framework, Device Runtime, and Collectives. Accelerator Platforms groups placeholder GPUs under Vendor A, B, and C. Arrows between the layers carry configuration dependency, not runtime data flow

2. 推理即部署空间

图1 将部署点转化为具体的约束链。模型决定支持的模态和执行路径，服务场景将模态和流量形状转化为 SLO，进而约束服务拓扑，最终通过版本化运行时配置文件在加速器平台上实现。

The infer-forge MonoRepo contains three groups: a Built-in Workspace; Inference Stack Repos centered on SGLang, including Dynamo, DeepGEMM, DeepEP, FlashMLA, FlashInfer, Humming, and Mooncake; and Harness Repos

3. MonoRepo

Infer-forge 通过 Git 子模块将相关仓库置于同一根目录，保留各自历史的同时提供跨仓库工程的稳定入口。

The Task Loop moves from Task Definition into Main Loop, uses Task Goal Met? to continue or satisfy Exit Criteria, preserves Task Memory, and draws on four Harness capabilities

4. Task Loop

任务循环从任务定义进入主循环，通过“任务目标是否达成”判断是否退出，同时保留任务记忆。

Task Definition consists of Task Type, Starting Context, Task Contract, and Exit Criteria

Loop Execution defines a Loop Block, uses Execution Routing to choose Model Tier and Agent Topology independently, executes the block, and uses Task Goal Met? to exit or continue while Task Memory carries the Current Loop Block, a Loop Block Handoff, and the Next Loop Block across iterations

Node Registry combines periodic runtime and GPU observations to determine claim cleanup eligibility.

Skills are organized across SGLang Upstream, Cross-lib, Task, and Ops, while Tools & CLI include Deploy, Build, Pull Weights, Sync Code, Evaluate, Profile, Diagnose Online, and Monitor

Journal uses LLM-wiki to connect records from multiple Tasks and Multi-dim Index fields such as Model, Type, and GPU to support Retrieve, Compare, and Filter

Safety Guard constrains execution through Push Guard, Traceable Path, Env Isolation, Production Read-only Access, Secrets, Data, Human Gate, and Cross-Model Adversarial Review

Task Graph defines Task node, Shared repo, and External system as graph elements, shows Verification before Handoff, and distinguishes Handoff edge, State edge, and Control edge

5. 实施现状

Infer-forge 已在内部持续使用，四个月内并发任务峰值从 2 升至 9，一个项目中协调了 38 个可验证任务节点。

© 2026 Winzheng.com 赢政天下 | 转载请注明来源并附原文链接

相关测评

LMSYS 突破极限：DeepSeek-V4-Pro 在 H20 上的服务优化

本文深入探讨1.6万亿参数MoE模型DeepSeek-V4-Pro在H20 GPU上的高效服务方案。通过硬件角色分配、Humming MXFP4AFP8量化及Online C128技术，结合PP2/P

DeepSeek MoE服务优化 08-29 15

LMSYS SGLang 亚秒级引擎恢复：权重缓存守护进程实现快速重启

针对SOTA大模型冷启动耗时过长的问题，Ant Ling Infra团队与SGLang团队推出Weight Cache Daemon。该守护进程通过CUDA IPC零拷贝映射，将量化后权重常驻GPU内

SGLang LLM服务 08-29 18

LMSYS 追逐Batch-1极限：Ling-3.0-flash在Blackwell上的推测解码优化

本文深入分析Ling-3.0-flash混合注意力MoE模型在4张NVIDIA Blackwell GPU上的Batch-1解码优化。通过NEXTN和DSpark两条推测解码路径，单请求吞吐从288

AI推理优化 推测解码 08-29 16

LMSYS Qwen3.8-Flash-Next：SGLang 实现 Day-0 支持

Qwen团队开源Qwen3.8-Flash-Next，这是一款多模态MoE模型，也是Qwen4架构的早期预览。SGLang与Qwen、NVIDIA及AMD团队合作，提供Day-0支持。该模型采用Gat

Qwen SGLang 08-29 12
本文来自 LMSYS 博客，赢政天下（winzheng.com）进行了全文翻译。 点击这里查看原文 如果转载中文，请注明出处，谢谢支持！