zai-org

/

GLM-5.3-Flash

Like 3.15k

Follow

Z.ai 23.4k

Thanks for the contribution. I reviewed commit 087acb5 and checked the input-template changes separately from the new response-parsing configuration.

The Jinja changes look reasonable, and I did not find a prompt-formatting or tokenization regression in the cases tested. The response_template addition introduces separate behavior changes that I would like to address before merging.

The validation covered:

These checks used Transformers 58d05ca5, llama.cpp 3259ab6e, and vLLM source at 1a983e2e. They cover template rendering and output parsing;

1. Please preserve whitespace in parsed content.

   With prefix="<|assistant|><think>", the generated text "</think> return x\n" currently produces content="return x", removing both indentation and the trailing newline.

   Similarly, before\n<tool_call>f</tool_call>\nafter produces content="beforeafter" because each text region is stripped before being joined.

   This comes from the text parser’s default stripping behavior combined with join: "". Adding "content_args": {"strip": false} to the content field preserved whitespace in the reproduced cases. Please add regression tests for leading indentation, trailing newlines, and text surrounding tool calls. The whitespace policy for reasoning_content should also be explicit.

   This affects existing Transformers chat-pipeline usage: the pipeline automatically calls parse_response when the tokenizer provides a response template. Separating reasoning and tool calls is expected, but losing content whitespace should be avoided.

2. Please clarify and test incomplete tool-call handling.

   When generation ends inside an <arg_value> without a closing </tool_call>, the response parser can still return a tool_calls entry containing only the arguments completed so far. The result can therefore look like a parsed function call while required arguments are missing.

   This differs from the current vLLM Rust GLM XML parser, whose finish() returns IncompleteToolCall for an unfinished call.

   Please define the intended contract for truncated output and cover it with tests. If this requires a change in the shared Transformers response parser, that dependency should be tracked explicitly. Until then, the claim of matching vLLM’s parser should be qualified for incomplete outputs.

3. The Rust compatibility scope should be documented.

   The response template contains lookahead expressions such as (?=...), which are unsupported by the regex engine used in the current vLLM Rust generic hf output parser. This configuration therefore cannot currently be used with that parser.

   This does not change the behavior of the existing glm47/glm45 output parsers: adding response_template does not automatically select the generic hf parser. The native Rust glm5_next renderer also bypasses the Jinja template.

   However, vllm serve alone does not imply that native path is active. In the reviewed vLLM revision, the Rust frontend is opt-in, and the Python frontend’s GLM renderer still uses the HF chat-template path. The equivalence tests above support the input-template change on that path.

I’m comfortable with the Jinja-only changes based on this validation. For the response-template addition, I would like to see the whitespace issue fixed, incomplete-call behavior covered, and the compatibility scope clarified.

Thanks again for improving compatibility and adding structured response parsing.

Hi @ShijinZhang , thank you for the detailed reply! Issue 1 has been fixed in the template here (and in other PRs with the same problem). Issues 2 and 3 are in vLLM, but there's active PR work - see https://github.com/vllm-project/vllm/pull/66222 and https://github.com/vllm-project/vllm/pull/66156

Thanks for the update, @Rocketknight1 , and for linking the upstream vLLM work.

I rechecked d06424af. The input-template changes and ordinary-content whitespace fix look good: all 1,608 rendering comparisons against the original base produced identical prompts and token IDs, and content_args: {"strip": false} now preserves leading indentation and trailing newlines in ordinary responses.

For truncated tool calls, I observe that complete arguments are retained while an unfinished argument is omitted—for example, returning {"n": 3} when generation stops inside the following x argument. The Transformers documentation states that generation ending finalizes open regions even without their closing delimiter. Could you confirm that retaining fully matched arguments while omitting an unfinished argument is also part of the intended, supported parsing contract that callers can rely on? If so, we can treat this as expected HF parsing behavior and document the difference from the native GLM parser.

One template-level clarification remains: the leading \s* in the tool-call opener consumes whitespace immediately before <tool_call>, even with strip: false on content. Using <|assistant|><think> as the prefix:

Generated: </think>before\n<tool_call>f</tool_call>
Parsed content: "before"

Here, \n denotes an actual newline. Is this whitespace intentionally considered part of the tool-call delimiter? If so, an explicit regression case would help establish that contract. If the intention is to preserve the preceding content exactly, the leading \s* needs adjustment.

I also tested the linked vLLM PR branches against their respective bases. These results verify the targeted compatibility fixes in the PR branches. The broader Python adapter comparison did expose an existing difference: it preserves surrounding reasoning whitespace that Transformers’ finalized result strips. Both affected scenarios also reproduce on the base adapter with legacy-compatible metadata, so this is not introduced by #66156.