Skip to content

test(e2e): add deepl conformance cases for ai-proxy#4121

Open
123123213weqw wants to merge 1 commit into
higress-group:mainfrom
123123213weqw:e2e-deepl-ai-proxy
Open

test(e2e): add deepl conformance cases for ai-proxy#4121
123123213weqw wants to merge 1 commit into
higress-group:mainfrom
123123213weqw:e2e-deepl-ai-proxy

Conversation

@123123213weqw

Copy link
Copy Markdown

Ⅰ. Describe what this PR did

Add a non-streaming e2e conformance case for the deepl provider of the ai-proxy wasm plugin. The case sends an OpenAI-format chat completion request (model Free) and verifies that ai-proxy transforms it to the DeepL /v2/translate request shape (text array + target_lang), routes to the Free host api-free.deepl.com, and converts the DeepL translation response back into an OpenAI chat.completion (with detected_source_language mapped to message.name).

It reuses the same conformance scaffolding as the existing openai/hunyuan/etc. cases:

  • New Ingress wasmplugin-ai-proxy-deepl (host api-free.deepl.com) pointing at llm-mock-service.
  • New matchRule with type: deepl, apiTokens: [fake_token], targetLang: EN (required by deeplProviderInitializer.ValidateConfig).
  • A single non-streaming test case asserting the OpenAI-format response, ignoring the volatile created and usage fields.

Ⅱ. Does this pull request fix one issue?

fixes #1723

Ⅲ. Why don't you add test cases (unit test/integration test)?

This PR itself adds the e2e conformance test case for the deepl provider.

Ⅳ. Describe how to verify it

Run the conformance suite against a cluster with the llm-mock (latest image, which ships the deepl mock) service and the ai-proxy wasm plugin built:

go test ./test/e2e/conformance/tests/... -run TestGoWasmAiProxy

The deepl case 1: non-streaming request case should pass.

Ⅴ. Special notes for reviews

  • Only a non-streaming case is added; DeepL's /v2/translate is non-streaming, so the deepl provider has no streaming path.
  • The request body uses model: "Free" (the deepl provider requires Free or Pro; Free selects the api-free.deepl.com host).
  • targetLang: EN is mandatory in the provider config and is therefore set in the matchRule.
  • JsonBodyIgnoreFields: ["created", "usage"] excludes the timestamp (time.Now()) and the always-null usage (deepl does not populate token usage) from the comparison.

Ⅵ. AI Coding Tool Usage Checklist (if applicable)

Please check all applicable items:

  • For new standalone features (e.g., new wasm plugin or golang-filter plugin):

    • I have created a design/ directory in the plugin folder
    • I have added the design document to the design/ directory
    • I have included the AI Coding summary below
  • For regular updates/changes (not new plugins):

    • I have provided the prompts/instructions I gave to the AI Coding tool below
    • I have included the AI Coding summary below

AI Coding Prompts (for regular updates)

AI Coding Summary

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
⚠️ Please upload report for BASE (main@1547826). Learn more about missing BASE report.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #4121   +/-   ##
=======================================
  Coverage        ?   50.70%           
=======================================
  Files           ?       88           
  Lines           ?    13398           
  Branches        ?        0           
=======================================
  Hits            ?     6794           
  Misses          ?     6156           
  Partials        ?      448           
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@CH3CHO

CH3CHO commented Jul 11, 2026

Copy link
Copy Markdown
Collaborator
2026-07-11T02:49:35.9073788Z     --- FAIL: TestHigressConformanceTests/WasmPluginAiProxy (387.23s)
2026-07-11T02:49:35.9075247Z         --- FAIL: TestHigressConformanceTests/WasmPluginAiProxy/WasmPlugins_ai-proxy (387.09s)

The test doesn't pass.

…test

The deepl conformance case's expected response body was invalid JSON:
`"finish_reason":null` sat outside the choice object but inside the
`choices` array (i.e. `...}},"finish_reason":null]` instead of
`...},"finish_reason":null}]`). The e2e HTTP comparator unmarshals the
expected body first, so the case always failed with "failed to unmarshall
ExpectedResponse body ... invalid character ':' after array element",
which is what surfaced as the higress-wasmplugin-test (GO) failure.
Move `"finish_reason":null` back inside the single choice object so the
document is well-formed. The corrected body now parses and is structurally
equal to the deepl provider's actual `chatCompletionResponse` output
(`index`, `message{name,role,content}`, `finish_reason:null`,
`logprobs:null`, `model`, `object`), with `created`/`usage` remaining in
`JsonBodyIgnoreFields` as before.
Verified by reproducing the provider's `responseDeepl2OpenAI` success
branch and comparing the marshaled output against the fixed expected body
(structural match), and with `gofmt -e` on the edited file.

Signed-off-by: 王越 <1939455790@qq.com>
@123123213weqw

Copy link
Copy Markdown
Author

Thanks for flagging the higress-wasmplugin-test (GO) failure. I tracked it down and pushed a fix.

Root cause

The deepl conformance case's expected response body was invalid JSON. The
"finish_reason":null field was misplaced: it ended up outside the single
choice object but inside the choices array:

- {"choices":[{"index":0,...,"message":{...}},"finish_reason":null],...}
+ {"choices":[{"index":0,...,"message":{...},"finish_reason":null}],...}

The e2e HTTP comparator (test/e2e/conformance/utils/http/http.go) parses the
expected body with json.Unmarshal before it even looks at the captured
response. Because the expected body failed to parse
(invalid character ':' after array element), the case failed immediately with
failed to unmarshall ExpectedResponse body ... — regardless of what the
gateway/mock returned. That is exactly what surfaced as the GO wasmplugin-test
failure.

Fix

Moved "finish_reason":null back inside the choice object so the document is
well-formed (test/e2e/conformance/tests/go-wasm-ai-proxy.go).

Verification

  • Reproduced the deepl provider's responseDeepl2OpenAI success branch and
    marshaled its chatCompletionResponse: the output is
    {"choices":[{"index":0,"message":{"name":"EN","role":"assistant","content":"..."},"finish_reason":null,"logprobs":null}],"created":...,"model":"Free","object":"chat.completion","usage":null}.
  • The corrected expected body now parses as valid JSON and is structurally
    equal to that output (after the existing JsonBodyIgnoreFields: ["created","usage"]),
    confirmed both via a focused provider test and an independent JSON parse/equality check.
  • gofmt -e reports no syntax errors on the edited file.

The provider/YAML wiring (Ingress api-free.deepl.comllm-mock-service and
the deepl provider config with targetLang: EN) is unchanged; only the
malformed expected-response literal was corrected.

@123123213weqw 123123213weqw force-pushed the e2e-deepl-ai-proxy branch 2 times, most recently from 59c8fb9 to 40ce0f0 Compare July 11, 2026 23:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement deepl LLM mock server & e2e test case

3 participants