Qwen Validation Debugger Synthetic Data E2E Contract

Date: July 15, 2026

Document Purpose

Qwen Validation Debugger is a tool that tests the entire flow with synthetic data before the production Retriever and validator are completed.

For a single test subject, create Godot 3 code and Godot 4 code, and generate JSONL that matches the code and JSONL that does not match. Then, substitute the same JSONL into each version's code to verify that the final judgment comes out correctly as Yes or No.

The existing response schema and judgment types were not changed.

Grammar scope: common, separate
JSONL tables: docs_chunks, api_mapping, label_prototypes
Slots: positive, negative
Final judgment: Yes, No

No third‑party judgment values, scores, or additional reason fields were created.

Overall Flow

TEST_ITEM
  -> common / separate classification
  -> Save classification results
  -> Generate target version code
  -> Actual Godot engine parser check
  -> Qwen yes/no code check
  -> Generate JSONL only with passing code
  -> Check existing JSON Schema
  -> Direct evidence check with current code
  -> Cross-validate JSONL and code

separate item creates the Godot 3 code first. The Godot 4 code receives that Godot 3 code as REFERENCE_GODOT3_CODE and generates a corresponding code that maintains the same functionality and execution flow.

Separation of Classification and Code Generation Input

The classification response retains the original five fields.

{
  "syntax_scope": "common 또는 separate",
  "label": "짧은 한국어 라벨",
  "reason": "분류 근거",
  "godot3_focus": "Godot 3 코드 생성 포인트",
  "godot4_focus": "Godot 4 코드 생성 포인트"
}

reason, godot3_focus, godot4_focus are kept for display and logging but are not passed to the code generation prompt. This is to ensure that even if the classification stage explains incorrect version syntax, that sentence does not become factual evidence for the next code generation.

Only the following inputs are used for code generation.

Test item
common or separate
Target version for generation
separate Godot 4 generation includes the Godot 3 code that passed first
On retry, the immediately rejected code and the actual diagnostic result

Composite Code Contract

The generated code must be a single GDScript file that the engine can parse, not a string that looks like a code snippet.

The first executable line is an extends declaration.
Use a class that actually owns the test API.
Regular functions are not nested inside other functions.
Use the actual API of the test item’s behavior.
Do not simulate the target behavior with pass, comments, or local call.
Do not mix version‑specific syntax in comments or code.
Godot 4 code must retain the functionality, inputs, target, and results of the Godot 3 code that passed first.

For items whose syntax differs clearly between versions, like RPC, the official syntax contract is provided in the prompt. This is not a post‑generation word substitution logic. The generated source string is not arbitrarily modified; it is fed directly to the engine parser as‑is.

Actual Godot Engine Inspection

Qwen could have let incorrect syntax pass simply by re‑evaluating its own code. In real testing, there was a case where Godot 3 code contained Godot 4’s @rpc, yet the Qwen checker answered “yes”.

Now the actual engine is run immediately after code generation.

Godot 3: --no-window --check-only --script  
Godot 4: --headless --check-only --script  
common: Check both Godot 3 and Godot 4

The executable can be specified as GODOT3_BIN or GODOT4_BIN. The local default installation is placed in a user‑data directory outside the repository. The code to be checked is created only as a temporary file and deleted after the check. In diagnostic logs, temporary paths and the user’s home path are removed.

Code generation and verification are performed up to three times. If they fail, change the seed so the same response is not repeated, and include the previous code and the actual engine diagnostics in the next generation prompt.

Preserve JSONL Schema

The top‑level fields are the same as the three existing table definitions.

docs_chunks

chunk_id, doc_version, source_url, source_file, source_sha256,
doc_type, symbol, section_path, heading, content, code_blocks,
language_tags, godot_version_tags, api_symbols, token_count, metadata

api_mapping

mapping_id, source_api, target_api, change_type, godot_from,
godot_to, confidence, evidence_chunk_ids, match_terms, notes,
negative_patterns

label_prototypes

prototype_id, label, task_type, input_pattern, expected_finding,
recommended_action, evidence_mapping_ids, evidence_chunk_ids,
severity, validator_rules

validator_rules also uses only the existing four fields.

{
  "required_when_seen_in_code": [],
  "before_code": "",
  "after_code": "",
  "reject_when_seen_in_code": []
}

After receiving a response, check the required fields, types, enums, and whether additional fields exist using the existing JSON Schema. Reject if the model writes confidence as a number or section_path as a string, and regenerate up to three times.

Generation of positive and negative

The positive JSONL uses direct evidence from the current synthesized code.

docs_chunks: directly describe the code’s purpose, target, input flow, and result.
api_mapping: Godot 3’s source_api and Godot 4’s target_api must each be present in the actual code.
label_prototypes: the before evidence must be in the Godot 3 code, and the after evidence must be in the Godot 4 code.
The negative_patterns and reject_when_seen_in_code of the positive are empty arrays.

When generating negative JSONL, do not provide the current code or the current item. Dynamically select the sample item following the current item as NEGATIVE_TEST_ITEM, and create the JSONL using only that other item. After generation, the server directly compares with the current code and rejects if the source/before evidence overlaps.

This process does not perform one‑to‑one substitution of specific API names or sentences. It only uses sample order selection and string comparison of the actual generation result.

Verification order

Verification is processed in the following order.

If the result is finalized by the version‑slot structure, judge “No” without calling the model.
If there is not a single piece of direct evidence before migration in the current code, judge “No”.
Only when the above two conditions are not decisive does Qwen compare meaning and usage.

The deterministic gate does not create a new state. The return value is, as before, “Yes” or “No”.

JSONL	Verification target	Expected response
same‑version `docs_chunks` positive	generation reference code	Yes
different‑version `docs_chunks` positive	cross‑version code	No
`docs_chunks` negative	any code	No
`3 -> 4` migration positive	Godot 3 code	Yes
`3 -> 4` migration positive	Godot 4 code	No
`3 -> 4` migration negative	any code	No

API request reliability

Disable thinking output.
Classification, code, and JSONL request responses that contain only the existing fields in the JSON Schema.
The final judgment allows only “Yes” or “No” in grammar.
Change the seed for code and JSONL regeneration.
HTTP 429, 502, 503, 504, connection termination, and request timeout are retried up to three times.

The API key is included in the request only from the current input value in the browser memory. It is not recorded in repositories, documents, logs, or commits.

Test results on July 15, 2026

Unit contract tests passed 27 out of 27.

In the actual Godot engine inspection, the following was observed.

Godot 3’s remote func code passed the Godot 3 parser.
Adding @rpc to Godot 3 code was rejected with “Unexpected '@'”.
Godot 4’s @rpc code passed the Godot 4 parser.

The actual RunPod Qwen test progress was as follows.

Initial full E2E: 5/12 passed. Incorrect Godot 3 synthesized code and overly broad relevance judgments were identified.
After applying engine inspection and generation contracts: 10/12 passed.
After applying dynamic negative items and direct string gate, migration negative: 4/4 passed.
The remaining 2 cases were instances where the model ignored the slot contract that prevented re‑applying migration positive to Godot 4. Those conditions were moved to a deterministic version gate and the unit tests passed.
A final full re‑test was started, but the RunPod service returned HTTP 502 for /v1/models and generation requests, causing a halt. This was a remote service non‑response, not a code or JSONL rejection.

When the remote service responds again, the same 12‑item E2E must be rerun from the beginning.