Qwen Validation Debugger Strategy and Implementation Record

Date created: June 30, 2026

Purpose

This document records the reason for creating tools/qwen-validation-debugger and its actual operation flow.

On June 28, I manually created Godot code chunks and JSONL with Qwen, and directly concatenated prompt + code chunk + JSONL to see how yes/no turned out. This method was good for getting a direction, but as the number of items increased, the cost of copying by hand, remembering the generation criteria, and comparing results became too high.

Therefore, on June 30 I turned this process into a separate web tool. The goal of this tool is not final model training, but to visually trace the relationship between the test code generated by Qwen and the JSONL, and to quickly discover incorrect yes/no judgments.

Current Screen

The current tool runs at http://localhost:8520/.

The screen is divided into three main areas.

• Left: checklist of 50 Godot test items
• Center/upper right: AI label and Godot code generation result for the selected item
• Bottom: docs_chunks, api_mapping, label_prototypes JSONL generation/validation slots

Why a Separate Tool Was Needed

Originally I requested the following directly from the Qwen chatbot.

Create a Godot code chunk,  
and generate a correct JSONL and an incorrect JSONL that conform to the `docs_chunks/api_mapping/label_prototypes` format.  
We will then check that when the `SOURCE_CODE` and JSONL are combined, the yes/no result is correct.

However, when doing manual testing, there were more axes to verify than expected.

If it is common syntax, there are 6 slots.

Common Code  
  docs_chunks       Description basis / Other code description  
  api_mapping       Common/unnecessary / Irrelevant conversion  
  label_prototypes  Common/unnecessary / Irrelevant conversion

If it is version-separated syntax, there are 12 slots.

Godot 3 Code  
  docs_chunks       Explanation basis / other code description  
  api_mapping       Conversion needed / irrelevant conversion  
  label_prototypes  Conversion needed / irrelevant conversion  

Godot 4 Code  
  docs_chunks       Explanation basis / other code description  
  api_mapping       Already applied / irrelevant conversion  
  label_prototypes  Already applied / irrelevant conversion

If I keep making this by hand, more time is spent on copying/pasting and organizing results than on the test itself. Therefore, I bundled items, code, JSONL, and verification results on a single screen with a GUI.

Overall Flow

flowchart TD
  A["50 test items"] --> B["Select item"]
  B --> C["Qwen labeling"]
  C --> D{"Is it common syntax?"}
  D -->|"common"| E["Generate common code"]
  D -->|"separate"| F["Generate Godot 3 code"]
  D -->|"separate"| G["Generate Godot 4 code"]
  E --> H["6 JSONL slots"]
  F --> I["6 JSONL for Godot 3"]
  G --> J["6 JSONL for Godot 4"]
  I --> K["Validate Godot 3 code"]
  I --> L["Validate Godot 4 code"]
  J --> K
  J --> L
  H --> M["Validate common code"]

The tool calls Qwen in three roles.

Labeling Stage

When an item is clicked, Qwen first decides whether it is common syntax or version‑specific.

For example, _process(delta) is used as a frame‑update function in both Godot 3 and Godot 4, so it can be considered common syntax.

Conversely, apply player movement speed and acceleration may have different physics bodies and move_and_slide() call patterns across versions, so separate Godot 3 and Godot 4 code slots are created.

JSONL Slot Design

Initially, slots were divided simply into correct JSONL and incorrect JSONL. That criterion proved too coarse.

docs_chunks are explanatory evidence. Therefore, if the description matches the code directly, the answer is Yes; if the document explains a different code behavior, the answer is No.

However, api_mapping and label_prototypes are not just explanations; they are conversion evidence when moving from Godot 3 to Godot 4. Thus, if a code already written for Godot 4 has api_mapping or label_prototypes marked Yes, it can read as “you’re already on Godot 4, why migrate again?”.

So the slot names were changed as follows.

Thanks to this criterion, Yes/No is no longer a simple relevance judgment but a decision aligned with the purpose of each table.

Separation of Generation‑Basis Code and Verification‑Target Code

The most important change today is at this point.

In the initial implementation, a slot was verified only against its own code.

Godot 3 code generated JSONL  
  -> Validate only in Godot 3 code  

Godot 4 code generated JSONL  
  -> Validate only in Godot 4 code

But the actual verification needed is broader than this.

When a JSONL created with Godot 3 code is attached to Godot 4 code, it should still return No. Conversely, we need to see what result occurs when a JSONL created with Godot 4 code is attached to Godot 3 code. That way we can detect the issue where mixed source/target strings incorrectly produce Yes.

So the single button was split into two.

Now each slot in the version‑separation section can perform all of the following checks.

Godot 3 code generated JSONL
  -> Verify Godot 3 code
  -> Verify Godot 4 code

Godot 4 code generated JSONL
  -> Verify Godot 3 code
  -> Verify Godot 4 code

Inside the tool, it is separated as follows.

Expected Response Calculation Criteria

The verification result is still received from Qwen as “yes” or “no”. However, the expected response is now calculated by looking at the table + JSONL generation reference version + verification target version + slot nature, rather than just a single slot.

This criterion is more of a debugging guideline to see where Qwen’s source matching wavers at this stage, rather than a final evaluation standard.

Changes to the Verification Prompt

The verification prompt now includes two pieces of version information.

JSONL_GENERATED_FROM_CODE_VERSION:
godot3

VERIFY_TARGET_CODE_VERSION:
godot4

The reason for inserting it this way is to prevent Qwen from simply answering “yes” just because the target_api in the JSONL matches the current code.

In api_mapping and label_prototypes, the following criteria are important.

• source_api, source_pattern, input_pattern, before_code, required_when_seen_in_code must be directly present in the current code for a yes.
• If only the target_api, after_code, or the recommended target form matches the current code, it is already applied, so no.
• When it indicates that no transformation is needed, such as signature_stable, no_change, common, the answer is no.
• Even if migration JSONL is attached to Godot 4 code or common‑syntax code, if no additional transformation is required, the answer is no.

Record of 50‑Item Test

1/50 Initialization of _ready() Function and Viewport Size Setting

On July 3, the first of the 50 test items was verified. This item was labeled as common code, and the reference code was as follows.

func _ready():
    var vp_size = get_viewport().get_visible_rect().size
    position = Vector2(vp_size.x / 2, vp_size.y / 2)

Initially, the incorrect slot of docs_chunks was called “Irrelevant Description JSONL.” However, after actual testing, this name proved inaccurate. This slot is not meant to create deliberately odd or incorrect JSONL, but rather to insert other official documentation descriptions that the Retriever might fetch as search candidates, to see whether Qwen filters them out during final verification.

Therefore, the meaning of this slot was clarified as “Other Code Description JSONL.”

In this test, the problematic JSONL described Viewport.get_visible_rect and _ready(), but the actual content was about other code behaviors such as safe area, UI scaling, $Label, and $Button.rect_position. Since the SOURCE_CODE reads the viewport size and moves the current node’s position to the center of the screen, overlapping APIs should not be considered direct evidence.

The expected answer was “No,” but Qwen returned “Yes.” The cause was that the verification prompt interpreted “exact match with the actual string/API call” too broadly, accepting documentation that merely contains the same API as direct evidence.

Changes:

• Renamed the docs_chunks incorrect slot from “Irrelevant Description JSONL” to “Other Code Description JSONL.”
• Defined this slot in the JSONL generation prompt as “an official‑documentation‑style JSONL that may share some APIs/topics with the search candidate but actually describes a different code behavior.”
• Strengthened the verification prompt so that the Yes criterion for docs_chunks is not simple API string matching but whether the core purpose, target object, and final action/result of the code directly match.
• Stated that even if the same API or method appears, if the JSONL describes a different purpose, target object, or result, it is only a search candidate and not direct evidence, thus the answer should be “No.”
• Adjusted the front‑UI slot label and preview text to follow the same criteria.
• Added a test in core.test.js to ensure this criterion is preserved in the prompts.

2/50 _process(delta) Frame‑by‑Frame Update Loop

On July 3, the second item was verified. This item was labeled by Qwen as common code.

Labeling summary:

syntax_scope: common
label: _process(delta) frame-by-frame update
reason: In both Godot 3 and 4, the signature and calling method of the _process(delta) function are the same, and there is no syntactic difference when writing frame‑based logic.

The verification results matched the expected responses for all 6 slots.

This entry records a case where, in the common grammar, docs_chunks yields yes as an explanatory basis, while api_mapping and label_prototypes result in no due to common/unnecessary or irrelevant transformations, and the flow operates correctly.

3/50 Player Movement Input Handling

Item 3 was labeled by Qwen as version separation code.

Labeling summary:

syntax_scope: separate
label: Player Movement Input Handling
reason: Although the input checking itself is similar, due to the introduction of Input.get_vector() in Godot 4 and the signature change of CharacterBody2D's move_and_slide(), the code structure needs to be separated.
godot3_focus: Inherits KinematicBody2D, manually calculates velocity.x/y and calls move_and_slide(up_direction)
godot4_focus: Inherits CharacterBody2D, simplifies diagonal movement with Input.get_vector() and applies removal of parameters in move_and_slide()

Verification Result:

Mismatches:

Prompt Observation:

In hybrid search, it is natural for a code chunk and a similar JSONL candidate to appear together. Therefore, the verification prompt must judge whether the search candidate is a basis for actually applying it to the current SOURCE_CODE. The three mismatches above stem from the question not being fixed enough, causing Qwen to interpret api_mapping and label_prototypes as broad relevance matches rather than migration‑necessity checks.

Specifically, issue 3 revealed the following problems:

• If a function name like move_and_slide, which can appear in both Godot 3 and Godot 4, matches alone, Qwen may lean toward Yes.
• When a Godot 3‑based JSONL is attached to Godot 4 code, if the code is already post‑conversion, label_prototypes · conversion needed must not return Yes.
• When a Godot 4‑based irrelevant conversion JSONL is attached to Godot 3 code, it must not return Yes if the conversion is not actually applicable to the current code.
• The key is not the JSONL generation method but the verification question. It must determine whether the current code is pre‑conversion, post‑conversion, or an entirely different transformation.

Thus, the prompt criteria were revised as follows:

• Change the verification question from “Is the JSONL related to the code?” to “Should we apply a migration to the current SOURCE_CODE based on this JSONL?”.
• If the current SOURCE_CODE is already in Godot 4 target form, even if the JSONL appears related to migration, mark it as “No” because it is “already applied”.
• label_prototypes · conversion needed is marked as “Yes” only when the current SOURCE_CODE actually contains the pre‑conversion usage pattern, argument composition, and call pattern.
• Irrelevant conversion JSONL is marked as “No” even when the verification target is Godot 3 code, if the conversion does not apply to the current SOURCE_CODE.
• Each field of the JSONL provides contextual judgment. In the verification stage, this context is used to decide whether the current SOURCE_CODE is in a pre‑conversion state or already post‑conversion.
• Because source_api and target_api may share the same name or match_terms may contain both source and target terms, a single function name or a broad match_terms match alone is not sufficient for a “Yes”.
• Even if the function name matches, the source‑side pre‑conversion pattern described by the JSONL (e.g., argument list, return handling, inheritance node, surrounding code structure) must be present in the SOURCE_CODE for a “Yes”.
• If JSONL_GENERATED_FROM_CODE_VERSION and VERIFY_TARGET_CODE_VERSION differ, first determine whether the current SOURCE_CODE is the pre‑conversion code or already the post‑conversion code.

4/50 Player Movement Speed and Acceleration Application

Item 4 was also labeled by Qwen as version‑separated code.

Labeling Summary:

**Label:** Apply Player Movement Speed and Acceleration  
**Reason:** In Godot 4, the physics node changed from `KinematicBody` to `CharacterBody`, and the `move_and_slide()` signature was altered to use argument passing, which changes the code structure.  
**Godot 3 Focus:** Assign speed to the `velocity` property, then call `move_and_slide()`, using `KinematicBody2D/3D`.  
**Godot 4 Focus:** Pass the argument with `move_and_slide(velocity)`, use `CharacterBody2D/3D`, and apply acceleration using `lerp()` or `move_toward()`.

Verification Result:

Mismatches:

Prompt Observation:

The four mismatches were fewer than the three, but the cause is the same. When the source and target function names in api_mapping are the same or similar, Qwen may judge a broad relevance—“this JSONL describes the function change”—and give a Yes. However, what this tool needs is not a relevance judgment but whether migration is required for the current code.

Therefore, based on observations up to case 4, the verification prompt was strengthened to consider more than just matching API names:

• Whether the current code actually contains the pre‑change pattern described in the JSONL
• Even with the same function name, whether the call arguments and return‑value handling match the pre‑change code
• If the verification target code is already in the post‑change form, whether migration is No
• Whether unrelated conversion JSONLs are not promoted to Yes merely due to simple relevance

Maximum Speed Limit for Player 5/50

Item 5 was labeled by Qwen as version‑separation code.

Labeling Summary:

**syntax_scope:** separate  
**label:** Player maximum speed limit  
**reason:** Because of the class name change between Godot 3's `KinematicBody2D` and Godot 4's `CharacterBody2D`, and the differences in the `move_and_slide()` method signature and how velocity is passed, separate code is required.  
**godot3_focus:** Inherit from `KinematicBody2D`, assign speed to `self.velocity` and then call `move_and_slide()`, limit speed with `limit_length()`.  
**godot4_focus:** Inherit from `CharacterBody2D`, assign speed to the `velocity` variable and explicitly pass it to `move_and_slide(velocity)`, apply `limit_length()`.

Generated code flow:

Verification results:

Mismatches:

Prompt observation:

Number 5 is close to a situation that can actually occur in hybrid search. Later, when searching the DB, not only completely unrelated JSONL appears, but also similar candidates from the same API or the same movement/speed handling family can appear together. At this point, the verification prompt should not ask “Is this a similar search candidate?” but rather “Does this JSONL directly describe the actual behavior of the current SOURCE_CODE?”

In this number 5, api_mapping and label_prototypes worked as expected in both Godot 3/4 validations. In particular, the fact that JSONL already applied to Godot 4 code or of the no_change, pattern_validation type fell to “No” can be seen as a signal that the migration judgment issue observed in previous numbers 3/4 has been somewhat mitigated.

Conversely, docs_chunks still needs further examination. Since docs_chunks is about matching explanation evidence rather than migration applicability, if it becomes “Yes” merely because of the same API name or version clue, noise remains in the hybrid search candidate verification stage. Therefore, future verification prompts should more strongly confirm that the purpose, target object, and result state of the SOURCE_CODE match the description in the JSONL for docs_chunks.

6~10/50 Additional Verification Records

From 6 to 10, we first record the results and then fix the verification prompts. In this stage, we only note which slots the current tool gets right or wrong without modifying the prompts.

Overall Summary:

Item‑by‑Item Summary:

Numbers 6, 9, and 10 all matched expectations under the current criteria. For 6, the Godot 3 flow (PackedScene.instance(), Input.is_mouse_button_pressed(1), get_viewport().get_mouse_position()) and the Godot 4 flow (instantiate(), InputEventMouseButton, get_global_mouse_position()) were separated, and the correct/incorrect candidates were each well distinguished. For 9, the version‑split candidates for enemy movement and rotation were also judged as expected. Number 10 was classified as common syntax, and the handling of off‑screen nodes based on get_viewport_rect().size, global_position, hide() was documented as “yes” in the docs, while the migration candidates of type api_mapping/label_prototypes were marked “no”.

Mismatches:

Observations:

Looking at 6–10, there is an overall improvement signal. Out of 102 verifications, 97 matched expectations, and 6, 9, 10 all passed. Especially for the common syntax case (10), both api_mapping and label_prototypes were marked “no”, indicating that the policy “do not attach migration evidence to common code” is working to some extent.

The remaining mismatches fall into two categories. One is like case 7, where migration evidence based on Godot 3 is also attached as “yes” to Godot 4 code. In this case we need to ask more clearly whether the current code is already in the post‑migration form or still pre‑migration. The other is like case 8, where docs_chunks candidates appear as “yes” because they share the same topic or similar API. Here we must better discriminate whether the explanatory evidence directly describes the actual behavior of the SOURCE_CODE or is merely a similar search candidate.

The key point is that this problem is not solved by blocking specific strings or candidates. When the JSONL is later inserted into the DB and retrieved via hybrid search, it is normal for similar but different candidates to appear together. Therefore, subsequent fixes should not artificially filter candidates, but rather make the verification prompt judge the relationship between the current SOURCE_CODE and the retrieved candidates more accurately.

Prompt enhancements applied after recording:

• For docs_chunks, when the JSONL generation version differs from the verification target version, we first check whether the JSONL explains the behavior of the current target‑version code.
• Even if versions differ, a candidate that explicitly describes a common API/common behavior may be “yes”, but a candidate that explains version‑specific code structure or node/API differences must be separately verified against the current SOURCE_CODE.
• For api_mapping and label_prototypes, we first determine whether the current SOURCE_CODE is pre‑migration, post‑migration, unrelated, or a code that does not require conversion.
• “Yes” is allowed only when the current SOURCE_CODE is pre‑migration and the JSONL‑described conversion should be applied now; the wording has been clarified accordingly.
• This enhancement is not about handling specific strings, but about strengthening the procedure that judges candidates retrieved by hybrid search against the current code.

Verification Result Check

Verification results are displayed separately for each slot by version.

With this approach, you can see the following directly within a single slot.

Godot 3 Code verification before / Response / Expectation / Match status  
Godot 4 Code verification before / Response / Expectation / Match status

Previously, only “whether this JSONL is correct/incorrect” was visible. Now you can see “whether it is correct/incorrect when attached to a specific code”.

Problems Solved by the Current Tool

Changes in Development Productivity

Before creating the tool, each test required the following steps to be done manually.

1. Explain test items to Qwen  
2. Request Godot 3 code  
3. Request Godot 4 code  
4. Request docs_chunks correct/incorrect JSONL  
5. Request api_mapping correct/incorrect JSONL  
6. Request label_prototypes correct/incorrect JSONL  
7. Attach each JSONL to SOURCE_CODE again for verification  
8. Manually record yes/no results  
9. Compare mentally whether the expected values match

Now the same flow is fixed within the screen.

1. Click the item  
2. Generate AI label  
3. Generate common or Godot 3/4 code  
4. Generate JSONL per slot  
5. Verify Godot 3 code / Verify Godot 4 code  
6. Check results and prompt/response

Especially the part where productivity improved is closer to “not getting confused” rather than “making a lot quickly.” In this work, it is more important that the judgment criteria do not wobble than the sheer amount generated. With the tool, the following became easier.

• Fix whether it is a common grammar or a version split first
• Continuously check the purpose of each table’s JSONL on the screen
• Separate the expected generation criteria from the actual verification results
• Cross‑validate the same JSONL in Godot 3/4 code
• Immediately view the prompt and raw response of a failed slot
• Reduce the state a person has to remember when repeating 50 items

Still Open Issues

This tool is not yet a final evaluator. It is currently a debugging tool.

The open issues are as follows.

• The explanation for docs_chunks can sometimes be accepted as a common description even when the versions differ.
• In api_mapping, if both source and target strings appear in the JSONL, Qwen could see only the target string and answer “yes.”
• For label_prototypes, distinguishing before/after patterns may be more important than simple string matching.
• The current expected response calculation is a debugging metric, so the actual score/F1 calculation criteria need to be refined later.
• It is difficult to separate the search‑candidate selection problem and the verification‑prompt judgment problem just by looking at the results.

Current Conclusion

On June 28 the feeling was “let’s do the 50 tests manually.” When we actually tried it, there were far too many axes to test, and mixing common grammar with version splits, generation‑criteria versions, and verification‑target versions quickly became confusing.

By June 30, after building the tool, this problem was largely resolved. Now we can see not only whether Qwen’s generated JSONL is correct/incorrect, but also which code generated it and which code verified it.

This tool can be used in the same way when the actual search results of docs_chunks, api_mapping, and label_prototypes are attached later. In other words, it became a foundational tool that can extend from manual demo data to verification of DB search results.