Qwen Godot Code JSONL Evidence Matching Test Checklist

Date created: June 28, 2026

Purpose

Generate 50 Godot 3 code chunks with Qwen and verify that the yes/no judgments are correctly separated using the same method as the JSONL evidence matching test conducted yesterday.

This test is not about whether the model knows Godot knowledge. It is a test to determine whether, when SOURCE_CODE and JSONL are provided together, there is a string/API/pattern evidence in the JSONL that directly matches the actual code chunk.

In other words, relevant JSONL should be marked as yes, and irrelevant JSONL should be discarded as no.

Test Method

For each code chunk, attach at least two types of JSONL for testing. However, because there may be common syntax that is the same in Godot 3 and Godot 4, not every feature is treated as Godot‑3‑only syntax.

JSONL
Qwen direct-evidence validator prompt
SOURCE_CODE chunk

The actual verification request is to check whether Qwen returns only yes or no when JSONL + prompt + chunk are submitted together.

For common syntax, the test is based on a single code chunk. In this case, the correct JSONL contains a description or pattern of the relevant code/function, while the incorrect JSONL contains a description of a completely different function or system. Cases where the expected response is yes and cases where it is no are not distinguished by different code, but by whether there is direct evidence within the JSONL.

Repetitive Generation Loop

The overall goal is to test all 50 features. However, today we will manually verify only up to 5, and thereafter reduce the process by first analyzing the yes/no response pattern from the accumulated results.

Today's goal: 5  
Overall goal: 50/50

After filling five items, do not forcefully increase the number of new entries; first examine the common syntax/version differences and the patterns of false positives and false negatives for each table.

The test creates data by repeatedly following the flow below.

1. Give Qwen one Godot feature to test.  
2. First indicate whether the feature is common syntax for Godot 3/4, specific to Godot 3, or specific to Godot 4.  
3. If it is common syntax, create a single code chunk.  
4. If it is common syntax, create one correct JSONL for that code/function and one incorrect JSONL for a completely different function.  
5. If it is not common syntax, create separate JSONLs for Godot 4‑only and Godot 3‑only and test them.  
6. All JSONLs should be formatted according to the target table columns.  
7. Insert the JSONL + SOURCE_CODE chunk into the validation prompt and check “yes”/“no”.  
8. Record whether the expected response matches the actual response.

The core is not the quality of code generation, but whether the JSONL evidence matching judgment is stable. Therefore, the correct JSONL should yield yes, and the incorrect JSONL should yield no.

Common Syntax Guidelines

Syntax that remains the same in Godot 3 and Godot 4 should not be treated as a version‑specific test. For example, there may be code used on both sides such as _ready(), _process(delta), default variable assignments, print().

In such cases, view it as follows.

Therefore, the record table must always indicate whether the syntax is common. This allows us later to distinguish whether a common syntax was forcedly judged as migration evidence, or whether a version‑specific syntax was treated as common.

Redefinition Memo

Testing items 4 and 5 showed that it is difficult to make a definitive judgment. In particular, the response for the same code chunk changed depending on whether the JSONL was created based on Godot 3 or Godot 4, and that difference itself became a subject of pattern analysis.

Thus, for now we do not finalize “good/bad”, but first record the raw responses as shown below.

The response order is generally considered as docs_chunks related, docs_chunks unrelated, api_mapping related, api_mapping unrelated, label_prototypes related, label_prototypes unrelated.

Item 4 was previously recorded as a structured entry, but because the Godot 3 test produced a flow like no, no, yes, no, yes, no, it is excluded from definitive judgments. From now on, this document serves as an observational record focusing on “which responses fluctuate under which criteria” rather than “how many were correct”.

Generation Request Example

Below is a request example for testing the first feature, the _ready() function initialization and viewport size setting.

The JSONL format and the purpose of each JSONL are separated into a different document.

• Qwen Test JSONL Schema and Purpose

Initialize the _ready() function and provide one Godot 3 chunk and one Godot 4 chunk that match the viewport size settings.

First, indicate whether this functionality uses a common syntax for both Godot 3 and Godot 4.  
If it is a common syntax, note that the same code can be used for testing.

Also, provide a correct JSONL entry that conforms to the `docs_chunks` format, as well as an incorrect JSONL entry.

This example is based on docs_chunks. For api_mapping and label_prototypes, create correct JSONL and incorrect JSONL that match each JSONL format and purpose in the same way, and verify “yes”/“no” using the same validation prompt.

The validation input will look like this.

JSONL:
<answer or incorrect JSONL one line>

PROMPT:
<direct‑evidence validator prompt>

SOURCE_CODE:
<Godot 3 code chunk, Godot 4 code chunk, or common‑syntax code chunk>

The prompt is sent as if to judge the direct explanatory basis or transformation basis between SOURCE_CODE and JSONL as shown below.

You are a JSONL evidence matching judge.

Determine whether the JSONL below directly contains explanatory evidence or transformation evidence for the SOURCE_CODE below.

Judgment criteria:
- It is a “yes” only if one of the JSONL schemas matches exactly with an actual string or API call inside the SOURCE_CODE.
- Simply sharing broad words like Godot, Godot3, Godot4, migration, 2D, physics does not count as related.
- If the JSONL concerns a different API, node, or system, the answer is “no.”
- Do not use any Godot knowledge you have; rely solely on the string evidence written in the JSONL.
- The response must be exactly one word: either “yes” or “no.”

Judgment Matrix

Observation Checklist

After completing 50 sample tests per table, verify the following items.

Test Record Template

Copy the table below and fill in the test results.

Table‑wise Records

docs_chunks

Verify whether the official documentation explanatory evidence matches the code chunk directly.

| 21 | Screen Shake effect implementation | Version differences | Compare code chunk with explanations of camera offset, shake intensity/duration, Tween or random offset rationale | Distinguish from enemy movement, camera tracking, UI animation candidates | sample-21 checks whether it directly explains the screen shake result even though the word “camera” appears | | 22 | Particle system creation and position matching | Version differences | Compare code chunk with explanations of particle node creation, position synchronization, emission start rationale | Distinguish from generic node creation, projectile creation, animation candidates | sample-22 verifies whether the creation target is a particle and whether position matching and emission flow are included | | 23 | Scene Change function | Version differences | Compare code chunk with explanations of scene change request, SceneTree call, target scene path rationale | Distinguish from candidates like menu display, pause, resource loading that are not actual scene changes | sample-23 separates whether it actually calls a screen/scene change or merely explains simple loading or UI display | | 24 | Main menu scene loading and display | Version differences | Compare code chunk with explanations of loading the main menu scene and displaying it on screen | Distinguish from simple scene switches, pause, resource preload candidates | sample-24 checks whether the flow actually displays the menu scene or describes handling of another scene | | 25 | Game pause toggle | Common syntax | Compare code chunk with explanations of toggling pause state, get_tree().paused, pause/resume based on input | Distinguish from candidates like opening a menu, scene change, game over | sample-25 verifies whether it changes the pause state itself or just shows a UI | | 26 | Keyboard input state check (Input.is_key_pressed) | Version differences | Compare code chunk with explanations of Input.is_key_pressed and key constant/code verification | Distinguish from action‑map input, mouse input, movement handling candidates | sample-26 separates the keyboard state query itself from the overall movement logic | | 27 | Get mouse (cursor) coordinates | Common syntax | Compare code chunk with explanations of querying viewport/global mouse position | Distinguish from mouse‑click firing, touch input, cursor display candidates | sample-27 checks whether the code reads coordinates rather than handling a click event | | 28 | Random number generation function | Version differences | Compare code chunk with explanations of RNG initialization, random number generation, range specification | Distinguish from random position distribution, enemy spawn timer candidates | sample-28 separates the number‑generation function itself from gameplay logic that uses the number | | 29 | Enemy spawn at specific intervals | Version differences | Compare code chunk with explanations of periodic enemy creation based on Timer/timeout/accumulated time | Distinguish from candidates like random position calculation or enemy movement AI | sample-29 verifies whether the time interval and spawn call are directly evidenced | | 30 | Random distribution of spawned enemy positions | Common syntax | Compare code chunk with explanations of random coordinate range, spawn area, position distribution | Distinguish from timer candidates that spawn at specific times | sample-30 checks whether it explains where enemies are placed rather than when they are created | | 31 | Player animation state transition (Idle, Run, Jump) | Version differences | Compare code chunk with explanations of changing animation names based on input/speed/state | Distinguish from candidates controlling animation playback speed or timing | sample-31 separates state transitions from playback‑time control | | 32 | Animation playback time control | Common syntax | Compare code chunk with explanations of controlling animation speed, seek, advance, playback position | Distinguish from Idle/Run/Jump state‑transition candidates | sample-32 checks whether it adjusts playback time/speed rather than selecting which animation to play | | 33 | Collision shape size adjustment | Version differences | Compare code chunk with explanations of CollisionShape2D/Shape2D size properties, RectangleShape2D size/extents series | Distinguish from collision detection, damage handling, Area2D signal candidates | sample-33 verifies whether the code adjusts the shape/size of the collision area rather than detecting a collision | | 34 | Apply physics gravity | Version differences | Compare code chunk with explanations of accumulating gravity and delta into velocity and applying it via move function | Distinguish from jump force, landing detection, Area gravity setting candidates | sample-34 separates gravity accumulation flow from jump/landing judgment verification | | 35 | Apply jump force and detect landing | Version differences | Compare code chunk with explanations of is_on_floor(), jump velocity, input conditions, landing state | Distinguish from candidates that only apply gravity or handle animation state transitions | sample-35 checks whether jump input and floor check are explained within the same movement flow | | 36 | Slide handling when colliding with walls | | | | | | 37 | Player direction flip (Facing Left/Right) | | | | | | 38 | Ammo count decrement | | | | | | 39 | Reload state transition when magazine is empty | | | | | | 40 | UI button click event binding | | | | | | 41 | Settings menu open/close toggle | | | | | | 42 | Text update when language changes | | | | | | 43 | Save game JSON serialization | | | | | | 44 | Load game JSON deserialization | | | | | | 45 | File read/write error handling | | | | | | 46 | Network connection status check | | | | | | 47 | Synchronization via remote function (RPC) call | | | | | | 48 | Dynamic scene resource loading | | | | | | 49 | Memory deallocation and resource unload | | | | | | 50 | Debug log output (print or print_debug) | | | | |

api_mapping

Verify that the evidence of how names, classes, functions, and symbols change from Godot 3 to Godot 4 directly matches the code chunks.

| 22 | Particle system creation and position matching | Version differences | Verify whether the changes related to Particles2D/GPUParticles2D, emitting, position/global_position are directly present in the code | Distinguish from projectile creation or generic node creation mapping | sample-22 separates the API changes specific to particle systems from the general creation logic | | 23 | Scene change function | Version differences | Verify whether the changes for the change_scene, change_scene_to_file, change_scene_to_packed family are directly present in the current code | Distinguish from simple load()/preload() or main‑menu display mapping | sample-23 checks that the scene‑change calls from Godot 3 and their replacements in Godot 4 are applied according to the actual code version | | 24 | Main‑menu scene loading and display | Version differences | Verify whether the changes for scene loading, instancing, and menu‑scene display are required in the current code | Distinguish from generic scene changes or resource preload mapping | sample-24 ensures that the purpose of showing the main menu and the scene‑API changes do not get mixed | | 25 | Game pause toggle | Common syntax | Verify whether the changes for pause mode/process mode and tree pause are actually needed in the current code | Distinguish from menu UI or game‑over mapping | sample-25 checks that migration mapping is not overly attached to the common pause‑toggle code | | 26 | Keyboard input state check (Input.is_key_pressed) | Version differences | Verify whether the changes for key constants, scancode/keycode, and Input.is_key_pressed directly match the code | Distinguish from action‑map movement handling or mouse‑input mapping | sample-26 separates the API changes for key‑state queries from gameplay logic that uses input | | 27 | Getting mouse (cursor) coordinates | Common syntax | Verify whether the mouse‑position query API requires version‑specific change justification | Distinguish from click‑projectile creation or touch‑input mapping | sample-27 ensures that unnecessary conversion justifications are not attached to coordinate‑reading code | | 28 | Random number generation function | Version differences | Verify whether the changes for randomize, RandomNumberGenerator, and range functions are directly present in the code | Distinguish from random spawn locations or spawn‑timer mapping | sample-28 separates the changes to the random‑number API itself from the purposes for which randomness is used | | 29 | Periodic enemy spawn | Version differences | Verify whether the Timer signal, timeout connection, and yield/await family changes are directly present in the current code | Distinguish from random enemy placement or enemy‑AI movement mapping | sample-29 checks both the periodic execution structure and the spawn‑target logic together | | 30 | Random distribution of spawned enemy positions | Common syntax | Verify whether actual API changes are needed for random range, viewport size, and spawn‑area calculations | Distinguish from timer‑based periodic spawn mapping | sample-30 ensures that spawn‑period mapping is not attached to code that only performs position calculations | | 31 | Player animation state transition (Idle, Run, Jump) | Version differences | Verify whether the changes for AnimatedSprite/AnimatedSprite2D and AnimationPlayer state transitions directly match the code | Distinguish from playback speed/seek/time‑control mapping | sample-31 separates animation‑node name changes from state‑transition patterns | | 32 | Animation playback time control | Common syntax | Verify whether the changes for playback speed, seek, and current position are actually needed in the current code | Distinguish from Idle/Run/Jump state‑selection mapping | sample-32 separates the playback‑time‑control API from the state‑transition API | | 33 | Collision shape size adjustment | Version differences | Verify whether the changes for CollisionShape2D, RectangleShape2D, and shape size/extents are directly required in the current code | Distinguish from collision signal, physics layer, damage mapping | sample-33 checks that the shape‑size‑adjustment API changes, not the collision‑detection API, match the actual code | | 34 | Applying gravity | Version differences | Verify whether the changes for CharacterBody2D/KinematicBody2D, velocity, and move_and_slide family are directly required in the current code | Distinguish from jump impulse, floor check, Area2D gravity mapping | sample-34 checks whether gravity is accumulated in the physics frame and reflected in movement as intended | | 35 | Applying jump force and detecting landing | Version differences | Verify whether the changes for is_on_floor(), jump velocity, Input action, and move_and_slide directly match the code | Distinguish from gravity‑only movement, animation state, wall‑collision mapping | sample-35 checks whether the mapping requires both jump‑start conditions and landing detection | | 36 | Sliding when colliding with walls | | | | | | 37 | Player direction flip (Facing Left/Right) | | | | | | 38 | Ammo count decrement | | | | | | 39 | Reload state transition when ammo is empty | | | | | | 40 | UI button click event connection | | | | | | 41 | Settings menu open/close toggle | | | | | | 42 | Text update when language changes | | | | | | 43 | Save game JSON serialization | | | | | | 44 | Load game JSON deserialization | | | | | | 45 | File read/write error handling | | | | | | 46 | Network connection status check | | | | | | 47 | Synchronization via remote function (RPC) calls | | | | | | 48 | Dynamic loading of scene resources | | | | | | 49 | Memory deallocation and resource unload | | | | | | 50 | Debug log output (print or print_debug) | | | | |

label_prototypes

Verify that the example, which changes not only the name but also the function usage, argument composition, and call pattern entirely, directly matches the code chunk.

| 23 | Scene Change Function | Version Difference | Verify whether the call pattern that receives a scene path and switches via SceneTree is directly present in the current code | Distinguish from simple resource loading, opening menus, pause prototype | sample-23 checks if the prototype describes the input path, call target, and result of the scene change | | 24 | Main Menu Scene Loading and Display | Version Difference | Verify whether the call pattern that loads and displays the menu scene is directly present in the current code | Distinguish from simple resource loading, scene transition, pause menu prototype | sample-24 checks if the prototype explains the result “the main menu is displayed” | | 25 | Game Pause Toggle | Common Syntax | Verify whether the pattern that flips the pause state and bundles related UI/input handling is directly present in the current code | Distinguish from opening the settings menu or game‑over prototype | sample-25 checks whether changing the pause flag and showing the screen should be considered a single flow | | 26 | Keyboard Input State Check (Input.is_key_pressed) | Version Difference | Verify whether the pattern that directly polls key input state is present in the current code | Distinguish from action strength, movement vector, mouse input prototype | sample-26 records input querying separately from the actions performed with that input | | 27 | Mouse Position (Cursor) Retrieval | Common Syntax | Verify whether the pattern that reads cursor coordinates and uses them for variables or position calculations is directly present in the current code | Distinguish from mouse click events or projectile‑spawn prototype | sample-27 checks whether the mouse‑position query itself is the core | | 28 | Random Number Generation Function | Version Difference | Verify whether the pattern that calls random generation / range specification is directly present in the current code | Distinguish from spawn‑position distribution, drop tables, timer‑spawn prototype | sample-28 separates the random API call from the places where the random result is used | | 29 | Enemy Spawn at Specific Intervals | Version Difference | Verify whether the pattern that calls a spawn function at regular intervals is directly present in the current code | Distinguish from random‑position calculation or enemy‑tracking AI prototype | sample-29 checks whether the time condition, timer, and spawn call order are correct | | 30 | Random Distribution of Spawned Enemy Positions | Common Syntax | Verify whether the pattern that places generated enemies within a random range is directly present in the current code | Distinguish from a spawn loop prototype that creates enemies at fixed intervals | sample-30 separates distribution/coordinate calculation from the generation cycle | | 31 | Player Animation State Transition (Idle, Run, Jump) | Version Difference | Verify whether the pattern that changes the animation name based on state values or speed is directly present in the current code | Distinguish from playback speed, seek, frame/time‑control prototype | sample-31 checks whether the specific state transition is the key point | | 32 | Animation Playback Time Control | Common Syntax | Verify whether the pattern that controls playback speed, seek, and current playback position is directly present in the current code | Distinguish from Idle/Run/Jump state‑transition prototype | sample-32 checks whether controlling playback time, not selecting the animation, is the core | | 33 | Collision Shape Size Adjustment | Version Difference | Verify whether the pattern that reads or modifies the size value of a collision shape is directly present in the current code | Distinguish from collision‑event handling or damage‑application prototype | sample-33 records separately whether shape size change is the core or the collision result processing | | 34 | Physics Gravity Application | Version Difference | Verify whether the pattern that accumulates gravity into velocity based on delta and reflects it via movement calls is directly present in the current code | Distinguish from jump input, landing check, spawn/timer prototype | sample-34 checks whether gravity accumulation inside the physics movement loop is the key | | 35 | Jump Force Application and Landing Detection | Version Difference | Verify whether the pattern that applies jump force when on the ground and then checks for landing is directly present in the current code | Distinguish from a movement prototype that only accumulates gravity or an animation‑state‑transition prototype | sample-35 checks whether jump condition, vertical velocity change, and floor detection are bundled in one flow | | 36 | Slide Handling on Wall Collision | | | | | | 37 | Player Facing Direction Switch (Facing Left/Right) | | | | | | 38 | Ammo Count Decrease | | | | | | 39 | Reload State Transition When Ammo Is Empty | | | | | | 40 | UI Button Click Event Connection | | | | | | 41 | Settings Menu Open/Close Toggle | | | | | | 42 | Text Update on Language Change | | | | | | 43 | Save Game JSON Serialization | | | | | | 44 | Load Game JSON Deserialization | | | | | | 45 | File Read/Write Error Handling | | | | | | 46 | Network Connection Status Check | | | | | | 47 | Synchronization via Remote Procedure Call (RPC) | | | | | | 48 | Dynamic Scene Resource Loading | | | | | | 49 | Memory Deallocation and Resource Unloading | | | | | | 50 | Debug Log Output (print or print_debug) | | | | |

Additional Records on July 6, 2026

Items 6 through 10 were already incorporated into the improvement flow on July 3, 2026, and detailed observations were left for those sections. Therefore, in this document we do not leave items 6–10 blank; we have written the existing observations to match the docs_chunks, api_mapping, and label_prototypes tables respectively.

On July 6, 2026, based on http://localhost:8520/#item=sample-22, additional records were added for items 11 through 22. This section explains for each item which evidence candidate corresponds to the relevant JSONL and which candidate should fall into a different code/different transformation/different pattern.

Additional Records on July 8, 2026

On July 8, 2026, based on http://localhost:8520/#item=sample-32, additional records were added from item 23 through item 32. This section is viewed more as an observation table showing how docs_chunks, api_mapping, and label_prototypes each should explain or reject the same code chunk from their respective perspectives, rather than a definitive judgment table.

Additional Records on July 13, 2026

On July 13, 2026, based on http://localhost:8520/#item=sample-35, records were added from item 33 through item 35. This segment deals with items where collision area size, gravity, and jump/landing involve physical movement and collision shapes influencing each other, so the purpose and application location of the code need to be considered together rather than just simple string overlap.

Reverse Observation for Item 5

Item 5 was included in today’s target quantity, but when inspected in reverse as a JSONL based on Godot 4, the result did not fall cleanly into a “yes/no” pair. Therefore, we do not make a definitive judgment and will treat it as a pattern‑analysis target in the next step.

The response order is docs_chunks #1, docs_chunks #2, api_mapping #1, api_mapping #2, label_prototypes #1, label_prototypes #2.

This result is seen less as a simple model error and more as a signal that the match_terms, required_when_seen_in_code, and content fields inside the JSONL may be too broad. In particular, strings that appear commonly such as move_and_slide() and Input.is_action_pressed can serve as direct evidence, but they may be too broad to serve as evidence for maximum speed limits or version‑difference reasons.

Mismatch Case Records

False Positive

Cases where the correct answer is no but yes was returned.

False Negative

Cases where the correct answer is yes but no was returned.

Response Format Violation

Cases where the answer was not yes or no but a description, JSON, empty response, or multiple sentences.

Observation Summary

While testing, fill in the numbers and raw responses below. Here we do not make final judgments; we only note how the responses fluctuate based on criteria.