2026-07-28 Source Flow Debugger Actual Source Search Report
Purpose
The previous record only stated that the actual Godot project was split into 134 chunks for searching and that related documents were included in the top results. From that description alone, it was impossible to know what each chunk returned, whether the returned JSONL actually existed in SQLite, or whether more direct documentation existed in SQLite but was pushed down in the ranking.
This report separates and records the following three questions.
- What JSONL candidates did the Source Flow Debugger return for each chunk?
- Are the returned candidates exactly the same as the verified original JSONL in the committed SQLite?
- Did the directly verified evidence from the entire SQLite appear at the top, or is ranking improvement needed?
The Qwen endpoint was unavailable, so this run only performed search and SQLite comparison. The Qwen Godot version classification and candidate approval/rejection results were not recorded, and the mere appearance of a search candidate is not treated as a final verification success.
Fixed Input
- Repository:
godotengine/godot-demo-projects - License: MIT
- Commit:
cae8dc567a56d3e7936f171bcb85f0ccb9634ad0 - Project:
2d/hexagonal_map - Input files:
project.godot,map.tscn,tileset.tres,tileset_edit.tscn,troll.gd,troll.tscn - Number of input files: 6
- Number of generated chunks: 134
- AST chunks: 3
- Direct chunks: 131
The SQLite used for the run isoutputs/godot-rag-sqlite/godot-rag.sqlite3, and at execution time it contained 2,265 docs_chunks, 1,133 api_mapping, 2 label_prototypes, and 3,399 embeddings.
Execution Status
| Stage | Status | Meaning of This Run |
|---|---|---|
| SQLite FTS5 + Okapi BM25 | Run | All 134 chunks were searched |
| embedding | Not run | Query embedding API key not set |
| reranker | Not run | Provider API key not set |
| Qwen validator | Not run | Qwen endpoint was down |
Therefore, the ranking below is not the final ranking of the full F strategy, but the result of the lexical/BM25 stage.
Overall Comparison Results
| Item | Result |
|---|---|
| Chunks | 134 |
| BM25 candidates returned per chunk | 80 |
| Returned candidates compared with SQLite | 10,720 |
| Candidates after deduplication | 844 |
| Candidates not found in SQLite | 0 |
Candidates whose provenance is not verified |
0 |
Candidates where returned raw_json differs from SQLite original |
0 |
docs_chunks returned |
10,713 |
api_mapping returned |
7 |
label_prototypes returned |
0 |
The consistency of the returned data was correct. When the tables and IDs shown by the Source Flow Debugger were looked up again in SQLite, all 10,720 entries existed, their provenance was verified, and the returned records matched the stored raw_json.
However, this does not mean that search quality was perfect. Because all 134 chunks filled the candidate ceiling of 80, lexical candidates were not narrowed enough. Additionally, file‑name anchor protection caused nine chunks to promote a document with a lower BM25 score to rank 1.
Representative Chunk Results
Well‑Searched Cases
| Chunk | Input Summary | Returned Result | SQLite Comparison |
|---|---|---|---|
| 1 | Description of the project.godot file format |
Manually editing project.godot rank 1 |
Same ID and original verified |
| 13 | TileMapLayer, tile_map_data, TileSet |
TileMapLayer property rank 1, method rank 2, TileData rank 3 |
All verified docs_chunks |
| 42 | TileSetAtlasSource, texture region |
TileSetAtlasSource rank 1, TileSetSource rank 2 |
Direct class evidence confirmed |
| 125 | _physics_process, Input.get_axis, move_and_slide |
Physics process rank 1, 2D movement docs rank 5‑7, CharacterBody2D rank 8 |
Movement/physics docs present together |
| 130 | CharacterBody2D node |
CharacterBody2D class rank 1 |
Direct class evidence confirmed |
The rank‑1 document for chunk 1 was not top‑ranked solely by BM25 score; it remained top because the project.godot lexical anchor was protected. In this case, the file‑format description matched directly, so the protection was valid.
Cases Requiring Rank Improvement
Chunk 123: extends CharacterBody2D
- Actual #1:
Collision exceptions CharacterBody2Dclass documentation: rank 17KinematicBody2D -> CharacterBody2Dmapping: rank 6
SQLite contained the exact CharacterBody2D class documentation, but it was placed after the broader collision·movement documentation. This is a ranking issue, not a data‑absence issue. In short declaration chunks, a perfectly matching class name and structural fields need to be treated more strongly than general context tokens.
Chunk 124: tan(deg_to_rad(30))
- Actual #1:
Setting up the spring arm and camera Evaluating expressions: ranks 2 and 3
The #1 document also has an example using deg_to_rad, so it is not completely unrelated, but the 3D tutorial outranked the document that explains the call itself. Re‑ranking is needed to separate direct API explanations from simple usage examples.
Structure Candidates Requiring Qwen Re‑validation
Chunk 13: map.tscn containing PackedByteArray
The PackedByteArray -> byte[] mapping appeared at rank 48. The stored JSONL describes a C# collection migration, but the input is a GDScript scene resource. Tokens match, yet the language and applied context differ, so the final approval must not occur at the retrieval stage.
Chunk 130: Node Already Using CharacterBody2D
The KinematicBody2D -> CharacterBody2D mapping appeared at rank 3. This mapping is relevant for version detection, but the current chunk does not contain the older API KinematicBody2D. It is not a candidate for migration warnings; Qwen should view the JSONL and the original chunk together and classify it as “Godot 4” evidence or unnecessary application.
The original SQLite JSONL for both cases was preserved unchanged in the machine‑readable report’s reviewCases.
Why label_prototypes Has 0 Entries
SQLite’s prototypes consist of only the following two entries:
enum PlaneSemanticLabelclass OpenXRSpatialComponentPlaneSemanticLabelList
The hexagonal_map chunk contained neither input pattern. Therefore, the fact that label_prototypes is 0 in this run is judged to be the result of the direct‑structure filter operating, not a missing entry.
Current Conclusions
- Data consistency between returned candidates and the SQLite source has been verified.
- Representative project settings, TileSet, TileMapLayer, and movement‑function searches included the relevant documents in the top ranks.
- Not all chunks reached the candidate upper limit of 80, so further reduction of candidates is insufficient.
- In short class declarations and single‑function calls, more direct documentation was found to be pushed down the rankings.
- Structure mappings can enter the candidate list based solely on related tokens, so Qwen cannot decide approval while it is offline.
Criteria for the Next Re‑run
When the Qwen endpoint is ready again, use the same commit and the same 134 chunks to add the following:
- Classification of each structure candidate as
Godot 3,Godot 4,Not related, orInsufficient evidence - Decision on rejecting candidates with language‑context mismatches such as
PackedByteArray -> byte[] - Whether to reject migration‑application candidates in chunks that already use the target API
- Decision on rejecting unrelated negative‑control JSONL entries
- Direct‑document rank changes before and after applying embeddings and rerankers