idea_world_labDEV JOURNAL
Tuesday, July 28, 2026

2026-07-28 Source Flow Debugger Actual Source Search Report

Purpose

The previous record only stated that the actual Godot project was split into 134 chunks for searching and that related documents were included in the top results. From that description alone, it was impossible to know what each chunk returned, whether the returned JSONL actually existed in SQLite, or whether more direct documentation existed in SQLite but was pushed down in the ranking.

This report separates and records the following three questions.

  1. What JSONL candidates did the Source Flow Debugger return for each chunk?
  2. Are the returned candidates exactly the same as the verified original JSONL in the committed SQLite?
  3. Did the directly verified evidence from the entire SQLite appear at the top, or is ranking improvement needed?

The Qwen endpoint was unavailable, so this run only performed search and SQLite comparison. The Qwen Godot version classification and candidate approval/rejection results were not recorded, and the mere appearance of a search candidate is not treated as a final verification success.

Fixed Input

  • Repository:
    godotengine/godot-demo-projects
  • License: MIT
  • Commit: cae8dc567a56d3e7936f171bcb85f0ccb9634ad0
  • Project: 2d/hexagonal_map
  • Input files: project.godot, map.tscn, tileset.tres,
    tileset_edit.tscn, troll.gd, troll.tscn
  • Number of input files: 6
  • Number of generated chunks: 134
  • AST chunks: 3
  • Direct chunks: 131

The SQLite used for the run is
outputs/godot-rag-sqlite/godot-rag.sqlite3, and at execution time it contained 2,265 docs_chunks, 1,133 api_mapping, 2 label_prototypes, and 3,399 embeddings.

Execution Status

Stage Status Meaning of This Run
SQLite FTS5 + Okapi BM25 Run All 134 chunks were searched
embedding Not run Query embedding API key not set
reranker Not run Provider API key not set
Qwen validator Not run Qwen endpoint was down

Therefore, the ranking below is not the final ranking of the full F strategy, but the result of the lexical/BM25 stage.

Overall Comparison Results

Item Result
Chunks 134
BM25 candidates returned per chunk 80
Returned candidates compared with SQLite 10,720
Candidates after deduplication 844
Candidates not found in SQLite 0
Candidates whose provenance is not verified 0
Candidates where returned raw_json differs from SQLite original 0
docs_chunks returned 10,713
api_mapping returned 7
label_prototypes returned 0

The consistency of the returned data was correct. When the tables and IDs shown by the Source Flow Debugger were looked up again in SQLite, all 10,720 entries existed, their provenance was verified, and the returned records matched the stored raw_json.

However, this does not mean that search quality was perfect. Because all 134 chunks filled the candidate ceiling of 80, lexical candidates were not narrowed enough. Additionally, file‑name anchor protection caused nine chunks to promote a document with a lower BM25 score to rank 1.

Representative Chunk Results

Well‑Searched Cases

Chunk Input Summary Returned Result SQLite Comparison
1 Description of the project.godot file format Manually editing project.godot rank 1 Same ID and original verified
13 TileMapLayer, tile_map_data, TileSet TileMapLayer property rank 1, method rank 2, TileData rank 3 All verified docs_chunks
42 TileSetAtlasSource, texture region TileSetAtlasSource rank 1, TileSetSource rank 2 Direct class evidence confirmed
125 _physics_process, Input.get_axis, move_and_slide Physics process rank 1, 2D movement docs rank 5‑7, CharacterBody2D rank 8 Movement/physics docs present together
130 CharacterBody2D node CharacterBody2D class rank 1 Direct class evidence confirmed

The rank‑1 document for chunk 1 was not top‑ranked solely by BM25 score; it remained top because the project.godot lexical anchor was protected. In this case, the file‑format description matched directly, so the protection was valid.

Cases Requiring Rank Improvement

Chunk 123: extends CharacterBody2D

  • Actual rank 1: Collision exceptions
  • CharacterBody2D class documentation: rank 17
  • KinematicBody2D -> CharacterBody2D mapping: rank 6

SQLite contained the exact CharacterBody2D class documentation, but it was placed after the broader collision‑movement documentation. This is a ranking issue, not a data‑absence issue. In short declaration chunks, a perfectly matching class name and structural fields need to be treated more strongly than generic contextual tokens.

Chunk 124: tan(deg_to_rad(30))

  • Actual rank 1: Setting up the spring arm and camera
  • Evaluating expressions: ranks 2 and 3

The rank‑1 document also includes a deg_to_rad usage example, so it is not completely unrelated, but a 3D tutorial outranked a document that actually explains the call itself. Re‑ranking to separate direct API explanations from simple usage examples is needed.

Structure Candidates Requiring Qwen Re‑validation

Chunk 13: map.tscn containing PackedByteArray

The PackedByteArray -> byte[] mapping appeared at rank 48. The stored JSONL describes a C# collection migration, but the input is a GDScript scene resource. The tokens match, yet the language and applied context differ, so the candidate should not be approved at the retrieval stage.

Chunk 130: Node Already Using CharacterBody2D

The KinematicBody2D -> CharacterBody2D mapping appeared at rank 3. This mapping is relevant for version detection, but the current chunk does not contain the older API KinematicBody2D. It is not a migration‑warning candidate; Qwen should view the JSONL and the original chunk together and classify it as “Godot 4 evidence” or “not applicable”.

The original SQLite JSONL for both cases was kept unchanged in the machine‑readable report’s reviewCases.

Why label_prototypes Has Zero Entries

SQLite’s prototypes consist of only two entries:

  • enum PlaneSemanticLabel
  • class OpenXRSpatialComponentPlaneSemanticLabelList

The hexagonal_map chunk contained neither of these input patterns. Therefore, the zero count for label_prototypes in this run is not a omission but the result of the direct‑structure filter operating as intended.

Current Conclusions

  • Data consistency between return candidates and the SQLite originals has been verified.
  • Representative project settings, TileSet, TileMapLayer, and movement‑function searches included the relevant documents in the top ranks.
  • Not all chunks reached the upper limit of 80 candidates, so candidate reduction is insufficient.
  • Short class declarations and single‑function calls show cases where more direct documentation is pushed down.
  • Structure mappings can enter the candidate list based solely on matching tokens, so Qwen cannot decide approval while offline.

Criteria for the Next Re‑run

When the Qwen endpoint is ready again, use the same commit and the same 134 chunks, adding the following items:

  1. Classify each structural candidate as Godot 3, Godot 4, Not related, or Insufficient evidence.
  2. Decide whether to reject language‑context‑mismatch candidates such as PackedByteArray -> byte[].
  3. Reject migration‑application candidates in chunks that already use the target API.
  4. Reject unrelated negative‑control JSONL entries.
  5. Observe direct‑document rank changes before and after applying embedding and reranker.

2026‑08‑01 F Strategy and Qwen Re‑validation

The same external‑repo commit and the six original files of 2d/hexagonal_map were re‑inserted into the Source Flow Debugger GUI. Based on the latest chunking, a total of 146 chunks were generated. In the full‑search audit that ran BM25, embedding, union, and reranker sequentially, all 146 chunks completed, and there were no missing return records, unverified provenance, or JSONL differing from the original in SQLite.

Actual file Chunk Verification scope
project.godot 20 Full hybrid search, full Qwen verification, individual re‑check of failed chunks
map.tscn 6 Full hybrid search and Qwen verification
tileset.tres 54 Full search audit, atlas source and final resource representative verification
tileset_edit.tscn 54 Full search audit, root node and sprite node representative verification
troll.gd 3 Full hybrid search and Qwen verification
troll.tscn 9 Full hybrid search and Qwen verification

Results of Separating Search and Verification

  • The CharacterBody2D declaration and nodes returned the class document and the KinematicBody2D -> CharacterBody2D mapping. Qwen adopted only the mapping that matched the target API and the actual chunk as version evidence, labeling it Godot 4.
  • PackedScene, Node2D, Sprite2D, TileSetAtlasSource, etc., returned the appropriate class documents, but those documents did not demonstrate major‑version exclusivity, so they remained classified as Insufficient evidence.
  • The run/main_scene setting matched the Setting the main scene document and the SQLite original exactly. It was initially rejected because the natural‑language description did not repeat the setting label verbatim; the verification rule was updated to treat a document that directly explains the key‑value pair as valid evidence.
  • doc_version or godot_version_tags are merely collection‑time version tags and do not automatically prove code version exclusivity. The rule was refined to confirm the major‑version difference only when the JSONL body explicitly explains it.

Data Gaps Confirmed by Full SQLite Comparison

The following expressions appeared in the GUI’s top candidates but were inaccurate, prompting a re‑search of SQLite’s docs_chunks, api_mapping, and label_prototypes raw JSON. All records containing the exact expressions were zero:

  • scene resource header
  • script external resource declaration
  • project format version setting
  • untyped declaration warning setting
  • window stretch aspect setting
  • default clear color setting

These cases show that correct JSONL exists in SQLite; the search simply missed them because the currently collected JSONL lacks direct evidence. Consequently, the verifier did not arbitrarily change a Not related return to Godot 3 or Godot 4.

Verification‑rule updates were split across commits d504f50 and 2222634 and passed all 33 Source Flow Debugger tests. The rules do not embed specific code syntax, API names, or setting keys as hard‑coded lists.

Deliverables