2026-06-17 Godot LLM/RAG Detector Construction Retrospective

One‑line Summary

Today I redefined the direction of a Godot‑specific coding model from a simple Q&A model to a repo‑level SWE Agent, and built the first concrete official‑document‑based RAG detector and data‑generation pipeline needed in front of it.

Major Flow of the Day

Initially, the goal was something like “let’s make an RAG chatbot using the Godot official documentation.” However, as the discussion and work progressed, the goal became clearer.

Collect official Godot documentation
-> RAG discriminator based on official documentation
-> GitHub Godot project labeling
-> SFT/DPO data generation
-> Qwen-based Godot 4 coding model
-> repo-level SWE Agent

The core of today was not to create a single chatbot, but to establish a data discrimination/generation foundation for building a model that can actually read and fix Godot projects later.

1. Judgment that a simple Q&A model is insufficient

The first thing organized today was the nature of the model's goal.

Previously, the plan was to create a Godot 4 question/answer dataset and train the model to provide good Godot 4 code answers. However, when considering a request like “make a map for me,” this is not a simple Q&A problem.

The actual request flow is closer to the following.

User request understanding
-> Explore repository structure
-> Find related scene/script/resource
-> Check asset paths and existing code style
-> Determine Godot 4 syntax/API
-> Modify code
-> Run/test/validate
-> Create patch

This flow is not a problem of generating a single answer, but a software engineering agent problem. Therefore, today I reoriented the model direction from the perspective of SWE‑agent trajectory training.

Key keywords recorded today:

Long-context repository-level software engineering agent training
SWE-agent trajectory training
Godot repo-level patch generation
long-context trajectory dataset

I also organized the reference cases.

SWE-agent trajectories
SWE-smith
SWE-Gym
CoderForge-Preview
ACC
RepoBench / CrossCodeEval / RepoCoder
aiXcoder CoLT
godot-dodo
wallstoneai/godot-gdscript-dataset

The conclusion of today was that a small instruction Q&A alone is insufficient. Ultimately, what is needed is a trajectory of exploring, judging, modifying, and verifying at the Godot project level.

Related notes:

docs/research-notes/2026-06-17-swe-agent-trajectory-keywords.md

2. Godot LLM Full Roadmap Summary

Next, we restructured the entire development roadmap.

The overall flow was organized as follows.

Data
-> First-stage RAG chatbot
-> SFT
-> DPO
-> SWE Agent

If the steps are broken down in more detail, they are as follows.

Stage 0. Preparation Phase  
Stage 1. Data Collection and Structuring  
Stage 2. First RAG Chatbot Development  
Stage 3. Data Labeling and Dataset Creation  
Stage 4. Model Training  
Stage 5. SWE Agent Development  
Stage 6. Continuous Improvement

In this roadmap, the most important judgment is that the first‑stage RAG chatbot should not be viewed merely as a question‑answering tool. First, we need to create a RAG discriminator that serves as an expert on the official Godot documentation, and then, using that discriminator, label and process GitHub data before expanding it with SFT/DPO and a SWE Agent.

In particular, the SWE Agent in Stage 5 should not be just a simple code generator; it must have the following capabilities.

Project Analysis
Search Required Files
Code Modification
Verify Godot CLI or Execution Result
Analyze Failure Causes
Create Patch

Related roadmap:

docs/roadmaps/2026-06-17-godot-llm-roadmap.md

3. Design of Data Generation Structure Based on RAG Discriminator

Another important judgment today was that “the final label decision should not be left to the LLM.”

At first, it is easy to think that a RAG chatbot could look at GitHub code or documentation and determine whether it is Godot 3/4. However, labeling directly affects the quality of training data, so entrusting it to the LLM’s spontaneous judgment is risky.

Therefore, the structure was set up as follows.

LLM is generation assistance  
The label is determined by the system  
The final JSONL is assembled/validated by the Python pipeline

What the system should be responsible for:

Symbol extraction  
Query API mapping DB  
Official document vector search  
Keyword search  
Label prototype search  
Label scoring  
confidence calculation  
Final JSONL assembly  
Validation

Things that can be handled by LLM:

Generate draft of revised code  
Generate description  
Generate SFT question/answer  
Generate DPO bad answer  
Generate patch draft  
Assist verification/problem description

The target dataset to be generated was also organized into eight categories.

version_classification
api_mapping
migration_fix
instruction_sft
dpo_preference
repo_explorer
patch_generation
metadata_verification

This theorem became the design principle for the RAG post‑processing code we created today. In particular, the reason we separated api_mapping, symbol_catalog, keyword_index, and search_text also comes from here.

Related notes:

docs/research-notes/2026-06-17-godot-rag-labeler-data-generation.md

4. MVP Flow from RAG Discriminator to Qwen 3.6 Model

The MVP flow diagram between the Godot RAG discriminator and the Qwen 3.6 coding model has been organized separately.

Organized flow:

Prepare the original document `godot_docs_full.zip`  
→ First chunking based on `chunk_docs.py`  
→ Godot‑specific post‑processing  
→ Build local search infrastructure  
→ Collect and structure GitHub data  
→ Run the RAG classifier  
→ Create the training dataset  
→ Qwen 3.6 SFT/DPO

Here, a local search infrastructure does not refer to only a single vector DB.

Required components:

Vector DB
Keyword Index
Reranker
API Mapping DB
Label Prototype DB

Also, GitHub data should be imported as a repository‑level structure, not just as simple code snippets.

Required input:

.gd
.tscn
.tres
project.godot
README
repo tree
metadata

The first SFT goal has been reorganized again.

Godot 4 Priority Thoughts  
GDScript Basic Output  
Godot 3 API Rejection  
Explanation of Reasons for Converting Godot 3 -> 4

Afterward, DPO needs to define the criteria for good answers and bad answers more clearly.

Bad answer: Answer mixed with Godot 3 API  
Good answer: Answer with pure Godot 4 code and supporting evidence

Related notes:

docs/research-notes/2026-06-17-godot-rag-to-qwen-coding-model-flow.md

5. v1 Official Document Chunking

After organizing the design, we created a form that can be input into RAG with the actual official document data.

The first deliverable we created is the following file.

work/godot_rag/chunks/docs_chunks.jsonl

Created 9,741 chunks by dividing the official documentation Markdown 1,570 items according to headings and length criteria.

Result:

Input pages: 1,570
Output chunks: 9,741
Max chunk chars: 2,800
Overlap chars: 350

v1 strengths:

The entire official documentation was turned into JSONL chunks.
JSONL parsing was stable.
Basic document types such as class_reference, tutorial, and migration were classified.

v1 issues:

A lot of leftover Sphinx wording remained.
There were many overly short noisy chunks.
The heading, section, method/property structure was not sufficiently preserved.
Class references were not finely split at the API level.

v1 was the “official documentation collection and basic chunking success” stage. It served as the starting point for the search MVP, but it was insufficient for a label classifier.

6. v2 post‑processing

Next, we created postprocess_chunks.py and generated v2 chunks.

Files:

work/godot_rag/postprocess_chunks.py
work/godot_rag/chunks/docs_chunks_v2.jsonl

What was processed in v2:

404 Chunk Removal  
Short Noise Chunk Removal  
Sphinx Residual Text Removal  
Extract symbols  
Extract class_name  
Enhance section/member_type/member_name  
Mark migration‑related chunks

Result:

Input chunks: 9,741
Output chunks: 8,778
Dropped 404 chunks: 3
Dropped short/noise chunks: 960
Migration-related chunks: 695

v2 has become usable as an MVP for RAG search based on official documentation. However, it was still insufficient to be considered a final label discriminator. In particular, syntax/API changes such as move_and_slide, velocity, @export, and await were not separated as independent evaluation criteria.

7. Incorrect v3 Direction and Reversal

The initially envisioned v3 was a method that hard‑coded some core APIs to create an api_focus chunk.

Example:

KinematicBody2D
CharacterBody2D
move_and_slide
@export
await

This method may seem like it quickly improves search performance. However, in reality it was risky.

Problem:

The search engine can become overfitted to the API selected by a person.
APIs outside the list become weaker.
Some representative APIs may appear to represent the entire Godot API.
Missing APIs can later lead to labeling accidents.

So the original v3 was reverted. This decision was important. It may look like a loss in the short term, but considering the final classifier quality, hard‑coding focus chunks were the wrong direction.

8. catalog‑based v3 redesign

After reverting, the direction was changed.

New principles:

v2 does not lose a single chunk.  
Do not arbitrarily create new focus chunks.  
Automatically extract symbol/catalog/index/mapping from the entire official documentation.  
Attach search reinforcement information as metadata.

Added code:

work/godot_rag/build_symbol_catalog.py
work/godot_rag/build_keyword_index.py
work/godot_rag/build_api_mapping.py
work/godot_rag/make_chunks_v3.py
work/godot_rag/validate_rag_artifacts.py
work/godot_rag/retrieval_smoke_test.py

Generated output:

work/godot_rag/catalog/symbol_catalog.jsonl
work/godot_rag/catalog/keyword_index.json
work/godot_rag/catalog/api_mapping.jsonl
work/godot_rag/chunks/docs_chunks_v3.part-*.jsonl
work/godot_rag/validation/validation_report.json
work/godot_rag/validation/retrieval_smoke_test.json

The entire docs_chunks_v3.jsonl is about 189 MB, which could exceed GitHub's single‑file size limit. Therefore, we uploaded split files that are preserved line‑by‑line in the repository, and if needed, they can be concatenated locally.

cat work/godot_rag/chunks/docs_chunks_v3.part-*.jsonl \
  > work/godot_rag/chunks/docs_chunks_v3.jsonl

9. v3 Consistency Verification

The most important thing considered in v3 was preventing omissions. If even a single v2 chunk is missing, the basis for the official documentation could disappear later.

Therefore, validate_rag_artifacts.py was created.

Verification criteria:

v2 chunk_id set == v3 chunk_id set
no missing or extra v3 chunks
catalog/index/mapping references point to existing chunks
search_text is present
no old hardcoded api_focus fields remain

Verification result:

status: pass
v2 chunks: 8,778
v3 chunks: 8,778
v2 unique chunk_id: 8,778
v3 unique chunk_id: 8,778
symbol catalog entries: 134,922
keyword index keys: 192,257
api mapping records: 144
v3 chunks with api mappings: 1,900
v3 migration-related chunks: 2,392

Also performed a lexical/keyword based smoke test before the embedding stage.

Representative query:

KinematicBody2D Godot 4 replacement
CharacterBody2D move_and_slide velocity
yield await Godot 4
export var @export Godot 4
onready var @onready Godot 4

The result passed all 5.

10. The biggest risk discovered today

The most important insight today was that “what is selected a lot” and “what is trusted and written on the label” are different.

If you select broadly across the entire official documentation, recall improves. However, if you treat all those results with the same level of confidence, the keyword index becomes polluted.

For example, the following is likely an actual API or syntax element.

CharacterBody2D
move_and_slide
FileAccess
@export
await

On the other hand, the following is likely a general document word or a table column.

Returns
See
Tip
MIT
Software
Type

In fact, during the process of checking the smoke test, we discovered an issue where a regular table column could be captured as an API mapping candidate, such as Type -> EditorSceneFormatImporterFBX2GLTF. This issue was corrected by excluding Type from the mapping candidates.

The conclusion drawn from this is clear.

Broad extraction is necessary.  
However, not everything extracted broadly should be used as a trusted fact.

11. Reorganized with a v3.1 Confidence Separation Structure

After identifying the above risk, it was decided that using v3 directly in the final label discriminator was not permissible. Therefore, in v3.1, the catalog is not combined into a single entity; instead, it is separated by confidence level and purpose.

New structure introduced in v3.1:

trusted_api_symbols
syntax_symbols
migration_mappings
mentioned_symbols
candidate_terms
rejected_terms
retrieval_keys
search_text

The meaning of each field has also been clearly separated.

trusted_api_symbols:
class/reference structures directly verified class/method/property/signal/constant

syntax_symbols:
syntax elements identified in the GDScript language documentation and migration documentation

migration_mappings:
old -> new mappings for Godot 3 -> 4 changes linked with supporting chunks

mentioned_symbols:
symbols mentioned in tutorial/body but already present in the trusted/syntax catalog

candidate_terms:
candidate words for search recall assistance

rejected_terms:
common words such as Returns, See, Tip, MIT, Software that should not be used as label references

Added code:

work/godot_rag/build_v31_artifacts.py
work/godot_rag/validate_v31_artifacts.py

Generated v3.1 deliverable:

work/godot_rag/chunks/docs_chunks_v3_1.jsonl
work/godot_rag/chunks/docs_chunks_v3_1.summary.json
work/godot_rag/catalog_v3_1/trusted_api_catalog.jsonl
work/godot_rag/catalog_v3_1/syntax_catalog.jsonl
work/godot_rag/catalog_v3_1/api_mapping.jsonl
work/godot_rag/catalog_v3_1/mention_index.json
work/godot_rag/catalog_v3_1/keyword_index.json
work/godot_rag/catalog_v3_1/v3_1.summary.json
work/godot_rag/validation_v3_1/validation_report.json
work/godot_rag/validation_v3_1/retrieval_smoke_test.json

v3.1 Result:

Source v2 chunks: 8,778
Output v3.1 chunks: 8,778
Trusted API catalog entries: 26,318
Syntax catalog entries: 68
API mapping records: 144
Mention index keys: 10,292
Keyword keys: 71,543
Validation status: pass
Retrieval smoke tests: 5 / 5 passed

There were also some important fixes.

In the initial v3.1 generation process, if you trusted the class_name directly from the v2 metadata, incorrect values such as Tutorials, All, String could end up in the trusted catalog. Therefore, we recreated the canonical class name based on the source URL and class reference path. After this fix, casing issues like Characterbody2d were corrected to CharacterBody2D.

Another crucial validation involved move_and_slide. In the broad extraction method, there was a risk of creating false trusted entries such as ProjectSettings.move_and_slide. In v3.1, we limited promotion to trusted only for members verified in the structured section of the class reference, and ultimately the move_and_slide trusted entry remained only for the actual Godot class, as shown below.

CharacterBody2D.move_and_slide
CharacterBody3D.move_and_slide

The work done in v3.1 today was not just adding fields, but changing to a structure that “uses a broad approach for search but a narrow, reliable basis for label determination.”

12. v3.1 Consistency Verification

In v3.1, omissions were especially risky. If even one of the 8,778 chunks from v2 is missing, the official documentation basis can disappear.

Verification criteria:

v2 chunk_id set == v3.1 chunk_id set  
v3.1 unique chunk_id count == 8,778  
Maintain doc_type distribution  
Remove legacy mixed fields  
Verify reference integrity of trusted/syntax/migration catalog  
Check existence of search_text

Verification result:

status: pass
errors: 0
warnings: 0
v2 chunks: 8,778
v3.1 chunks: 8,778

Ran the search smoke test again.

KinematicBody2D Godot 4 replacement
CharacterBody2D move_and_slide velocity
yield await Godot 4
export var @export Godot 4
onready var @onready Godot 4

All five results passed. This stage does not mean “the final labeler is completed,” but rather that the v3.1 architecture is sound enough to at least transition to an MVP vector/keyword index.

13. GitHub Integration and Record Organization

Today, I not only created the code and data artifacts but also organized the GitHub integration.

Tasks performed:

Remote configuration  
Large v3 file splitting  
Push work branch  
Merge remote main with local history  
Push main  
Organize document directory

docs_chunks_v3.jsonl is about 189 MB, so uploading it as a single file could exceed GitHub’s single‑file size limit. Therefore, it was split into docs_chunks_v3.part-000.jsonl, docs_chunks_v3.part-001.jsonl, and docs_chunks_v3.part-002.jsonl.

Also, initially the retrospectives were placed in the root retrospectives/, but that didn’t match the existing repository structure. So the documentation hierarchy was organized as follows.

docs/research-notes/   Design notes  
docs/roadmaps/         Full roadmap  
docs/retrospectives/   Retrospectives by date  
work/godot_rag/        RAG code and deliverables  
outputs/godot_docs_full/ Official documentation crawling results

Additionally, created docs/README.md to also organize the documentation directory role.

14. Organizing Git author/email

As recorded in the README today, the GitHub contribution graph reflection issue was also resolved.

Identified issues:

main history's author/committer emails are mixed  
local host email  
naver email  
GitHub noreply email

Processed direction:

Change the global Git configuration to yyeongjin <appsky1888@gmail.com>  
Unify the main history author/committer  
Reflect to the remote repository  
Store a backup branch before rewriting

Backup branch:

backup/before-author-email-rewrite-2026-06-17

This task is not directly linked to RAG itself, but it was one of the important整理 tasks today. It was work to lay the foundation so that subsequent records and commits are properly reflected in the GitHub profile.

15. Documents Created/Organized Today

Main documents created or organized today:

docs/research-notes/2026-06-17-swe-agent-trajectory-keywords.md
docs/research-notes/2026-06-17-godot-rag-labeler-data-generation.md
docs/research-notes/2026-06-17-godot-rag-to-qwen-coding-model-flow.md
docs/roadmaps/2026-06-17-godot-llm-roadmap.md
docs/retrospectives/2026-06-17-godot-rag-judge.md
docs/README.md

README also added a development log for June 17 and a RAG data preparation section.

16. Current Conclusion

The status as of today can be summarized as follows.

Direction:
It should go to the Godot SWE Agent, not the Godot Q&A model.

Data:
The official documentation RAG should serve as a discriminator for generating training data.

Label:
It should be decided by a Python pipeline, not an LLM.

RAG:
v2 is a safe default chunk set.  
v3 is a broadly extracted intermediate catalog deliverable.  
v3.1 is the currently recommended RAG deliverable, applying confidence separation.

Risk:
You must not use all extracted symbols with the same confidence level.

Today’s work was not just about creating a few files, but a day where we turned away from directions that were easy to go wrong several times. In particular, the decisions to revert the hard‑coded API focus chunk, to identify the noise risk in broad v3, and to separate trusted/syntax/migration/mention/candidate/rejected in v3.1 were important.

17. Next Tasks

The next tasks are not immediately GitHub code labeling.

Things to do first:

Generate embeddings based on the search_text of docs_chunks_v3_1.jsonl
Test hybrid retrieval that combines vector search + keyword search
Design a Python pipeline that determines labels based on trusted_api_symbols, syntax_symbols, migration_mappings
Define the input format for structured GitHub repo data
Design a repo‑level determination flow that looks at .gd, .tscn, .tres, project.godot, and README together
Connect the RAG classifier results to a pipeline that creates an 8‑type JSONL dataset
When a remote LLM endpoint is ready, link it to assist with generating modified code / explanations / SFT / DPO candidates
Afterwards, expand to training with Qwen‑based SFT/DPO and SWE Agent trajectory learning

The final lesson of today is this.

Collecting many documents is less important than separating the knowledge you have gathered by the level of trust with which you will use it.