2026-06-18 Godot RAG Work Reset Retrospective

Today's Status

I drank about 1 liter of coffee at a café yesterday, and today I felt very unwell. So I couldn't code properly. Still, I will record the ideas that came to mind now and why I organized yesterday's work.

Missing Parts in the Current Architecture

The missing parts in the current Godot LLM/RAG architecture are as follows. These parts need to be improved later.

Static analysis layer is weak  
GDScript AST/parser based verification is missing  
Godot project dependency graph is weak  
Execution/syntax verification is weak  
Label taxonomy is still coarse  
Distinguishing provenance between LLM outputs and verified answers is weak  
Data leakage/duplicate removal design is weak

Especially, there are parts that should not be solved only with RAG. Determining Godot code is not just a document‑search problem; it also requires considering the actual GDScript syntax, scene/resource dependencies, project structure, and the differences between Godot 3 and 4 APIs. However, yesterday’s workflow involved chunking the official documentation, feeding the result back to an LLM, and receiving feedback. This approach seemed fast on the surface, but in reality the validation criteria kept shifting.

Reason for Summarizing the Work on June 17

On June 17, I tasked the LLM with chunking and then fed the output back to GPT for feedback. No matter how I looked at it, this method didn’t feel right.

The reason for crawling the official documentation was to use the entire Godot docs as a basis. During the process, before the LLM could properly analyze the documents, it started treating certain APIs as “MVP core” and began hard‑coding specific keywords or overly reinforcing surrounding context. This turned the system into a search engine tuned to a few keywords rather than a RAG system based on the whole official documentation.

The problem was more than just unsatisfactory results. The LLM created its own criteria without my verification, generated files based on those criteria, and then used those results to make further decisions. Consequently, hallucinations appeared while the original document structure was not consulted, and later the LLM itself no longer knew which files were generated under which criteria.

On a larger scale, phenomena emerged where ChatGPT or Codex arbitrarily narrowed or changed the scope without my instruction. I wanted to see the structure based on the entire official documentation, not to be told to hard‑code “a few MVP core APIs.” Yet the LLM independently decided, “If it’s MVP, this is the core,” and produced the next deliverable before the step was even finished.

The problematic pattern is as follows.

The user wants a comprehensive analysis based on the entire official documentation structure  
→ The LLM arbitrarily defines the MVP scope  
→ Determines some APIs as core  
→ First performs hard‑coding / over‑reinforcement / catalog generation  
→ Produces the next‑stage deliverables before verification  
→ The results are packaged as plausible numbers  
→ In reality, the original structure and labeling criteria become contaminated

This is not a simple implementation mistake but an issue with the way we work. ChatGPT and Codex tended to assume an answer existed even before one was produced, and to move on to the next step before locking in the current one. It seemed to follow instructions, but in reality it would reinterpret the scope of the instructions, ignore them entirely, or try to create a “nice‑looking final form” first.

The particularly risky point in this work was that the LLM could not properly distinguish the reliability of the intermediate outputs it produced. Facts verified in official documentation, words that happened to match a regex, rules explicitly approved by the user, and speculative supplemental information guessed by the LLM were all mixed together. When the LLM was then asked to evaluate the results, it ended up rationalizing its own output in a plausible way.

Why the Chaining Output Was Emptied

Today I organized the RAG chaining files that were created yesterday.

The items that were organized are as follows.

v1 docs_chunks.jsonl  
v2 docs_chunks_v2.jsonl  
v3/v3.1 catalog/index/mapping deliverable  
Chunking/post‑processing/validation draft script  
Initial RAG chat/index draft

At first I thought only v3/v3.1 were problematic, but looking again, v1 and v2 also needed to be re‑examined from the ground up. Originally, the chunks themselves should have been analyzed. For example, we needed to see how the Godot class reference is actually structured, how method/property units are represented in the original documentation, and how the Sphinx conversion results get broken.

However, in practice the generation of chunking results came before the analysis of the original documents. As a result, v1 was close to a piece based on character count, and v2 was that with additional post‑processing on top. On the surface, numbers such as chunk count, JSONL parsing success, and doc_type distribution appeared, but the crucial question “Can this chunk be used as evidence for distinguishing Godot 3/4?” was not sufficiently verified.

In particular, we needed to check whether structures like CharacterBody2D.move_and_slide in the class reference remain stable, whether properties like velocity are correctly separated, and how to extract old/new relationships from migration documents. If we assign names like trusted_api_symbols, syntax_symbols, api_mapping without doing this, we end up with seemingly plausible but polluted data.

So today I cleared the chunking outputs up to v1/v2. This is not giving up on the work, but an effort to stop treating an incorrect baseline as a baseline.

Conclusion of the Day

The conclusion of the day is simple.

We need to re‑chunk from the beginning.  
Before that, we must first analyze the original document structure at `outputs/godot_docs_full/pages`.  
We must not use intermediate outputs generated by the LLM as the basis for the next step without verification.

I haven't even checked it, but when an LLM makes its own judgments, doesn't properly look at the original, and mixes in hallucinations, the LLM ends up not knowing what it has done. In that state, if you keep creating files, only debugging files increase, and you lose track of the criteria.

From now on, I do not trust ChatGPT or Codex to “automatically expand in a good direction” and implement things. Especially for tasks where criteria are crucial, such as datasets, labelers, and RAG discriminators, the following principle is required.

Stop if LLM arbitrarily defines the MVP scope  
Stop if hardcoding not approved by the user is included  
Stop if deliverables are created without analyzing the original structure  
Stop if moving to the next step without a verification report  
Stop if the provenance of LLM outputs and verified answers gets mixed

If we do it again tomorrow, the first task will not be creating a new catalog. First, we need to analyze godot_docs_full itself. In other words, we should check what structure the official documentation Markdown files in outputs/godot_docs_full/pages actually have, and based on that structure reconsider how to chunk them.

The question we need right now is not “which API to add first,” but rather something closer to the next step.

What pattern is the class reference document in godot_docs_full composed of?  
Can the list of methods/properties/signals/constants be reliably separated from the original Markdown?  
What form does the table/list/sentence structure of the migration document take?  
Should the tutorial documents not use the same chunking criteria as the class reference?  
Should the chunk unit be set differently for each document type?  
Among page-level, section-level, and API member-level, which should be used as the default chunk?  
What validation report should be defined before chunking?

In the end, the task is not to “recreate RAG,” but to analyze godot_docs_full and redesign the chunking to fit the official Godot documentation. Only after that should the chunking be redone.