2026-06-14 Retrospective

Thoughts on the Bus Going Up to Seoul

Today, while on the bus heading to Seoul, I revisited the Godot 4 dataset/RAG pipeline I designed last time. On June 12, I thought about crawling the official documentation, collecting Godot projects from GitHub, and placing a RAG chatbot at the front to determine whether it was Godot 3/4 and generate instruction data.

But today, thinking again, there were quite a few unresolved questions that need to be addressed for this structure to work well in practice. In particular, the gap between “collecting documents” and “the model effectively using those documents” seemed larger than expected and kept nagging at me.

First Question: Can a RAG Chatbot Handle the Entire Context?

Initially, I thought that if we crawled the official Godot documentation to create a RAG chatbot, we could use it to classify or verify the collected GitHub projects. However, when I imagined a scenario where we scrape GitHub projects, convert them to md or jsonl format, and then feed all that content to the chatbot at once, doubts arose.

Can the model receive the entire input context?

Even a single Godot project includes a README, directory structure, multiple .gd files, scene files, and resource paths. Adding the official documentation fragments retrieved by RAG would quickly make the input large. Just because the crawled documents exist does not mean the model can reliably read the entire project and related documents for every request.

In the end, building a RAG chatbot does not mean inserting whole documents; it means creating a search/selection structure that decides which document fragments to retrieve for a given question. If this part is weak, the model may miss important files or waste time reading unnecessary documents, losing the essential project context.

Second Question: Is One Answer Code Sufficient for One Question?

The second question concerned the shape of the instruction dataset. For example, consider the question “Create a map for me.” The answer to this question may not be a single block of code.

For an LLM to actually produce an answer, it must go through several steps: examine the project structure, identify which scenes and scripts exist, check available assets, read the existing code style, verify compliance with Godot 4 syntax, and finally decide which files and code to modify.

Thus, the question “Create a map for me” appears as a single request on the surface, but internally it requires multiple inference layers. If the training data only contains the final answer code, it is questionable whether the model can learn the intermediate decision‑making process.

The correct data might need to include not only the finished code but also which files to read, what information was used as a basis for the decision, why a particular API was chosen, and so on. Especially if we want the model to act like a coding agent, it may be more appropriate to keep a flow such as “question → search → decide → modify → verify” in the data rather than just “question → answer.”

Third Question: Won’t Python Weight Resurface When Processing Chunks?

The third question was about language weighting. Even if we prompt “Create a map with Godot 4,” I wondered whether the model’s tendency to favor Python code or Python‑style problem‑solving patterns might become stronger during the chunk‑by‑chunk reading of existing projects or documents.

General‑purpose coding models have likely been trained much more heavily on Python. Therefore, merely prefixing the prompt with the word Godot may not be enough. If the retrieved chunks do not contain a strong enough Godot 4 context, or if the intermediate reasoning triggers generic coding patterns, the answer could drift toward a Python‑style response.

So it seems we need to enforce the Godot 4 context much more strongly during prompting and data preprocessing, not just say “Write it in Godot 4.” For example, we could attach a Godot 4‑specific tag to the input data, filter out mixed Godot 3/other‑language code more aggressively, or repeatedly inject a constraint like “This task follows Godot 4 GDScript conventions” before generating the answer.

Today’s Summary

The key insight today is that RAG or crawling alone are not the solution. More important than gathering many documents is designing how the model reads context and makes decisions for actual requests.

The current structure seems to need to be broken down further:

• Instead of inserting the entire project at once, we should build a searchable structure based on file/role/dependency criteria.
• We need to examine whether instruction data should include not only the final code but also the search and decision‑making process.
• We must design prompts, tags, filtering, and preferred‑data criteria more strongly so that the Godot 4 context does not get diluted during inference.

In short, creating a “Godot 4 coding model” requires more than just collecting Godot 4 code. The data must reflect the order in which the model reads the project, how it decides something is Godot 4, and how it selects a good answer.

Retrospective

Today wasn’t a day of writing a lot of code, but it was a day of re‑examining gaps in the previous design. While zoning out on the bus, I noticed problems that I had missed at my desk. Saying “let’s build a RAG chatbot” sounds easy, but in practice the search granularity, context size, question/answer data structure, and language‑weight control are all intertwined.

There is no answer yet. However, I think the emergence of questions is a good sign. Recording these doubts at this stage will serve as a benchmark for what to validate when implementing the pipeline later.

Next time, based on these questions, I’ll need to specify how to split the project context, how much of the inference process to include in the instruction data, and how to create preprocessing rules that keep the Godot 4 context strong.