2026-06-12 Retrospective

Decided to Write a Retrospective Again

I’m leaving a retrospective after a long time. The reason I haven’t recorded anything for a while seems less about being busy and more about wanting to leave only perfectly crafted writing. I wanted to record only after the results were clearly organized, and I felt that documenting failures or mid‑process direction changes would be unnecessary.

But looking back, that very process was the most meaningful. Over the past 10 days the finished outputs weren’t many, but I investigated a lot about building a Godot‑specific coding model, where I got stuck, and what judgments I changed, so a lot accumulated. Therefore, today I decided to organize the flow so far.

Local Model Experiments

At first I tried to avoid the RunPod fee by loading a Qwen‑series model on my local PC. Since I’m using an RTX 3060, I thought a ~9B model could probably run on WSL.

However, it didn’t work as expected. Network latency and response delays were severe, and it took more than 5 minutes just to reach the inference stage before receiving a single answer. At that speed the experiment was too slow, and attaching it to development tools seemed impractical, so I abandoned the local model run immediately.

What I learned from this experiment was clear: in my environment it’s more important right now to design the dataset structure and training pipeline than to force a large model to run locally.

Investigating Godot Datasets

To train a model I first needed data. While looking for a dataset usable for learning Godot 4 code, I found Hugging Face’s wallstoneai/godot-gdscript-dataset.

• https://huggingface.co/datasets/wallstoneai/godot-gdscript-dataset

At first I thought it was simply a collection of many .gd files, but after analyzing it with Gemini I realized the structure mattered more. The key is not extracting files one by one from a GitHub repository, but expanding an entire repository to preserve project context.

This dataset appears to merge a repository’s README.md, .gd files, and directory structure into a single text, then organizes the project name, Godot version, file tree, and per‑file contents in markdown format. This way the model sees not only isolated code snippets but also the overall project structure and file relationships.

A particularly important part was the method for distinguishing Godot 3 vs 4 versions. Godot does not have a typical web‑project layout with package.json or requirements.txt. Instead you need to read values like config_version or config/features inside the project.godot file, or use differences in GDScript syntax.

For example, Godot 3 uses expressions such as onready var, KinematicBody, while Godot 4 uses @onready, CharacterBody3D. By capturing these syntax differences with regexes or parsers, you can achieve a degree of version filtering even for niche languages that lack a JSON‑based version file.

The Dataset Alone Was Not Enough

Later I watched a YouTube video about fine‑tuning for the classic programming language OPL.

• https://www.youtube.com/watch?v=5wGD92ktQL4

But the video didn’t give me a complete sense of the workflow. The dataset collection process and the fine‑tuning process need to be connected, yet the question “If I only have a dataset, what and how do I train?” kept lingering.

So I asked my SSAFY coach how to most effectively collect data for a specific version of a niche language. I received an important answer.

The first point was that the dataset I was looking at is more of a raw code dataset than an assistant‑training dataset. If I want a chatbot‑style output, simply feeding raw code isn’t sufficient.

The second point was that, based on the raw dataset, I need to have the LLM generate question/answer pairs. Hearing that clarified what I need to do: I must not stop at gathering only Godot 4 code, but also generate question data and answer data to create an instruction dataset.

Without this step, a request like “design a map” could be answered in Python rather than Godot, because existing models have been trained far more on Python data. If the answer data isn’t clear, older Godot code or unrelated languages could get mixed in.

Instruction Dataset and Prompting Considerations

I also looked at Magicoder-Evol-Instruct-110K as a candidate instruction dataset.

• https://huggingface.co/datasets/ise-uiuc/Magicoder-Evol-Instruct-110K

However, most of it was written with Python in mind. Using it as‑is would not be suitable for building a Godot 4‑specific model.

That raised the question, “Do I really need to explicitly mention Godot?” I thought that if I feed enough Godot data, the model might answer in Godot code even without the user explicitly saying “Godot”.

I discussed this with a senior at school, which helped clarify the direction. Nowadays, designing a good prompting or document‑based retrieval structure can be more effective than pure RAG, and creating vector‑based markdown documents for the model to retrieve needed information seems practical.

In the end, at the current stage, giving a clear context like “Create a map with Godot” rather than just “Create a map” is more likely to yield accurate answers. Because existing models have large Python weights, it’s necessary to strongly provide the “Godot” condition early in the prompt.

Architecture Redesign

Initially I thought of the architecture in a very simple way.

Dataset collection → Question/Answer dataset creation → Model training

But soon I felt that this structure was too risky. If I collect data without fully knowing what changed from Godot 3 to 4, incorrect code could end up as answer data. Even when creating a question/answer dataset, Python code or Godot 3 code could get mixed in.

So I thought of a structure that uses the official documentation as the primary reference point. By crawling the migration documents from Godot 3 to 4 and the official Godot 4 documentation, I create a RAG chatbot, place this chatbot at the front end, and have it determine whether the collected code is for Godot 3 or 4, whether it uses an older API, and whether it is suitable as Godot 4 code.

The attached diagram is a draft of the pipeline organized at that time.

The overall flow is as follows.

• Crawl the official Godot documentation and migration documents to build a RAG knowledge base.
• Collect Godot projects on GitHub on a repository basis.
• Merge README, file structure, and GDScript code into a single context.
• Perform first‑stage filtering using the project.godot configuration file and GDScript syntax differences.
• Use the RAG chatbot to further determine Godot 3/4 status and whether old‑version APIs are used.
• Create an instruction/response dataset only from data refined for Godot 4.
• Train a Godot 4 coding model with SFT and DPO/Preference data.

SFT and DPO direction

Later, by asking ChatGPT additional questions, I also obtained unexpected directions. I had only thought of a “model that generates Godot 4 code,” but I could split the training tasks more concretely.

In SFT, the following tasks can be created.

• Godot 3/4 code classification
• Convert Godot 3 code to Godot 4 code
• Generate Godot 4 code
• Fix Godot 4 errors
• Reject or correct Godot 3 API usage

In DPO/Preference, we can clearly define bad answers and good answers. For example, a response that mixes Godot 3 syntax is treated as a bad answer, while a response written purely in Godot 4 code is treated as a good answer.

Having this perspective made me realize that we should first decide “what counts as a good answer” rather than simply “let’s gather a lot of data.”

Crawling the official Godot documentation

To try this approach, I found a crawling tool and used crawl4ai.

• https://github.com/unclecode/crawl4ai
• https://docs.godotengine.org/en/stable/tutorials/migrating/upgrading_to_godot_4.html

I crawled and documented about 1,500 pages, including the Godot 4 official documentation, the Godot 3 → 4 migration guide, the Godot 4 class reference, and Godot 4 tutorial docs.

At first I wondered if I had crawled too much, but compared to the total model context size it amounts to roughly 3 % of the scale. If the data is used in a RAG or document‑search structure rather than fed entirely into the model at once, it is a manageable amount.

Data pipeline and I/O considerations

When redesigning the architecture, I also had to think about disk I/O and training timing. If data collection, preprocessing, post‑processing, storage, re‑indexing, and training all happen sequentially, bottlenecks are inevitable.

I consulted a senior at school about this part. The answer was that it is better to handle data acquisition and processing quickly and run training in batch mode rather than trying to fine‑tune in real time.

For example, only trigger reinforcement learning or fine‑tuning when the dataset exceeds a certain size. Treating the re‑indexing slot like a threshold reduces I/O bottlenecks, and the parts where real‑time performance matters are data acquisition and processing, not training.

Re‑indexing costs cannot be eliminated completely by design. Therefore, instead of trying to remove this cost, it is more realistic to set criteria for when to re‑index and when to train.

Today’s summary

Summarizing the flow over the past ten days: it started vaguely with “I need to create a Godot coding model.” Now a more concrete structure has emerged.

Raw code datasets alone are insufficient. We need refined code data based on Godot 4, a judgment structure built on official documentation, instruction/response data, SFT tasks, and DPO/Preference criteria.

Also, given my current skill level, building a coding model from scratch without a large model like Qwen is realistically difficult. Comparing a local initial model with ChatGPT’s coding ability made me realize that differences in model size and pre‑training volume cannot be ignored.

Nevertheless, the direction is much clearer than before. We can try building a knowledge base from the official Godot 4 docs, filtering collected data, defining good vs. bad answer criteria, and turning those into training data.

Reflection

Writing this reflection, the biggest thing I realized is that the bar for recording should be lowered. If you only record perfect results, the truly important process disappears. Failed experiments, ambiguous decisions, and mid‑project architecture changes are the material for future decisions; if they aren’t left behind, you won’t know later why you made certain choices.

Going forward, I’ll try to record the process itself more often rather than only posting finished results. Instead of postponing reflection because I feel I’ve failed, I think it’s better to include failures and concerns in the record.

I want to use today as a chance to consistently leave reflections again. It doesn’t have to be a perfect article, and it can be a work‑in‑progress. What matters is documenting what I tried, what I learned, and what criteria I’ll use for future actions.