Decided to write a development retrospective after a long time
Because I kept thinking I should leave only perfect records, I ended up postponing documentation
To meaningfully capture the attempts and thoughts from the past ~10 days, I organized the research and architecture design process for creating a Godot‑specific coding model
Ran an experiment to place a Qwen‑family model on the local PC to reduce RunPod costs
Tried to run a 9B model on WSL in an RTX 3060 environment
However, network speed and latency issues were severe, and even the reasoning stage before generating an answer took more than 5 minutes, so the local‑run experiment was halted
Investigated dataset collection methods for training a Godot‑specific model
The core idea was to merge, per GitHub repository, the README.md, .gd files, and project structure into a single text, then use the project.godot configuration file and GDScript syntax differences to classify Godot 3 vs 4 versions
Notably, by leveraging version‑specific clues such as config_version, config/features, onready var, @onready, KinematicBody, CharacterBody3D, even non‑mainstream languages without JSON‑based dependency files can be version‑filtered
Watched a fine‑tuning video about the classic programming language OPL to understand the fine‑tuning workflow
Realized that collecting a dataset and turning it into a trainable format are separate challenges
Asked an SSAFY coach how to efficiently collect data for a specific version of a non‑mainstream language
Received feedback that the current Godot dataset is closer to a raw code dataset than an assistant‑training Q&A dataset
Concluded that to build a chatbot‑style product, it is better to generate question/answer pairs with an LLM and convert them into an instruction dataset rather than feeding raw code directly
Without this step, a request like “design a map for me” would likely trigger a Python‑centric answer that the base model has heavily learned
Determined it is mostly Python‑centric and not suitable as‑is for Godot 4‑specific training
Considered whether the model could answer Godot questions without explicitly mentioning “Godot,” but decided that providing the context word Godot increases the chance of a correct answer because the base model’s weights are heavily biased toward Python
Consulted a senior from school about RAG and prompting strategies
Advised that instead of injecting all data into the model, building a markdown‑document‑based vector search structure and guiding the model to retrieve needed data may be more realistic
Since re‑indexing large documents is costly and time‑consuming, a retrieval/prompting‑based approach may be more appropriate at the current stage than full‑scale training
Designed an initial architecture for a Godot‑specific coding model
Initially imagined a simple pipeline: dataset collection → question/answer dataset generation → model training
Realized that accurate filtering and answer generation require a solid understanding of the changes from Godot 3 to 4
Worried that Godot 3 code, Python code, or legacy APIs could leak into the answer set, so the architecture was revisited
Planned a front‑end RAG chatbot based on official documentation to handle Godot 3/4 version classification and conversion
Crawl the official Godot migration guide and Godot 4 docs to build a RAG chatbot, then use it to classify whether collected data belongs to Godot 3 or 4
Only data identified as Godot 4 will be processed into an instruction dataset
Gained additional insights on SFT/DPO training directions via ChatGPT
For SFT, tasks can include Godot 3/4 classification, Godot 3 → 4 conversion, Godot 4 code generation, Godot 4 error fixing, and Godot 3 API rejection/correction
For DPO/Preference, preference data can be constructed as bad answer = response containing Godot 3 code, good answer = pure Godot 4 code
Crawled and documented roughly 1,500 pages, including the Godot 4 official docs, the Godot 3 → 4 migration guide, the Godot 4 class reference, and tutorial docs
Although the volume seemed large, it represents about 3 % of the total model context size
Received further advice from the senior about disk I/O bottlenecks in the data storage and training pipeline
Suggested handling data acquisition and preprocessing/post‑processing near‑real‑time, while keeping the actual training batch‑oriented
Decided to trigger reinforcement learning or fine‑tuning once the dataset exceeds a certain size, using metric‑based batch processing
Since re‑indexing costs cannot be eliminated structurally, pipeline stability for data acquisition and processing is more critical than real‑time training
Consolidated current overall direction
Crawl official docs to build a Godot 4‑centric RAG knowledge base
Collect GitHub Godot projects and merge README, project structure, and GDScript files per repository
Perform first‑pass filtering using project.godot settings and Godot 3/4 syntax differences
Use the RAG chatbot to further determine Godot 3/4 status, legacy API usage, and suitability for Godot 4
Generate instruction/response pairs from the refined Godot 4 data
Train a Godot 4 coding model with SFT and DPO/Preference data
Retrospective
Because I aimed to keep only finished results over the past 10 days, I failed to record the process
Yet the failed experiments, roadblocks, and mid‑course decisions are precisely the records that guide the next steps
Going forward, I will focus on continuously documenting attempts and reasoning rather than striving for perfect outcomes