2026-07-03 Retrospective

What I Did Today

Today I simply ran the collection while keeping a 24‑hour YouTube stream on, tried a translation test, and proceeded with verification by clicking through the validation tool created on June 30.

The Markdown → JSONL Converter side saw the official documentation JSONL conversion count rise from 737 to 850. Today, rather than pushing a big new implementation, I continued collecting while RunPod was running and intermittently checked the verification and translation flows.

For translation, I am experimenting with a pipeline that uses NVIDIA’s free model API. Through GitHub workflow secrets I saw it work to some extent, and I also saw screens where Japanese/Chinese READMEs were generated. It’s not yet at the stage where I can say it’s fully adopted, but the more documentation we have, the stronger the need for an automatic translation flow becomes.

The Reward of Building the Validation Tool

The biggest thing I felt today is how great it was to have built the validation tool on June 30. Without that tool, things could have gone badly wrong.

In the old method, you would have Qwen generate code and JSONL separately, then paste them into a prompt and compare the expected response in your head. Now you can click an item, generate code and JSONL, run validation on both Godot 3 code and Godot 4 code, and view the raw prompt and raw response all on one screen. The manual work is still there, but at least you can see exactly what went wrong without missing anything.

Today I checked up to item 10 out of 50 test items by clicking through. In the process, I found cases where the LLM gave incorrect outputs.

• In item 1, a docs_chunks candidate was marked “yes” because it shared the same _ready() or viewport‑related clue, even though it actually described different code.
• In items 3, 4, and 7, the migration JSONL for Godot 3 was attached to Godot 4‑style code, yet the tool still returned “yes.”
• In item 8, a docs_chunks candidate was judged “yes” because it dealt with the same tracking/movement topic or similar API, even though the actual behavior of the current SOURCE_CODE differed.
• Items 6, 9, and 10 all passed as expected, showing that the direction of improvement is not completely wrong.

These issues are less about a single malformed JSONL and more about whether Qwen, when a similar candidate appears later via hybrid search, will treat it as evidence or filter it out as merely a search result. Therefore, instead of blocking specific tokens, the verification prompt needs to be refined so it can more accurately judge the relationship between the current code and the JSONL candidate.

Thoughts

Today wasn’t a day of flashy implementations. I ran automatic collection, manually clicked through verification, and looked at JSONL and verification results, so time passed more slowly than expected. Still, I feel satisfied that I got something done.

Verification takes longer than expected. Even if it feels like looking at a single code file and a single JSONL, the nature of docs_chunks, api_mapping, and label_prototypes differs, and the code differs between Godot 3 and Godot 4. Doing just ten items already yields a fair number of cases.

However, if you catch the error points in advance as I did today, you can reduce the chance of crashes later during actual DB search and Retriever verification flows. I want to keep catching problems early in future tests as well.

If I were to improve, using free APIs alongside RunPod could be an option rather than relying solely on RunPod. If NVIDIA’s free model API or other free endpoints can reliably handle translation, auxiliary verification, and document synchronization tasks, the dependence on RunPod might be reduced.