2026-06-27 Retrospective

Today it seems I set a goal of improving the consistency of the source-to-AST input flow document that I wrote on the 26th.

Until yesterday, the whole layers were distinguished in my head, but in the document or code they remained at a too abstract stage. I could distinguish terms like AST, Retriever, LLM verification, JSONL evidence, but when an actual project came in, it was not specified which files unfolded in what order, which text went into the AST Parser, and which chunk was passed directly to the Retriever.

I thought I understood it in my head, but the anxiety that the code I would start writing in that state might not be implemented in the direction I wanted was quite strong. So today I first restructured the whole architecture and tried to design it in a GUI form that a person could visually verify.

Why Concrete Details Were Needed

The roles of the AST and the Retriever were actually distinguished.

.gd files go into the AST Parser and must be split by function or declaration unit. The Retriever receives the resulting code snippets and must search the DB for related JSONL evidence. Then LLM verification looks at the prompt, the current chunk, and the retrieved JSONL to decide whether that evidence is actually relevant.

But when I tried to organize this only in words, it kept getting fuzzy. Especially in an actual project flow, files first unfold in the form # <relative path>, then the text beneath is divided by file, a specific function or declaration in a .gd file becomes a chunk, and only the chunkText of that chunk should be passed to the Retriever.

I wanted to see this flow clearly. I wanted to directly verify on a web screen which part of which file became which chunk, whether the input the Retriever received was exactly that chunk, and whether file paths, line numbers, or prompts got mixed into the search input.

Therefore I thought a tool was needed not just to write documentation, but to let users check it in an easy‑to‑read GUI.

Source Flow Debugger

As a result, by creating the Source Flow Debugger I was able to view the AST and Retriever flows on a single screen.

At first I tried to organize how to pass source code to the AST side, but it ended up with AST chunk, direct chunk, Retriever input, DB search button, and Qwen verification preview all attached on one screen. Consequently the AST, Retriever, and LLM verification became an integrated structure, and it turned out to be more convenient than I expected.

A point where I used my brain well was that I didn’t try to build a finished system from the start, but made it an observation tool. At this stage what was needed was not a “tool that automatically gives the correct answer,” but a tool to confirm that data moves according to the flow I intended.

What I verified today is as follows.

• Expand a Godot project in the form # <relative path>.
• .gd files are split into AST‑type chunks.
• Text resources like .godot, .tscn are split into direct chunks.
• Documentation files such as README.md are excluded in source‑analysis mode, and the reason for exclusion is left on the screen.
• The Retriever input contains only chunkText, without file path, line number, or prompt.
• Under each chunk, allow searching docs_chunks, api_mapping, label_prototypes tables separately.
• Afterwards, Qwen verification proceeds with the structure prompt + chunkText + retrieved JSONL.

When a small Godot project was added, it was confirmed to be broken down into 5 files, 14 chunks, AST 9, Direct 5. At this level, the chunk‑by‑chunk splitting flow seems to have succeeded to some extent.

GPT Demo Test

Originally I wanted to find a related GitHub repository based on the collected JSONL set, clone that repository, and test it.

But then I realized it wasn’t necessary to find an actual GitHub repo from the start. What I wanted to check now was “whether the LLM correctly judges that the retrieved JSONL is truly related to the current Godot code chunk.” So I thought it would be faster to have GPT generate a demo Godot chunk together with related JSONL and unrelated JSONL, and test that first.

Thus I deliberately created a Godot 3 code chunk, and also created a matching Godot 3 → Godot 4 conversion JSONL and a completely unrelated JSONL for testing.

At first I simply asked like this.

Does this JSONL contain content corresponding to source code? Answer only yes / no.

It turned out that the related JSONL was also yes, and the unrelated JSONL was also yes.

It might seem disappointing at first, but I actually thought it was a relief. If this issue hadn't surfaced at the current stage, the LLM could have plausibly stitched together unrelated JSONL with its own knowledge when actual DB queries were later added.

So I made the prompt stronger.

You are a JSONL evidence matching judge.

Determine whether the JSONL below directly contains evidence that matches the SOURCE_CODE provided.

Judgment criteria:
- It is a “yes” only if one of the JSONL’s `source_api`, `source_pattern`, `match_terms`, `required_when_seen_in_code`, or `before_code` matches an actual string or API call in the SOURCE_CODE exactly.
- Broad terms such as Godot, Godot3, Godot4, migration, 2D, physics, etc., do not count as related.
- Negative statements in the JSONL such as “does not describe”, “not related”, “unrelated”, “does not apply”, etc., are not accepted as supporting evidence.
- If the JSONL refers to a different API, node, or system, the answer is “no”.
- Do not use any Godot knowledge you have; rely only on the string evidence written in the JSONL.
- The response must be exactly one word: either “yes” or “no”.

Let's change it like this: related JSONL is yes, unrelated JSONL is no.

The criteria obtained from today's experiment are important. When the Retriever brings a JSONL candidate and it is verified by the LLM, you must not look at a broad semantic similarity. First, you need to check whether fields inside the JSONL such as source_api, source_pattern, match_terms, required_when_seen_in_code, before_code directly match the actual string/API call of the current chunk.

Tasks for Tomorrow

It seems we need to create more demo sets tomorrow.

We need to create several related JSONL and unrelated JSONL for the Godot chunk, and test many times how Qwen derives answers. It is still too early to set a standard after one or two successes.

In particular, the following cases need further verification.

• When multiple related JSONL are mixed, does Qwen correctly choose the actual evidence?
• When unrelated JSONL have only similar keywords, does it correctly discard them with no?
• Does Qwen distinguish the difference between api_mapping and label_prototypes based on string evidence?
• Can explanation‑type JSONL like docs_chunks be verified in the same way?
• When it looks like Godot 3 code but there is no JSONL evidence, does it avoid arbitrary migration?

Today was a day of pulling abstract structures into actual screens and input flows. Things that I thought I already knew in my head became much clearer when I unfolded them on the web, broke them apart, and displayed them as search inputs.

The next step is to repeatedly perform actual DB searches and Qwen verification, confirming that this flow remains stable.