2026-06-28 Retrospective

Today I left the task of collecting Markdown into JSONL with Qwen running, and thought about how to search and retrieve it when that JSONL is later pushed into a DB.

The AST debugging program we built extracts code chunks from a Godot project. Based on those chunks, we needed a strategy for how to search and retrieve the docs_chunks, api_mapping, label_prototypes JSONL that went into the DB. So I documented search strategies from A to F, and as of now I have adopted strategy F.

Right now I am still collecting Markdown into JSONL, so it is difficult to immediately test inserting it into a real DB and searching with a retriever. Therefore today I created a checklist of 50 items and asked Qwen to write arbitrary Godot code. I also generated matching arbitrary JSONL and non‑matching arbitrary JSONL, and when I fed jsonl + prompt + chunk together, I checked how Qwen judged them. Even after trying only about five cases today, I could immediately see that this work is more labor‑intensive than I expected.

50‑Item Test Plan and Reality

Originally I intended to test all 50 Godot features today.

But the reality turned out to be different. First I made a list of 50 features that are likely to appear as functions or small code blocks in Godot 3, organized that list into a document, and also wrote a prompt template to see whether Qwen would correctly judge the evidence matching with a “yes” or “no” when jsonl + prompt + chunk are fed together.

At first it seemed like I could just create and verify them one by one. However, testing a single feature required far more JSONL than I anticipated.

I had to look at all three tables—docs_chunks, api_mapping, label_prototypes—and for each table create both related JSONL and completely unrelated JSONL. In addition, I had to separate cases where the syntax is the same in Godot 3 and Godot 4 from cases that differ by version.

If the syntax is common, I can use a single piece of code as a basis and put the successful and failing JSONL for the three tables together for verification. Conversely, if the syntax is not common, I have to create both Godot 3 and Godot 4 code. Then I need to compare the results when the JSONL made for Godot 3 is used with Godot 3 code, with Godot 4 code, and when the JSONL made for Godot 4 is used with Godot 4 code, and vice versa.

In the end I only tested about five of the 50 items today. It’s much less than the original plan, but I think it’s fortunate to have stopped at this point. If I had manually pushed all 50 and then discovered that the criteria were ambiguous, things would have gotten much more tangled.

Common Syntax and Version Basis

The biggest insight from today’s testing was that it is difficult to simply split Godot 3 and Godot 4 into two categories.

Some features have almost identical syntax in both versions. In those cases it is absurd to force a distinction of “Godot 3‑only” or “Godot 4‑only” just by looking at a single piece of code. If it is common syntax, we should treat it as such and check whether the related and unrelated JSONL for docs_chunks, api_mapping, and label_prototypes are correctly separated.

Conversely, features that have version differences are more complex. A JSONL created for Godot 3 can still yield a “yes” when used with Godot 4 code, and a JSONL created for Godot 4 can yield a “yes” with Godot 3 code because of certain fields. Especially migration‑related JSONL contain both source and target, so simple string matching can lead to unstable results.

Therefore I realized that going forward we need to record separately which version the JSONL was created for and which version the code being inspected belongs to. Even a “yes” must be distinguished as a natural “yes” for common syntax or an ambiguous “yes” caused by mixed source/target strings.

Need to Record F1‑Score‑Style Metrics

If test results are recorded only as “success” or “failure,” it becomes hard to see patterns later.

The direction I thought of today is not to calculate the F1‑score immediately, but to keep the raw judgment results so that later we can view them like precision and recall. For example, we need to note whether a “yes” should have been a “no,” whether a “no” should have been a “yes,” whether a “yes” on both sides is correct for common syntax, or whether a “yes” is ambiguous because of migration JSONL.

If we accumulate data this way, we can later determine whether the prompt, the JSONL generation criteria, or the table‑field design is the problem. Conversely, if we continue to jot things down loosely by hand, it will become difficult to trace why a particular response was produced.

Even with only five cases today, several ambiguous points became apparent. So from the sixth case onward, instead of pushing the test strategy as is, we should first adjust the recording method and prompting strategy.

Limits of Manual Testing

Copying prompts, attaching code chunks, inserting JSONL one by one, and re‑recording Qwen’s responses is too inefficient.

Doing the first few manually helped set a baseline. In practice, I discovered that a single feature requires far more JSONL than expected, that we need to separate common syntax from version‑specific syntax, and that the meaning of “yes” and “no” changes with context.

However, repeating this process for all 50 items is not realistic. Manual entry can lead to mistakes, and comparing results later becomes cumbersome.

Therefore I started thinking about building a debugging tool. If we could input the code chunk, table type, JSONL generation version, inspection code version, expected response, and actual response on a single screen, and have Qwen’s call results automatically logged, it would be much better. Just as we inspected the AST and Retriever flow on the web, it would be great if JSONL matching tests could be repeated in a GUI or debugging page.

Qwen Chatbot Test Data Generation Considerations

What I wrestled with today was the process of generating arbitrary test data with the Qwen chatbot.

What I wanted to create was a small demo set that could shake up the verification method itself. For example, I would ask the Qwen chatbot to generate a Godot 3 code chunk, a Godot 4 code chunk, and related and unrelated JSONL that fit the docs_chunks, api_mapping, label_prototypes formats.

The problem is that if you simply ask the chatbot, “Give me correct JSONL and incorrect JSONL,” the results can become too plausibly mixed. Whether something is common syntax, Godot 3‑only, Godot 4‑only, or a migration source/target affects whether the answer is “yes” or “no.” Therefore, when creating arbitrary data, we also need to specify the base version, the inspection code version, the table type, and the expected response.

What I realized today is that we need to define the generation method and prompts for arbitrary test data with the Qwen chatbot more precisely. Only then can we verify, even within this test, whether Qwen is merely stitching together its own knowledge or actually judging based on the explicit string evidence inside the JSONL.

Conclusion of the Day

Today was not so much a day of testing large amounts, but rather a day that confirmed that both the retriever search strategy and the Qwen chatbot‑based JSONL matching experiment require more effort than expected.

I initially thought I could run all 50 tests. However, in practice, even just 5 tests revealed plenty of issues. There are cases where Godot 3 and Godot 4 diverge completely, but there are also common syntaxes, and situations where the judgment wavers because source and target appear together, as in migration JSONL.

Therefore, what is needed now is not to arbitrarily increase the number of tests, but to have a structure that properly records test results. We need to log the JSONL generation criteria version, inspection code version, table type, expected response, actual response, and ambiguous reason. That way, when we finish all 50 tests, we can view the outcomes as patterns rather than mere impressions.

I started the day thinking I could just create arbitrary data and test without much thought, but I ended up seeing the need for a method to generate test data with the Qwen chatbot, a prompting strategy, and even an automatically invoked debugging tool. Starting tomorrow, from the sixth test onward, I’ll adjust the criteria slightly and first find a way to reduce the manually repeated workflow.