Today the goal is to decide which flow to choose among various search/validation alternatives
While keeping chunkText as the input unchanged, we need to determine how to combine BM25, PostgreSQL full-text search, embedding, reranker, and Qwen direct‑evidence validator
Initially it was not a final design; we had gathered the pros and cons of each search method obtained via ChatGPT and the yes/no response flow as material for judgment
Afterwards, based on simulations for each alternative and a comprehensive comparison table, we decided to adopt strategy F first
In other words, we keep chunkText as the search input, gather a broad set of candidates with BM25 and embedding, reorder them with the reranker, and finally have Qwen verify whether the retrieved JSONL directly matches the current code chunk
The next validation will be done with Qwen testing 50 items per table for docs_chunks, api_mapping, and label_prototypes
For each table we will insert about 50 samples divided into relevant and irrelevant cases, and check how the responses differ for cases that should yield yes versus those that should yield no
This test is not only to see whether the search works, but also to confirm whether the Qwen direct‑evidence validator accepts or discards the retrieved JSONL as direct evidence for the actual code chunk
Because it is being done manually, the amount of actual testing is much larger than expected
We created 50 Godot test items, and each item requires checking all three tables: docs_chunks, api_mapping, label_prototypes
If it is a common function/syntax, we create one Godot code chunk, then produce expected yes/no data for docs_chunks, expected yes/no data for api_mapping, and expected yes/no data for label_prototypes; after that we put the prompt + test code + six data items together to see how the response pattern varies
If it is not a common function, we have to create separate code for Godot 3 and Godot 4, effectively doubling the workload
To later summarize the results with a classification metric such as F1‑score, we must manually record the true/false outcome for every case
Therefore we judged that completing all 50 items in a single day is impossible, and set today’s target to 5 items
After completing the 5 items, we will shift from simply increasing the number of tests to first analyzing the yes/no response patterns observed so far
In practice we processed 5 out of the 50 items
Even with only 5 items, we observed that the response flow differs between JSONL generated for Godot 3 and JSONL generated for Godot 4
Especially for code with version differences, using a Godot 3‑based JSONL to check Godot 4 code can still produce a yes because of common strings or migration evidence, and conversely, using a Godot 4‑based JSONL to check Godot 3 code can yield ambiguous results when source/target strings are mixed
Hence, from the next test onward we should not only check whether the six responses are yes/no, but first separate whether the issue is common syntax or version difference, then record raw responses by separating the JSONL generation version from the code version being checked
This process revealed that we need to adjust not only the verification prompt but also the prompting strategy and dataset collection strategy
Going forward, when creating JSONL we should further separate the collection/generation criteria so that common syntax, Godot 3‑specific evidence, Godot 4‑specific evidence, and bidirectional migration evidence do not get mixed
Meanwhile, the official documentation Markdown → JSONL conversion continues, and as of today about 600 of the total 1,570 Markdown files have been converted to JSONL