Today the goal is to decide which flow to choose among various search/validation alternatives
While keeping chunkText unchanged as input, we need to determine how to combine BM25, PostgreSQL full-text search, embedding, reranker, and Qwen direct‑evidence validator
Initially it was not a final design, but a collection of pros and cons for each search method obtained via ChatGPT and the yes/no response flow as material for judgment
Afterwards, based on simulations for each alternative and a comprehensive comparison table, we decided to adopt strategy F first
In other words, we keep chunkText as the search input, gather a wide set of candidates with BM25 and embedding, reorder them with the reranker, and finally have Qwen verify whether the retrieved JSONL directly matches the current code chunk
The next validation will be done by testing 50 items per table (docs_chunks, api_mapping, label_prototypes) with Qwen
For each table we will insert about 50 samples split into relevant and irrelevant cases, and check how the responses differ for cases that should yield yes versus those that should yield no
This test is not only to see if the search works, but also to verify whether Qwen direct‑evidence validator accepts or discards the retrieved JSONL as direct evidence for the actual code chunk
Because it is done manually, the amount of actual testing is much larger than expected
We created 50 Godot test items, and each item requires checking all three tables: docs_chunks, api_mapping, label_prototypes
For a common function/syntax we create one Godot code chunk, then generate expected yes/no data for docs_chunks, expected yes/no data for api_mapping, and expected yes/no data for label_prototypes; we then put the prompt + test code + six data items together to see how the response pattern varies
If it is not a common function, we have to create separate code for Godot 3 and Godot 4, effectively doubling the workload
To later summarize the results with a classification metric such as F1‑score, we must manually record the true/false outcome of every case
Therefore we judged that completing all 50 in a single day is impossible, and set today’s target to 5 items
After completing the 5, we will shift from simply increasing the number of tests to first analyzing the yes/no response patterns observed so far
In practice we processed 5 out of the 50
Even with only 5, we observed that the response flow differs between JSONL generated for Godot 3 and JSONL generated for Godot 4
Particularly for code with version differences, using a Godot 3‑based JSONL to check Godot 4 code can still yield yes because of common strings or migration evidence, and conversely, using a Godot 4‑based JSONL to check Godot 3 code can become ambiguous if source/target strings are mixed
Hence, from the next test onward we should not only check whether the six responses are yes/no, but first separate whether it is a common syntax or a version difference, then record raw responses by separating JSONL generation version and code version under test
This process reveals that we need to change not only the verification prompt but also the prompting strategy and dataset collection strategy
Going forward, when creating JSONL we should further separate collection/generation criteria so that common syntax, Godot 3‑specific evidence, Godot 4‑specific evidence, and bidirectional migration evidence do not mix
Meanwhile, the official documentation Markdown → JSONL conversion continues, and as of today about 600 of the total 1,570 Markdown files have been converted to JSONL