From the Conclusion
It is correct. I misunderstood the intention.
If we capture it precisely again, it is this.
1. The zip contains the original markdown, schema, debugger, and design documents.
2. Your local environment already has the markdown → JSONL → DB conversion.
3. What I can see is the “pre‑conversion markdown and DB schema/search flow.”
4. Therefore, the PoC must split the markdown inside the zip into JSONL‑like chunks and simulate the search targets that would have been inserted into your local DB.
5. What you want is not the code skeleton but a trace of how each search method generates scores and why it succeeds or fails when a chunk is inserted.And one important correction.
The /api/retrieve endpoint in the current project is not BM25.
The current code is:
chunkText
-> plainto_tsquery('simple', chunkText)
-> search_tsv @@ query
-> ts_rank_cd(...)It is. PostgreSQL's ts_rank_cd is a full-text ranking function, not BM25 itself. The PostgreSQL documentation also describes ts_rank/ts_rank_cd as ranking functions that receive a document-length normalization option. plainto_tsquery tokenizes the input sentence, ignores punctuation, and creates a query from the non‑stopwords. (PostgreSQL)
From the Conclusion
The best solution that satisfies your conditions is this.
Input:
raw chunkText as is
Retriever internal:
1. Search BM25 candidates with raw chunk as is
2. Search code embedding candidates with raw chunk as is
3. Union of candidates
4. Reorder with reranker
5. Qwen or validator directly verifies JSONL evidenceThis is a version that does not make Godot API signal extraction mandatory.
In other words, the “Godot API signal extractor” I mentioned earlier should not be mandatory. As you said, that smells like hardcoding. It can be used, but if you make it a core of the basic design, maintenance will suffer.
The final recommendation is:
First candidate: BM25
Second candidate: code embedding
Final ranking: reranker
Verification: Qwen direct-evidence validatorProject Understanding Verification
The actual structure observed in the zip is as follows.
JSONL Design
The document defines the deliverables as follows.
work/godot_rag/jsonl/docs_chunks.jsonl
work/godot_rag/jsonl/api_mapping.jsonl
work/godot_rag/jsonl/label_prototypes.jsonl
work/godot_rag/jsonl/ingest_report.jsonlThe DB schema is not structured to put an entire JSONL line into payload jsonb.
The JSONL fields are expanded into table columns.
docs_chunks:
chunk_id
doc_version
source_url
source_file
source_sha256
doc_type
symbol
section_path
heading
content
code_blocks
language_tags
godot_version_tags
api_symbols
token_count
metadata
embedding
search_tsv
api_mapping:
mapping_id
source_api
target_api
change_type
godot_from
godot_to
confidence
evidence_chunk_ids
match_terms
notes
negative_patterns
label_prototypes:
prototype_id
label
task_type
input_pattern
expected_finding
recommended_action
evidence_mapping_ids
evidence_chunk_ids
severity
validator_rules
embedding
search_tsvIt means that your local DB would contain something like this.
{"chunk_id":"...","source_file":"outputs/godot_docs_full/pages/getting_started__first_2d_game__03.coding_the_player__161d377b.md","doc_type":"tutorial","heading":"Coding the player","content":"...","code_blocks":["func _process(delta): ..."],"api_symbols":["Input.is_action_pressed","Vector2.ZERO","AnimatedSprite2D","position.clamp"]}Your chunk input
Assume the input is always exactly this.
func _process(delta):
var velocity = Vector2.ZERO
if Input.is_action_pressed(&"move_right"):
velocity.x += 1
if Input.is_action_pressed(&"move_left"):
velocity.x -= 1
if velocity.length() > 0:
velocity = velocity.normalized() * speed
$AnimatedSprite2D.play()
else:
$AnimatedSprite2D.stop()
position += velocity * delta
position = position.clamp(Vector2.ZERO, screen_size)If you look at this chunk with a code tokenizer, the unique tokens appear as follows.
_process
animatedsprite2d
clamp
delta
else
func
if
input
is_action_pressed
length
move_left
move_right
normalized
play
position
screen_size
speed
stop
var
vector2
velocity
x
zero
0
1Total 43 tokens, 25 unique.
The important token here is this.
Strong tokens:
animatedsprite2d
is_action_pressed
move_left
move_right
normalized
clamp
screen_size
vector2
zero
Weak tokens:
func
var
if
else
velocity
position
delta
x
0
1BM25 gives high scores to strong tokens.
This is because the rarer a word appears throughout the document, the larger its IDF becomes.
How BM25 Works to Produce Results
BM25 roughly calculates “how well the query terms are embedded in the document.”
Formally, three elements are key.
TF: How many times does a word appear in this document
IDF: How rare this word is in the entire corpus
Length normalization: Adjust so that a long document does not automatically have an advantageElasticsearch also uses BM25 as its default relevance algorithm, describing term frequency, inverse document frequency, and field-length normalization as core components. (Elastic)
The formula looks like this.
score(query, doc)
= Σ over query terms [
IDF(term)
*
TF_boost(term frequency in doc, doc length)
]If a rare term such as animatedsprite2d appears many times in the document, the score increases significantly.
Results Simulated as Real
Original Godot markdown inside the zip:
outputs/godot_docs_full/pages/Split this into roughly 1800‑character chunks like JSONL docs_chunks, and fed the raw chunks directly as queries into BM25.
Candidate #1
Input.is_action_pressed("move_right")
Input.is_action_pressed("move_left")
velocity.length()
velocity.normalized() * speed
$AnimatedSprite2D.play()
$AnimatedSprite2D.stop()Why it's number one, these tokens matched directly.
animatedsprite2d
is_action_pressed
move_left
move_right
normalized
play
stop
input
speed
length
velocity
vector2
zeroThe top contributing tokens that significantly increased the score are as follows.
animatedsprite2d df=18 tf=7 contribution=11.975
is_action_pressed df=27 tf=2 contribution=8.609
move_left df=13 tf=1 contribution=7.479
velocity df=180 tf=18 contribution=7.467
move_right df=15 tf=1 contribution=7.299
normalized df=67 tf=2 contribution=7.069
stop df=133 tf=2 contribution=5.900
play df=144 tf=2 contribution=5.764
input df=323 tf=7 contribution=5.650
speed df=219 tf=3 contribution=5.637Here, df is the number of chunks that contain the word out of the total 4165 chunks.
In other words:
animatedsprite2d appears in only 18 out of a total of 4165 chunks
move_left appears in only 13 chunks
move_right appears in only 15 chunks
is_action_pressed appears in only 27 chunksSo documents containing these words rise strongly.
This is how BM25 created the #1 ranking.
Candidate #2
AnimatedSprite2D description
$AnimatedSprite2D.play()
get_node("AnimatedSprite2D").play()
position += velocity * delta
position = position.clamp(Vector2.ZERO, screen_size)Matched token:
animatedsprite2d
clamp
screen_size
play
delta
_process
velocity
position
stop
vector2
zeroThe reason this chunk ranks second is that, although there is no input handling part, clamp, screen_size, and AnimatedSprite2D match strongly.
In particular:
screen_size df=3
clamp df=32
animatedsprite2d df=18screen_size is a rarely used term, so even a single occurrence yields a high score.
Candidate 3rd Place: Failure Candidate
3D player movement
Input.is_action_pressed("move_right")
Input.is_action_pressed("move_left")
Vector3.ZERO
direction.normalized()
Chunk content:Matched token:
is_action_pressed
move_left
move_right
normalized
speed
velocity
delta
input
x
var
func
zeroThis seems related, but it is not the exact answer.
Why it was posted:
Input.is_action_pressed
move_right
move_left
normalized
speedThis common token for such movement code overlaps strongly.
However, this document is 3D. The chunk is 2D, and AnimatedSprite2D, Vector2, position.clamp, screen_size are key.
In other words, using only BM25 leads to these false positives.
Why the current method fails
The approach close to /api/retrieve right now is plainto_tsquery(raw_chunk).
This method roughly becomes this query.
func & process & delta & var & velocity & vector2 & zero
& input & is_action_pressed & move_right & move_left
& length & normalized & speed & animatedsprite2d
& play & stop & position & clamp & screen_sizeIf you do this, a single JSONL chunk will require too many tokens.
In my simulation:
strict raw AND hits: 0That is, if a chunk does not satisfy all unique tokens simultaneously, it counts as 0.
Because the actual official documentation is also divided like this.
chunk A:
Input.is_action_pressed
velocity.normalized
AnimatedSprite2D.play/stop
chunk B:
AnimatedSprite2D description
position.clamp
screen_sizeBoth are related, but not everything is contained in a single row.
Therefore, the current approach needs to be overhauled.
Alternatives by Method
Alternative A. Current Method: raw chunk + PostgreSQL plainto_tsquery
Flow:
raw chunkText
-> plainto_tsquery
-> search_tsv @@ query
-> ts_rank_cdYes, chunk result:
**High likelihood of failure**
Even a slight change in chunk splitting results in 0 cases.Advantages:
Implementation already completed
Possible with PostgreSQL alone
Simple infrastructureDisadvantages:
Weak with long code chunks
Excessive AND conditions
Lots of code token noise
Not BM25
Semantic search not possibleJudgment:
DiscardAlternative B. raw chunk + BM25 only
Flow:
raw chunkText
-> tokenizer
-> BM25
-> return top JSONLYes, chunk result:
1st place: first_2d_game / coding_the_player
2nd place: first_2d_game / coding_the_player / clamp section
3rd place: first_3d_game / player_movement_codeWhy did it succeed?
AnimatedSprite2D
move_right
move_left
is_action_pressed
screen_size
clampThese words are rare throughout the Godot docs, so they received a high score.
Advantages:
Principle Transparency
Easy Debugging
No Model Cost
Can Use Raw Chunk As Is
Code/API Strong on Exact StringsDisadvantages:
Weakness in synonyms/descriptions
If the API name does not appear directly, it cannot be found
Similar code gets mixed in, like 3D movement
If the documentation is expressed differently, it is missedJudgment:
Must be used
But it is insufficient on its ownAlternative C. raw chunk + embedding only
Flow:
raw chunkText
-> embedding vector
-> docs_chunks/api_mapping/label_prototypes embedding and cosine search
-> return top JSONLReasons for using the model:
To find documents with similar meaning even if the strings do not match exactly.For example, the query contains:
position.clamp(Vector2.ZERO, screen_size)There is, and in the document:
prevent the player from leaving the screen
clamping a value means restricting it to a given rangeIf it is described like that, BM25 can become weak. Embedding captures this kind of semantic connection.
Advantages:
Find even if the expression is different
Strong with document descriptive content
Easy to insert raw chunk as is
Can handle long chunksDisadvantages:
Accurate API judgment is weak
Can mix subtle differences such as 3D/2D, Godot3/Godot4
It is difficult to explain why this result occurred
In api_mapping, the risk of false positives is highYour chunk prediction:
Success:
first_2d_game / coding_the_player likely to be found
Failure:
first_3d_game movement may also appear with high probability
because “player movement” has a similar meaningJudgment:
Do not use alone
Use for supplementing BM25 candidatesAlternative D. raw chunk + BM25 + embedding parallel
Flow:
raw chunkText
-> BM25 top 50
-> embedding top 50
-> Merge
-> Mix scores
-> Return top JSONLThis method is quite good.
Your chunk flow:
What BM25 captures:
Input.is_action_pressed
AnimatedSprite2D
position.clamp
screen_size
What embedding captures:
player movement
2D movement tutorial
moving inside screen
animation based on movementAdvantages:
String search and semantic search complement each other's weaknesses
Quality improves in the first pass even without Qwen
Maintain raw chunk conditionDisadvantages:
Score combination tuning is needed
BM25 score and vector score have different scales
False positives cannot be completely preventedJudgment:
Practical Minimum Recommended ThresholdAlternative E. raw chunk + Qwen query JSON generation + search
If you want to avoid hard‑coding, this method is also possible.
Flow:
-> Request Qwen to generate JSON for search
-> Search with BM25/vector using the generated JSON
-> Return JSONL candidatesFor example, Qwen output:
{
"search_intent": "Godot 4 2D player movement tutorial",
"important_literals": [
"Input.is_action_pressed",
"Vector2.ZERO",
"AnimatedSprite2D",
"position.clamp",
"screen_size"
],
"likely_doc_topics": [
"first 2D game",
"coding the player",
"player movement",
"clamp position to screen",
"play and stop AnimatedSprite2D"
],
"migration_signals": []
}This is a method that does not hardcode the Godot API signal extractor directly.
Instead, Qwen creates a query profile.
Advantages:
Hardcoding is minimal
Intent summarization possible in complex chunks
Search terms can be made more readable for humans
Specialized Godot judgment can be delegated to QwenDisadvantages:
Slowness
Cost incurred
Can generate clues without Qwen
Hallucination possible even before the search stage
Raw chunk verification is absolutely necessaryYes chunk success:
Qwen succeeds when extracting AnimatedSprite2D / Input.is_action_pressed / position.clamp.Yes chunk failure:
If Qwen creates a non-existent intention like “migration from Godot 3 to 4”, it will pull the api_mapping incorrectly.Judgment:
Good for search quality experiments
Be cautious as the primary production search engineAlternative F. raw chunk + BM25 + embedding + reranker
This is the best.
Flow:
-> BM25 top 80
-> embedding top 80
-> candidate union
-> reranker directly compares the raw chunk with each JSONL candidate
-> returns top JSONL
-> Qwen validator directly verifies evidenceThe reranker reads the query and candidate documents together and re‑evaluates relevance. The Voyage documentation also describes the reranker as a model that takes first‑stage search result candidates such as embedding/BM25 and reorders them by relevance score. rerank-2.5 is a quality‑optimized reranker with a 32K context. (Voyage AI)
What the reranker does in the three chunks:
Candidate 1:
first_2d_game / coding_the_player
raw chunk and the actual code flow are almost the same
=> Very high
Candidate 2:
same page / clamp section
position.clamp and AnimatedSprite2D are described
=> High
Candidate 3:
first_3d_game / player_movement
Input.is_action_pressed is the same but in a Vector3/3D context
No AnimatedSprite2D
No screen_size clamp
=> LowAdvantages:
Best quality
Significantly reduces BM25 false positives
Also reduces embedding false positives
Maintains raw chunk conditions
Low reliance on hard‑codingDisadvantages:
Cost exists
Latency exists
If you put in too many candidates, it becomes slow
Since the reranker is not a justification verifier, a final validator is neededJudgment:
Final RecommendationWhy use the model and why not use it
There is no model in BM25
BM25 is statistical retrieval.
This word is in the query.
This word is also in the document.
This word is rare in the corpus.
Then increase the score.So it is strong at code search like your chunk.
AnimatedSprite2D
Input.is_action_pressed
Vector2.ZERO
position.clampThis kind of thing values the string more than the meaning.
However, BM25 has a weak semantic connection such as “prevent the clamp from going off-screen”.
Why use an embedding model
Embedding converts sentences into vectors.
raw chunk
-> [0.12, -0.03, ...]
docs_chunk
-> [0.11, -0.02, ...]
cosine similaritySo it finds the closest content even when the strings do not exactly overlap.
If embedding is required in this project:
1. When the document is explanatory and the code differs from the words
2. When the API names do not overlap but represent the same concept
3. When tutorial chunks are distributed across multiple expressions
4. When BM25 returns zero results and a fallback is neededWhen you need to exclude embedding:
1. api_mapping source_api exact matching
2. Godot3/Godot4 version determination
3. Remove false positives that only match target_api
4. Confirm migration ruleIn other words:
embedding = recall expansion
BM25/exact = direct evidence
reranker/Qwen validator = candidate organization and verificationCharacteristics by Model
voyage-code-3
It is the most suitable for this project.
Reason:
The query is not a natural language question but a GDScript code chunk.
The target JSONL for search also contains mixed code_blocks, API names, and documentation descriptions.Voyage official documentation states that voyage-code-3 is a code retrieval optimization model with a 32K context, a default dimension of 1024, and supports 256/512/1024/2048 dimensions. The Voyage announcement also explains that it achieved higher average performance compared to OpenAI text-embedding-3-large and CodeSage-large on 32 code retrieval datasets. (Voyage AI)
Features:
Advantages:
strong in code query
can handle long chunks
selectable 1024/2048
suitable for code → docs search
Disadvantages:
depends on external APIs
incurs cost
not a Godot‑specific modelRecommended use:
voyage-code-3 1024 floatIf you only consider quality, 2048 is also possible, but 1024 + reranker is more realistic.
OpenAI text-embedding-3-large
A general-purpose high‑quality embedding.
According to the official OpenAI documentation, text-embedding-3-large has a default dimension of 3072, and text-embedding-3-small has a default dimension of 1536. (OpenAI Developer)
Features:
Advantages:
Strong universal semantic retrieval
Stable for document description search
Good OpenAI ecosystem
Disadvantages:
Not dedicated to code retrieval
3072 dimensions so storage/index cost is large
In this project's “GDScript chunk -> JSONL” it is less direct than voyage-code-3Recommendation:
Second priorityGemini Embedding
Google Gemini embedding is fundamentally 3072-dimensional, and you can select sizes such as 768/1536/3072 using output_dimensionality. Google documentation also explains that using a smaller dimension can reduce storage space and computational cost while keeping quality loss minimal. (Google AI for Developers)
Features:
Advantages:
Strong universal/multilingual semantic retrieval
Good for document description-type retrieval
Dimensionality reduction possible
Disadvantages:
Not a dedicated option for code retrieval
Focuses on semantic similarity rather than API code accuracy
Risky to use alone for exact judgment in Godot migrationRecommendation:
It would be good if it focuses on document QA.
If it focuses on code chunk search, place it under **voyage-code-3**.Jina embeddings v4
Jina embeddings v4 is a model that excels in complex document search, multilingual, multimodal, and visually rich document retrieval such as tables/charts/images. According to Jina's description, it emphasizes long inputs and multimodal document retrieval. (jina.ai)
Features:
Advantages:
Wide document search scope
Strong in multimodal/complex documents
There is also a code adapter series
Disadvantages:
Your project is currently markdown/code centric
Image/table retrieval is not core
Might be an excessive choiceRecommendation:
Currently, the priority is low.When Using Qwen for Search
Qwen, as you mentioned, is currently focused on JSONL generation/validation.
It can also be used for search.
There are two ways to use it.
1. Qwen query profile generator
2. Qwen validator / rerankerQwen query profile generator
- Generate JSON for search with Qwen
- BM25/vector searchAdvantages:
## Reduce Hardcoding
Able to infer Godot context
Can summarize complex code chunksDisadvantages:
hallucination possible
Possible injection of incorrect intent in the pre‑search stage
Slow
Cost incurredQwen validator
raw chunk
+ retrieved JSONL
-> Determine whether this JSONL is direct evidenceThis is powerful.
The project observation document already sets the criterion that “instead of broad topic similarity, the JSONL field and the chunk string must match directly.”
Recommendation:
Using Qwen as a direct evidence validator after search is safer than generating queries before search.Success/Failure Flow Based on Chunk
Success Flow: BM25 + Embedding + Reranker
Input:
raw chunk as isFirst BM25 is noise:
first_2d_game / coding_the_player
Reference:
Input.is_action_pressed
move_right
move_left
velocity.normalized
AnimatedSprite2D.play
AnimatedSprite2D.stopSecond BM25 is noise:
same page / clamp section
Reference:
AnimatedSprite2D
position.clamp
Vector2.ZERO
screen_sizeEmbedding is supplemented:
player movement
2D movement
animation based on movement
screen boundsReranker summary:
1st:
first_2d_game / coding_the_player
2nd:
same page / clamp and AnimatedSprite2D explanation
Lower:
first_3d_game / player_movement_codeFinal JSONL:
{"table":"docs_chunks","payload":{"source_file":"outputs/godot_docs_full/pages/getting_started__first_2d_game__03.coding_the_player__161d377b.md","doc_type":"tutorial","heading":"Coding the player","content":"...Input.is_action_pressed...velocity.normalized...$AnimatedSprite2D.play()...$AnimatedSprite2D.stop()...","api_symbols":["Input.is_action_pressed","Vector2.ZERO","AnimatedSprite2D"]}}{"table":"docs_chunks","payload":{"source_file":"outputs/godot_docs_full/pages/getting_started__first_2d_game__03.coding_the_player__161d377b.md","doc_type":"tutorial","heading":"Coding the player","content":"...position += velocity * delta...position = position.clamp(Vector2.ZERO, screen_size)...","api_symbols":["Vector2.ZERO","position.clamp","AnimatedSprite2D"]}}This is success.
Failure Flow 1: Current raw tsquery
Input:
raw chunk as isInternal query:
func & process & delta & var & velocity & vector2 & zero
& input & is_action_pressed & move_right & move_left
& length & normalized & speed & animatedsprite2d
& play & stop & position & clamp & screen_sizeResult:
0 possibleReason for failure:
Fail if any word is missing in a JSONL chunk.This needs to be rewritten.
Failure Flow 2: BM25 only
Result:
1st first_2d_game / coding_the_player
2nd same page / clamp section
3rd first_3d_game / player_movement_codeReason for failure:
first_3d_game also shares Input.is_action_pressed, move_left, move_right, normalized.BM25 does not fully understand whether it is “2D or 3D”.
It simply calculates word scores.
Therefore, BM25 only is not sufficient.
Failure Flow 3: embedding only
Expected result:
2D player movement
3D player movement
scripting player input
vector math
animation player docsReason for failure:
In terms of meaning, they are all player movement, so it is close.embedding finds closely related items well, but the API accuracy is low.
Failure Flow 4: Qwen query profile only
If Qwen extracts it well like this, it succeeds.
{"important_terms":["Input.is_action_pressed","AnimatedSprite2D","position.clamp","Vector2.ZERO"]}But it fails when extracted this way.
{"migration_intent":"Godot 3 to Godot 4 migration","important_terms":["AnimatedSprite","AnimatedSprite2D"]}Reason for failure:
The raw chunk does not have AnimatedSprite, only AnimatedSprite2D.
It's valid Godot 4 code, but you can bring over the migration mapping.So Qwen is risky if used only as a pre‑search query generator.
A post‑search validator is safer.
Three Final Recommendations
Option 1: Lowest Complexity
raw chunk
-> BM25
-> top JSONLUsage:
PoC Phase 1
Quickly verify search quality visuallyAdvantages:
Simple
Fast
Transparency
No model costDisadvantages:
false positive exists
meaning search does not work
If the document representation differs, it is missedIn the chunk:
Success is a function
but 3D movement is mixedOption 2: Balanced Type
-> BM25 top 50
-> embedding top 50
-> union
-> weighted score
-> return JSONL
-> Qwen direct-evidence verificationUsage:
The most realistic best intermediate stage for the current projectAdvantages:
BM25 and embedding complement
Remove unrelated JSONL with Qwen
Less hardcodingDisadvantages:
Score tuning required
If there is no reranker, the ranking is slightly unstableIn the chunk:
first_2d_game is reliably superior
3D movement decreases in Qwen verificationOption 3: Quality First
- BM25 top 80
- voyage-code-3 embedding top 80
- union
- rerank-2.5
- Qwen direct-evidence validator
- final JSONLUsage:
Final RecommendationAdvantages:
Best quality
Minimize hardcoding
Maintain raw chunk conditions
The reranker reduces BM25/embedding false positives
Qwen validator removes JSONL without direct evidenceDisadvantages:
Cost
latency
Many implementation stagesIn the chunk:
Final accept:
docs_chunks / first_2d_game / coding_the_player
Final reject or lower-rank:
first_3d_game / player_movement_code
unrelated api_mapping
migration label_prototypesFinal Judgment
If I were to overhaul this project now, I would proceed as follows.
The input maintains only one `chunkText`.
The first search uses BM25.
Reason: It is the most transparent and powerful for code/API string search.
The second search uses the voyage-code-3 embedding.
Reason: Since the query is a code chunk, a code retrieval model is appropriate.
The third ranking uses rerank-2.5.
Reason: To promote the true chunk and matching JSONL to the top among similar candidates retrieved by BM25 and embedding.
The final validation uses the Qwen direct‑evidence validator.
Reason: If there is no string/pattern evidence directly matching the chunk inside the JSONL, it must be discarded.One‑line summary:
Enter the raw chunk as‑is → generate parallel candidates with BM25 + code embedding → reranker re‑ordering → Qwen direct evidence verification is the correct answer.
It is appropriate not to include the Godot API signal extractor.
Even if you add it, it should only be used as a “score bonus.” The primary search should follow your intent by using raw‑chunk‑based BM25/vector.