idea_world_labDEV JOURNAL
Sunday, June 28, 2026

Retriever Search Alternative Decomposition Document

Date: June 28, 2026

This directory is a set of documents split for easier viewing of the Retriever Search Alternative ChatGPT Original Memo.

The original memo is preserved as‑is. This README combines the common context, current approach, model candidates, and selection guide. Sub‑documents contain only the alternative‑specific documentation.

Comprehensive Comparison Table

These tables are not final choices; they are observation tables for judging the search alternatives that need to be compared on June 28 on a single screen. The criterion is “when the code chunk is fed directly to the Retriever, which JSONL candidates appear, and can those candidates be accepted as direct evidence in the Qwen verification stage.”

One‑Glance Conclusions

Alternative Location Expected Role Ready‑to‑Use Level Core Advantage Core Risk Current Assessment
A. PostgreSQL full‑text maintenance Current /api/retrieve baseline Check how far the currently implemented search can go High Already attached and logs are immediately visible Long code chunks become queries and related documents may be omitted Keep as baseline, but do not declare it the final search method
B. BM25 only First‑stage search candidate Find documents that contain the code/API strings directly Medium Strong on literal strings such as KinematicBody2D, move_and_slide, AnimatedSprite2D Broad terms like 2D/3D cause false positives Best for the initial PoC because it is easy to observe
C. embedding only Meaning‑based recall expansion Find relevant explanatory documents even when the strings are not identical Medium Captures sentence meaning, tutorial explanations, and similar usage contexts broadly High risk when used alone for version/API accuracy judgments Use as a supplemental recall rather than standalone
D. BM25 + embedding Pragmatic middle ground Combine literal and semantic evidence Medium Leverages BM25 precision and embedding recall together Requires score combination, deduplication, and top‑candidate criteria Closest to a minimal viable production setup
E. Qwen query profile Search‑assist experiment Generate a JSON query intent from a code chunk Low Human‑readable query profiles are easy to understand LLM may fabricate APIs/intents that do not exist before the search Use as an experiment/assist rather than the primary first‑stage search
F. BM25 + embedding + reranker + validator Final candidate architecture Connect candidate generation, re‑ranking, and direct evidence verification Low Offers the best quality and observability Cost, latency, and many stages Designated as the ultimate target candidate

PoC Simulation Result Comparison

Alternative Input chunk How search candidates are produced Likelihood of relevant JSONL appearing Likelihood of irrelevant JSONL mixing in Expected state before Qwen verification Expected state after Qwen verification
A Entire chunkText Generates a long query with plainto_tsquery('simple', chunkText) Low or unstable Could be low, but if 0 results there are no observable candidates No candidates or only partial returns May be omitted before direct‑evidence verification
B Entire chunkText Ranks documents by token‑wise BM25 scores High Medium Documents such as Input.is_action_pressed, AnimatedSprite2D, clamp may appear Dropped if the exact string is absent
C Entire chunkText Code/document embedding similarity Medium High Broad documents like “movement”, “2D”, “player” explanations appear Dropped if the JSONL lacks the exact string
D Entire chunkText Merges BM25 and embedding candidates High Medium Both literal and semantic candidates are shown together Only candidates with direct evidence remain
E Entire chunkText interpreted by Qwen Search using the profile Qwen creates Medium if profile matches High APIs not in the profile may mix in The profile itself must be validated against the actual code
F Entire chunkText Reranker re‑orders BM25 + embedding candidates High Low Many candidates, but top ones are curated Qwen leaves only “yes/no” candidates with direct evidence

Differences Revealed in Demo Judgments

Observation Situation Initial Simple Prompt Improved Direct‑Evidence Prompt Meaning for Search Structure
Godot 3 code + related JSONL “Yes” could be returned “Yes” should be returned Passes if the supporting evidence matches the actual string/API
Godot 3 code + unrelated JSONL Broad Godot terms could yield “Yes” “No” should be returned Risky if candidates appear without direct‑evidence verification
KinematicBody2D code + 3D Spatial mapping Might be considered related under the broad notion of Godot migration “No” is correct Embedding‑only or loose LLM judgments can cause false positives
yield(...) code + await prototype “Yes” when the exact pattern exists “Yes” label_prototypes can serve as evidence for whole‑pattern changes
AnimatedSprite2D.play() code + generic animation docs May seem semantically related “No” if the JSONL lacks the exact string/pattern Docs chunks also need direct‑evidence checks for final decisions

Candidate Return Pattern Comparison

Alternative Candidates that rank high Candidates easily missed Candidates prone to mixing incorrectly Visual check points
A Documents with many query tokens within a single row When related tokens are split across multiple documents Almost none if the query is too strict Why 0 results? Which token broke the match?
B Rows containing the API name, function name, or node name directly Re‑phrased explanatory documents Documents from other systems that share the same term Whether match_terms match the actual source chunk string
C Tutorials/explanations with similar meaning Short exact API mappings Docs where Godot version or 2D/3D differs Whether direct string evidence exists beyond cosine score
D Exact candidates + semantic candidates Candidates that lose out in score combination Candidates that match weakly on both searches Whether the candidate came from BM25 or embedding
E Candidates matching the intent Qwen extracted Real code clues that Qwen omitted from the profile Candidates about imagined APIs/intents Whether each field of the profile actually exists in the chunk
F Candidates the reranker deems matching the chunk Godot‑specific edge cases unknown to the reranker model High reranker score but no direct JSONL evidence Points where rerank score and validator judgment diverge

Comparison of False Positive / False Negative

Alternative false positive example false negative example Mitigation method
A Rarely, if the query is too broad, general movement documents get mixed in If related documents are split across multiple rows, 0 results Change the query generation method or add BM25 candidates
B movement, player, 2d cause other movement documents to be mixed in If the API name appears only descriptively in the document, it is missed Separate stopwords/weak tokens, weight API tokens directly
C Documents of similar meaning such as 3D/other versions get mixed in Short API rename rows are not captured Use alongside BM25 and attach a validator
D Candidates that match weakly on both BM25 and embedding get mixed in If the fusion criteria are wrong, one side’s candidates get suppressed Keep a union source flag and a direct evidence log
E Qwen creates a migration_intent that does not exist, contaminating candidates Qwen omits actual code clues from the profile Do not trust the profile before search; use it only as a supplement
F The reranker overestimates plausible explanations Documents that were never retrieved in the candidate generation stage Secure enough candidates and place Qwen verification at the end

Suitability per Table

Alternative docs_chunks api_mapping label_prototypes Remarks
A Unstable with long explanatory documents May miss if the exact‑mapping row is short May miss if the pattern row is short Baseline for confirming the limits of the current approach
B Good for API/heading/token search Good for exact source/target API search Good for source/target pattern search All three tables are useful as first‑search candidates
C Good for recall of descriptive documents Weak for short rows like API rename Can supplement when usage patterns are similar Most natural as a supplement to docs_chunks
D Captures both descriptive and string candidates Captures exact‑mapping and semantic candidates together Broadly captures whole‑pattern candidates Common structure across the three tables, most straightforward
E If the query profile is good, explanatory document search is possible Risky if hallucinated APIs appear Can create prototype intent but may be contaminated More of an observation/supplement experiment than core search
F Good for final explanation evidence refinement Good for migration evidence refinement Good for usage‑change evidence refinement Convenient for handling all three tables in the same pipeline

Logs That Must Be Checked

Alternative Pre‑search log In‑search log Post‑search log Failure‑diagnosis points
A raw chunkText, generated tsquery Number of hits for search_tsv @@ query, ts_rank_cd Returned rows, scores Whether the query became too long, which tokens are missing from rows
B raw chunkText, token list Term frequency, matched terms, BM25 score Top‑k per table Whether strong and weak tokens are distinguished
C raw chunkText, embedding model Vector distance / cosine score Semantic top‑k Whether the candidate matches only in meaning without actual strings
D BM25 top‑k, embedding top‑k Union, duplicate merge, score fusion Candidate list with source flags Why candidates retrieved by only one search survived
E Raw chunk sent to Qwen Qwen query profile JSON Profile‑based search results Whether the profile fields actually exist in the source
F All BM25/embedding candidates Rerank score, rerank order Qwen “yes/no”, reject reason Why candidates that passed the reranker failed the direct‑evidence verification

Implementation Complexity and Operational Cost

Alternative Implementation effort DB changes External model cost Latency Debugging difficulty Ease of fixing when it fails
A Low Almost none None Low Low Low, though performance limits may be structural
B Medium Requires BM25 search engine / index selection None or low Low Low High – token/weight adjustments are possible
C Medium Needs embedding column / index Yes Medium Medium Medium – model/dimension/threshold adjustments needed
D Medium‑high Needs both BM25 and vector support Yes Medium Medium High – can isolate which search path failed
E Medium Storing the profile is optional Yes High High Low – reproducing why an LLM profile was wrong can be hard
F High Requires BM25 + vector + rerank logs Yes High High High – step‑by‑step logs reveal where to fix

Where to Place Qwen

Position Advantages Risks Currently more suitable usage
Query generation before search Can create search intent without hard‑coding May generate nonexistent clues and contaminate search Use only for experiments
Direct‑evidence verification after candidate retrieval Can filter results that do not match actual code Incur call cost Considered most important
Final answer generation Can produce code explanations / migration responses Wrong evidence leads to wrong answers Place after verified JSONL candidates
Integrated overall judgment Can produce a conclusion that is easy for humans to read May hide intermediate failures Use after sufficient logs have accumulated and are reviewed

Selection Priority

Priority Choice Reason Questions to verify at this stage
1 Measure against A as baseline Need to know where the current implementation breaks How many rows does the current method actually return?
2 Attach B as a separate PoC Code/API string search is the most explainable Does it find the correct rows for KinematicBody2D, yield, move_and_slide?
3 Compare C as a supplemental recall Can broaden descriptive candidates from docs_chunks How many meaning‑based candidates are rejected by direct‑evidence verification?
4 Generate candidates in parallel with D Realistic minimal structure Does quality improve when BM25 and embedding candidates are combined?
5 Add reranker/validator with F Verify final quality Does Qwen’s direct‑evidence judgment reliably discard unrelated JSONL?
Hold E LLM before search carries high contamination risk Is the profile created solely from actual code strings?

PoC Checklist

Check Item Screen/Log to View Success Criteria
Put the same chunkText into multiple alternatives Chunk input area of the web debugger The input does not change across alternatives
Table-specific search button docs_chunks, api_mapping, label_prototypes buttons Differences in candidates per table are visible for the same chunk
Verify raw candidates Returned JSONL payload Fields matching the actual string inside the source chunk are shown
Verify Qwen validation prompt + chunk + jsonl debug area Relevant candidates show “yes”, unrelated candidates show “no”
Verify reject reason Validator result log Candidates that only match broad words like Godot, migration, 2D are rejected
Collect false positives List of rejected candidates Can be used later to improve query/token/validator rules
Save comparison per alternative Observation log or result JSON Differences from A‑F remain for the same input

Alternative Documents

Current Premise

The current criteria are as follows.

1. Inside the zip there are the original markdown, schema, debugger, and design documents.  
2. The local environment already contains the markdown → JSONL → DB conversion.  
3. What can be seen now is the pre-conversion markdown and the DB schema/search flow.  
4. The PoC should split the markdown inside the zip into JSONL-like chunks and simulate search targets that would have been inserted into the local DB.  
5. What is needed is not the code skeleton, but tracking how each search method generates scores and why it succeeds or fails when a chunk is inserted.

JSONL/DB Criteria

The deliverables based on the document are as follows.

work/godot_rag/jsonl/docs_chunks.jsonl
work/godot_rag/jsonl/api_mapping.jsonl
work/godot_rag/jsonl/label_prototypes.jsonl
work/godot_rag/jsonl/ingest_report.jsonl

The DB is understood not as a structure where an entire JSONL line is placed into payload jsonb, but as a structure where JSONL fields are expanded into table columns.

docs_chunks

chunk_id
doc_version
source_url
source_file
source_sha256
doc_type
symbol
section_path
heading
content
code_blocks
language_tags
godot_version_tags
api_symbols
token_count
metadata
embedding
search_tsv

api_mapping

mapping_id
source_api
target_api
change_type
godot_from
godot_to
confidence
evidence_chunk_ids
match_terms
notes
negative_patterns

label_prototypes

prototype_id
label
task_type
input_pattern
expected_finding
recommended_action
evidence_mapping_ids
evidence_chunk_ids
severity
validator_rules
embedding
search_tsv

Standard chunk

The chunk used as a standard in the alternative comparison is the following Godot code.

func _process(delta):
	var velocity = Vector2.ZERO
	if Input.is_action_pressed(&"move_right"):
		velocity.x += 1
	if Input.is_action_pressed(&"move_left"):
		velocity.x -= 1

	if velocity.length() > 0:
		velocity = velocity.normalized() * speed
		$AnimatedSprite2D.play()
	else:
		$AnimatedSprite2D.stop()

	position += velocity * delta
	position = position.clamp(Vector2.ZERO, screen_size)

Strong token:

animatedsprite2d
is_action_pressed
move_left
move_right
normalized
clamp
screen_size
vector2
zero

Weak token:

func
var
if
else
velocity
position
delta
x
0
1

Search input remains exactly chunkText. The approach of extracting required Godot API signals before the search and converting them into a separate query is not considered a core part of the basic design. If needed, it will be reviewed later only as an auxiliary score.

Current /api/retrieve method

The current project's /api/retrieve is not BM25.

The flow is close to the following.

chunkText
  -> plainto_tsquery('simple', chunkText)
  -> search_tsv @@ query
  -> ts_rank_cd(...)

PostgreSQL's ts_rank_cd is a full-text ranking function. Unlike the search-engine style BM25 itself, which combines term frequency, inverse document frequency, and field length normalization, it should be viewed differently.

plainto_tsquery(raw_chunk) extracts tokens from the input chunk to create a query. The reference chunk can become a long query as follows.

func & process & delta & var & velocity & vector2 & zero
& input & is_action_pressed & move_right & move_left
& length & normalized & speed & animatedsprite2d
& play & stop & position & clamp & screen_size

This method may require too many tokens within a single row.

If the official documentation chunk is divided as follows, a problem arises.

chunk A:
  Input.is_action_pressed
  velocity.normalized
  AnimatedSprite2D.play/stop

chunk B:
  AnimatedSprite2D description
  position.clamp
  screen_size

Both are related, but a single row may not contain all tokens at the same time.

In the original note, it is considered that a strict raw AND type search could fail as follows.

strict raw AND hits: 0

The current method can be seen as a baseline for the PoC, but it is weak to maintain as the final search strategy. In the next step, the current method and BM25 should be separated and compared.

Model Candidates

voyage-code-3

It was mentioned as the embedding candidate that best fits the current project.

Reason:

The query is not a natural language question but a GDScript code chunk.  
The target JSONL for search also contains mixed code_blocks, API names, and documentation descriptions.

Advantages:

  • Strong at code retrieval.
  • Advantageous for processing long chunks.
  • Suitable for code → docs search.

Disadvantages:

  • Depends on external APIs.
  • incurs cost.
  • Not a Godot‑specific model.

Memory‑based recommendation:

voyage-code-3 1024 float

OpenAI text-embedding-3-large

A versatile high‑quality embedding candidate.

Pros:

  • Strong for general semantic retrieval.
  • Stable for document description search.
  • Easy to integrate with the OpenAI ecosystem.

Cons:

  • Not dedicated to code retrieval.
  • The default dimensionality is large, which can increase storage/indexing costs.
  • For GDScript chunk → JSONL search, it may be less direct than voyage-code-3.

Memory‑based recommendation level:

Second priority

Gemini Embedding

General/multilingual semantic retrieval candidate.

Pros:

  • Good for document‑description style retrieval.
  • Offers dimensionality‑reduction options.
  • May have strengths in multilingual document search.

Cons:

  • Not a dedicated option for code retrieval.
  • Prioritizes semantic similarity over API code accuracy.
  • Using it alone for exact Godot migration decisions can be risky.

Jina embeddings v4

A candidate strong in complex, multimodal, visually rich document retrieval.

Pros:

  • Broad coverage for document search.
  • Strong with multimodal/compound documents.
  • Code‑adapter series can also be considered.

Cons:

  • The current project focuses on markdown/code.
  • Image/table retrieval is not central.
  • At this stage it may be an excessive choice.

Recommendation based on memo:

Currently, the priority is low.

rerank-2.5

It was mentioned as a reranker candidate.

Role:

BM25 and embedding compare the candidates again with the raw chunk.  
Reorder the candidates based on relevance scores.

This project expects the following effects:

  • Reduce false positives for 2D movement and 3D movement.
  • Better reflect direct context such as AnimatedSprite2D, Vector2, screen_size, position.clamp.
  • Organize the candidates brought by BM25 and embedding into the final candidates.

Note:

  • The reranker is not a validator.
  • Direct evidence verification must be done again by the Qwen direct‑evidence validator.

Summary of Options

Alternative Flow Advantages Risks
A Keep PostgreSQL full‑text Already implemented May return zero results for long chunks, not BM25
B BM25 only Transparent, strong on strings/APIs False positives such as 3D movement
C embedding only Enables semantic search API/version accuracy is weak
D BM25 + embedding Realistic balance Score combination tuning required
E Qwen query profile Reduces hard‑coding, can summarize intent Hallucination, cost, pre‑search contamination
F BM25 + embedding + reranker + validator Quality first Cost, latency, many stages

Current Interim Judgment

The conclusion of the original memo is close to the following.

The input retains only one `chunkText`.

The first search uses BM25.  
  Reason: It is transparent and strong for code/API string search.

The second search uses the voyage-code-3 embedding.  
  Reason: Because the query is a code chunk, a code retrieval model is appropriate.

The third ranking uses rerank-2.5.  
  Reason: To lift the truly matching JSONL of the actual chunk to the top among similar candidates brought by BM25 and embedding.

The final verification uses the Qwen direct‑evidence validator.  
  Reason: If there is no string/pattern evidence in the JSONL that directly matches the chunk, it must be discarded.

One-line summary:

Enter raw chunk as is
  -> Generate parallel candidates with BM25 + code embedding
  -> Rerank reordering
  -> Qwen direct evidence verification

What to actually compare next

  • How many results the current PostgreSQL full-text method returns from the reference chunk
  • Whether BM25 actually ranks first_2d_game at the top
  • How much 3D movement false positives are mixed in with BM25 only
  • What explanatory documents embedding only retrieves additionally
  • How to combine duplicates and scores in the BM25 + embedding union
  • Whether the reranker properly reorders 2D/3D candidates
  • Whether the Qwen direct-evidence validator discards unrelated JSONL as No in practice