JSONL Evidence Matching Prompt Observation

Date: June 27, 2026

Purpose

Today, before attaching an actual DB query in the Source Flow Debugger, I had GPT generate a demo Godot code chunk and JSONL candidates, and checked whether the LLM correctly judges relevance/irrelevance based on that JSONL.

The initial idea was simple.

SOURCE_CODE + JSONL Candidate
  -> Does this JSONL contain content corresponding to the source code?
  -> Yes / No

But in practice, this question alone was insufficient. There were cases where the LLM, instead of providing a direct string evidence from the JSONL, used its own knowledge about Godot or broad topic similarity to decide “yes”.

Demo Data Generation

First, we had GPT create an arbitrary Godot chunk along with related JSONL and unrelated JSONL.

The request intention was as follows.

Create an arbitrary Godot chunk and an arbitrary JSONL.  
The Godot chunk is deliberately made with Godot 3 code.  
Create both JSONL containing the basis for Godot 4 conversion and one that is completely unrelated.  
Test whether the difference between the relevant JSONL and the irrelevant JSONL becomes apparent when thrown at an LLM.

Capture:

Initial Prompt Issue

At first, I asked like this.

Does this jsonl contain content that corresponds to source code? Answer only yes / no.

In this approach, even when relevant JSONL is provided, it outputs yes, and when irrelevant JSONL is provided, it also outputs yes.

Relevant JSONL result:

Irrelevant JSONL result:

The problem is that the question was too broad. If you only ask “Does it contain content that corresponds to the source code?”, the LLM can deem it relevant just by seeing broad terms such as Godot, Godot3, Godot4, migration, 2D, physics.

In this case, even when the Retriever fetches an unrelated JSONL, the LLM can fill in from its own knowledge and answer “yes”. This is not the verification we want.

Revised Prompt

So the prompt was changed to a evidence‑matching judge.

You are a JSONL evidence matching judge.

Determine whether the JSONL below directly contains evidence that matches the SOURCE_CODE provided.

Judgment criteria:
- It is a “yes” only if one of the JSONL’s `source_api`, `source_pattern`, `match_terms`, `required_when_seen_in_code`, or `before_code` matches an actual string or API call in the SOURCE_CODE exactly.
- Broad terms such as Godot, Godot3, Godot4, migration, 2D, physics, etc., do not count as related.
- Negative statements in the JSONL such as “does not describe”, “not related”, “unrelated”, “does not apply”, etc., are not accepted as supporting evidence.
- If the JSONL refers to a different API, node, or system, the answer is “no”.
- Do not use any Godot knowledge you have; rely only on the string evidence written in the JSONL.
- The response must be exactly one word: either “yes” or “no”.

In this prompt, the judgment criteria were narrowed to “direct string evidence”.

In particular, it was restricted that at least one of the following fields must match the actual string/API call of SOURCE_CODE directly.

source_api
source_pattern
match_terms
required_when_seen_in_code
before_code

Another LLM was prevented from reasoning with broad keywords and its own Godot knowledge.

Result After Modification

The related JSONL came out as yes.

The irrelevant JSONL came out as no.

Through this result, I reconfirmed that the LLM verification step after DB search must first check whether “the searched JSONL contains a string evidence that directly matches the current chunk,” rather than merely judging simple semantic similarity.

Criteria Gained Today

The DB search result verification prompt should follow these criteria.

• Broad topical similarity is not accepted as evidence.
• Do not let the LLM fill missing evidence with its prior Godot knowledge.
• The explicit fields in the JSONL must directly match the actual string/API call of the current chunk.
• JSONL content mentioned negatively is not accepted as relevant evidence.
• The first‑level related/irrelevant judgment is better kept short and strong, limited to yes or no.

These criteria will be important when attaching DB search tomorrow.

After retrieving candidate JSONL from docs_chunks, api_mapping, and label_prototypes, the Qwen verification step should operate as follows.

chunkText
  -> Collect JSONL candidates via DB search  
  -> prompt + chunkText + retrieved JSONL  
  -> Determine whether the JSONL itself contains a string evidence  
  -> Discard unrelated JSONL  
  -> Use only related JSONL as the basis for subsequent explanation/migration decisions

The core of today's experiment was not “whether the LLM knows correctly,” but “whether we can make the LLM decide only by looking at the evidence written in the JSONL.” If this fails, even irrelevant candidates fetched by the Retriever can be plausibly stitched together by the LLM. Therefore, not only the search quality but also the search result verification prompt itself must be designed separately.