idea_world_labDEV JOURNAL
Wednesday, June 17, 2026

2026-06-17 Godot RAG Classifier-Based Data Generation Architecture Memo

Architecture Image

Godot RAG Classifier-Based Data Generation Overall Architecture

Core Idea

The core of this architecture is not to entrust the final label decision to the LLM. The LLM takes on a generation‑assistance role such as revised code, explanations, SFT questions/answers, DPO bad answers, patch drafts, while the actual label and the final JSONL assembly/validation are determined by the local system pipeline.

In summary, the principles are as follows.

LLM is generation assistance  
The label is determined by the system  
The final JSONL is assembled/validated by the pipeline

Overall Flow

Document Preparation
-> Build local DB
-> Input source data from GitHub
-> Extract symbols
-> Search rules/vectors/keywords
-> Label scoring and decision
-> Assist LLM generation
-> Generate final JSONL

This structure is a draft for creating Godot 3/4 classification, API mapping, migration fix, instruction SFT, DPO preference, repo explorer, patch generation, and metadata verification data in a single pipeline.

1. Document Preparation

Collect and refine the official Godot documentation in the offline stage.

  • Collect official Godot documentation
  • Remove unnecessary wording
  • Classify document types
  • Parse based on structure
  • Generate document chunks
  • Build embeddings and index

Generated basic deliverables:

docs_chunks.jsonl

2. Three Core Databases Being Built

API Mapping DB

Stores the change relationships between Godot 3 API and Godot 4 API.

Example:

KinematicBody2D -> CharacterBody2D
yield -> await
export var -> @export
move_and_slide(v) -> move_and_slide()

Saved file:

api_mapping.jsonl

Official Document Vector DB

Embed document chunks to create a vector DB for evidence search.

Usage:

  • Search related official document chunks
  • Provide transformation evidence
  • Explain reasons for API changes
  • Reduce hallucinations

Label Prototype DB

Store prototypes for label candidates and similarity search.

Example labels:

godot3_code
godot4_code
mixed_code
broken_code

Saved file:

label_prototypes.jsonl

3. Search/Label Determination

When the source data collected from GitHub arrives, the system first analyzes the code and documentation.

Example input:

repo: owner/repo
file_path: scripts/Player.gd
content: ...
repo_tree: ...

The system extracts symbols.

Example:

KinematicBody2D
move_and_slide(velocity)
export var
yield

After that, perform the following searches.

  • Query the API mapping DB
  • Search the official documentation vector DB
  • Search the label prototype DB

The final label is determined by system scoring, not by the LLM.

Example:

label: godot3
confidence: 0.93
bad_apis:
  - KinematicBody2D
  - move_and_slide(v)
  - export var
replacement_apis:
  - CharacterBody2D
  - move_and_slide()
  - @export

4. LLM Generation Assistance

LLM does not decide the label directly, but assists the generation task by receiving the label and rationale determined by the system as input.

Possible generation tasks:

  • Generate corrected code
  • Generate explanations/rationales
  • Generate SFT questions/answers
  • Generate DPO bad answers
  • Generate file navigation answers
  • Assist in patch generation
  • Validation/problem analysis

The important points are as follows.

LLM generation results are drafts.  
Labels, final schema, confidence, and verification status are managed by the system.

5. Final JSONL Generation

The pipeline assembles system results and LLM-generated results into a single JSON object.

In the validation stage, the following are checked:

  • Presence of required fields
  • Label consistency
  • Residual incorrect APIs
  • Recalculation of confidence
  • Connection between document evidence and output

8 Generated Datasets

1. Version Classification Data

File:

version_classification.jsonl

Content:

  • Classification of Godot 3/4/mixed/broken
  • Determine valid_for_godot4
  • Extract bad_apis

2. API Mapping Data

File:

api_mapping.jsonl

Content:

  • old_api → new_api mapping
  • change_type, category, etc.

3. Transformation/Modification Answer Data

File:

migration_fix.jsonl

Content:

  • before/after code
  • Reason for change
  • Configuration list

4. Question/Answer SFT Data

File:

instruction_sft.jsonl

Content:

  • instruction/input/output
  • samples of various patterns

5. DPO Preferred Data

File:

dpo_preference.jsonl

Content:

  • chosen
  • rejected
  • Reason/Condition

6. Repo Explorer Data

File:

repo_explorer.jsonl

Content:

  • Predict files to read for task/error resolution
  • Reason for reading

7. Patch Data

File:

patch_generation.jsonl
  • before/after
  • unified diff / patch
  • Reason for application

8. Meta/Verification Information

File:

metadata_verification.jsonl

Content:

  • confidence
  • score
  • basis
  • source document chunk ID
  • quality/risk information

Execution Summary Flow

GitHub source data
-> Symbol extraction
-> Rule/DB search
-> Label scoring
-> LLM generation assistance
-> Final JSON assembly and storage

Core Principles

  • The label is determined by the system.
  • The LLM only serves as a generation assistant.
  • The final JSONL is assembled and validated by the pipeline.
  • Store supporting documents, scores, confidence, and source information together.
  • Be sure to check for any remaining incorrect Godot 3 API references.