Godot Official Documentation RAG Classifier Schema and Architecture

Date: June 21, 2026

Purpose

Summarize the reference architecture for connecting the full collection of Godot official documentation through the flow JSONL -> PostgreSQL -> Retriever -> Validator -> Qwen 3.6. This document is not an implementation guide for creating training data, but a design guideline that defines how to structure the already‑collected official documentation Markdown in outputs/godot_docs_full/pages and turn it into a searchable evidence database.

Current Input Data

The results of the documentation collection are located under the repository’s outputs/godot_docs_full.

Current baseline:

Full Pipeline

1. The crawler collects the Godot official documentation from the internet.
2. The collection results are saved as page‑level Markdown files.
3. The Markdown is normalized and metadata is attached, then converted to JSONL records.
4. The JSONL is injected into PostgreSQL.
5. PostgreSQL stores document chunks, API mappings, and label prototypes separately.
6. When a user requests Godot source‑code analysis, the AST Parser structures the project code.
7. The Retriever searches for evidence in PostgreSQL based on the user question and AST analysis results.
8. The Validator bundles the question, AST results, and retrieved evidence and sends them to Qwen 3.6.
9. Qwen 3.6 is not the final judge; it compiles an answer using the verified evidence.
10. After the Validator checks the response’s evidence/format/forbidden patterns, it returns the result to the user.

Converting Markdown to JSONL

pages/*.md are human‑readable page sources, which are too large for RAG. Therefore, the conversion script splits pages into section/API‑member/example units and preserves the original URL and document type.

Common Normalization Rules

JSONL Outputs

docs_chunks.jsonl Schema

{
  "chunk_id": "godot-stable:classes/class_node.html#description:0001",
  "doc_version": "stable",
  "source_url": "https://docs.godotengine.org/en/stable/classes/class_node.html",
  "source_file": "outputs/godot_docs_full/pages/classes__class_node__....md",
  "source_sha256": "...",
  "doc_type": "class_reference",
  "symbol": "Node",
  "section_path": ["Node", "Description"],
  "heading": "Description",
  "content": "Nodes are Godot's building blocks...",
  "code_blocks": [],
  "language_tags": ["gdscript"],
  "godot_version_tags": ["4.x", "stable"],
  "api_symbols": ["Node", "_ready", "_process", "queue_free"],
  "token_count": 420,
  "metadata": {
    "status": "copied_old",
    "bytes": 12345
  }
}

Essential fields:

api_mapping.jsonl schema

{
  "mapping_id": "godot3-to-4:kinematicbody2d-to-characterbody2d",
  "source_api": "KinematicBody2D",
  "target_api": "CharacterBody2D",
  "change_type": "rename_or_replacement",
  "godot_from": "3.x",
  "godot_to": "4.x",
  "confidence": "verified_from_docs",
  "evidence_chunk_ids": [
    "godot-stable:tutorials/migrating/upgrading_to_godot_4.html#..."
  ],
  "match_terms": ["KinematicBody2D", "CharacterBody2D"],
  "notes": "Godot 4 character movement node replacement candidate.",
  "negative_patterns": ["do not suggest KinematicBody2D for Godot 4 projects"]
}

Principles:

label_prototypes.jsonl schema

{
  "prototype_id": "label:godot3-api-in-godot4:kinematicbody2d",
  "label": "godot3_api_in_godot4",
  "task_type": "version_classification",
  "input_pattern": "extends KinematicBody2D",
  "expected_finding": "Godot 3 style physics body API detected.",
  "recommended_action": "Use CharacterBody2D or CharacterBody3D depending on project dimension.",
  "evidence_mapping_ids": [
    "godot3-to-4:kinematicbody2d-to-characterbody2d"
  ],
  "evidence_chunk_ids": [],
  "severity": "high",
  "validator_rules": {
    "requires_ast_symbol": "KinematicBody2D",
    "forbidden_answer_terms": ["KinematicBody2D is recommended in Godot 4"]
  }
}

Label candidates:

Draft PostgreSQL schema

PostgreSQL assumes the use of pgvector. Keyword search uses tsvector or a trigram index.

docs_chunks

Indexes:

api_mapping

Indexes:

label_prototypes

Indexes:

AST Parser input/output

The AST Parser converts user source code into a searchable structure before passing it directly to an LLM. Initial targets are .gd, .tscn, project.godot.

Input

Output schema

{
  "project_id": "local-analysis-...",
  "godot_project": {
    "config_version": 5,
    "features": ["4.4", "Forward Plus"]
  },
  "files": [
    {
      "path": "scripts/player.gd",
      "language": "gdscript",
      "extends": "CharacterBody2D",
      "class_name": "Player",
      "symbols": ["CharacterBody2D", "Input", "move_and_slide"],
      "annotations": ["@onready"],
      "version_signals": ["godot4_annotation_syntax"],
      "diagnostics": []
    }
  ],
  "version_evidence": {
    "godot4": ["config_version=5", "@onready"],
    "godot3": []
  }
}

Initial extraction fields:

Retriever operation

The Retriever is a layer that selects evidence before the LLM.

1. Extract the intent and target task from the user question.
2. Retrieve API symbols, version signals, and file paths from the AST Parser results.
3. Perform an exact lookup in api_mapping first.
4. Perform API symbol filtering + keyword search + vector search together in docs_chunks.
5. Retrieve similar labels and validation rules from label_prototypes.
6. Sort the search results into evidence bundles.

Evidence bundles:

{
  "query_id": "analysis-...",
  "task_type": "version_classification",
  "ast_summary": {},
  "doc_evidence": [],
  "api_mapping_evidence": [],
  "label_evidence": [],
  "retrieval_scores": {
    "exact_api_hits": 2,
    "keyword_hits": 8,
    "vector_hits": 12
  }
}

Separation of Roles between Validator and Qwen 3.6

Qwen 3.6 is a model that reads retrieved evidence and organizes the answer. The final label, evidence adoption, and prohibited‑pattern verification are handled by the Validator.

Items the Validator checks:

Injection Order

1. Check outputs/godot_docs_full/summary.json and failed.json.
2. Load manifest.json and pages/*.md.
3. Create a Markdown normalization report.
4. Generate docs_chunks.jsonl.
5. Produce api_mapping.jsonl candidates from migration/class documents.
6. Create label_prototypes.jsonl from approved rules and representative cases.
7. Upsert only records that pass JSONL schema validation into PostgreSQL.
8. Generate embeddings and update the vector index.
9. Refresh the keyword index and the exact index.
10. Validate Retriever results with sample questions.

Quality Assurance Checklist

Implementation Priorities

1. Write a report analyzing the structure of pages/*.md.
2. Write a conversion script for docs_chunks.jsonl.
3. Write a JSONL schema validation script.
4. Write PostgreSQL DDL.
5. Inject docs_chunks and verify search.
6. Generate api_mapping candidates and draft the manual approval flow.
7. Draft the initial label set for label_prototypes.
8. Extract minimal fields with the AST Parser.
9. Output Retriever evidence bundles.
10. Connect the Validator + Qwen 3.6 response‑organizing loop.

Core Principles

• Preserve the original Markdown of official documents without modification.
• Treat JSONL as a reproducible intermediate artifact.
• The database must retain the original path, original URL, hash, and conversion script version.
• Prevent the LLM from inventing labels.
• Godot 3/4 determination is made by combining AST signals, API mappings, and official‑document evidence.
• Uncertain candidates are kept in a candidate state rather than being used directly as training data.
• Qwen 3.6 is the answer organizer; the judgment criteria belong to the Retriever and Validator.

Next Tasks

The next step is to perform a sample analysis of the Markdown structure in outputs/godot_docs_full/pages. Because class reference, migration, and tutorial documents have different structures, we need to first decide chunking strategies per document type rather than forcing a single chunking rule.