idea_world_labDEV JOURNAL
Thursday, June 25, 2026

Godot Source Analysis Scoring Architecture

Date: June 25, 2026

Purpose

Organize the workflow that uses the Godot official documentation RAG to classify GitHub source code by project, and then proceeds to SFT/DPO data generation.

The core of this organization is to treat docs_chunks, api_mapping, and label_prototypes all as the same official‑documentation Markdown → JSONL classification target. Depending on the nature of the document, explanatory chunks go to docs_chunks, function/class/symbol name changes go to api_mapping, and changes in function usage/argument composition/call patterns go to label_prototypes.

Current Conclusion

The three key tables to prepare at the initial official‑documentation collection stage are as follows.

Table Role
docs_chunks Target for storing official‑documentation explanatory text, tutorials, class reference chunks
api_mapping Target for storing how function names, class names, and symbol names changed from Godot 3 → Godot 4
label_prototypes Target for storing how to write when function usage, argument composition, or call patterns change entirely
Item Summary
Existing confusion Initially thought the three tables had separate workflow roles.
New boundary All three tables are storage targets used when classifying official‑documentation Markdown into JSONL.
Current priority Keep the three tables as originally planned and store one or more JSONL files according to the document’s nature.

Final Analysis Unit

Input is a GitHub project or a local project directory.

The analyzer scans the filesystem and splits Godot‑related files into AST or code‑snippet units.

Initial targets:

File Purpose
.gd GDScript AST/line/function‑level analysis
.tscn, .tres Extract clues about class, script, node type inside scenes/resources
project.godot Extract project clues such as version, features, renderer, autoload, etc.
README/documentation Assist with project description and Godot version hints

On‑demand LLM Invocation Flow

The LLM is not kept running continuously; it is invoked on demand for each needed AST/code snippet.

Each AST snippet is passed to the LLM together with the following rationale.

AST/code chunk
+ Search results of required official documentation JSONL
+ Output JSONL schema
+ Project/file identification information

LLM response schema and score DB storage columns have not been finalized yet. In this document, we only record which tables are searched in which flow and how the results are used for filesystem classification.

Code Explanation Generation Data Flow

Code Explanation Generation Data Flow

In code explanation generation, docs_chunks is the core evidence.

Flow:

  1. The user inputs “What does this mean?” and source code.
  2. The AST Parser splits the code into fragments.
  3. The Retriever searches docs_chunks for related official documents.
  4. Qwen 3.6 verifies whether the prompt, code fragment, and retrieved evidence are related.
  5. Unrelated search results are discarded.
  6. Related evidence and code fragments are sent back to Qwen 3.6 to generate an explanation JSONL.
  7. The Validator verifies the response.
  8. The verified result is stored in the score DB.
  9. The score DB results are used to classify the filesystem.

Godot 3 → Godot 4 Function Name Conversion Flow

Godot 3 to Godot 4 Function Name Conversion Flow

In migration, api_mapping is the core evidence.

Flow:

  1. The user inputs “Does this need conversion?” and source code.
  2. The AST Parser extracts candidate Godot API symbols.
  3. The Retriever searches api_mapping for source API, target API, and change type.
  4. Qwen 3.6 verifies whether the search results are actually related to the code fragment.
  5. Unrelated mappings are discarded.
  6. Only related mappings are used to generate a Godot 3 → 4 migration JSONL.
  7. The Validator verifies the conversion response.
  8. The verified result is stored in the score DB.
  9. The score DB results are used to classify the filesystem.

Godot 3 → Godot 4 Usage Pattern Conversion Flow

Godot 3 to Godot 4 Usage Pattern Conversion Flow

In usage pattern conversion, label_prototypes is the core evidence.

Flow:

  1. The user inputs “Does this need conversion?” and source code.
  2. The AST Parser extracts function calls, argument structures, and candidate call patterns.
  3. The Retriever searches label_prototypes for usage/argument/call pattern conversion evidence.
  4. Qwen 3.6 verifies whether the search results are actually related to the code fragment.
  5. Unrelated prototypes are discarded.
  6. Only related prototypes are used to generate a Godot 3 → 4 usage pattern conversion JSONL.
  7. The Validator verifies the conversion response.
  8. The verified result is stored in the score DB.
  9. The score DB results are used to classify the filesystem.

Score DB

Retriever search results per AST fragment, LLM verification results, and Validator pass results are ultimately stored in the score DB.

The score DB is not a source of training data but a repository for judgment results used to classify the filesystem. It records the Retriever/LLM/Validator outcomes produced during source analysis.

  • docs_chunks search results: verification results of whether official document evidence for a code explanation request is actually related to the code
  • api_mapping search results: verification results of whether Godot 3 → Godot 4 function/symbol conversion evidence is actually related to the code
  • label_prototypes search results: verification results of whether function usage, argument structure, and call pattern conversion evidence are actually related to the code

The columns, aggregation method, and classification labels of the score DB have not been decided yet. This document only fixes the role of the score DB as a “repository that gathers pre‑classification judgment results.”

Final filesystem classification:

project source
  -> AST/code chunks
  -> official docs JSONL retrieval
  -> on-demand LLM verification
  -> score DB
  -> classified filesystem

SFT and DPO Generation

The detailed design of SFT and DPO has not been decided yet. What has been confirmed so far is that the score DB itself is not a training data source. First, we will classify the file system, and then we plan to create a source for designing SFT and DPO based on the classified file system.

Git Upload Safety Guidelines

In this documentation, we will not upload the web UI, local conversion scripts, AST analysis scripts, API keys, JSONL outputs, or test source code.

Allowed uploads:

  • Architecture documents
  • Flowchart images
  • Retrospective/roadmap text that can be made public

Prohibited uploads:

  • Streamlit web app code
  • Markdown → JSONL conversion experiment scripts
  • Retriever/AST analysis experiment scripts
  • API key, endpoint, .env
  • Collected full Markdown source zip
  • AST/code chunk analysis results
  • Intermediate JSONL outputs
  • External Godot project source cloned for testing