Reorganized the roles of docs_chunks, api_mapping, and label_prototypes in the Godot source‑code analysis pipeline
docs_chunks is used in the data flow for generating code explanations. It searches official‑document chunks based on AST/code fragments and prompts, discards irrelevant evidence, and creates explanation JSONL.
api_mapping stores how function names, class names, and symbol names changed from Godot 3 to Godot 4.
label_prototypes stores transformation patterns for cases where not only the name changed but the usage, argument composition, and call patterns changed as a whole.
Organized a structure that analyzes a GitHub project or local filesystem by AST/code‑fragment units, calls the necessary Retriever, and then invokes Qwen 3.6 on demand
Official‑document Markdown is classified according to content type and saved as JSONL in one or more of docs_chunks (explanatory text), api_mapping (name/symbol changes), or label_prototypes (usage/call‑pattern changes).
For each fragment, Retriever search results, LLM verification results, and Validator pass results are stored in a JSONL/score‑DB flow.
The columns, aggregation method, and classification labels of the score DB have not been decided yet; currently it is fixed only as a repository for pre‑classification judgment results.
Later, the classified filesystem will be used as source material for designing SFT and DPO.
While documenting the LLM classification method, I discovered that, contrary to the original requirement, the LLM was sending only the first 3,000 characters of the Markdown instead of the whole document, and corrected it to transmit the entire Markdown as originally requested.
This work reaffirmed that documentation is not merely a record but a process for verifying the actual behavior of code.
During Markdown → JSONL conversion, after the RunPod server stopped unexpectedly and the Streamlit app was relaunched, I observed whether the previous progress was retained; it was not reset and continued.
After the app restart, the file that was being processed returned to pending, and when restarted, the existing classification results were reused, allowing processing to resume from the same file.
For long‑running conversion jobs, I noted that an early‑warning notification setting is needed to detect RunPod server termination or unresponsiveness.