Reorganized the roles of docs_chunks, api_mapping, and label_prototypes in the Godot source‑code analysis pipeline
docs_chunks is used in the data flow for generating code explanations. It searches official‑document chunks based on AST/code fragments and prompts, discards irrelevant evidence, and creates explanation JSONL.
api_mapping stores how function names, class names, and symbol names changed from Godot 3 to Godot 4.
label_prototypes stores transformation patterns for cases where not only the name changed but the usage style, argument composition, and call pattern changed as a whole.
Organized a structure that analyzes a GitHub project or local filesystem by AST/code‑fragment units, calls the necessary Retriever, and then invokes Qwen 3.6 on demand
Official‑document Markdown is classified according to content type and saved as JSONL in one or more of: docs_chunks for explanatory text, api_mapping for name/symbol changes, label_prototypes for usage/call‑pattern changes.
For each fragment, Retriever search results, LLM verification results, and Validator‑pass results are stored in a JSONL/score‑DB flow.
The columns, aggregation method, and classification labels of the score DB have not been decided yet; for now it is fixed as a storage for pre‑classification judgment results.
Later, the classified filesystem will be used as source material for designing SFT and DPO.
While documenting the LLM classification method, we discovered that, contrary to the original requirement, the LLM was sending only the first 3,000 characters of the Markdown instead of the whole document, and we fixed it to transmit the entire Markdown as originally requested.
This work reaffirmed that documentation is not merely a record but a process for verifying the actual behavior of code.
During Markdown → JSONL conversion, after the RunPod server stopped unexpectedly, we observed whether the existing progress persisted when the Streamlit app was relaunched.
After restarting the app, the file that was being processed returned to pending, and when restarted again the existing classification results were reused, allowing processing to continue from the same file.
For long‑running conversion jobs, we noted that an early‑warning notification setting is needed to detect RunPod server shutdown or unresponsiveness.