2026-06-26 Retrospective | idea_world

Today, while documenting the source-to-AST input flow, I organized “how to actually pass the source code to LLM judgment.”

It wasn’t just a day of writing documentation. I’m also collecting Godot‑related data in real time, so the document couldn’t remain an abstract design diagram. I needed a criterion to verify which path the collected data takes, how it is chunked, and what request it should lead to.

Why I Wanted to Organize This Design First

The reason I wanted to document this design first was the anxiety that, in previous work, the AI could write code strangely or arbitrarily change the scope of a request.

In particular, I was worried that the implementation might send only a part of the beginning while the whole Markdown needs to be sent, that it might insert hard‑coded parts I didn’t want, or that it might require looking at the entire file but end up sending only the “core code.”

Therefore, instead of jumping straight into coding, I wanted to fix in a document: “what request to give the AI,” “how to split the files,” “what to feed the AST Parser and what to feed the Retriever directly,” and “what combination should go into the LLM call.”

These days, I find myself thinking more about how to instruct the AI so that the intention is less distorted than about the implementation itself. Even for the same requirement, the result varies greatly depending on how the scope is fixed, which expressions are prohibited, and what evidence the AI is asked to verify.

This document is less a simple development plan and more a baseline to prevent the request from becoming vague when later delegating implementation to the AI. It is a pre‑emptive organization of thoughts to keep the AI from arbitrarily reducing input, sending only core code, or modifying files that were not requested.

Current Standard

I concluded that source code needs to be expanded using headers of the form # <relative‑path>/<relative‑path>. The purpose of these headers is not decorative in the prompt but to track where each file came from and how it is chunked before reaching the Retriever.

From the start, I imagined a structure that unfolds a GitHub repository in path order, excludes files that should be excluded while showing what was excluded in a web UI or log, then splits the remaining files into readable units and continuously tracks which original file each chunk originated from.

During documentation, the AI repeatedly misinterpreted a binary choice: “send the whole file at once or only the core code?” My approach is different: expand the file by path, then cut it into ordered chunks before sending.

In my standard, .gd files are the targets for the AST Parser. The AST Parser must sequentially cut out parseable units such as the a function, b function, c function, d function inside a .gd file. When function boundaries are ambiguous, the file can be cut into code‑snippet units in order. The key is not to “arbitrarily pick only core functions,” but to chunk while preserving the original order and path, and to track the chunk’s passage to the Retriever.

Conversely, files like .md that we decided to exclude from this source‑code analysis must appear on an exclusion list. The important point is that the fact of exclusion must not disappear. The UI or logs must show why a file was excluded. If there are textual configuration files that should not be excluded, it is natural to bypass the AST Parser and split them into lines, paragraphs, key‑value pairs, nodes/resources/connections, etc., and feed them directly to the Retriever.

In summary, the current flow looks roughly like this.

repository path order
  -> "# <relative/path>" file expansion for tracking
  -> excluded file list visible in UI/log
  -> .gd files to AST Parser
  -> AST Parser emits ordered function/code chunks
  -> excluded files are recorded with reason
  -> non-.gd allowed text/config files emit ordered chunks
  -> each chunk goes to Retriever with source path metadata
  -> prompt + current chunk + retrieved evidence
  -> LLM judgment
  -> validation
  -> accumulated project-level result

Parts of AI Interpretation That Were Fixed

In the middle, the AI created a separate structure like repository_file_manifest, which was not the direction I had in mind. What is needed is not a separate manifest repository, but to track, based on the file flow expanded with a # <relative path> header, which files were excluded, which files were split into which pieces, and in what order those pieces entered the Retriever.

It would be odd to say that binaries or assets are “not read.” The point is that the original bytes are not fed to the LLM; it does not mean the file disappears completely from the project's assessment. If an asset path appears inside a text piece such as .tscn, .import, or README, you can make the judgment based on that piece.

Another aspect I felt was needed is transmission debugging. The AI should not flow by picking only the core functions from a .gd file and sending them. The file expanded with # <relative path> must be split by the AST Parser into functions/code pieces, and it should be possible to verify, using diff or sha256‑like methods, which piece went to the Retriever and in which order.

Points Summarized While Reviewing the PR

While looking at the PR review, I realized that llm_judgment_request was defined differently in two places in the documentation. One side focused on chunk_text, chunk_kind, retrieved_evidence, while the other also included chunk_code, surrounding_context, and judgment_contract.

When the same request object is defined inconsistently in the docs, it will inevitably cause confusion during implementation. The request schema must be unified. In particular, to distinguish AST chunks from direct‑retrieval chunks, chunk_kind is required, and to verify LLM judgments, both retrieved_evidence and judgment_contract need to be documented together.

In the Qwen review, I also saw comments about a README link, a typo related to the June 25 document, and link issues. However, the June 25 document and workflow appearing in the PR diff were not changes I intended. Arbitrary modifications made by the AI were mixed in, and after discovering them I demanded a rollback.

Issue of Mixed Scope

In today’s PR diff I discovered that the June 25 document and .github/workflows/qwen-code-pr-review.yml were included. Those were not changes I requested. It looked as if the AI had arbitrarily modified or deleted files, and the PR view showed those files as changed or removed.

.github/workflows/qwen-code-pr-review.yml
docs/observations/2026-06-25-qwen-markdown-classification-observation.md
docs/retrospectives/2026-06-25-source-analysis-scoring.md
docs/roadmaps/2026-06-25-qwen-pr-review-workflow.md

I therefore asked for those changes to be reverted. The files were restored to the origin/main state, and the PR diff was narrowed to focus on README.md and docs/roadmaps/2026-06-26-source-to-ast-input-flow.md.

Seeing this process made me realize that even when assigning documentation work to an AI, the change scope must be locked down firmly from the start. Writing retrospectives or roadmap documents should be managed like code changes, with a clear PR scope.

Today’s Conclusion

What I re‑confirmed today is that the core question is “how a file expanded by path is excluded, how it is split into pieces, and how those pieces flow— in what order—through the Retriever and LLM judgment.” It is not about dumping the whole file at once, nor about arbitrarily picking only the core code.

The current guidelines are set as follows:

Files are expanded with a # <relative path> header, which is used for the original path and piece‑tracking.
Excluded files must be visible in the web UI or logs.
.gd files are sent to the AST Parser, which creates functions or code pieces in the original order.
Files that are to be excluded, such as .md, remain on an exclusion list and must be visible in the web UI or logs.
Text‑based configuration files that are not excluded are chunked without passing through an AST, using units like lines, paragraphs, or config blocks.
Each chunk carries tracking information such as source path and chunk order when it goes to the Retriever.
LLM calls are performed repeatedly in the unit of prompt + current chunk + Retriever search results.
Project judgment is considered possible only after the results of multiple chunks have been accumulated from the start.

Data collection is ongoing. Therefore, future documentation should not be merely a “plausible architecture” but a criterion that shows how actual collected data is fed in and how it is validated.