Observation Log for Selecting Test Repository Based on JSONL Collection Status

Date: June 27, 2026

Purpose

Based on the JSONL generated so far, determine which type of Godot repository should be cloned first for testing.

The goal here is not “to pick any famous repository.” First, we examine which areas the evidence currently accumulated in the DB is strong in and which areas are still weak, and then choose a repository type that the Retriever can actually find evidence for.

The analysis criteria are the following two files.

/Users/joyeongjin/make_md/work/web_jsonl/docs_chunks/docs_chunks.jsonl
/Users/joyeongjin/make_md/work/web_jsonl/api_mapping/api_mapping.jsonl

state.json contains conversion status, classification prompts, and original response debug, but it may also be mixed with runtime settings, so the original text is not copied into public documentation. This observation log only retains JSONL statistics and judgments.

Snapshot Status

The conversion as of the observation point is still in progress.

docs_chunks.jsonl: 153 records  
api_mapping.jsonl: 117 records  
label_prototypes.jsonl: No file created yet

Web conversion status roughly followed this flow.

done: 132 files
pending: 1425 files
deferred: 11 files
converting: 2 files
classified: 144 files

Therefore, this document is not a final conclusion but the “first selection criteria for reports based on the JSONL collected so far.”

Observation of docs_chunks

docs_chunks had 153 records generated.

The composition is close to the following.

class_reference: 127
about: 26

Chunks were created from 95 original Markdown files, and 129 records contained an API symbol. Records with a code block were 36.

So far, the areas that stand out strongly in docs_chunks are as follows.

Animation series  
Audio series  
Camera series  
Canvas/CanvasItem series  
Basic APIs such as Array/Callable/GlobalScope  
Some AnimatedSprite2D  
Some AStar/AStarGrid2D

Conversely, the areas that still appear weak are as follows.

KinematicBody / CharacterBody migration  
move_and_slide call method  
yield → await conversion  
Color8 → Color.from_rgba8 conversion  
TileMap migration  
RigidBody / physics body migration  
HTTPRequest / networking

Immediately, the current docs_chunks can be used to some extent as “evidence from the official Godot 4 documentation,” but it is still difficult to say that sufficient evidence has been accumulated for a typical Godot 3 → Godot 4 2D action game migration.

Observation of api_mapping

api_mapping had 117 records created.

The types of changes are divided as follows.

rename_or_replacement: 82
behavior_change: 24
deprecated: 11

All records' confidence was entered as verified_from_docs.

The strongly evident mapping areas so far are as follows.

Array methods and behavior differences  
AnimationPlayer / AnimationMixer / AnimationTree / AnimationNode series  
AudioStream / AudioEffect / AudioEffectSpectrumAnalyzer series  
Camera / Camera3D / CameraAttributes series  
CanvasItem draw/modulate series  
Some functions and deprecated constants in GlobalScope  
Some in AStarGrid2D

For example, the current JSONL contains evidence of the following nature.

Camera -> Camera3D  
Camera.get_transform -> Camera3D.get_camera_transform  
WorldEnvironment exposure setting -> CameraAttributes series  
Some AnimationPlayer features -> Split into AnimationMixer  
AnimationNode._process -> AnimationNodeExtension._process_animation_node  
AudioEffectLimiter -> AudioEffectHardLimiter  
Array.sort behavior change  
C# name difference of Array.has  
CanvasItem.modulate behavior change  
CanvasItem.draw_string / draw_rect behavior change

This means that if you run a migration test right now, the code related to Camera, Animation, Audio, CanvasItem, and Array is more likely to break than KinematicBody2D or move_and_slide in a 2D platformer.

Observing label_prototypes

In the current snapshot, there is no label_prototypes.jsonl file.

Therefore, at this stage there is insufficient evidence to test the “example where the entire function usage pattern changes” with Retriever. Although api_mapping does contain a behavior_change, it is more akin to evidence of API changes extracted from documentation rather than a full‑scale usage pattern example.

Thus, when choosing a repository now, it is better to select one that first validates the areas already covered by docs_chunks and api_mapping rather than a repository that aggressively tests label_prototypes.

Repository characteristics that are good to clone right now

Based on the JSONL criteria, the best repositories to clone first are “small ones with a clear Godot version hint, and that actually use the mappings and overlapping APIs in real code.”

The priorities are as follows.

First priority: Small Camera/WorldEnvironment/Animation example based on Godot 3

The best initial test candidate is a small example repository among Godot 3 projects that actually uses features such as Camera, WorldEnvironment, AnimationPlayer, AnimationTree, CanvasItem drawing, and AudioEffect.

The reason is that the following evidence is already present in api_mapping.

- Camera → Camera3D  
- Camera.get_transform → Camera3D.get_camera_transform  
- WorldEnvironment exposure setting → CameraAttributes  
- AnimationPlayer → AnimationMixer related changes  
- AnimationNode / AnimationTree series changes  
- CanvasItem draw/modulate behavior changes  
- AudioEffect / AudioStream series changes

If you clone this repo, clues such as Camera, WorldEnvironment, AnimationPlayer, CanvasItem, and AudioEffect appear in the AST fragments, and it is likely that the Retriever will find evidence in the current JSONL.

The purpose of the test is also clear.

Godot 3 code snippet
-> AST snippet
-> api_mapping search
-> docs_chunks auxiliary search
-> LLM judgment prompt preview

So you can immediately check with the current data whether “this code uses the Godot 3 API, and what justification is needed for it to change in Godot 4.”

Priority 2: Small Animation/Audio/Camera Example Based on Godot 4

The second candidate is a small example repository written in Godot 4.

This repo is good for verifying “normal Godot 4 code explanations” and ensuring that the “Retriever does not pull in unrelated Godot 3 migrations,” rather than for migration verification.

Currently, docs_chunks contains quite a bit of official documentation for Animation, Audio, Camera, CanvasItem, and Array. Therefore, when you insert a Godot 4 code snippet, it’s easy to see whether the explanation can be anchored around docs_chunks.

It would be ideal if this repo has the following characteristics.

Completed small project with `project.godot`
About 5–30 `.gd` files
Includes `.tscn` files as well
Uses `AnimationPlayer` or `AnimationTree`
Uses `AudioStreamPlayer` or `AudioEffect`
Uses `Camera2D`/`Camera3D`
Uses basic APIs such as `Array`/`Callable`
Does not contain many large assets

3rd Priority: Examples of the Same Topic Split Between Godot 3 and Godot 4 Versions

If possible, examples that provide the same topic divided into Godot 3 version and Godot 4 version are the best.

For instance, if the same camera/animation/audio example has separate Godot 3 and Godot 4 branches, it is easier to compare Retriever and LLM judgments.

Such repositories are good to check out.

Whether api_mapping works well in Godot 3 snippets  
Whether the explanation basis of docs_chunks works well in Godot 4 snippets  
How to show version-specific code differences of the same functionality in an LLM prompt  
How to accumulate godot3/godot4/mixed judgments on a per-project basis

Clone It Right Now: Ambiguous Repository Types

Looking only at the current JSONL, repositories of the following types are still ambiguous for the first test.

Typical Godot 3 2D Platformer

A Godot 3 2D platformer with many clues such as KinematicBody2D, move_and_slide, yield, TileMap, Color8 will inevitably be needed later.

However, in the current snapshot those clues are almost absent from the JSONL. Adding such a repository right now may cause the Retriever to fail to find evidence, or to pull unrelated Animation/Audio/Camera evidence.

Thus, this type of repository is a “good final‑test candidate,” but not a “good first candidate for immediate verification with the current JSONL.”

Large Full‑Game Repository

A large game repository with many assets and hundreds of .gd files is still premature.

What is needed now is not to throw a huge project at the model, but to trace exactly which JSONL evidence a code snippet retrieves. Large repositories make the debugging target too broad, making it hard to see why certain evidence was retrieved.

The first test should use a small, clearly functional example.

C#‑Centric Repository

The current JSONL contains some mentions of C# and a few C# naming differences, but the overall flow is designed around GDScript AST and .gd snippets.

Therefore, a C#‑centric Godot repository is not suitable as a first‑test target at this stage.

GDExtension / Native‑Plugin‑Centric Repository

Repositories focused on GDExtension, C++, Rust bindings, or native plugins are also premature.

The current goal is to split a Godot project’s .gd, .tscn, project.godot into snippets and observe the Retriever finding official‑documentation evidence. Native‑extension‑centric repositories shift the analysis axis.

Repository Selection Criteria

Based on the current standards, a clone candidate should satisfy the following conditions.

The Godot version must be verifiable in the README or project.godot  
A project.godot file must be present  
A .gd file must be present  
It is preferable to have a .tscn file as well  
Large assets should be minimal  
The scope of functionality should be small and clear  
There must be code that overlaps with the APIs currently in JSONL

The keywords that match well with the current JSONL are as follows.

Camera
Camera3D
WorldEnvironment
CameraAttributes
AnimationPlayer
AnimationTree
AnimationNode
AnimationMixer
AudioStream
AudioEffect
CanvasItem
draw_string
draw_rect
modulate
Array.sort
Array.has
AnimatedSprite2D
AStar2D
AStarGrid2D

Conversely, the keywords that are still too early to use as core criteria in the first test are as follows.

KinematicBody2D
CharacterBody2D
move_and_slide
yield
await
Color8
TileMap
RigidBody2D
HTTPRequest
multiplayer
GDExtension

Judgment

Based on the JSONL collected so far, the first clone target should be a “small Godot 3 migration candidate repo.” However, a project that actually uses the Camera/WorldEnvironment/Animation/Audio/CanvasItem families, rather than a common 2D platformer, would be more suitable.

The reason is simple.

Currently, api_mapping already has references for Camera, Animation, Audio, CanvasItem, and Array. In contrast, the key clues for Godot 3 2D physics migration—KinematicBody2D, move_and_slide, yield, TileMap—are hardly present in the current JSONL.

Therefore, the most realistic first test flow is the following.

Small Godot 3 Camera/Animation/Audio example clone  
-> Fragmented .gd and .tscn files  
-> Currently searching Retriever with JSONL  
-> Verify whether the api_mapping hit is actually related to the code  
-> Preview the prompt that will go into the LLM

The next step is to clone a small Animation/Audio/Camera example written in Godot 4 and verify the docs_chunks‑based description generation flow.

Ultimately, when label_prototypes have been created and evidence for KinematicBody2D, move_and_slide, yield, Color8, and the TileMap family has accumulated, it will then be appropriate to clone a typical Godot 3 2D platformer or action‑game repository to test migration decisions.

Conclusion

The type of repository to clone right now is decided as follows.

**First candidate:**  
- Small and clearly versioned Godot 3 example  
- Actually uses some of Camera / WorldEnvironment / AnimationPlayer / AnimationTree / Audio / CanvasItem  
- Includes .gd, .tscn, project.godot files  
- Assets are not excessively large  

**Second candidate:**  
- Godot 4 example that uses the same area  
- Used for verifying code explanations and docs_chunks search quality rather than migration  

**On hold:**  
- Godot 3 2D platformer centered on KinematicBody2D / move_and_slide / yield / TileMap  
- Test after more migration evidence related to label_prototypes is accumulated

This order appears to be the most waste‑free way to validate the currently collected JSONL.