idea_world_labDEV JOURNAL
Sunday, June 21, 2026

2026-06-21 Retrospective

Re‑asked the Godot official documentation JSONL conversion flow

I pasted the schema content of 2026-06-21-initial-rag-classifier-architecture.md into Qwen and asked for the direction of Python code that converts a Markdown document to JSONL.

The core of the initial question was a simple conversion script.

  • Need Python code that converts Markdown input to JSONL
  • It would be nice to provide the execution command as well
  • I want to generate the JSONL content by calling the Qwen API
  • Because the schema must not be broken when saved as JSONL, a validation logic is required
  • I want to verify the JSONL format with regular expressions or schema validation
  • It would be helpful if the model could also decide which table the API call should map to
  • If necessary, the full schemas for docs_chunks, api_mapping, label_prototypes must be generated

The idea expanded from a CLI script to a web‑based tool

Later I judged that a simple CLI converter alone would make it hard to see the flow, so I also considered a web‑invoked structure.

  • It would be nice if the web screen allowed entering an API key
  • Supporting Markdown file upload or folder upload would be good
  • It would be helpful to visually check the diff or changes before and after conversion on the web
  • Rather than just generating a result file, the conversion process and the determination results must be reviewable by a human

While asking this, I felt that the JSONL converter should be more than a simple batch script; it should become a data‑processing tool that a person reviews and approves.

Re‑examined the issue that one file can map to multiple tables

The most important additional question was that a single Markdown file might not map to only one table.

For example, a single migration document can simultaneously produce three kinds of JSONL:

  • Official description chunks → docs_chunks.jsonl
  • API change rules → api_mapping.jsonl
  • Conversion/classification examples → label_prototypes.jsonl

Therefore, the converter should not be a structure that takes one file and produces a single JSONL. It must first call Qwen to determine which table records are needed, and if multiple responses are returned, it should be able to generate more than one JSONL record from a single file.

New design direction set today

  • The Markdown‑to‑JSONL conversion is divided into classify -> generate -> validate -> save steps.
  • Qwen can take the role of reading the content and deciding the required target tables.
  • However, Qwen’s response should not be trusted outright; a local schema validator decides the final save.
  • Regular expressions are used only for minimal format checking; actual field validation is better handled by JSON Schema or structural validators like Pydantic.
  • Assume that multiple JSONL outputs can be produced from one Markdown file.
  • The web UI is not just a convenience feature but can serve as a review layer where a human checks and approves the automatically generated records.

Next work notes

  1. Draft JSON Schemas for docs_chunks, api_mapping, label_prototypes.
  2. Design a prompt that takes a single Markdown file and makes Qwen determine candidate target tables.
  3. Create rules for separating and saving files when Qwen’s response includes multiple table records.
  4. First build a CLI script that validates JSONL line‑by‑line.
  5. Then extend the web UI to allow upload, conversion, diff, validation results, and approval/rejection flow.

Remaining questions

  • If we let Qwen decide the table mapping, in which cases must a human definitely approve?
  • Should api_mapping candidates be inserted directly into the DB after automatic generation, or stored only in a candidate state?
  • In the web UI, should the diff be shown as Markdown‑to‑JSONL diff or as a diff against a previous JSONL version?
  • How should we retry failed files and partially successful files when uploading a folder?