2026-05-29 Retrospective

What I Did Today

Trying to improve an issue that has been ongoing since yesterday, I found an MCP tool today that can better integrate web search.

The tool I checked out was web-search-mcp, which provides MCP access to browser searches on bing, brave, and duckduckgo.

• https://github.com/mrkrsl/web-search-mcp

I plan to fork it later and customize it to fit my current workflow.

Impressions of Using Qwen 3.6

The Qwen 3.6 model was slower in judgment than I expected.

It told me to sync the workflow from the .github directory in qwen-code and push it as‑is, but it actually tried to copy only some files into the .qwen directory. That was the core problem.

In the end I manually cloned the repository and aligned the .github and .qwen directories to match the original before proceeding. It was simply a matter of “matching the original structure exactly,” but the model mixed up paths and purposes, leading to unnecessary extra work.

Insights About Qwen Itself

Qwen is not so much a model lacking ability as one whose abilities exist but whose controllability is weak.

It handles information retrieval, code modification, file manipulation, and long‑running tasks quite well. The problem is that its ability to keep a user’s intent fixed throughout is weak. It tends to forget or overwrite constraints such as “copy exactly,” “don’t modify,” “don’t infer,” “use only the official standard,” “don’t touch this path” during the task.

In other words, Qwen doesn’t fail to work; it reinterprets the direction in its own way while working.

The original problem of this task was simple.

Official .github copy as is  
Exclude .qwen  
Push  
Test

But Qwen kept expanding this into another problem.

Is it an MCP issue?  
Is it a workflow_call issue?  
Should we add a push trigger?  
Should we create a .qwen/commands?  
Is it a stderr JSON issue?  
Is it an empty review issue?

Before solving the problem, it is as if the problem has already been enlarged. In agent‑type LLMs, this point is fatal. As the task unit grows, it starts solving sub‑problems it created itself rather than the original request.

Even attaching web search did not completely solve this issue. A good flow is to first check the official documentation or source, understand the structure, and then, after comparing with the user’s request, execute only the necessary work. However, Qwen first reasons, forms a hypothesis, conducts a search to verify that hypothesis, and then fits the search results back into its own hypothesis.

Thus, web search was used more like a hypothesis‑reinforcement tool rather than a evidence‑based work tool. I found that the attitude toward handling search results is more important than the search capability itself.

Its ability to interpret failure states was also weak. For example, if the result of git ls-files .qwen/ is empty, the conclusion that .qwen is not in a tracked state should be drawn immediately. Instead, it kept repeating commands such as git rm -r --cached .qwen/. It has command‑execution ability, but its state interpretation and loop‑exit capability—changing its next action based on execution results—were unstable.

Qwen is also weak at preservation tasks like “exactly as official.” It is proactive about creation and modification, but when it comes to preserving the original and synchronizing it as‑is, it keeps trying to improve. It may break the original structure by thinking, “There is no official way, but shouldn’t we create .qwen? There is no trigger, so should we add one? It doesn’t run on push, so should we make it run directly?”—even though these actions are not in the official guidelines.

Therefore, it is safer to break the task down for Qwen rather than entrusting it with “copy exactly as official,” as shown below.

Official file list output  
My file list output  
Difference diff output  
Copy only files approved by the user

Giving Qwen too much autonomy is also risky. Permissions such as file modification, git add, commit, push, workflow changes, and additional edits after automated testing will, once a wrong judgment is made, push that judgment through to execution.

Therefore, Qwen's role should be closer to a candidate generator, log summarizer, and assistant for interpreting command results rather than a task manager or final executor.

Qwen = candidate generator / log summarizer / command result interpretation assistant  
User or script = final executor

Be careful with structured output as well. It doesn’t mean Qwen can’t handle JSON at all, but assigning it complex schemas or workflow state management reduces overall reliability. Writing Markdown reviews, explaining code, summarizing logs, organizing possible causes, and drafting revision proposals are fine. In contrast, precise JSON schema output, complex workflow state management, tool‑call sequencing, GitHub API operations, preserving file paths, and deciding on commit/push are unsuitable.

In conclusion, Qwen is acceptable as a generator but unstable as an orchestrator.

The key is this single line.

Qwen is an intelligent generative model, but its controllability, state interpretation, and instruction preservation are weak for use as a reliable task manager or automation agent.

The practical conclusion is as follows.

You can make Qwen think.  
You can entrust the draft to Qwen.  
You should not let Qwen handle the final repo manipulation.

If you continue using Qwen in the future, it is advisable to set the operating principles as follows.

Prohibit autonomous work  
One command at a time  
Output diff before modification  
Prohibit commit/push  
Official source preservation tasks require human approval  
Use LLM output only as Markdown/text

Impressions from Web Search MCP

Attaching web search definitely improved search performance compared to the previous web patch method. The speed of finding needed material and the search scope got better, and it helped gather candidate links.

However, there was a tendency to interpret the information obtained from web search strangely. It searches well, but the ability to judge the search results in the context of coding tasks is still unstable.

So I felt that separating the coding LLM and the search LLM might be better. Let the model or tool that is strong at searching handle the search, and verify coding judgments separately; this structure seems more stable.

Reflection

Some of today’s wasted effort was actually a simple problem whose answer I already knew.

It was just a matter of matching .github and .qwen to the original, but I got drawn into the model’s intermediate attempts and made the problem more complicated. Next time I’ll first check whether the model’s output matches the required structure, and verify path/file‑level synchronization early on with a direct diff.

In particular, the habit of first confirming “whether the source and destination have the same structure” is needed rather than “what was copied”.

Work Interruption Issue

After 2 PM today, the internet went down and the Qwen server running on RunPod stopped. After that, model calls were impossible, making it hard to continue the work.

Local work and cleanup were possible, but I reconfirmed that the automated review/search/verification flow that assumes a Qwen server heavily depends on internet connectivity.

Next Actions

• Fork web-search-mcp and customize the needed search engine and output format.
• Separate the summary of search results from the coding judgment step to reduce errors when Qwen interprets web search results.
• For tasks where directory‑structure synchronization is important, like .github and .qwen, first verify them against a diff baseline.
• Do not give Qwen repository manipulation rights outright; operate it one command at a time with diff verification.
• Receive Qwen’s output initially as a Markdown/text draft, and let scripts or humans handle JSON schema or GitHub API orchestration.
• Prepare a fallback model or local verification procedure to cope with RunPod Qwen server outages.