When AI Gets Stuck in a Local-Optimum Deadlock: Why the Core Skill of the Future Isn't "Solving Problems" But "Defining the Problem Space"

In the previous post, I talked about how the industry has lately taken a liking to mapping the L1-to-L5 framework from autonomous driving directly onto AI autonomy in software development and machine learning (ML).

A lot of people are optimistic that we're rapidly transitioning from L2 (using Cursor or Copilot as an assistant) to L3 or even L4 — the agent era — where in the future you'll just give a general description, and AI will assemble its own team of PM, QA, and SWE to get things done, with humans just standing by as observers.

Sounds great, right? But honestly, reality is usually a lot more bare-bones than we imagine.

When the AIs Start Talking Among Themselves, It's a Very Confident Apples-to-Oranges Comparison¶

A lot of people assume that if you just throw a bunch of AI agents together and give them a vague goal like "a machine learning project," they'll figure everything out on their own.

But in practice — and I've run into this myself recently, and I'm sure you have too — once you let that group of PM AI, QA AI, and SWE AI start hashing out a feasible approach on their own, the moment they "get going," they collectively fall into a strange blind spot. They start comparing apples to oranges, all while sounding extremely professional and confidently making things up.

In machine-learning terms, this is getting stuck in a "local optimum" deadlock.

At this point, you still need a human. You have to play the strict manager — point out the mistakes yourself, challenge the AI's conclusions — before it's willing to regenerate.

This is why, for projects today, we still prefer to follow an honest, human-driven workflow: discuss the framework, design, write tests, implement, and verify — locking down each step along the way. It's more time-consuming, but at this stage, it's the only way to get predictable, reliable quality.

My Daily Routine Now: "Senior Abstraction Translator"¶

If you ask me what takes up most of my time in day-to-day development now — is it reviewing AI-written code for bugs?

Actually, what takes the most thought every day is figuring out how to take all sorts of abstract or scattered requirements and translate them into KPIs and objective functions that AI can actually act on.

So when you get a request, you can't just hand it straight to AI — you first need to sketch out the architecture in your head:

• How should this realistically be implemented?
• Which design choices are less likely to break down in terms of performance or operations?
• Most importantly: where do you need to leave room for flexibility, to handle new requirements that will inevitably pop up later?

AI is just running errands to find answers within the "problem space" that humans have already defined. In a world where everyone has access to AI, if you tell it "build me a model with 99% accuracy," there's a good chance it'll quietly cut corners just to hit that number, handing you an answer that's mathematically sound but a real-world business disaster. Only an experienced professional knows where the walls need to go — and that's the real know-how of this L3 era.

Right now everyone's anxious about whether they should take a course on feature engineering, or learn to write models by hand.

I'm still on that learning journey myself, but I think of code as something static — for the basic functions, it's enough to roughly know they exist and what they're called. When it comes to actually running training and test data, language models have already left humans in the dust when it comes to writing code and tests.

But that doesn't mean engineers are out of a job. On the contrary, the real fundamentals for humans now come down to two things:

1. Understand the logic and concepts behind ML: You don't need to write it by hand, but you need to understand the principles. That way, during training, you can spot at a glance whether AI is cutting corners, or whether some result looks off.
2. Understand the meaning behind the data: Data itself is just numbers — only humans know what that data represents in the real world. Knowing when to apply what kind of processing, and asking AI when something seems off or unclear. You have to work closely with domain experts, because only by truly understanding the "metadata behind the data" can you draw the right problem space and train a good model.

As a lot of people have been saying, "solving problems" keeps getting cheaper, while "defining problems" keeps getting more valuable. AI can never decide on its own what matters most to a company. On this path, the people who can reframe the problem and draw the right boundaries will always be valuable.