More Lessons Building with Claude Code
Part 2 of lessons I learned building an AI tool with Claude Code.
Run tests in Terminal console - it’s helpful to see each step taken. I also like to include metrics as part of the output, makes it easier to compare runs. I keep a cheat sheet of test scripts in my docs folder.
Challenging Claude’s thinking (constructively) when I don’t have the ability to read an entire repo in seconds is difficult. I can ask it to review the codebase comprehensively, evaluate the results from the last test, and come up with its hypothesis, but its conclusions are far from being acceptable at face value.
Not that they were bad - but they were just starting points. We sparred back and forth on recommendations, what the observed patterns mean, where the root cause of the issue might be. Each of us have a separate context of what the system looks like - neither of us have the full picture perfectly. Eventually we land on something that worked, but I’m not sure if there was a faster path towards debugging these kinds of issues.
The issues I’m talking about have little to do with ‘something broke’ and more to do with ‘why aren’t we getting the results we want?’
It required thinking through the whole system and how each part might affect the whole, and trying to understand ‘where’ the problem is.
I’m the one asking clarifying, probing questions - are we sure about that conclusion? Wouldn’t this have to be true if that were the case? Have we checked this datapoint to confirm or run a test to verify?
The hard part is, keeping enough of the context in my own head. I do have documentation I ask Claude to update for me - I study our architecture docs, the mermaid diagrams, to understand what’s happening under the hood. But they’re not perfect or as comprehensive as I’d like - I’d want every piece of functionality accounted for. Claude gives me digestible abstracts.
My best defense are my questions - treating things Claude says as hypotheses, not blindly accepting its every interpretation and turn. I trust it enough to understand how to code, but I still need to be in control when it comes to deciding what to do and why.
The hardest thing about vibe coding now is the planning - reading through every generated requirement and recommendation still takes time. The AI doesn’t get it right immediately.
For example, it recommended we created a threshold when accuracy is < 60% to count it as ‘trustworthy’ and I had to call it out. “My bad,” Claude says and admits it was an error. Elementary school math concepts here. But to be fair, I asked it for a full integration of a framework and it’s doing its best.
Claude and I go back and forth, evolving planning docs across multiple chats and iterations. It writes 1000x faster than I can read. Whatever I don’t understand, I look up separately on Comet/Perplexity (cause I don’t want to clutter up my Claude Code thread with separate questions).
So a lot of my time has been studying our planning docs, codebase, asking questions and getting clarification. Once you let the AI run with it it’s hard to stop it early enough to prevent an unforeseen drift.
These are doable strategies, albeit time consuming. Again, my biggest enemy is myself - my habit of running multiple tasks in parallel and switching context quickly is a huge detriment to solo-building. The day job of a product manager requires me to juggle multiple tasks and address many surface level issues quickly, but I have the benefit of human-in-the-loop teammates to catch problems before things drift too far.
As a solo AI builder, it’s just me. I’ve had to force myself to single task in the evenings and weekends to switch to builder mode.
I get confused about when to turn on the ‘thinking mode’ icon in Claude Code. I usually forget to toggle it off, which leads to ‘creative’ work when I just wanted a straightforward task implemented.
Turns out, you don’t really need to use the button - you can just tell Claude to ‘think’ and ‘think harder’ or ‘ultra think’ for maximized thinking/reasoning effort. Typing it out in the chat feels a lot more fluid and intuitive compared to ‘remember to hit that button to think better’ and I wish I knew this sooner.