I turn black boxes into knowledge by not quitting.

Past Attempt

Last year, around the middle of the year, I tried to build a chatbot with an LLM for WhatsApp. The idea was to use it for customer support. I created the first version in N8N, a no‑code tool, because I wasn’t confident writing the AI agent in code. At the time I was comfortable with front‑end development, but not with building AI agents from scratch.

The no‑code approach gave me templates and made the initial setup easier, but I quickly ran into problems controlling the AI. I could run a test with a client, yet every real interaction behaved differently than I expected. I kept tweaking the prompts, but I couldn’t validate which parts of the prompts or which actions of the agent were failing. It felt like a black box, I could see the output, but I didn’t understand what was happening inside.

Because the system required a lot of maintenance and I needed to focus on projects that could bring money faster, I abandoned the chatbot. It was frustrating to build something I didn’t fully grasp, especially when I felt it was within my reach but remained elusive.

Learning Observability

A few weeks ago I attended a talk in Curitiba about using LangFuse for observability in LLM‑based systems. The talk explained how to get insight into the actions of an LLM and how to control it more effectively.

That sparked the idea to revisit my chatbot, this time with proper observability tools. I decided to build an AI agent using LLM models, but now I’m writing the code myself. Cloud Code handles most of the implementation, while I focus on the architecture and system design, what engineers do these days.

Current Build

During today’s work the AI stopped responding well to the prompt. I tried to change the architecture to improve it, but the changes didn’t help. The agent was running as a separate microservice, which added complexity.

Because I had set up LangFuse, I could look at the traces saved as history. Those traces let me see exactly which action was causing the failure. I discovered that the problem lay in the harness, the framework that the agent was running in, not in the model itself.

Debugging with LangFuse

Using the information from LangFuse, I adjusted the harness and let Cloud Code implement the fix. I also considered switching to a more powerful model, which would be more expensive but could solve the issue. In the end, the cheapest model, once the harness was corrected, performed the instructions I had given it.

Takeaways

What was a black box for me last year is now something I can open, understand, and improve. The project that failed taught me the lessons that make the current one work. If you let the tools you’re using remain opaque, it’s frustrating to translate that black box into knowledge. Consistency, time, and incremental learning turn opacity into clarity.

There are still black boxes I’m working on, and I know it will take time to understand them. But as long as I keep trying and learning from each project, I’ll keep expanding my grasp of these systems.

That’s what I wanted to share today. This afternoon I finally got past a morning of being stuck, and by the end of the day everything worked better than I anticipated.