Allan Nascimento

Hey everyone, this is Allan and today I will talk about how AI is a lot harder to work with in real projects than people really think.

Project Context

Since Monday, two years ago I quoted a project and today my client asked if the price I gave would fit an AI feature, a feature that relies on using AI to do a determined task. I explained that AI is not something you ship and forget; it needs a lot more maintenance and monitoring than people think. There is a lot of hype around AI agents and implementing AI in businesses, but people forget, or don’t know, how hard it really is.

AI Is Not Deterministic

AI is not deterministic like code. When you apply AI you get different behaviors in each interaction, and you can’t predict all the outcomes in production. If you change the order of the things you ask about, or the words you use, the AI will answer in a different way. If you ask a question that you didn’t train the AI on or didn’t prompt it to handle, the AI will hallucinate and give an answer that doesn’t make sense. AI is not something you build once and then fix bugs and be done. You have to test it and monitor it all the time.

Observability Lesson

Last year I tried to use AI in a local tool for an ATM and built an AI agent, but I didn’t know about observability at the time and it didn’t work as I thought it would. I had to change the prompt every day and I didn’t price correctly for the amount of hours it took me daily. I learned a lot from that project, but I made many mistakes in planning and pricing because I didn’t account for the daily execution needed to keep it working. I had to stop the project and spend more time studying AI before it could work.

A few weeks ago I attended a talk about observability, specifically using logs and metrics. I am now implementing those ideas in my projects because I learned my lesson. There are better tools and technologies for monitoring and logging every AI action, understanding where problems arise, and making fixes.

Client Prompt Issue

A big problem I faced last year was that my client kept wanting to change the prompt, but the prompt was stored inside my environment and the client didn’t have access. I learned that it’s sometimes better to make it easier for the client to edit. Things that sound obvious to experienced people can be new to someone learning on their own from zero. I started learning by doing real, small‑scale projects, and even simple‑looking issues can be overlooked in the hype around AI.

Pricing and Maintenance

When the client asked if an AI feature would fit inside the price I quoted without taking AI into account, I told him that AI is not done from the get‑go. It needs a lot of maintenance and turns a one‑time delivery into recurring work. If you ship it and walk away, the AI will produce errors, stop responding as planned, and exhibit unpredictable behavior. There are also API costs to running AI. People want AI to solve every problem and be cheap, but the cost of using AI scales. Better models that actually do the work are also a lot more expensive.

Final Thoughts

The main point is that AI doesn’t end at delivery. You have to keep improving it, test it many times, correct mistakes as they happen, and do it quickly. The closer you are to the problems, the better you can observe them as they happen, and the better the AI system will be. You also need guardrails, such as AI as a judge, humans in the loop on the agent’s actions, and a lot more work than people think, so the AI doesn’t drift from its objective.