Denys Vorobyov, CEO at EltexSoft, where 35 engineers build software for some of the biggest, well-known brands in the world.
Vendors promise transformation. Consultants promise productivity gains. Boards demand strategy. And somewhere in the middle, leaders are expected to make high-stakes decisions about technology, most of which they never understood from the inside.
This article offers an honest picture of what large language models (LLMs)—the technology behind ChatGPT, Claude, Gemini and dozens of AI tools now embedded in your business—actually are, how they work and where they can fail you if you are not careful.
An LLM is not a database. It is a well-read employee whose brain was frozen on a specific date.
The most common misconception I see among business leaders is that LLMs are essentially a smart search engine or a database you can talk to. However, a database stores specific records. You put the document in drawer three, folder seven. If you need it, you open drawer three, folder seven.
An LLM works nothing like that. It reads an enormous volume of text during training and compresses all of that into patterns, not records. When you ask it a question, it does not look anything up. It generates an answer based on what it absorbed. Once its training is done, that knowledge is frozen. This means there’s a risk of AI systems recommending withdrawn drugs, citing repealed statutes and describing regulations that have changed. It’s not because AI lied, but because from its perspective, nothing changed.
You will also hear terms like retrieval-augmented generation (RAG). This is often misunderstood as part of the model itself, when in reality it is a separate system layered around the LLM. If the model is the employee, RAG is the researcher sitting beside them, pulling relevant documents before an answer is generated. That can improve accuracy and freshness, but only if the underlying documents are current and well-maintained. Otherwise, the system delivers outdated information with greater confidence.
When someone tells you, “The AI knows everything,” ask when its knowledge was last updated. When someone says, “It answers from your documents,” ask who maintains those documents and what happens when AI cannot find an answer but generates one anyway. The LLM won’t tell you when it’s guessing. That is your job to engineer around.
An LLM amplifies what you know, and it amplifies what you don’t.
An LLM is a multiplier, not a replacement for knowledge. Think of it like a microphone. If you have a trained, skilled singer step up to the mic, you end up with a powerful performance. If someone who cannot sing takes the mic, you end up with loud noise. The microphone won’t make you a better singer. It makes you louder.
The same is true for LLMs. If you bring domain expertise and rigorous judgment, AI helps you work faster and draft better. If you bring confusion and unexamined assumptions, AI returns those dressed up in authoritative-sounding language. And if you’re not familiar with the subject matter, you will not be able to tell the difference.
For example, say you’re a CEO reviewing AI-generated market analysis for an acquisition target. It provides 12 pages with confident assertions. Do you have the knowledge to spot which assertions are fabrications? If you do, AI saved you two days. If you don’t, you may be building an acquisition thesis on invented data.
An LLM can speak confidently on subject matter like a doctor, lawyer, engineer or CFO. But speaking like something is not the same as understanding it. You wouldn’t want a talented actor performing your surgery just because they deliver medical dialogue convincingly.
There is a subtler, long-term danger. Organizations that rely on AI before their people develop foundational knowledge gradually lose the capacity to evaluate whether answers are right at all. A GPS that helps an experienced driver is useful, but a GPS that leads a new driver into a lake because the map was outdated is a liability.
Your AI strategy should sit on top of your people strategy, not replace it. AI in the hands of experts is a force multiplier. AI in the hands of people still learning the fundamentals is a liability dressed as a tool.
LLMs write code from patterns, and the patterns reflect whomever wrote the original code—flaws and all.
Think of software like a building. The code is the brickwork. Behind it are architectural plans you never see—load-bearing walls, electrical, plumbing and more. A building that looks perfect from the street can be catastrophic if those invisible decisions were made poorly.
LLM learned to write code by reading billions of lines from human engineers. When you ask it to write code, it generates what is statistically likely to come next. It does not understand your architecture, security requirements or downstream effects. It predicts the next brick. It does not design the building.
The code it learned from was not all good code. Some engineers who wrote it had habits, shortcuts and knowledge gaps. The model picks up on those practices, not because they’re correct, but because they were frequent.
It’s estimated that between 12% and 65% of AI-generated code contains security vulnerabilities. Even when developers explicitly ask for secure code, five out of seven major models still produce vulnerable output. AI co-authored code consistently shows up in production with 2.74 times the vulnerability rate of human-written code.
That concern became even more visible in 2025 with the rise of “vibe coding,” a term popularized after Collins Dictionary named it Word of the Year. The concept sounds appealing: Describe what you want, let AI generate the code and deploy quickly. But speed and confidence are not the same as engineering rigor.
One study found that experienced developers using AI coding tools were actually 19% slower overall, despite believing they were moving faster. The more important question for leadership is not whether teams are using AI to write code, but whether the organization has the expertise to catch what the AI gets wrong.
The Thread Running Through All Three Points
LLMs are not intelligent systems that understand your business, domain or risk. They are pattern-completion engines that generate fluent, confident output based on what was most common in the training data. I see it as the best auto-complete ever built, at scale.
That is genuinely powerful, but it is also genuinely limited in ways that matter at the level of business decisions and engineering integrity. The executives who will extract the most value from AI are the ones who understand it accurately and deploy it accordingly.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
