It's not magic

In the fall of 1999 I was studying computer science at NCSU. One of my professors, the great Dana Lasher, had a saying he repeated so often it became part of the furniture: “There is no magic in this world.”

It was an assembly language class and he was teaching us to debug. Debugging, at bottom, is the disciplined refusal to believe in ghosts. The program is deterministic and does everything it does for a reason. Certainly when the bug is in a toy program you built for class, chances are excellent the bug is on you. If you don’t see how, think about it some more!

We students needed to be told this repeatedly. I will freely confess that on a number of occasions I was absolutely convinced there must be something wrong with the compiler. Eventually, of course, I would find a missing bracket, or something like that.

Any technology seems like magic if you don’t understand it. I’ve never fully wrapped my head around basic electronics; electricity still feels like magic in a certain sense, and I’ve been using it all my life. On one occasion when I tried to meddle in these arts, I carelessly touched a capacitor and found myself halfway across the room: I was thrown, I imagine, rather than teleported. Electricity sure felt like magic that afternoon! Still, we can agree the poor capacitor was just doing its job and that someone who understood electricity better could explain what happened.

Now, software systems can get hideously complex, and analyzing one while it runs can be genuinely hard. The root cause may turn out to be subtle, astronomically improbable, or bizarre. But the system is still deterministic, and if you can gather the right information, you can reason your way there.

This is good, because we want technology to do what we want, and not to do what we don’t want. We want it to start when we say and stop when we say. We do not want to be Mickey Mouse in “The Sorcerer’s Apprentice,” who enchants a broom to haul his water and then cannot make it stop: chasing around as automation makes messes more efficiently than any human could. (Goethe told this story in 1797 — anxiety about runaway automation is older than electricity.) Unwanted chaos happens, with complex software systems, more often than we would like. Even if, like my capacitor and Mickey’s brooms, the code is just doing its job.

And now we are all eyeing this new technology that … thinks. There may be a lot of ways LLMs do not “think like us,” but there is one way they do think like us: they just think what they think, and they don’t know why. They can be asked to reconstruct “why they probably thought of that,” but they don’t know for sure. They have to guess, just like we are left guessing why thoughts come into our own heads. For us it’s the result of a bunch of neurons firing; for the LLM it’s an enormous amount of arithmetic — arithmetic it can no more inspect than we can inspect our neurons. We both get to have thoughts; neither of us gets to watch.

I don’t think I have to trot out examples; by now we are all aware that LLMs can be brilliant, but they can also be bananas. The engineering challenge is harnessing that intelligence to do routine work with consistently acceptable results. (Remember: ChatGPT is not a bare model. It is a model sitting inside a harness, and the harness took considerable engineering.)

There is naturally a certain gold-rush mentality around AI, and gold rushes are not famous for careful thinking. Adoption deserves it anyway, from the engineering side and the business side at once — because the disciplinary lines are blurring, and an engineering solution can look remarkably like a management solution. You may set one model to produce results and a different model to check its work, because LLMs, like people, sometimes space out and forget their instructions.

It is a weird new way of engineering, but it is not magic. We need to stop thinking about it magically — the way we have, too often, with classical software.

Software doesn’t maintain itself

Software invites magical thinking because it is invisible. You cannot walk the factory floor. You cannot kick the tires, because there are no tires. And software seems exempt from time: it doesn’t corrode, get eaten by termites, or lose its pliancy in the hot summer sun. If it worked today, it should work the same way tomorrow — or so we think.

Alas, unless we are talking about software embedded on a chip in a talking teddy bear, software needs ongoing maintenance. The teddy bear gets a pass only because it never connects to the internet. (Maybe yours does, but mine doesn’t.) The internet is an ocean of emerging security threats — threats that, by definition, the system’s original creators did not foresee. Often the new vulnerability isn’t even in your code, but in a third-party framework, extension, or library. Still, it’s not going to fix itself!

Any software engineer will tell you that this kind of maintenance has often been given short shrift. Our civilization is littered with legacy software systems doing very important work while running years and years behind on critical updates. We are doubtless going to be taught some painful lessons about this by evildoers armed with frontier models. Lesson #1: the longer you’ve deferred your updates, the more painful the remaining lessons.

And AI does not get an exemption. You may rent your model from Anthropic or OpenAI, but the harness you build around it is … just software. Like all software, it will need to be updated and maintained.

The models themselves add to the challenge, because they keep evolving. A newer model may be “better” in general yet do a worse job than yesterday’s model at your specific task. Sometimes smarter is not better: a smarter model can overthink a problem that a dumber model would nail. So for the foreseeable future, you will need to keep evaluating models inside your specific harnesses. That is part of the total cost of ownership of these solutions — a cost to control, not underestimate.

So, about that “llchemy” name

After all this you can be forgiven for raising an eyebrow at the name I chose for my practice: Llchemy, LLC.

In 2017, a researcher named Ali Rahimi accepted a major award at the field’s biggest conference and told the room that machine learning had become alchemy. He meant it as an alarm. Yann LeCun, one of the field’s godfathers, pushed back: useful inventions routinely precede the theory that explains them — the steam engine came before thermodynamics, and worked fine the whole time.

Although some alchemists were definitely kind of grifters, getting free room and board from princes on the promise of boundless riches, alchemy was not entirely fraud. It was disciplined, empirical, occasionally brilliant work that produced real results — along with a wealth of nonsense and doubtless no shortage of unfortunate accidents.

Hence the name of this practice. Though it connotes magical transformation, Llchemy is a joke with a straight face. The plan is to practice the part of alchemy that became chemistry — measure, keep honest notes, discard what doesn’t reproduce — and leave the incantations to others. That’s how alchemy (which is mostly useless) turns into industrial chemistry.

This is what Llchemy is for: helping small and midsize organizations find the few AI priorities that are real — the ones that address a real pain or build a new capability — and skip the ones that are incantation. Sometimes the most valuable finding is “not this, and not yet,” which is an odd thing for an AI consultancy to say, and exactly why somebody should be saying it.

Professor Lasher was right, and I have spent twenty-seven years failing to find a counterexample. There is no magic in this world. What there is, on a good day, is better: things that work, for reasons you can know, that stop when you tell them to.