Natural language, SQL and the decades-long drive to talk to computers like humans

Self-serve analytics have been a buzzword in the world of data for years at this point. It’s been brought on by the combination of irritants; waiting for data analysts to answer your questions is annoying, and -- as the analyst -- it is annoying to write all those SQL queries.

It’s always been a little unclear to me, a data person, what exactly was meant by self-serve analytics. Maybe with the right series of drop down filters in the right dashboard design our pesky non-technical co-works won’t reach out to collaborate with us via Jira tickets, I suppose. For years, it’s been clear that there was going to need to be someone upstream of the question to clean the data, organize the data and present it correctly. 

Now, self-serve analytics is rearing its head once more. And like anything else these days the innovation here is simple: bolt a Large Language Model to it. It’s slightly more complex than that, but not really. Gone are the days of an analyst writing SQL. The right prompt in natural language will get you all the business intelligence you need straight from the database, according to the companies selling these LLMs.

“The real headache with self-serve agentic analytics is that data is just inherently messy and ambiguous,” reads an Anthropic blog post from earlier this year. “It usually boils down to whether or not you can actually connect a colleague’s vague query to the right entities in your current data model while knowing the proper way to handle them. Solve that mapping puzzle, and the subsequent SQL generation or execution is basically an afterthought.”

Anthropic lays out the best practices for making this work well and it basically boils down to having good, well-organized data that you detail yourself in plain language with docstrings to explain functions and semantic layers to define metrics. The new world is built on yaml and markdown files. An agent can access these details and help to guide its query writing. It apparently works well enough that Anthropic has automated “95% of business analytics queries.” (It’d be interesting to know more about those 5% of queries that aren’t automated, but the company doesn’t get into that.)

Jerry Nixon, a product manager at Microsoft who has focused on Azure and SQL Server, recently noted that while these AI agents can produce some amazing queries, they’re not infallible.

“The problem is this weird determinism that doesn’t exist inside of AI,” Nixon said at a recent Microsoft conference. “I can take a model and give it a prompt and it’ll give me back a query that if I review it is beautiful. I mean it's textbook.”

The issue is consistency: the same model with the same prompt can provide a different query on a second pass, he noted.

Nixon noted this process is referred to as NL2SQL. “It stands for bad idea,” Nixon joked before noting it means natural language to SQL.

Now would be a good time to point out that both Anthropic and Microsoft have a financial incentive to have people move toward using these LLM tools to write more code. They’ve poured vast sums of money into the development of these systems. If they can convince the corporate world to adopt them -- along with a hefty enterprise subscription cost -- then they point investors and analysts toward these wins.

But this desire to use "natural language” to communicate with a computer rather than complicated code is nothing new. It’s an idea that goes back decades at this point.

In fact, some researchers at one of those fancy Boston schools were working on this in the middle of the 20th century. Bert F. Green, who was a professor emeritus in the Department of Psychological and Brain Sciences at Johns Hopkins University when he passed away in 2019, was the lead author on a 1961 paper called “Baseball: an automatic question-answerer.”

“Baseball is a computer program that answers questions posed in ordinary English about data in its store,” the paper’s introduction reads.

And the problem and solution were honestly kind of similar to what we see Anthropic talking about in their blog post. The problem then and now is translation from human to machine language, and the solution context. In 1961, that context was a structure of definitions that defined terms like teams, dates and standings. 

As the computer's need for context has remained consistent, so too has our realization that it can get things wrong. “The program usually rejects questions it cannot handle, in which case the questioner may rephrase his question,” the Baseball paper notes in its discussion. “He can also check the printed spec list to see if the computer is on the right track, in case the linguistic program has erred and failed to detect its own error.” Now, the non-deterministic reality of LLM-driven work mentioned by Nixon means errors rather than rejections can sneak in -- even on a task they usually handle just fine.

Critics of using natural language to generate code have been around as long as the researchers trying to make it work. In 1978, the short essay "On the Foolishness of "Natural Language Programming"" by Edsger W. Dijkstra, argued that the human language did not evolve to communicate with computers. Machine code allows for increased complexity that isn’t able to be captured within the words and symbols we use to communicate with each other.

“Instead of regarding the obligation to use formal symbols as a burden, we should regard the convenience of using them as a privilege: thanks to them, school children can learn to do what in earlier days only genius could achieve,” Dijkstra wrote.

There are echoes of what Dijkstra said in a book published four years later, “Orality and Literacy: The Technologizing of the Word” by Walter Ong. Ong points to the ways in which literacy increased analytic reasoning within the population, led us toward more categorical thinking and provided more permanence to our thoughts and ideas. Different ways of communicating change the way we see the world around us, Ong demonstrates.

So then, what does it mean if humans are no longer the ones in the weeds of machine code. If Claude can write the SQL and a Databricks agent can build an end-to-end ETL pipeline with the input of one human user, who generated some natural language instructions in a markdown file, then what will happen to our ability to understand and think about these systems?

Ezra Klein, a columnist and podcaster with The New York Times, recently asked Jack Clark, a co-founder and the head of policy at Anthropic, about this issue after noting that the company has publicly said Claude is writing most of its internal code.

“Making sense of it is going to require building many technologies that you might think of as oversight technologies,” Clark said.

As our understanding of the digital infrastructure crumbles we’ll build more tools -- which we don’t understand fully -- to monitor it and ensure everything is working correctly, I guess is the argument. It’s black boxes all the way down.

So then we have to wrestle with the question: are we optimizing ourselves to understand the world or to make it efficient? And if we’re optimizing for efficiency, which I think we are, then what, exactly, are we trying to be efficient at?

Data professionals like myself are not alone in this new world. Mathematicians are watching this all unfold and have wildly different opinions within the field on what LLMs means for the future of the trade, according to a recent piece in Scientific American.

One big difference between data professionals and mathematicians, though, is who our employers tend to be. 

“For centuries, mathematicians have made an uneasy arrangement with society to skirt this “problem,”” Joseph Howlett wrote in the Scientific American piece. “Students of engineering and medicine need to learn math, so as long as mathematicians teach, they’re funded and can spend the rest of their time on proofs. AI and politics are challenging that arrangement, but it’s not likely to collapse soon.”

Data professionals, meanwhile, tend to be corporate cogs, a cost to be cut. I think data and data technology are fascinating. Unfortunately, though, data engineers and data scientists became well-paying and readily available jobs not because it’s interesting, but because they’re valuable skills for making money in the digital economy. Do business owners care about understanding the parsing logic of Postgres or the partitioning strategy used for cold storage? Or do they just care that the internal data tool works as expected and it’s cheaper than this time last year. Probably the latter.