(This note was presented to my team in an internal talk. It refers to a working example from an internal project the team was involved in.)

This note is about the use of AI when doing any kind of exploratory analysis, targeted specifically to engineers starting their careers.

Our recent conversations should have highlighted a problem that has plagued the software engineering and programming languages community for several decades now: the limitation has never been in our ability to crank out code. It’s always been in specifying what we need the code to do.

Being able to generate code for a given specification is mostly a solved problem today with AI: for a tightly defined spec (more on this below), it’s good at generating what’s statistically the most common way to solve the problem. Imagine writing a captcha solver, where the inputs, outputs, and functionality are mostly well defined. This problem ought to have been solved by the community a while ago, and we’re invoking that communal knowledge that’s baked into a statistical predictor (AI model) to predict the sequence of characters (code) that will solve the problem.

The real challenge appears when we don’t know what we want: exploring data we haven’t seen before, looking for knowledge/facts/citations/resources that may not even exist out there, writing code for a problem we don’t fully understand or appreciate, looking for medical knowledge we don’t understand, and so on.

What should be our mental model, our approach in such cases? The pursuit of science and the tools of reasoning developed over the last several centuries should provide some guidance.

There is significant literature that suggests a common limitation in human reasoning: when developing a hypothesis (a mental model of a phenomenon: for example, how are states represented in the Drupal data dump we’re analyzing), we as a species tend to exhibit two harmful behaviors:

1. I believe a hypothesis H. I then find observations consistent with H: here you go into the problem with a biased model H, and seek data and observations that conveniently only explain H.
2. I am presented with a hypothesis H. I accept it and form other hypotheses based on H: it’s a variant of #1 but different. You do not challenge a hypothesis presented to you, and accept it at face value.

Modern science has progressed by challenging just these biases. When presented with H, scientific reasoning often starts by asking what evidence could falsify H, rather than looking for evidence that confirms H. And in the search to falsify H, you will gather information about H.

A simple example:

Your Wi-Fi technician tells you that the Wi-Fi is slow because too many devices are connected. To falsify this claim, you disconnect every other device, connect only one device, and test your speed.

• If the speed remains bad, the technician is giving you bs. You have falsified their hypothesis.
• If the speed gets better, there could still be alternate explanations for the improvement. But there is merit to what the technician claims.

Note: This is related to the intuition behind null hypothesis testing.

Here’s some foundational literature from more than 40 years ago:

• Wason’s experiments showing our tendency to seek evidence that confirms what we already believe, rather than evidence that could prove us wrong. Link
• Klayman and Ha’s 1987 work on why we default to seeking confirming evidence when testing hypotheses, and the conditions under which that habit misleads us. Link
• A more recent talk by Ben Recht tracing how scientists made sense of data before modern statistics and computing, using the discovery of vitamins as a case study. Link

This may seem obvious to all of you. But here’s the catch: very few of us actually apply this mindset when working with an LLM, or in life in general. A language model is a hypothesis explorer, statistically picking the most likely hypothesis for the problem you’re specifying, and generating code and results accordingly. But pay attention to the communication game being played here:

• the true hypothesis is H
• your understanding of H is an incomplete representation of it: H_you
• you prompt the LLM, and the LLM forms its own interpretation of what you mean: H_llm
• the code C it produces is conditioned on H_llm, not directly on H

The generated code likely pertains faithfully to H_llm: no issues there. That’s what LLMs are trained to do well. But notice: H_you and H_llm are latent—never written down, never made concrete. You are just observing the artifact C and implicitly judging the goodness of H_llm, H_you, and H.

This is where the problem arises: you are blind to how far H_llm is from H_you, and importantly, how far H_you is from H. And worse, the artifact C “looks” correct—it’s formatted nicely, it has some numbers that seem right at first glance, and has explanations that look right. It likely has references to citations that may be legit, and may even have your favorite emoji. But does C faithfully represent H? Very likely no.

This is the real game. I want you to seriously exercise your muscle of invalidating working hypotheses.

A code or results artifact produced by an LLM should be treated as incorrect by default, unless you can validate it by writing tests, cross-referencing it with other data, or manually checking whether the calculations hold up. If you can’t explain a result, assume—and I will too—that it’s false. The onus is on you to prove me wrong, and in the process, prove yourself wrong.

Please invest in this muscle; LLMs can’t substitute for it. I don’t want to be impressed by how quickly you finish a task, I want to be impressed by how deeply you understand it. AI drastically speeds up work we already understand well, but in exploratory or research work, depth beats pace. Train yourself on bite-sized problems first: when you get an LLM to generate a result for something you don’t fully understand, pause and ask what information could falsify it. Building that intuition takes months, and that’s fine—you’ll see yourself getting better and quicker at it. Once you’re comfortable, move on to the autopilot/assisted modes LLMs provide.

The point isn’t to avoid using LLMs. Quite the opposite. Use them aggressively when you understand the problem. But when you’re exploring something unfamiliar, don’t outsource the exploration itself. Build the habit of asking what you believe, what you don’t know, and what evidence would prove you wrong. That’s the muscle that will compound over the course of your career.

P.S. The em dashes in this post were added by me.