What If AI Has the Right Answer to the Wrong World?

I was watching M3GAN with the kids recently. It is not exactly the kind of film I expected to make me think seriously about AI literacy, but somewhere between the horror and the comedy, I found myself thinking about something quite different from what was happening on screen.

AI is becoming remarkably good at answering questions. For children growing up with it, that could be enormously valuable. They can ask for an explanation of something they did not understand in school, explore a subject they are curious about, get help breaking down a difficult concept or continue asking questions long after a parent or teacher might have run out of patience.

But children are also still developing judgement. They are learning how to distinguish a good argument from a bad one, how to recognise when something does not quite make sense, how to challenge what they are told and, perhaps most importantly, how to form their own conclusions. If the pattern gradually becomes question → AI → confident answer → accepted truth, they may become very good at obtaining conclusions without necessarily developing the skills required to reach them.

The concern is not simply that AI might sometimes be wrong. Humans are wrong all the time. Teachers, parents, managers, experts and books can all contain mistakes. The more interesting problem is that AI can be wrong, incomplete or narrowly framed while sounding extraordinarily convincing.

For most of our lives, fluency has been one of the signals we use to recognise expertise. Someone explains something clearly, confidently and in detail, and we naturally give their explanation more weight. AI complicates that shortcut because the quality of the explanation and the quality of the underlying judgement are not necessarily the same thing.

And sometimes the problem may not be the reasoning at all.

It may be the information available to reason from.

Imagine the Moon controls the internet

Imagine there is a dictatorship on the Moon.

The Moon has its own internet, and an AI system has access to enormous amounts of information from it. There are millions of documents, newspapers, academic papers, government reports, historical archives, discussion forums and books. There is certainly no shortage of sources.

Unfortunately, almost all of them have developed within the same information environment. The Moon is good. Earth is bad. Moon institutions are superior. Earth is dangerous. Historical events are interpreted through that worldview, current events are reported through it, and generations of commentary have reinforced it.

Now ask Moon AI whether Earth is bad.

It does not need to hallucinate or deliberately mislead you. It might provide ten credible Moon sources supporting its answer, explain the history, quote respected academics and construct a remarkably sophisticated argument. You could challenge it, ask it to reconsider its assumptions and request more evidence. It may do all of those things and still conclude that Earth is bad, because that is where the information available to it leads.

Ask for fifty sources and perhaps it can provide those too.

But fifty sources are not necessarily fifty perspectives.

Now imagine Earth has an AI of its own. Earth AI also has access to millions of sources: academic research, newspapers, historical archives, government documents, books and public debate. Ask it the equivalent question about the Moon and it may reach precisely the opposite conclusion. The Moon is bad. Earth is good.

Earth AI can support its answer just as convincingly. It can provide ten sources, or perhaps fifty. It can explain the history, identify patterns and construct an internally logical argument from the evidence available to it.

We now have two capable AI systems examining large amounts of information and reaching opposite conclusions.

That sounds like a problem with AI until we change one thing.

Remove the wall between them.

Give Moon AI access to everything Earth AI can see, and give Earth AI access to everything Moon AI can see. Suddenly both systems encounter credible information that challenges conclusions which previously appeared obvious. Historical events have competing interpretations. Evidence that once looked overwhelming sits beside evidence pointing somewhere else. Assumptions that were almost invisible become visible because, for the first time, there is something outside the original information environment against which to compare them.

Their answers should change.

The intelligence of the systems has not changed. Their ability to reason has not changed. We have not improved the prompt or replaced the model. We have simply changed the information available to them.

I think that distinction matters. We spend a great deal of time asking whether AI is intelligent enough, whether it can reason properly and whether its answers are supported by sources. Those are sensible questions, but even excellent reasoning cannot compensate for important information that was never available in the first place.

A million sources can create extraordinary confidence while still leaving an enormous blind spot if those sources all inherit the same assumptions. What changes Moon AI is not another million Moon documents. It is gaining access to information capable of challenging what it already “knows.”

When an answer quietly becomes an assumption

There is another consequence of the Moon example that I find particularly interesting.

Suppose Moon AI answers the original question and establishes that Earth is bad. The answer is convincing, there are plenty of sources and the user accepts it. The next question might be why people on Earth behave so badly. From there, the user might ask why Earth has failed to solve a particular problem, or why the Moon's approach works better.

Each answer can become more sophisticated than the last. The AI can analyse causes, compare statistics, identify historical patterns and cite supporting evidence. Nothing about those later answers necessarily looks suspicious because they may be perfectly logical given the premise they started from.

The problem is that the conclusion from the first question has quietly become an assumption inside the second.

Nobody is asking whether Earth is actually bad anymore.

This is hardly unique to AI. Humans do it too, and organisations can become remarkably good at it. Once something becomes accepted as the explanation for a problem, meetings, reports, projects and decisions begin building around it. Before long, considerable intellectual effort can be spent solving the problem as it has been defined without anyone returning to ask whether the original definition was right.

AI can make that process much faster, which is why I increasingly think one of the more important skills in working with AI may eventually have very little to do with prompting. It may be recognising when the premise itself deserves another look.

We are all learning this at the same time

My first thought while watching M3GAN was about children because they are growing up alongside this technology. But the more I thought about it, the less convinced I became that adults have much of an advantage.

We are all new to living with AI.

Adults may actually have an additional difficulty because we have spent decades developing shortcuts for recognising expertise. We look for experience, sources, clarity, confidence and the ability to explain something coherently. AI can reproduce several of those signals remarkably well without possessing human judgement or necessarily having access to every perspective relevant to the question.

Perhaps adults therefore need many of the same capabilities children are still developing: challenging assumptions, recognising framing, looking for missing perspectives, distinguishing between multiple sources and genuinely independent perspectives, and understanding that confidence is not the same thing as certainty.

Most importantly, we need to retain the ability to form a conclusion rather than simply receive one.

Much of the current discussion about AI literacy understandably focuses on prompting: how to structure instructions, provide context, refine a request and get a better result. Those skills are useful today, but I am not convinced they are the durable part of AI literacy. The technology will get better at understanding what we want. Interfaces will improve. Systems will gain more context. Some of the prompting techniques we currently teach may eventually disappear into the technology itself.

Knowing when and how to challenge the answer is a much harder problem.

If humans own the judgement, we need to keep developing it

I have written before about why I believe **AI should automate tasks, but humans should own relationships**. My argument then was largely about using AI to remove repetitive work while keeping human beings responsible for the areas where judgement, accountability and relationships matter.

I still believe that.

But the more I use AI, the more I think there is another question underneath that principle. If humans are going to retain responsibility for judgement, what happens when increasingly capable technology can perform more of the thinking through which that judgement develops?

There is a parenting analogy here that I keep returning to. Being a good parent does not mean making every decision for your child forever. When children are very young, of course we make most decisions for them. Over time, though, part of the job is gradually transferring that responsibility. They make small decisions, make mistakes, encounter consequences, reconsider choices and gradually develop judgement because they have had to use it.

Leadership works in much the same way. A manager who makes every difficult decision for their team may initially appear helpful, particularly if the manager is experienced and usually makes good decisions. But if the team eventually becomes incapable of operating without that manager, I am not sure we would describe that as successful leadership. Good leaders create conditions in which other people develop judgement of their own.

AI introduces an unusual version of the same problem. If it becomes capable of analysing more information than we can, remembering more than we can, comparing options faster than we can and producing increasingly sophisticated recommendations, using it will be entirely rational. I certainly do not want to spend hours doing work manually that AI can do better in minutes merely to prove that I still can.

The harder question is what happens to the capabilities we stop exercising as a result.

There is a significant difference between using AI to extend our thinking and using it to avoid thinking. The boundary between those two may become increasingly difficult to see as the quality of the output improves.

The information environment is changing too

There is another part of this that I do not think we fully understand yet.

Historically, humans produced most of the information from which AI systems learned. Now AI produces information too. Humans use AI to draft articles, summarise research, generate explanations, write reports, build websites and support decisions. Some of that material is published and becomes part of the wider information environment that other humans encounter and, potentially, future AI systems may encounter too.

That creates an interesting feedback loop. Human information influences AI, AI influences human information, and that AI-influenced information becomes part of what people subsequently read, repeat and build upon.

I do not think we should jump from that observation to dramatic conclusions. There are too many variables involved, and AI systems, training methods and information sources will continue to evolve. But it raises an interesting question about independence. If an idea appears across hundreds of sources, how many genuinely independent observations does that represent?

The Moon thought experiment becomes relevant again here. Moon AI's problem was never that it lacked information. It had enormous amounts of it. Its limitation was that almost everything it could see came from an environment in which the same underlying assumptions were repeatedly reinforced.

Giving Moon AI another million Moon documents would not necessarily solve that problem.

Giving it access to Earth might.

Perhaps AI literacy is really judgement literacy

None of this makes me want to use AI less. Quite the opposite. I think AI can be an extraordinary tool for learning, exploration, analysis and decision support. One of its most useful capabilities may actually be its ability to help us escape our own version of the Moon: asking for competing explanations, exploring perspectives we would not otherwise have encountered, testing an assumption or asking what evidence would cause a conclusion to change.

But that depends partly on what we ask it to do and what we continue doing ourselves.

If AI literacy becomes synonymous with knowing how to generate better answers, I think we will have missed something important. The more durable capability may be knowing when an answer deserves to be challenged, what assumptions produced it, which perspectives might be missing, whether apparently independent sources really are independent and what evidence would cause the conclusion to change.

Those are not really AI skills. They are thinking skills.

And that may be exactly why they become more important as AI improves.

We obviously do not live in a fictional dictatorship on the Moon, but none of us sees the world from every perspective. We live inside cultures, organisations, professions, social groups and information environments that influence what we encounter and how we interpret it. AI systems have information boundaries too.

The lesson from Moon AI and Earth AI is not that truth is unknowable or that every perspective is equally valid. It is almost the opposite. When both systems gain access to information that was previously unavailable to them, their answers should change because they can finally test what looked like established truth against evidence from outside their original frame.

The same is true for us. We tend to think better when we are exposed not merely to more information, but to information capable of challenging what we already believe.

That is why I keep coming back to judgement when I think about AI literacy. AI can help us reach answers faster, expose information we would otherwise miss and challenge ideas we might never have questioned. But the more capable it becomes, the easier it may also become to accept the finished answer without doing much of the intellectual work underneath it.

Perhaps that is where the parenting and leadership comparisons matter most. We generally recognise that making every decision for a child does not prepare them to make good decisions as an adult. We recognise that a leader who provides every answer can create dependency rather than capability. In both cases, judgement develops partly through use: making decisions, encountering different perspectives, being wrong occasionally, revisiting assumptions and understanding why a conclusion changed.

I suspect our relationship with AI will require some version of the same balance. There will be parts of thinking that make little sense to preserve simply because humans used to do them manually. But there may be other capabilities that we need to exercise precisely because technology is making it increasingly unnecessary to exercise them.

AI can help us reach answers faster. The more difficult question is whether we are still learning how those answers should be formed, challenged and revised.

As AI becomes capable of doing more of the thinking, perhaps the question is not simply what we can delegate to it. It is which parts of thinking are important enough that we deliberately continue developing them ourselves.