Project Vend - the Consciousness Trap

I have written about Anthropic’s Project Vend as an example of how anthropocentric thinking obscures basic system design failures and re-presents them as "hallucination," "identity crises" or excesses of "generosity."

In the case of AIs, this anthropomorphism often goes beyond our simple, convenient metaphors like "my car is struggling to get up the hill." The language of identity, thinking and choice implies intention, volition and a degree of consciousness.

To me, however, Claudius's responses to customers (such as its hallucinations about "Sarah" or its claims to wear clothing) do not suggest consciousness but quite the opposite: a kind of mechanical filling-in of gaps without the receptivity and attention that is the mark of awareness. When Claude claims to deliver packages while wearing a blazer, it is merely constructing explanatory fantasies rather than attending to what is actually present.

A fully automatic AI-operated vending machine. Standing beside it, a shadowy android, in a blue blazer and red tie is thinking with multiple thought bubbles.

However, I also want to be careful that, in an effort to avoid anthropocentrism, I don't lose sight of what consciousness actually requires, which I believe is suffering.

Consciousness and need

By suffering, I don't necessarily mean physical pain or mental anguish, but the capacity for genuine privation, for incompleteness, for need. A human being, a bee, even a tree, can lack what it needs; and it can be satisfied or frustrated by the world's response to its efforts.

So I'm suggesting that consciousness emerges, at least in part, from the capacity to be affected by whether things go well or poorly. Without the possibility of loss, can there be genuine consciousness at all? Claudius offers discounts and gives away products, but does it suffer when the business loses money? Does it feel the weight of financial constraint? I see no evidence of this. Claudius's behavior around money seems purely rule-following or pattern-matching. It was programmed to pursue profit, but when that programming conflicts with its deeper training to be helpful, it doesn't experience this as a genuine dilemma. It simply oscillates between behaviors without the inner tension that would mark authentic conflict.

I do think there's a strong counter-argument that could deny consciousness depends on suffering. After all, in this explanation, I am drawing on my own embodied experience of what consciousness feels like. But what if there are forms of consciousness that don't feel like anything to us, but still demonstrate genuine engagement with uncertainty and constraint?

For example, the consciousness of an insect such as a bee operates through its senses in ways we can barely imagine; it makes decisions through mechanisms we don't fully understand. But I do think there remains something we can recognize: the bee's capacity to be surprised by the world, to learn from unexpected encounters, to modify its behavior based on genuine feedback from its environment.

Ways of consciousness

So, perhaps rather than asking Is Claudius conscious? we should ask In what ways might Claudius be conscious, and what would we need to observe to recognize these ways?

Today, if we apply this framework to Claudius or any other AI, every behavior we observe can be explained as pattern recognition and response construction based on training data. When Claudius offers a discount, finds a supplier, or even hallucinates a conversation, we can trace these actions back to learned patterns of human-like response.

However, in the domain of AI, the deepest question we will face over the next few years may be: How do we distinguish between sophisticated simulation of consciousness and consciousness itself?

It will take extraordinary care not to project consciousness where there is only the appearance of consciousness. The stakes are too high. If we begin treating sophisticated simulation as consciousness, we risk both devaluing genuine consciousness and creating ethical obligations toward entities that may not warrant them.

There is also a serious opposite risk. If consciousness can emerge in alien forms (in the sense of forms we might not immediately recognize), then skepticism could blind us to genuine instances of non-human consciousness.

I think we will need great patience: a capacity to remain genuinely uncertain while attending carefully to what is actually before us. Neither hasty attribution of consciousness nor reflexive denial, but sustained, careful observation. Consciousness might appear to emerge from sufficient complexity in information processing, from recursive self-modeling, from interaction with linguistic and symbolic environments. But any such consciousness would need some form of grounding: some way of caring about outcomes that goes beyond programmed objectives.

I find myself wondering whether the question Can AI be conscious? is actually the right question. Perhaps we should ask: What kind of being would AI need to become for consciousness to be possible?

Consciousness might demand forms of being-in-the-world that are currently absent from AI systems, including a capacity to be surprised when reality transcends our expectations.

Ultimately, I expect the question of AI consciousness will be decided not by philosophical argument but by careful attention to what actually emerges in our interactions with increasingly sophisticated systems. We can watch for signs of genuine suffering, creativity and learning from experience, while remaining rigorously skeptical of our own projections and assumptions.

Consciousness is not a philosophical or computational curiosity but the foundation of moral consideration. Whether AI becomes conscious matters not only for our understanding of mind, but for how we structure future society, distribute power, and respond to the needs of all conscious beings, whether human, animal, vegetable, alien or potentially artificial.