I don't get this either. I used to think about these experiments and my conclusions are completely opposite. Any physical medium that runs the right program would be conscious. And the right program is probably a very broad category.
>It already seems implausible to me that a vast desert of rocks being manipulated into various patterns is conscious. What exactly is conscious here? What happens if I accidentally kick a rock to the side — have I killed whatever ghostly being inhabits these rocks?
If you kick a rock to the side it's probably analogous to making someone's neuron misfire. If the pebble computer is as sturdy as a human brain then there would be probably no noticeable effect.
When I ask "why bother evolve consciousness, versus being just an instinctive reacter to stimuli?" It usually seems to come down to decision making. There's something about having an experience that helps with certain kinds of problem solving.
So if the evolved consciousness is going to actually be useful, it needs to be able to do something that typical instinct-driven computing can't. What might that be?
Maybe it circumvents the error-correcting mechanisms that you might otherwise expect to exist: Open the door to a small bit of environmental noise, and attempt to make meaning from it. If the meaning survives enough feedback iterations after you shut the door, you've just had an idea. This, I imagine, is how creativity works.
So what I'm saying is that most rock-moves would be error-corrected and inconsequential, but if you move the right rock you're tipping the scales between whether today feels like a eggs day or an oatmeal day--the kind of decision that doesn't make sense as the output of an algorithm.
>It already seems implausible to me that a vast desert of rocks being manipulated into various patterns is conscious. What exactly is conscious here? What happens if I accidentally kick a rock to the side — have I killed whatever ghostly being inhabits these rocks?
If you kick a rock to the side it's probably analogous to making someone's neuron misfire. If the pebble computer is as sturdy as a human brain then there would be probably no noticeable effect.