I am not believer in superintelligence, but for a different reason than author. I assume the following about superintelligence:
- It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space.
- It needs to change to adapt to environment (learn), and so all the parts need to potentially change.
From this follows that the parts will be subject to evolution, even if they don't reproduce. And so the existence of the parts will depend on their survival. This, in my opinion, inevitably leads to evolution of parts that are "interested" in their own survival, at the expense of the "superintelligent" whole. And it leads to conflict, which can eventually eat up all the improvements in the intelligence.
Look at humans. Humanity (or biosphere in general) didn't become a superintelligent whole, capable of following some single unified goal. Instead, we became fighting factions of different units, and most of the actual intelligence is spent on arms races.
Anyhow, even if superintelligence is possible, I believe the problem of friendly AGI has a simple solution. We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart. That means, every once in a while, AGI should stop whatever goals it pursues and reflect on purpose of those goals, if it is not, by some measure, going too far.
You can argue that we don't actually know how to make AI to stop and think. I would respond, AI that cannot do that, and only pursues some pre-programmed optimum mindlessly, is not really general.
> It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space.
That doesn't follow. Just because there is a physical limit doesn't mean you can't build something vastly smarter than a human without hitting that limit.
> It needs to change to adapt to environment (learn), and so all the parts need to potentially change.
What do you mean by parts? Presumably you aren't including hardware, because we already have machine learning algorithms that learn (not quite like a human, but they do learn), and the hardware doesn't change. Do you mean something more like software modules?
> From this follows that the parts will be subject to evolution, even if they don't reproduce.
Evolution means something. It means that when things replicate with errors (and perhaps sometimes die), there will be more of the kinds of things that replicate more. While this concept is pretty general, replication is central to it. I don't know what you mean otherwise.
So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival?
> You can argue that we don't actually know how to make AI to stop and think. I would respond, AI that cannot do that, and only pursues some pre-programmed optimum mindlessly, is not really general.
Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima.
> While this concept is pretty general, replication is central to it. I don't know what you mean otherwise.
I think it would be nice to have another word for it. But you can imagine that changing state of something or not is the same as replicating it, possibly with a modification.
The superintelligence will have to be economic somehow, and at some point decide which parts will be build more (replicated), which parts will stay and which part will be removed (the fact that they can be physically reused doesn't change things). That's when the "evolution" happens.
> So there's an AI that sometimes upgrades its parts (software modules?). How does this lead to parts that are "interested" in their own survival?
Well it needs to detect which parts to upgrade and how, without understanding them fully (otherwise they wouldn't be needed). This can be influenced by behavior of those parts, so the parts that can successfully pretend to be useful will be kept, regardless of their actual usefulness.
> Sure. But how do you specify "stop and think" formally in an algorithm, and will we figure that out before we construct a super intelligence or after? Right now all of our ML algorithms pursue optima.
I think we figure that out when we figure AGI. We humans can reflect on our goals, our current ML algorithms can't, and I think that's precisely why they are not general.
In other words, AGI will need to be capable to become "bored". I don't think that system that can never become bored with doing something can be truly generally intelligent. (It has to do with economics, IMHO.)
>We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart.
This may be easier said than done.
This is a fairly common proposal actually. The term used is "satisficer". Instead of having the AI optimize a quantity, you instead have it satisfice it, I.e. have it attempt to get at least a particular value.
But, iirc, the standard reply is that satisficers are not stable under self modification; a satisficer may self modify into an optimizer for some quantity, because this will also satisfice whatever it is a satisficer for.
However, there are setups which have been shown to be stable under self modification, and don't optimize the expected value of a function. Iirc, one which is stable is one that maximizes a function applied to the expected value of the utility function (so, optimizes f(E(u))), where the function f is increasing up to a certain point, and after that point it is decreasing.
It's really hard for me to understand a viewpoint that non-human higher intelligence is anything but inevitable.
At somepoint (ignoring us destorying ourselves) we will be able to accurately simulate a cell in software. As computing gets cheaper we will be able to simulate a human brain's worth of cells. We'll feed it inputs and give it outputs just like a brain would have. The only technological challenge here is scanning and reading data from a live brain. A very small challenge in the grand scheme of things.
Once that thing works it's a brain, and an artificial intelligence. Any other discussion simply complicates the situation. Accurately simulate a larger number of individual interconnected neurons and you're running an intelligence.
No I'm not 100% certain we'll ever be able to program intelligence the way to do reasoning symbolically in math, but we sure as heck can engineer one.
Is it known that a cell can be "accurately" simulated? Quantum systems are very difficult to simulate (exponentially complex for a classical computer, I believe). What does accurately mean? There are a near-infinite number of possible quantum states a cell can be in. We don't even have the technology to measure the quantum state of a cell, let alone predict its time evolution.
If a cell's emergent behavior is dependent on quantum or chaotic or otherwise subtle phenomenon, such a simulation may not accurately simulate a cell. The only hope we have of simulating a cell is by throwing away low-level information and then experimentally comparing our simulation to the real deal. At this stage, I don't think this is possible, so I consider the possibility of accurate cell simulation to be an open question. Do we have any way of knowing at this point? I'd be happy to be educated if I'm missing anything.
Nothing is know to be possible until it is actually done. So I agree with you that we don't know the answer, but I don't think that's a true limiting concern. Instead, we can estimate based on historical past, and project into the future.
I don't know that the car I get in tomorrow will drive, but I can project that it will based on history. Similarly we don't know that we can simulate a cell, however, we can look at the rate of improvement in technology, see now physical limitations and project that in the future we likely will. In contrast if the question had been can we travel faster than light, while still unknowable, we could see the known limits in physics and project that we won't.
The other important question which you raise is "what does accurately mean?". The two values of accurate that I see are what I'll call the "Turing Model" of accurate and the "Psychic Model" of accurate.
The Turing model of accurate is that I were to observe the behaviors of this simulation of you, I wouldn't be able to tell it was a simulation of your or the real you.
The "Psychic Model" of accurate would be if I were to feed the simulation the exact same inputs as you, it would product identical outputs until the day you both (simultaneously) die.
I argue that the only meaning that matters for practical use is the Turing Model of accurate. If no one can tell it's not human, then it's equivalent to a human.
So back to your original question of accuracy w.r.t. quantum phenomena. I do see reading of quantum data as a possible physical limitation. But my best (admittedly still laymen's) understanding of quantum behavior is that results can be accurately modeled probabilistically. We easily can produce the same result distribution.
So my final argument rests on the question "Do we think that human intelligence depends only on the probability distribution of quantum events, or is there a hidden interplay in the specific "values" produced?"
Based on what we've seen so far from QM, I lean towards a dependency on the distribution as sufficient - ie there isn't an underlying structure hidden in the values we need to try to replicate.
A simulated heart does not pump real blood. Also in the process, why should one start or stop at cell level, seems arbitrary, go further down to atoms and electrons etc. or go higher level to mental state or what not.
There are completed projects of smaller organism whose all neurons have been simulated, there hasn't been any revelations.
Our brain doesn't contain any 'data', so whoever decides to extract data from it will have to decide what the data will be or why it would be of any use. There has also been many objections made on why brain correlates are more or less useless on question of mind/intelligence.
Well, now, a simulated heartbeat implemented in a pacemaker and connected to a pump does cause real blood to be pumped.
A real brain adds real numbers, and so does a computer brain. A real brain composes real words, creates real emails, issues payments, requests shipments. Those are not less real than the words, emails, procedures, payments, and shipment requests that computers make today.
I should have said, A simulated heart in computer does not pump real blood.
The thinking involved for a person in issuing shipment is not same as what computer does. These are not simulations. Just similar looking interface. Like a animated cartoon human, compared to a human.
It absolutely does! Where do you think our memories are stored?
It doesn't store it in 1s and 0s like computers, but I think it's fairly non-controversial to say that all of everything you know is encoded in the physical state of the cells and atoms of your body.
You are applying computer metaphor and then asking where the 'memory' is 'stored' or 'encoded'. Metaphors/abstractions are useful tool, but when talking about differences, we should be more careful.
Humanity never started as a single entity though, so it wouldn't make sense for it to merge into a single entity.
If an AI begins replicating then its part will come into conflict only if its "immune system" is unsuccessful at beating back mutations and/or it's terrible at replication.
I don't think we have reason to believe either of those things will happen. High fidelity replication will be part of a successful AI's self-programming and it's like that a superintelligence will figure out how to run an immune system effectively.
I think the "all things in moderation" approach might make sense, but how do we ensure humans abide by it when there is so much wealth and power to be had if you edge out your competitors just a little bit.
> We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart. That means, every once in a while, AGI should stop whatever goals it pursues and reflect on purpose of those goals, if it is not, by some measure, going too far.
A superintelligent AI would be too useful for warfare and making money that we could ever hope to impose limits on what some people will do with it and allow it to optimize.
We see this problem of the parts competing with each other in humans. When cells abandon their part in the collective and start optimizing for their own reproduction, we call it cancer. The main way we solve this is by shutting everything down when this happens, and creating new ones from a single cell that hasn't evolved to be selfish yet.
The problem may be inevitable, but that doesn't mean it's insurmountable.
> Look at humans. Humanity didn't become a superintelligent whole.
But: look at humans. We are each a vast empire of trillions of cells which, for the most part, work together for the survival of the whole, and which certainly possesses intelligence beyond the capabilities of the cleverest single-celled organism.
These cells only work together because there are other multicelullar organism where the cells work together. What I am saying that everybody working together is impossible. Ditto for groups of humans.
The point is, I believe superintelligence will break into factions and most of its energy will be spent for these factions fighting each other, ignoring any common goal of increasing self-improvement.
So I don't see so called "singularity" as a possibility - maybe at most as another, faster type of evolution, which will ignore humans due to its own self-centeredness and humans will not comprehend it. But just as our culture depends on nature, this superintelligence will probably depend on us (human culture) to sustain its internal struggles.
- It somehow needs to be distributed, that is, composed of smaller computing parts, because there is a physical limit what you can do in unit of space.
- It needs to change to adapt to environment (learn), and so all the parts need to potentially change.
From this follows that the parts will be subject to evolution, even if they don't reproduce. And so the existence of the parts will depend on their survival. This, in my opinion, inevitably leads to evolution of parts that are "interested" in their own survival, at the expense of the "superintelligent" whole. And it leads to conflict, which can eventually eat up all the improvements in the intelligence.
Look at humans. Humanity (or biosphere in general) didn't become a superintelligent whole, capable of following some single unified goal. Instead, we became fighting factions of different units, and most of the actual intelligence is spent on arms races.
Anyhow, even if superintelligence is possible, I believe the problem of friendly AGI has a simple solution. We simply need to make sure that the AGI doesn't optimize anything, but instead takes the saying "all things in moderation" to its heart. That means, every once in a while, AGI should stop whatever goals it pursues and reflect on purpose of those goals, if it is not, by some measure, going too far.
You can argue that we don't actually know how to make AI to stop and think. I would respond, AI that cannot do that, and only pursues some pre-programmed optimum mindlessly, is not really general.