The Sky’s the Limit, But …
At this point of this blog, the “winner takes all” outcome should be obvious.
Once an ultra-intelligent, artificial general intelligence exists and is able to self-improve and operate beyond human understanding, it may be directed not just to solve a single problem but all (solvable) problems. It can invent novel ways to improve itself.
As IJ Good reasoned in 1965:
Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever. Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control.
With AI, the possibilities for improving society are limitless.
At the same time, clearly there is great risk for unintended consequence and bad actors. Since winner takes all, corporations, governments and others all will race each other to be the first. As an industry and a society, we need to design and implement safeguards with equal urgency.
It’s unlikely that the meaningful protections will be as simple and elegant as Asimov’s Laws (also known as the Three Laws of Robotics), which have been both widely popularized by Hollywood and widely criticized by experts as too limited.
I suspect we will discover that the only way to protect the human race from an ultra-intelligent AI is… you guessed it: an ultra-intelligent AI.