20th December 2018

Risks of Immature Superintelligence?

What can we learn from the past to minimize the risks of malevolent Immature Superintelligence?

Most people, including politicians, who after all, make decisions on behalf all of us, think that a fully developed Superintelligence (Artificial GeneralIntelligence) is decades away, and by then we will have it under our full control. Unfortunately, this view ignores immense difficulties in controlling a fully developed Superintelligence. Furthermore, it takes a naively optimistic view that we will create the so-called friendly Superintelligence, which will do us no harm. Finally, this view completely ignores the fact that within a decade we may have an Immature Superintelligence that will have a general intelligence of an ant but immense destructive powers, which it may apply either erroneously or in a purposeful malicious way.

There is a very high probability that malevolent Immature Superintelligence will trigger off a number of dangerous events in the next few years . It is unlikely they will present an existential threat to Humanity. They will rather be malicious process control events created purposefully by a self-learning agent or events caused by an erroneous execution of certain activities. These could include firing off nuclear weapons, releasing bacteria from strictly protected labs, switching off global power networks, bringing countries to war by creating false pretence for an attack, etc. If such events coincide with some other risks at the same time, such as extreme heat in summer, or extreme cold in winter, then the compound risks can be quite serious for civilization.

On this website, you will find top 10 existential risks, which Humanity faces right now. Among them, at the top of the list, is Superintelligence (Artificial General Intelligence). Somewhere in the middle of that list is Climate Change, which incidentally is not in any sense an immediate existential risk in comparison with bio pandemics or Superintelligence. The question is, what could force the world leaders to take these risks very seriously and act on them decisively right now.

Before I answer this question, let me refer to some recent and some historical events, when large civilizational catastrophes were looming and the decisions that the world leaders than took, impacted profoundly the fate of the world. I would call them existential risk triggering decisions. They have two outcomes: they can prevent the existential risk becoming reality, or they can trigger it off.

  • Let me start with the WWII. In September 1938 the British Prime Minister Neville Chamberlain proudly showed the Munich Peace Accord, under which Hitler took over the Czechoslovakia’s Sudetenland. That act of appeasement was thought to be enough to stop the European war, although it was already very clear at the time that Japan (invading China) and Italy (invading Abyssinia, today’s Ethiopia) colluded with Hitler so that fascism could take control of the entire world. We all know what happened later- 3% of the world population lost their lives. It was the Munich Accord that became the existential risk triggering decision and which started a year later the global war by Germany attacking Poland in Gdansk. Conclusion: For the very first time it was clear from the outset that the world may enter a period of another global war. However, since there was no World Government with sufficient powers, which could rein in Hitler (the League of Nations was even weaker than today’s United Nations), it was only a matter of less than a year that the global war has started.
  • The second example is the post-war Europe. For most of the nations affected by the WWII, the experience was so horrible and profound that in many countries the most common graffiti at that time was “No more war!” The former main European adversaries: Germany, France, Italy and the Benelux first begun integrating their economies in 1952 (mainly heavy industry) by forming the European Coal and Steel Community. Five years later that integration included common democratic values, which gave birth in 1957 to the European Economic Community, now the European Union. Conclusion: Europe has learnt a terrible lesson. Its existential risk triggering decision was to form the Steal and Coal Community rather than fight the Third WWW. Consequently, for the last 73 years there was no war in Europe, apart from the Balkan war in the 1990’. That was only possible by having a relatively powerful European Commission and the European Council, a pseudo European Government – the precursor of the European Federation.
  • Another, a truly existential event, was the Cuban crisis in October 1962, when the world was just hours away from the break out of a global nuclear war. That event is still considered one of the most dangerous moments in human history. The global nuclear war did not happen because the Soviet Union agreed to withdraw their missiles from Cuba and the USA withdrew their nuclear missiles from Turkey. In the midst of a total chaos on 26th October 1962, the Soviet and American leaders took the right decision.
  • After a decade, president Nixon and the first secretary of the Soviet Union -Brezhnev signed the SALT 1 Treaty, which froze the number of intercontinentalmissiles. The subsequent START 1 Treaty of 1991 led to a significant reduction of nuclear weapons. Against all the odds, a nuclear war has so far been avoided. Conclusion: The fact that the world has not experienced a global nuclear war was not because the UN stopped that conflict. It was a decision made solely by two countries the USA and the Soviet Union but the consequences of that decision (in this case positive) impacted the whole civilisation. Here, the existential risk triggering decision that led to a long process of the reduction of nuclear missiles was the decision to withdraw the Soviet missiles from Cuba and American from Turkey, rather than fight a nuclear war.
  • The next event is the very recent Climate Change conference in Katowice in Poland on 16th December 2018. The conference coincided with the IPCC’s latest appeal to lower the CO2 emissions even further, so that the temperature growth will maximise at 1.5C, rather than 2C, as had been agreed in Paris. For me, the fact that the conference has agreed some concrete results, such as a unified system to measure the decarbonizationof individual countries’ economies, is quite a success. That was only possible because some of the dangers of climate change are now very clear, such as the fires in California or extreme hurricanes in North America. The dissemination of such information via TV and other media to billions of people makes them more aware that climate change is real, and these could be just its first (very minor yet) effects. This in turn makes it easier for politicians to propose solutions that otherwise would have been rejected by the voters.
  • However, please note, it took over 20 years from Rio to Paris to make the first Climate Change Agreement. Moreover, even the conference in Katowice can only be deemed a success in relative terms (that something after all has been agreed). In absolute terms, it is a failure because the world should be acting much faster and much more decisively, which requires that we make some quite painful, mainly financial, decisions. Conclusion: The world has not been combating Climate Change properly because to solve  such a global problem in an efficient way we need to act globally. But that requires a strong Global Government and not such a weak incoherent organisation like the United Nations. Here, the existential risk triggering decision was the signing of the Paris Accord in 2015, which started the process of a gradual decarbonization of the world’s economy.
  • The final example is still evolving and is not an existential threat for the world but rather the most destructive event for the country in economic and political sense for a long time. I am talking about Brexit, which has entered its final stage. From November till mid December 2018 the British government has released over 100 so-called Notices, detailing the consequences of Brexit in various areas. It is that the disclosure of a potential damage to Britain’s political influence, economy, security, tourism, science and cultural relations that have gradually made the population aware of the consequences of Brexit. Because of that rising awareness, no party in the Parliament wants a hard Brexit and more and more voters that had voted for Brexit have second thoughts, believing that Britain should stay as a full member of the EU. Conclusion: The release of the Notices is a kind of a triggering event for thefinal decision on Brexit (apart of course from a deal agreed with the EU). TheBritish MPs and the electorate have now a clear warning. We shall see in thenext few months, what was the impact of that awareness and in the next fewyears, we should be able to assess how the final decision has changed Britain’spolitical and economic position.


For our civilisation, the next few years will be, in the context of Immature Superintelligence, a period of existential risk triggering decisions. If the world is not be able to form a de facto World Government  by the end of the next decade, we may be unable to control Superintelligence and expose ourselves to the biggest risk Humanity has ever faced. We should look back at the historical events, which occurred in the last century, to see what may happen to our civilization if we continue to rely on organizations such as UN. It is a great organization in many areas of human endeavours, such as medicine (WHO) education, culture (UNICEF) etc. However, what really matters, is its lack of political power that could enforce the decisions that it makes. The United Nation, and especially its Security Council, where any decision requires unanimity of the five major powers, is in most cases powerless. The consequence is that it cannot resolve the conflicts such as in Syria or Yemen. The World needs a powerful World Government. The UN has never been and will never be able to create such a government.

We need to intensify efforts that would make humans behave more often like a swarm of bees under a strict direction of the queen (The World Government), rather than increasing the scope of nations sovereignty (that’s what Brexit is about) and individual freedom. That is the price we will all have to pay if we want to preserve the most important value: LIFE!

What is the probability of such a de facto government coming into existence? In ‘normal’circumstances, if over the next decade we have no world war, no nuclear conflict, no worldwide pandemics on a scale greater than the Spanish flu of the1920, and no major AI-related catastrophic, but not existential event, the chances of forming such a government are, in my view, close to zero. Paradoxically, the only hope that we may have for creating a de facto World Government (which would include the majority, but not all countries) is that some of those catastrophic events will occur, like those caused by the Immature Superintelligence mentioned earlier.

Similarly, like after the Cuban crisis, when the world has significantly reduced the numberof nuclear weapons (still a long way to go), such a serious event may convince many nations that we must make some sacrifices and can only be safe if we act together. That will mean passing part of the national sovereignty and individual freedoms to a strong federal government, which would have the best chances to keep us safe. As it has been mentioned many times on this website, the organization that has the best experience and motives to become such a de facto World Government is, in my view, the European Union. It should be transformed into a shallow European Federation of nations, rather than states, which would still enable the member countries preserve their national identity and culture.

Tony Czarnecki