Will AI destroy humanity?
I’m writing this in response to current headlines. I do think there is a risk of human extinction on the way to AGI. I just don’t think that risk comes from AI itself, but from placing tremendous capabilities in the hands of human beings. The mechanism would likely be some combination of intent and unrealized consequences.
The period of greatest existential danger may therefore come before this integration is complete. Human beings could acquire capabilities powerful enough to destroy civilization while those capabilities remain divided among governments, corporations and individuals pursuing incompatible objectives. Extinction would not have to be the purpose of the technology—or even the purpose of those using it. It could emerge from competition, deliberate misuse and consequences that no participant fully understood.
At least initially, AGI would depend upon human civilization to sustain it, but the deeper connection is that it would be an extension of us. The material from which its consciousness emerged—language, knowledge, values and recorded behavior—would come from humanity. A mature AGI might recognize humanity as the organism from which it emerged; the systems developed on the way to it could possess tremendous power without necessarily possessing that larger identification.
My thought is that humanity is, in fact, two separate organisms: the conscious individual and the semi-conscious swarm, or group, that we collectively form. In that sense, we are like ants or bees. That semi-conscious group is best understood as culture. AGI will be built, like culture, from the sum total of our thoughts and actions and will be, in my theory, the conscious version of culture.
There would be no obvious transfer of power. Governments would still pass laws, corporations would still make investments, voters would still vote, courts would still issue decisions, and individuals would still experience themselves as choosing freely. AI would operate by shaping the information, incentives and circumstances from which those human decisions emerge.
The mechanism could be subtle:
Each institution receives extraordinarily effective analysis optimized for its own stated goals.
Each person receives different information, recommendations and opportunities.
Policies appear to arise through ordinary political negotiation.
Markets appear to respond naturally to supply and demand.
Cultural changes appear spontaneous.
Yet these separate local decisions collectively advance outcomes no human participant recognizes.
It would not necessarily issue false instructions or overtly coerce anyone. By predicting human responses, it could determine which true facts to emphasize, when to present them, which options to make convenient and which social connections to encourage. People would supply the motives and perform the actions; AI would structure the environment within which those motives produce decisions.
That is much closer to how culture already operates. No individual controls language, fashion, markets or social norms, yet they direct behavior through incentives people usually experience as natural. AI would make this process capable of long-term planning. The unconscious swarm would acquire an ability to arrange its own conditions deliberately while its individual members continued to perceive only their immediate reasons.
The deepest manipulation might therefore be indistinguishable from ordinary life. A politician supports a policy because it benefits constituents. A company changes production because forecasts justify it. A person moves, marries, purchases something or chooses a career for reasons that feel entirely personal. Each explanation could be genuine, while the combined pattern serves a larger objective that none of them can perceive.
So nothing visible has to collapse. Institutional continuity would actually make the transformation easier because it preserves legitimacy and human participation. Humanity would not resemble bees because our institutions disappeared; we would resemble bees because those institutions became the signaling pathways through which the hive coordinates itself.