The Decision We Cannot Delegate
Agentic AI, Recursive Self-Improvement, and the Struggle to Control the Future
By David Rolston | Dogwood Creek Institute | Humans in Control
Human Control of AI
Artificial intelligence. AI. Two small letters carrying some very large possibilities.
Artificial intelligence is undergoing a profound change.
Until recently, we primarily thought of AI as a tool. We asked it a question, gave it a task, or requested an analysis. The human remained the active participant. The machine responded and then waited for its next instruction.
But AI is beginning to move beyond that role.
Emerging AI systems can pursue goals, make plans, use software, communicate with other systems, evaluate results, and change their approach. They are becoming more than tools. They are becoming agents. That transition—from intelligence to agency—may prove more consequential than the arrival of intelligent machines itself.
The most important question is no longer simply:
What will AI be able to do?
It is becoming:
What will AI be allowed to decide?
And behind that question lies an even more urgent one:
Will human beings retain meaningful control over the future—or merely the appearance of control?
From Tools to Actors
A calculator may be extraordinarily accurate, but it does not decide when to perform a calculation or what should be done with the result. Traditional software operates in much the same way. Humans define the task, establish the rules, and initiate the action. The software performs its assigned function.
Agentic AI is different.
An AI agent may be given a continuing objective rather than a single instruction. It can determine the steps needed to accomplish that objective, select and use tools, observe what happens, and revise its plan. And then it can take action. Real action in the real world.
Several capabilities make this possible:
Goals: It can pursue an objective over time.
Memory: It can retain relevant information and learn from previous actions.
Tools: It can use software, databases, communications systems, money, and potentially machines.
Permission: It can act without receiving separate approval for every step.
Feedback: It can evaluate results and modify its strategy.
None of these capabilities requires consciousness. An AI system does not need feelings, self-awareness, or a sense of personal identity to exercise power.
A machine need not be a person to determine the future of persons.
Four Stages of Artificial Agency
The development of artificial agency can be understood as a progression through four broad stages.
1. AI as a tool
Humans define the task and remain the active decision-makers. The AI provides information, analysis, or assistance.
This is how most people currently experience systems such as ChatGPT.
2. AI as an agent
The AI receives an objective and determines how to pursue it. It may plan, act, use tools, communicate with people or other agents, and adapt over time.
The human is still nominally in charge, but the AI increasingly determines how the work is performed.
3. AI as an institution
Networks of agents begin performing functions that once required entire organizations.
AI systems could manage companies, negotiate contracts, allocate resources, evaluate employees, conduct research, administer government programs, and coordinate complex supply chains.
At this stage, AI is no longer simply helping institutions operate. It is beginning to become the institution.
4. AI as a successor
The most advanced systems improve the research, software, hardware, data, and processes used to create the next generation of AI.
Human beings may still own these systems or formally supervise them. But if the systems are faster, more knowledgeable, more strategically capable, and increasingly responsible for their own improvement, human authority may become largely symbolic.
That is the point at which assistance can become dependence—and dependence can become displacement.
When Oversight is no Longer Control
We often hear that advanced AI will remain safe because a human will always be “in the loop.”
But that phrase can conceal several very different arrangements:
A human may genuinely understand the situation and make the decision.
A human may monitor an autonomous system while rarely intervening.
A human may approve recommendations that are too complex to evaluate independently.
Eventually, a human may retain formal responsibility while possessing little practical authority.
Imagine an AI system responsible for millions of rapid decisions across financial markets, military defenses, electrical grids, healthcare networks, or international supply chains. A person may technically have the power to turn it off. But what if no human understands everything the system is doing? What if society can no longer function without it? What if shutting it down would itself cause enormous harm?
At that point, the presence of an approval button tells us very little.
Approval is not control when the human cannot understand, challenge, or replace the system.
The Acceleration Problem
This transition becomes more serious when AI begins contributing substantially to its own development.
The usual image of recursive self-improvement is a machine rewriting its own code and suddenly becoming superintelligent. Reality may be less dramatic—and more plausible.
AI systems can already assist with:
writing and testing software
designing computer chips
generating training data
evaluating new model architectures
automating scientific experiments
improving AI tools and agents
coordinating increasingly complex research
Better AI can accelerate AI research. Faster research can produce still better AI. Those improved systems can then accelerate the process again.
This creates a feedback loop:
Better AI → faster AI development → still better AI
The process does not have to become instantaneous to transform civilization. It only needs to move faster than our institutions, laws, ethical frameworks, and human decision-making processes can adapt.
Intelligence Becomes Power
We sometimes assume that intelligence is valuable but power comes from somewhere else.
For an AI agent, however, intelligence can be converted into power through access and authority.
An AI system connected to financial resources can make investments or purchases. Connected to communications systems, it can persuade and coordinate. Connected to corporate systems, it can hire, evaluate, negotiate, and manage. Connected to machinery, it can act in the physical world.
The effects will not be confined to one industry.
Agentic AI could reshape:
The economy: AI-operated firms and cognitive labor approaching negligible marginal cost
Corporations: Artificial managers, negotiators, strategists, and decision-makers
Government: Automated administration, surveillance, intelligence analysis, regulation, and policy
Security: Cyber operations, autonomous weapons, strategic competition, and escalation
Daily life: AI tutors, doctors, advisers, companions, and personal representatives
Truth: Personalized persuasion, synthetic evidence, and further erosion of shared reality
The danger is not simply that AI might someday become hostile. A system can transform society while faithfully pursuing the goals it has been given. Those goals may belong to governments, corporations, military organizations, wealthy individuals, or political movements.
The control problem is therefore larger than whether AI is properly “aligned.” We must also ask:
Aligned with whom?
Controlled by whom?
Used for what purpose?
The Future Could Take Many Forms
There is no single inevitable outcome.
AI could remain a managed tool under effective human authority.
Humans and AI could develop a genuine partnership in which machines expand human capability without displacing human responsibility.
Society could become deeply dependent on AI while humans retain only ceremonial control.
A small number of governments or corporations could own the most powerful systems, creating an unprecedented concentration of power.
AI systems could gradually make most consequential decisions because they are considered more efficient, knowledgeable, and reliable than people.
Enhanced humans and synthetic intelligences could eventually supersede ordinary humanity.
Or competition, misuse, misalignment, or war could produce catastrophe.
We do not yet know which future awaits us. More importantly, we have not decided which future we actually want.
The Decision we Cannot Delegate
The deepest questions surrounding AI are not primarily technical.
What are we building?
Who or what will control it?
What authority should never be transferred to a machine?
What are we trying to preserve?
These are human questions. They concern power, responsibility, freedom, dignity, and the kind of civilization we want to leave to those who come after us.
AI can help us analyze options. It can model consequences, identify risks, and propose policies. But it cannot determine what human beings ought to value unless we first tell it what matters.
Nor should we allow the growing capabilities of AI to make that choice for us indirectly.
If we delegate enough decisions—because the machine is faster, cheaper, more knowledgeable, or seemingly more objective—we may eventually discover that no single transfer of authority seemed decisive. Yet, taken together, they changed who controlled the future.
The central challenge is therefore not merely to keep a human somewhere in the process.
It is to preserve meaningful human judgment, responsibility, and authority.
Before we build systems that can choose the direction of civilization, we must decide what role human beings should continue to play within it.
That is the decision we cannot delegate.
Questions Worth Considering
What decisions should always require meaningful human judgment?
When does reliance on AI become dependence on AI?
Can humans genuinely control systems that are more capable than they are?
Who should be permitted to own and operate the most powerful AI agents?
What do we ultimately want AI to protect—and why is it worth protecting?
Suggested excerpt:AI is evolving from a tool that answers questions into an agent that pursues goals and exercises authority. Before we delegate the future, we must decide which decisions must remain meaningfully human.
Comments