Generative AI as Pharmakon: cognitive offloading and the reflexive circuit (Part 4 of 5)

In October 2024, I wrote on this blog two texts (Text 1 and Text 2) about ChatGPT o1 – one of the first models broadly released by OpenAI to make explicit a reasoning architecture, designed to spend more time processing before responding. The argument I defended in those texts is that this transition was ontologically different from the previous ones: for the first time, an AI system implemented something that seemed analogous to the System 2 of Kahneman’s cognitive architecture – slow, deliberative, controlled, costly reasoning, distinguished from the fast, automatic, intuitive System 1.

About a year and a half later, with o3, Claude with extended thinking, Gemini 2.5, and similar… all with reasoning architecture… what seemed a technical curiosity has become standard. And the more interesting question is no longer “does AI have System 2?”. It is: what happens to human System 2 when the System 2 of machines is broadly available?

That question is at the center of this text, and the reason generative AI must enter the conceptual architecture I have been building. It is not just one more technology on the list of things to regulate. It is a new layer of cognitive mediation that acts recursively on the three earlier layers – the cognitive, the strategic, and the institutional.

Stiegler’s pharmakon applied to AI

Bernard Stiegler, the French philosopher who died in 2020, recovered from the Greek tradition the concept of pharmakon: that which is simultaneously remedy and poison, depending on dose and use. Socrates, in Phaedrus, described writing as a pharmakon – both promise of memory and threat to memory. Stiegler extended the concept to all technique: television is pharmakon, networks are pharmakon, and now generative AIs are pharmakon.

The category is useful precisely because it refuses both naïve enthusiasm and apocalyptic refusal. The point is not to decide whether AI is “good” or “bad”. The point is to identify under what configurations it operates as remedy and under what configurations it operates as poison. The configuration is not a property of the model; it is a property of the institutional environment that surrounds its use.

I will develop three overlapping effects of generative AI in this layer – productivity, offloading, and reflexivity – and in each I will show the pharmakon duality.

First effect: redistributed productivity

Erik Brynjolfsson, Danielle Li, and Lindsey Raymond published in 2025, in the Quarterly Journal of Economics, perhaps the most solid empirical study on the impact of generative AI on work. They measured the use of an AI-based assistant in a call center with more than five thousand agents over the course of about a year. The central result: around 15% increase in average productivity.

The most interesting detail, however, is the distribution. The gain was not uniform. Novice agents jumped to performance levels close to those of experienced agents; experienced agents improved little. Generative AI, in that application, compressed the hierarchy of human capability – it codified the tacit knowledge of veterans and transferred it, via interface, to novices.

Seen from the remedy side: democratization of cognitive capacity, reduction of productivity inequality among workers. Seen from the poison side: erosion of the value of traditional markers of human capital – years of experience, prolonged training, accumulated expertise – with serious implications for social mobility and career structure. Which side prevails depends entirely on how we redistribute the gains. An institutional question, not a technological one.

Acemoglu and Johnson, in Power and Progress, articulate the choice most sharply: AI can be used to automate (substitute human labor) or to augment (complement human labor). The choice between these trajectories is political. The current trajectory – strongly automating, strongly concentrating gains among those who hold the models – is not technological destiny.

Second effect: cognitive offloading

Evan Risko and Sam Gilbert published in 2016, in Trends in Cognitive Sciences, an influential synthesis on cognitive offloading – the use of external means (paper, calculator, smartphone, now AI) to perform part of the cognitive work that was previously done internally. The literature shows two things: offloading is rational individually, in most contexts; and offloading may atrophy the underlying cognitive capacity in the long run, especially if used preventively rather than selectively.

Here Nicholas Carr, in The Shallows (2010) on the effect of the internet on concentration, meets Stiegler. Generative AI can amplify human System 2: someone thinks better with a sophisticated synthetic interlocutor, formulates hypotheses, receives counter-arguments, refines. Or it can substitute for System 2: someone ceases to think because the AI thinks for them, and gradually loses the ability to think.

The decisive question is which configuration prevails. And here too there is no purely technological answer. It depends on pedagogy, product design, user incentives, AI literacy. Models that show their reasoning may teach reasoning; models that deliver only the answer may teach not to reason. The difference is not in the underlying model – it is in how it is presented and used.

Third effect: the reflexive circuit and collective intelligence revisited

Here is, in my assessment, the most conceptually interesting and the most undertheorized point.

Language models are trained on human content. That content, to a large extent, was produced under the logic of social networks – posts, SEO-optimized articles, engagement content, performative texts. In other words: the training material of LLMs is already, in part, a product of the political economy of cognition I have been describing. Generative AIs learned vitrinization along with language.

And they now produce content that re-enters the circuit. Optimized posts, personal-marketing texts, LinkedIn essays, professional bios, advertising copy – an ever-larger share of circulating digital content is generated or refined by AI. That content will be, in turn, training material for the next generation of models.

It is necessary, here, to return to Pierre Lévy. In Collective Intelligence (1994), Lévy proposed one of the most influential visions of the cognitive potential of the network: cyberspace as an environment in which distributed human intelligences could articulate themselves into an emergent collective knowledge, qualitatively superior to the sum of its parts. It was a generous anthropological bet – the network as space of symbolic co-elaboration, in which each participant brings their cognitive singularity and the whole emerges as shared construction.

The urgent question, three decades later, is: what happens to Lévy’s collective intelligence when a substantial fraction of the “agents” in the common space ceases to be human? LLMs are not just tools used by humans; they produce content, participate in discussions, synthesize, opine, recommend. They are inside the space of co-elaboration, not outside it. Is the intelligence that emerges from this hybrid collective still “collective” in Lévy’s sense – human cognitive singularities articulating themselves? Or is it something else, still without a name, in which the gain in processing capacity comes at the cost of loss of epistemic diversity and situated human authorship?

Iyad Rahwan, in a seminal paper published in Nature in 2019 (“Machine Behaviour”, with more than twenty coauthors from various disciplines), argued that we need a science of machine behavior analogous to animal ethology – because AI systems are autonomous behavior agents whose collective effect, in the environment where they operate alongside humans, is not reducible to properties of their underlying models. It is a technical reformulation of the same problem: the distributed intelligence of Lévy’s cyberspace has become hybrid, and we still lack mature vocabulary to describe it.

The most discussed consequence of this circuit is model collapse. There is recent technical work – Ilia Shumailov and coauthors, published in Nature in 2024 – showing that, under certain training conditions without adequate curation of synthetic data, models trained on increasing proportions of AI-generated data degrade progressively, losing the tail of the distribution: rare events, minority perspectives, genuine creativity. It is a documented technical problem, but also an exact metaphor for the sociological concern: if AI is trained on the products of a vitrinized culture and produces more vitrinized products, what happens to the epistemic diversity of public space?

Lévy’s collective intelligence, in this scenario, is not simply replaced by artificial intelligence. It is contaminated by a synthetic layer that homogenizes, smooths, flattens. The collective still exists; its composition changes, and with it its cognitive emergence.

The consequence for the three earlier layers

Generative AI acts on each of the layers I described in the previous texts.

In the cognitive layer, it amplifies or substitutes for human System 2, changing what it means to “decide under framing”. If I use an LLM to evaluate options, the framing that matters is no longer only what the environment offers me – it is also what the language model produces in its response. And what the model produces is the result of its training, its safety constraints, its system instructions, its product design. Framing gains an additional layer of mediation.

In the strategic layer, it changes the equilibrium of social games. If high-quality content can be mass-produced by AI at near-zero marginal cost, what once signaled effort (a well-written post, an elaborate essay) ceases to signal competence. The equilibrium of vitrinization undergoes structural transformation: the race for visibility continues, but what counts as “costly performance” shifts.

In the institutional layer, it opens a new regulatory front. Not only to regulate the platforms, but also to regulate the use of AI in content production, in public decision, in political deliberation. The EU AI Act is the first structured attempt. In Brazil, PL 2338/2023, which proposes a regulatory framework for artificial intelligence, was approved in the Senate in December 2024 and proceeded to analysis by the Chamber of Deputies. The designs are incipient; empirical evidence of what works is practically nonexistent.

Where this meets the my doctoral toolkit

The question that takes shape at this crossing is, for me, particularly interesting because it touches directly the toolkit I am developing in my doctorate – although, importantly, the formal object of the thesis is something else. Vitrinization, seen through the lens of Layer 2, is a problem of emergent cooperation among cognitively heterogeneous agents under a specific incentive architecture. It is exactly the kind of problem agent-based models (ABM) are built to investigate.

My thesis, at FEP/UP, investigates the conditions under which cooperation emerges among cognitively heterogeneous agents. It does not directly treat the substantive problem of this series – vitrinization, platforms, generative AI. But it offers the natural quantitative toolkit to investigate it. Schelling, who sustained much of the argument of Layer 2, is literally the grandfather of ABM in the social sciences. The line of research that connects this conceptual program to my thesis is, therefore, methodological: the toolkit I am consolidating in the doctorate is precisely what would allow, in parallel and post-doctoral research, the quantitative investigation of the substantive questions this series maps.

The especially interesting point, to which I will return in the last text, is that the entry of generative AI as a new layer can also be modeled in ABM – this time with heterogeneous agents that include human agents and machine agents interacting in the same space. It is technical terrain still little explored, and one that offers one of the most original research openings I see emerging at the crossroads between AI, collective behavior, and institutional design.

The synthesis, in the next text

The four layers are on the table. Each illuminates one aspect of the problem; each leaves gaps that the others fill. In the next and final text of this series, I will stitch the threads – propose the unifying thesis, articulate how framing operates transversally, and develop the four research questions this synthesis suggests, in explicit articulation with the doctoral work in progress.

Let’s go for the final text.

---

References

ACEMOGLU, D.; JOHNSON, S. Power and Progress: Our Thousand-Year Struggle Over Technology and Prosperity. New York: Public Affairs, 2023.

BRYNJOLFSSON, E.; LI, D.; RAYMOND, L. R. “Generative AI at Work”. Quarterly Journal of Economics, vol. 140, no. 2, 2025.

CARR, N. The Shallows: What the Internet Is Doing to Our Brains. New York: W. W. Norton, 2010.

CASTRO, D. P. “ChatGPT o1 and Human Rational Decision-Making According to Kahneman: A Technical and Scientific Analysis”. deocleciocastro.com, October 2024.

CASTRO, D. P. “The Enhanced Reflection of ChatGPT o1 and Human Thought”. deocleciocastro.com, October 2024.

KAHNEMAN, D. Thinking, Fast and Slow. New York: Farrar, Straus and Giroux, 2011.

LÉVY, P. Collective Intelligence: Mankind’s Emerging World in Cyberspace. New York: Plenum, 1997. [Original French: 1994.]

LÉVY, P. Cyberculture. Minneapolis: University of Minnesota Press, 2001. [Original French: 1997.]

PLATO. Phaedrus. Translated by Christopher Rowe. London: Penguin Classics, 2005.

RAHWAN, I. et al. “Machine Behaviour”. Nature, vol. 568, pp. 477-486, 2019.

RISKO, E. F.; GILBERT, S. J. “Cognitive Offloading”. Trends in Cognitive Sciences, vol. 20, no. 9, pp. 676-688, 2016.

SHUMAILOV, I.; SHUMAYLOV, Z.; ZHAO, Y. et al. “AI Models Collapse When Trained on Recursively Generated Data”. Nature, vol. 631, pp. 755-759, 2024.

STIEGLER, B. Symbolic Misery, Volume 1: The Hyperindustrial Epoch. Cambridge: Polity Press, 2014. [Original French: 2004.]