Conjuring a magic encyclopedia

For now, assume with me that it can be done. Wish away with me the various technical and financial challenges. Great! Now we can rapidly conjure up a deeply, fully researched knowledge base on any topic. All claims point back to who’s said them, in what context, and (importantly) with what justifications and evidence (if any). Any quibbles or nuances which have been expressed on a point are similarly readily available. It’s not opinionated: all competing viewpoints with their associated justifications are associated and comparable.

That’s way, way too much information! Imagine trying to read everything ever about diet or shipping or taxes or microbes. It’s not happening. So as well as this, we now magically have tools which gather similar points together, summarise, and can make a decent stab at which points we’ll consider most or least relevant. We can dig deeper (or send AI agents to scout deeper) as desired. And when new interesting and informative content arises, or in contexts where nuance and clarification are helpful, it can be bubbled up to our attention.

All this is doable today: enough web searches, enough cross-referencing of tweets, articles, journals, following of citation chains, gathering and comparing of hypotheses and points of view, etc. will make progress. But it’s exhausting. When someone does go to those lengths, their partial — but heroic — efforts to map out what’s been said often languish either unpublished or unrecognised.

Don’t we already have this? A shining example is Wikipedia, where the collective curatorial effort of a wide range of editors gradually maps out an expanding core of topics and commentary. But Wikipedia has lags, biases, and (perhaps most importantly) huge gaps, especially on important frontier questions. (Let’s not talk about Grokipedia.1)

Meanwhile, the tech to ‘smartly browse’ and bubble up informative pieces is nascent too, in bits and pieces like AI chatbots and community notes: already useful in their ways, but faltering, unreliable.

It’s these comparison points and the early progress I see which gives me some excitement that the grander vision is viable and that we can take steps towards it now.

Who cares?

I’m not naive. I know that many (most?) humans a lot of the time aren’t actually interested in finding out or sharing what’s true; mainly they want to say what makes themselves and their friends seem popular and cool… and enemies seem dastardly and disgusting. We all have these impulses to greater or lesser extent. Yes, you too! Sometimes those impulses seem deranged (they’re not designed for the modern world); other times they might even make sense, at least selfishly.

Nevertheless (and perhaps mysteriously!), a lot of the time, some people actually want to find out true things and share them. (I do! Do you?) Hence journalism, science… even hearsay and rumour (at their — perhaps rare — finest). We recognise that, when they’re actually anchored and doing their best to be right (or at least less wrong), those are absolutely foundational to wellbeing and prosperity in a modern society. Without (good) journalism, politics runs astray and tyrants abound. Without (grounded) science and technology, public health suffers, food supply and shelter and infrastructure decay, and progress falters.

As Ben Goldhaber and I previously wrote:

Knowledge is integral to living life well, at all scales:

• Individuals manage their life choices: health, career, investment, and others on the basis of what they understand about themselves and their environments.

• Institutions and governments (ideally) regulate economies, provide security, and uphold the conditions for flourishing under their jurisdictions, only if they can make requisite sense of the systems involved.

• Technologists and scientists push the boundaries of the known, generating insights and techniques judged valuable by combining a vision for what is possible with a conception of what is desirable (or as proxy, demanded).

• More broadly, societies negotiate their paths forward through discourse which rests on some reliable, broadly shared access to a body of knowledge and situational awareness about the biggest stakes, people’s varied interests in them, and our shared prospects.

  • (We’re especially interested in how societies and humanity as a whole can navigate the many challenges of the 21st century, most immediately AI, automation, and biotechnology.)

But our knowledge-producing institutions are plagued by publish-or-perish and clickbait incentives alike2 — and the social media landscape is even worse, riddled with misinformation and brainrot from all political quarters. I care about this. So do you, I daresay.

I especially care now, as society is poised before a series of important decisions about our future relationship with technology, especially AI. It could be ruinous, with tyranny, neo-feudalism, or extinction real prospects. Or it could be fantastic. Just wanting it to be OK isn’t enough: we have to seek, generate, share, and defend important knowledge — about developments in technology, as well as about trends in politics and power — and act on it.

How do we actually help?

There’s no single path or silver bullet. But the incredibly high-level picture is: better communication of knowledge is usually good. It helps people be more informed and make better decisions according to their needs. A better shared understanding makes it easier for people to work together toward shared goals (even if they don’t agree on all priorities). On average if people make better decisions and can work together better, we’ll get more flourishing and less catastrophic risk.

We’re trying to stimulate one piece of this picture with the knowledge-base direction. Heavy-handedly adjudicating what’s true rarely works.3 Instead, equip people with the fullest picture possible, as accessibly as possible, and we find our way: evidence adds up, and when it doesn’t, that means we need to look for more.

As Scott Alexander wrote years ago (emphasis partly mine),

Logical debate has one advantage over narrative, rhetoric, and violence: it’s an asymmetric weapon. That is, it’s a weapon which is stronger in the hands of the good guys than in the hands of the bad guys. In ideal conditions (which may or may not ever happen in real life)... when done right, it can only prove things that are true. … Unless you use asymmetric weapons, the best you can hope for is to win by coincidence.

I’m not focused on logical debate per se, and in any case I wouldn’t be so Manichean about it — we’re all ‘good guys’ sometimes and ‘bad guys’ sometimes (whether we mean it or not) — but the articulation is compelling. Humanity has accrued a slowly-growing arsenal of these asymmetric weapons: libraries, citation, scientific review4, databases, encyclopedias, web search, to name a few. Today they’re creaking under the weight of a confusing information deluge and assaulted by powerfully-vested interests.

I earnestly believe that an upgrade to truth-seekers’ ability to find and scrutinise information, to build and share fuller pictures of topics at hand, can be ‘infectious’: more people more of the time can see a little further, pierce a little more of the fog of confusion and misinformation, be better epistemically defended, and embody — and exemplify — truth-seeking cognition. When people (through malice or negligence) spread confusion and falsehoods, they’re that bit more likely to face scrutiny and consequences. After all, we’re making that scrutiny cheaper, easier, and more accessible.

This applies whether the ‘wielders’ of these new weapons are curious members of the public, scientists, analysts in public institutions, business leaders and technologists, or even the AI assistants those folks recruit to accelerate their work.

In politics, that can mean that people engage more often in collaborative, truth-seeking cognition and less in tribal cognition. And in technology, it can mean that more people can stay better abreast of the important shifts and prospects that will shape our future — helping the public hold decisionmakers to account (and choose better ones), and helping those decisionmakers sincerely and deeply engage with the topics at hand. I want these kinds of epistemic heroics to become commonplace, and I want the epistemic giants among us to stride further still.

Let’s do it!

You can express interest right away to receive commentary, information, and updates — whether you’d like to participate or are just interested in the outcomes of the competition.

The heights of human epistemic investigation are impressive and valuable, but rare and difficult to reach — see our abridged collection of strong examples.1 The limiting factor is rarely exquisite insight (though this helps!), and more often diligence, a curious and open mindset, and the time and effort needed to do the thorough work investigating background on a topic: activities AI is well placed to assist with.

Existing AI-assisted knowledge base work demonstrates real pieces of this — agent memory (e.g., Claude Code’s memory and skills), LLM-curated personal wikis (Karpathy’s perhaps the highest-profile), and deep-research tools. But these mostly produce single-user artifacts tuned to one investigator’s context, not the kind that travel, combine, or survive (especially adversarial) scrutiny.

We’re particularly excited by the compounding potential — if structured analyses2 become reusable, refineable artifacts, every serious investigation enables future work, on the same or related topics, and by the same or different people, to reach further from a more solid epistemic foundation. Who knows, you might even solve debate!

This competition provides three challenging case studies — with deliberately varied challenge profiles — and invites you to produce tooling and techniques to help people navigate them. First, the debated and impactful question of COVID-19 origins. Second, the risk that the Large Hadron Collider (LHC) creates synthetic black holes (perhaps destroying the Earth). Third, the health impact of eggs (as a human food source). The tooling should be general: we’ll judge against these and also other difficult case studies.

What we’re looking for

We want to see workflows and methodologies using AI that advance the state of the art in carrying out epistemic investigations and producing compounding knowledge bases. We aren’t asking you to build an entire, robust, fully-featured system. Instead, we’re excited by any submission that advances the state-of-the-art on a component.3

We’ve found it useful to think of these investigations as being split into several different layers: ingestion, structure, and assessment (more here). When stacked together and operating in concert, they’d create useful trusted artifacts. Something like a superior deep research, generating and interacting with a structured knowledge base, aimed at the truly epistemically discerning consumer.

Below are a set of ideas for potential desiderata for a workflow. We’d expect most submissions to not be solely focused on a single layer, as we’re guessing for something to be useful it needs to work across the layers — but some discipline in separating these responsibilities may be useful for producing interoperable, shareable, compounding benefits.

Ingestion

How do you take a messy, multi-source evidence base and turn it into something structured enough to reason over?

• Extract and attribute claims to specific sources, with provenance metadata (who said what, when, in what context).

• Identify when the same claim appears across multiple sources in different forms.

• Search for resources with bearing on topics and subtopics at hand.

• Capture useful metadata tags. For example relating sources and claims to topics and other sources (toward structure) or about methodologies, deference, and assumptions (toward assessment).

Structure

How do you document the relationships between claims so that the full shape of the argument becomes navigable?

• Resolve the inference structure: which claims and evidence are offered as support for which other claims.

• Represent the discourse structure: where people are addressing different sub-questions and perhaps how they are tracking those relating to an overall inquiry — there may be explicit, and sometimes implicit, differences of emphasis.

• Capture relationships regarding “similar but not identical” claims. These could be different ways of framing conditions or caveats to statements, or different estimates of uncertainty for quantities or propositions.

• Track how the structure evolves over time.

Assessment

How do you evaluate what to actually believe, or what to look at next, given everything above?

• Identify rhetorical moves that carry more persuasive weight than evidential weight.

• Flag correlated evidence being treated as independent.

• Identify cruxes, i.e. the specific factual or inferential disagreements that, if resolved, would most change the overall picture (perhaps drawing on debate structure).

• Surface what’s missing — important sources or perspectives that aren’t represented in the working knowledge base, toward further data collection (or hinting at additional primary information collection and reasoning).

• Provide frameworks for calibrating confidence that account for out-of-model error, adversarial information environments, and the limits of any single analyst’s expertise.

• Distinguish what the debate settled from what it merely performed settling.

What a good entry looks like

We’ll offer a minimum of $5k to entries which we judge to meaningfully improve on the state of the art in faithful, scalable AI-assisted investigations, and up to $50k for entries which are truly inspiring to us. This might be by (for example) reliably producing accessible, thorough, highly-interoperable knowledge-enabling content across diverse domains which is readily shared and expanded on by others.

We aren’t prescribing a single, specific type of submission4. A couple shapes we’d be excited to see:

• A spec describing a step-by-step process of a human-AI workflow for producing a structured epistemic analysis of a complex dispute. Demonstrate it on multiple part(s) of at least two cases. The workflow can incorporate human steering and be subjective in places, but should let others (even with differing beliefs and preferences) usefully pick up where another left off, and it should gracefully scale to mostly-or-entirely ‘hands free’. Make clear where your design choices are uncertain, and be transparent about where you’re making tradeoffs, and why.

• A prototype tool (most likely a pipeline involving LLMs) that implements one or multiple layers of the stack, demonstrated in a repeatable way on each of the case studies. Minimally, it should substantially accelerate users’ investigation of a topic, and ideally it should produce reusable, shareable knowledge artefacts which stand up to adversarial pressure.

• A protocol enabling interoperability and compounding without flattening the underlying material, demonstrated with reference to our cases. How can we navigate the tension between interoperability and nuance? What does a format look like that’s flexible and general enough to link diverse subtopics and complex, multi-perspective investigations while preserving important detail? How can it be maintained over time in a way that plays well with newly-emerging sources, a diverse and changing user base, and an expanding frontier of AI capability for tooling?

A submission might be of a different shape, look like one of these, or may combine these (for example a spec including protocol discussion and a reference prototype). Some stepping-stone alternatives which could contribute to putting a team in a great position to achieve the biggest wins (but which we expect are unlikely to win the biggest prizes without follow-up work):

• A comparative analysis repeatably5 applying two or more different AI assessment methodologies to the same (sub-)questions from the topic, with explicit discussion of where they agree and diverge. What downstream considerations do they best enable? What are their strengths and shortcomings? What kinds of supporting epistemic metadata would help them to work better?

• A critique with counterexamples of an otherwise promising approach, demonstrating the importance of further work or indicating less tractability than we might have thought.

Optionally, submit a description of your plan or a briefer, less complete implementation of it by Jun 21, 2026, and we will weigh in on whether the work seems on track for a prize (and potentially provide feedback). Use the main submission form and check the early feedback box.

What we care about most: Would this actually help someone reason better about this case? Does it generalize? Does it scale with improvements to AI or more compute? Does it compound, with multiple people or teams building on each others’ work?

We’ll ask judges to use the following criteria when assessing submissions: Epistemic Case Study Competition - Judging Criteria.

In addition to the potential prizes, strong entries that demonstrate real promise may also lead to an offer for further funded work with us (we estimate an 75% chance that a $50k-winning entry receives an offer like this).6

Please use this linked form to submit your entry; entries are due by Jul 19, 2026.

FLF’s general contest rules apply.

Prize structure

We’ve allocated roughly $200k for this competition with the size of any individual award reflecting how much an entry moves us. We’d rather award fewer, larger prizes for entries that genuinely impress us than spread the pool out. If a wave of strong work arrives, we’ll happily expand the total prize pool.

Concretely, we expect to award up to:

• $50k → for an entry or entries we find truly inspiring. The kind of submission that changes how we think about the problem or that we’d want to point to as a new reference point for AI-assisted epistemic work. We may not award it at all – or – we may also award it more than once if multiple entries clear that bar.

• $5k to $50k → for entries that meaningfully advance the state of the art, whether across the full stack or on a well-defined piece of it (ingestion, structure, or assessment). The size of each award reflects how far the work pushes the field and we expect several entries to land somewhere in this range.

• Continuation funding → beyond prize money, we expect to fund individuals or teams to keep building, on terms agreed case by case. For the strongest entries this may be the real prize: an ongoing relationship with FLF and a path to sustained work on the stack. We’ll raise this with finalists after judging.

Interested in participating or following along?

Want to compete, follow along, or join the conversation? Express interest to receive updates, commentary, and see how you can participate as the competition unfolds.

Why we’re doing this

We’re building toward what we call a full epistemic stack, layered infrastructure for making the provenance, structure, and assessment of knowledge transparent and traversable at scale. We think recent AI advances make this newly tractable, but the hard problems are in methodology and workflow design, as well as usability, not just capability.

Not only do we expect these tools to be of widespread benefit, but we expect some organizations like ours to be eager early adopters. FLF hopes to meaningfully inform its strategy and prioritisation based on insights from these tools, meaning that great work here could move millions of dollars per year and help us (and others) be more effective.

We’re excited to see what you build.

Much gratitude to Ben Goldhaber (formerly FLF), Joel Chan, Saif Haobsh, Austin Chen, Andreas Stuhlmüller, and Dustin Kimmel for contributions and feedback.

The case studies

COVID

In early 2024, a $100,000 judged debate took place between Saar Wilf (founder of Rootclaim) and Peter Miller on the origins of COVID-19. Over 15 hours of structured argument, two smart people marshalled epidemiological data, viral genetics, Bayesian inference, and institutional analysis to reach opposite conclusions. Two expert judges ruled decisively for zoonosis. Six independent Bayesian analyses of the same evidence spanned 23 orders of magnitude.

For more read Scott Alexander’s detailed writeup. We feel that the debate videos, judge decisions, and comment threads it links to form one of the richest publicly available records of a complex real-world epistemic dispute on an important issue.

And yet all this information is still incredibly difficult to navigate, interrogate, and use to inform one’s beliefs.

• It requires significant background expertise to understand the state of play of the debate and make a considered judgement. The debate was overseen by two judges with PhDs who work as a professional microbiologist and applied mathematician, respectively.

• The format, a live video debate, may not be the optimal way for a judge to interact with the material.7

Further, this intense epistemic effort represents a point in time in a conversation which continues to evolve.

We feel this makes it a strong stress test for tools and methods that aim to make reasoning more transparent, traversable, updateable, and trustworthy.

Your job: craft the AI-assisted methodologies that build a structure to help people navigate this topic successfully.8

Starting material

• Scott Alexander’s writeup of the COVID origins debate (the core case material)

• Judge Will’s decision | Judge Eric’s decision

• Michael Weissman’s Bayesian analysis (an example of a more rigorous independent analysis)

• Rootclaim’s response

• The debate videos: Session 1 | Session 2 | Session 3

Black holes

CERN, home of the world’s largest particle accelerator, the Large Hadron Collider (LHC), has a frequently asked question: Will CERN generate a black hole?

What??

As in some previous science experiments, noting that novel circumstances might produce unprecedented outcomes, some participants had apocalyptic concerns. How were these put to rest? (Were they truly? What does that hinge on?)

Unlike COVID, this is (we hope!) essentially a closed case, and uncontested. It nevertheless rests on a huge body of accumulated and interacting knowledge which enabled scientists (and the officials and public supporting them) to move forward with confidence.

The key challenge here may be in probing this argument for its dependencies and key considerations, and perhaps noting the weakest or most speculative points — all in an accessible way.

Eggs

Are eggs good to eat? Bad to eat? Great in moderation? How can we tell? Does it vary across people, and what predicts this? What else should we be paying attention to here?

This vague and open-ended topic, though mundane, is representative of a huge number of everyday questions — and hopefully also a microcosm of many more impactful debates. Sometimes getting resolution on what are the important things to answer and what are the appropriate ways of knowing is (more than) half of the challenge.

an ultraintelligent machine could design even better machines; there would then unquestionably be an ‘intelligence explosion’, and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make. — Good, 1965, Speculations concerning the first ultraintelligent machine.

Since the 2000s, a few researchers (recently increasingly many) have taken seriously the prospect that some kind of intelligence feedback mechanism could kick in around the time that AI becomes competent enough to contribute to its ‘own’ (or rather, its successors’) development.

This is increasingly recognised as a prospect we should take seriously. It’s been discussed somewhat furtively by leading AI developers for many years, but in late 2025 it became a public talking point. Sam Altman, CEO of OpenAI, recently set a target of fully automated AI-improving-AI by 2028. Jack Clark, co-founder of Anthropic, writes ‘AI systems are about to start building themselves’.

I’m not going to address the resulting prospect of speedup here, which I think is real, though perhaps surprisingly modest.1 (If, like some of my friends, you think I’m underweighting this prospect, you should probably be even more concerned about concentration of influence.2)

Rather, this note is briefly making the rather simple (but I think rather overlooked) observation: plausibly automated AI R&D would enable a (perhaps severe) concentration of influence… and this could be the most important effect to be paying attention to.

It’s a fairly obvious point once made. A more thorough analysis might shed light on how likely this is, the most important consequences, and any leading indicators to pay attention to. I’ll only make some starting gestures toward these.

As I presaged in Engineering a Safer World,

R&D automation…: fewer human participants means fewer whistleblowers, less internal scrutiny, less governance and decisionmaking robustness. More concentration of that influence. That could mean more single points of failure. It’s famously difficult to maintain a conspiracy of more than one or two people: ‘two can keep a secret if one is dead’, as they say! And besides conspiracy, compared to larger teams, individuals and small groups may be far more susceptible to capture, coercion, corruption, or plain foolishness and rash decisionmaking.

Whose influence?

To be clear, you don’t need to be concerned about any specific people currently in positions of particular influence to think that concentration could be very impactful (and concerning).

The wisest supreme leader is still a single point of failure.3 Especially during a period of tumult, leaders can be displaced and reporting structures revolutionised. Concentration of influence — even the prospect of that concentration — simultaneously raises the incentives for corruption and power struggle (while raising the stakes for the rest of us). Meanwhile, power corrupts and a centralised decisionmaker can rarely account for all the relevant considerations, even if they mean to. In AI, we’ve already seen company leadership falling out over differences of opinion, fighting it out in the boardroom and in court, and even clashing with governments. Turf war over the means of production of AI (which may increasingly equal the means of production full stop).

The point is that losing checks and balances can be a problem whoever is at the top.4 As in politics, so in the heights of industry — which, today, and increasingly, firmly includes AI.

Suffice to say that I imagine the kinds of roles in the AI oversight chain which might be (nominally, initially) preserved longest are senior research directors, company executives, and closely-involved regulatory or executive government officials.

Perhaps more important: whose influence lost? Directly, the skilled human labour which today is involved in researching, engineering, and deploying frontier AI systems. Via them, their wider research networks, social ties, and any societal oversight they might have provided (e.g. whistleblowing to journalists or authorities). Plausibly also the marketers, account managers, even lawyers and others who interface with what a company is doing with its AI products, and their networks. (If not themselves replaced, they’d lose collegial understanding and influence via the replaced researcher-implementers). In short: everyone else.

Concentration without (overt) malice

Why shed the researcher-implementers, the deployment engineers, and others (or sideline them from the important ‘real’ responsibilities)? It doesn’t have to be a power-grab, at first (or even at all) — it’s just good business! Human employees (especially software experts) are incredibly expensive compared with AI.5

Beyond this, analogously to areas where AI already surpasses humans (like chess), human participants may positively get in the way, slowing and compromising what would not only be a cheaper, but a more effective AI-only workflow.

A company might choose to instead eat this cost — perhaps it prefers to exhibit loyalty to current staff, acknowledges concerns around erosion of human oversight, or is inherently conservative in that way. But in a competitive environment (either commercially or in strategic and security terms), that may be tantamount to inviting defeat or irrelevance, if competitors are charging ahead.

At some point shareholders and other stakeholders might think it irresponsible not to embrace a replacement AI workforce. Ultimately this could add up to far fewer people having insight into what’s going on or a say in its direction. Even if you can’t (or don’t want to) immediately fire them, you might deliberately or even passively sideline your human employees as more and more work is carried out by machines.

Influence over what?

Notably, board, investors, auditors, journalists, and regulators (who already often suffer an information scarcity) — and the broader society they represent — could also be substantially cut out if logging, analysis, PR, and incident reporting are automated… This might only require the usual levels of corporate paltering rather than special malice. It might even be prosocially-motivated if the relevant actors are (justifiably or not) concerned about a power-grab by government. (It could of course also be deliberately selfish.)

Conversely, a government or regulator which strong-arms its way into the midst of the development process might itself become the locus of concentrated influence. (This in turn might be superficially or genuinely motivated by national security concerns.) The more automated a firm, the easier it may be to seize, either overtly or through more subtle channels. When employees are critical pieces in a production process, that’s a taller order.6

I’m not suggesting any of this leads (immediately) to influence over all the world (though if you do take over the world, please consider my friend Owen’s advice for what to do if you find yourself in that situation). Nor even over an entire economy or state. But minimally, this looks like highly concentrated influence over a frontier AI company: presently a rare, increasingly lucrative, and burgeoningly politically-influential machine.

Activity at that frontier — the decisionmaking by leading AI developers — stands a chance of being among the most economically, politically, and strategically important activity in the coming years. If the cat is out of the bag with AI, that frontier, steered wisely, might be an important part of a societal resilience and readiness process (compare the recent Project Glasswing in cybersecurity). Used foolishly or even maliciously, the frontier might degrade societal capacity right at a time when it could be most important.

Influence… or power?

To repeat: I don’t think it’s given (or even particularly likely) that the starting possessors of a hypothetical concentration of influence would remain in control. From the point of view of a rogue AI, for example, control over a frontier AI company might be a delicious opportunity for expansion, defence, and societal dependency — ideal routes to escalated un-unpluggability. Likewise from the point of view of a power-hungry politician or corporate psychopath. Notice that corruptibility and coercibility are perhaps more plausible with fewer hands: it’s less practical to threaten, buy off, or muscle out a whole team or company than a single person.

So if this hypothetical automation-driven concentration of influence is a concentration of power, I don’t expect the ultimate wielder to be worthy of it. This marks a true difference of opinion with some others I know, who might think it more likely that, say, some current AI company leader could both accrue and keep hold of this level of power, and also use it wisely.

The obvious paths to hard power via AI are in surveillance — newly scalable with AI analysts scrutinising as many people’s movements as desired — and in development and automated production of flexible, high-range tools of coercion like weaponised drones. Other kinds of power might come from positions of economic leverage or narrative and propaganda dominance, conceivable with advanced-enough AI and without societal defences. That's one point at which concentration could become entrenchment of power.

Concretely, what might be different?

Let’s really quickly gesture at two cases which might be substantially different in a world where AI production is largely automated.

Imagine a near future: frontier AI company revenues are in the 100s of $billions or $trillions, AI services are used in civil services, executive decisionmaking, faith leadership, … That’s a lot of points of leverage for subtle manipulation. Suppose a leader (selfishly or under pressure) intends to inject secret loyalties into their products. If they’re sole controller without checks and balances, that could be trivial, with obvious concerning consequences. Without automation-replacement, there are perhaps dozens or hundreds of people with reasonable oversight per major release, and several people even for any given small change. That’s a far more difficult system to twist.

Alternatively, consider a company on the brink of truly autonomous, general, human-surpassing AI: a huge responsibility. Many ambitious but underwrought alignment targets have been suggested for such a project, many of them foolish, perhaps fatally so. A leader hubristic without serious checks might withdraw from critique, double down (perhaps sycophantically egged on by AI toadies), and drive through a ruinous agenda. Society at large might be none the wiser (or at least insufficiently alert and able to intervene). A leader willingly taking risks (perhaps with a selfish or misguided vision of unfathomable rewards) would be even more of a problem. On the other hand, a leadership more dependent on staff and teams may be subject to more psychological and cognitive support, scrutiny, moral encouragement, and so on. A literal healthcheck. Of course, human teams can be subject to concerning groupthink (not unheard of at AI development companies!), but the more off-base a suggestion is (ethically or pragmatically), the more likely it is that the collective intelligence of the team corrects it.

Any number of scenarios can be considered; rarely do they appear more promising from a societal perspective if a single or a few leaders are insulated from scrutiny and critique.

What’s the outlook?

It looks very unclear to me. Plausibly it’s in the balance and decisions and conversations now will shape how this plays out!

There are some reasons to think that (near-) automation of AI production could instead produce equal or even improved societal oversight.

For one, the sometimes unintuitive effect of automation is to increase per-worker productivity, making it more worthwhile to bring more humans into the process (even on myopic financial grounds). Historically this effect has anyway often presaged a sharp decline once automation reaches a sufficient level. This might make concentration actually more difficult for a time. I don’t expect this to play out for very long in AI frontier research7, but I’m somewhat uncertain about this.

Directed sensibly, a glut of researcher-grade AI worker-equivalents could be tasked with analysis, scrutiny, decisionmaking robustness, even auditing and whistleblowing. There’s nothing in principle preventing this, for any given level of trustability in our AI. Perhaps we’ll end up bringing more people into the process in the increasingly important, complementary roles of oversight, decisionmaking, and direction-setting. These folks could be made more effective with AI tools and assistance. That’s a decision to be made.

As I suggested earlier, leaders perhaps soon faced with these temptations may balk, either at the moral prospect… or at the pragmatic prospect of being subsequently replaced, captured, or corrupted in the ways I hinted at.

Perhaps it’s a tenuous balancing act. Internally to a given AI producer, we might want to guarantee a large enough ‘in the loop’ workforce that we can trust internal deliberation and whistleblowing and so on to keep things on the rails. Outside that, we might want to avoid no-holds-barred proliferation of AI-producing capability — but we probably want at least a few developers to keep each other in check and reduce monopoly or kingmaker dynamics.

I don’t expect any such concentrations of influence to play out overnight. Loss of control is a process, not a moment. AI development organisations — and society at large — should be paying attention and having the necessary conversations. Much to look out for, and perhaps much to look forward to!

For those in the know, ‘the METR graph’1 is unusually compelling because it achieves what so few measures of AI progress have achieved: a somewhat meaningful Y axis (‘time horizon’2) as well as a somewhat predictable trend over time! (This is remarkably rare!)

Frustratingly, the only superficially available takeaway is something like, ‘the line goes up straight-ish over time’. This is better than nothing, but it’s very dissatisfactory from the point of view of getting confidence in the predictions, because it exposes no deeper mechanism. This drives a lot of confusion and argument about the implications.

A deeper mechanism would be good for two reasons:

• It enables a sanity check on the trend, perhaps enabling more confidence in its predictions than we would sensibly allow with only the surface understanding.

• It gives some way to interrogate when and how the trend might change (because if the deeper mechanism gets deflected, the superficial projection would be broken, but a prediction based on the deeper mechanism might stay viable for longer).

  • (A sub-reason: if we want the trend to change, knowing some more mechanism might shed light on some levers to pull rather than sitting around to wait and see.)

As an analogy, a similarly superficial trend, Moore’s Law, can be a little better mechanistically explained by the more general Wright’s Law3. This is great, because that law covers more cases, and it can handle some deflection from the trend, or give some idea of when (and under what conditions) the trend might break. Important when looking at plausible futures, and how to steer toward desirable ones!

Attempting to find some mechanism in the METR graph

Warning: mild maths incoming (consider reading on LessWrong for better rendering)

Task ‘length’ and success modelling

Why did METR focus on ‘task length’?

First, it’s not how long the AI agent takes. It’s how long the task in question takes a panel of sampled human experts, on average.4 So in their ‘time horizon’ measurements, METR is capturing the effective hours of human-expert-equivalent activity that AI agents can carry out.5

One way to think about the time it takes human experts to complete a task is that, for each subtask they had to know how to do (or be able to figure out how to do) and then successfully execute, the overall task takes incrementally longer. By how much? That depends on exactly what ‘subtasks’ we’re imagining breaking things down into.6 But on average longer tasks correspond to more distinct challenges, all else equal.7

This is the first piece of mechanism we should take into account. ‘Time’ is not agent time: it’s a noisy estimate for ‘number of somewhat challenging requirements necessary to complete the task’.8

This is treating overall tasks as formed by something like drawing ‘subtasks’ out of a large collection of possible requirements. Given the agent’s general competence, specific knowledge, tools available, and opportunity to retry or learn on the fly, sometimes the agent can meet these requirements. Other times it can’t.9 ‘Longer’ tasks simply draw more subtasks (that’s why they’re ‘longer’, in this model: expert humans had more subtasks they needed to carry out).10

Toby Ord demonstrates one way to take this intuition further, noting that if we explicitly model overall success S according to a simple model where chance of failure compounds with task ‘length’, t, we get a reasonable fit for the data METR collected. (Interestingly Toby mainly seems to continue treating this as ‘agent time’. I’ll instead take as given that we’re talking about a proxy for number of subtasks.)

In other words, for a given AI agent and task domain, there’s something like a ‘hazard rate’, P (per-subtask probability of failure), which reasonably well summarises (and predicts) the AI’s level of success in that domain:

(i.e. to succeed at a t-step task, the agent must not fail — must avoid the ‘hazard’ P — t times.)

This enables us to translate back and forth between an estimate of this hazard rate P and an estimate of a ‘half-life’ or 50% success horizon — how ‘long’ (i.e. complex) a task needs to be before the agent fails more often than not — and also to extrapolate to ‘durations’ corresponding to other reliability levels, like 99% or 99.9%.11

In this formulation, the hazard rate, P, stands in for what fraction of our ‘subtask’ pool the agent can’t (yet) succeed at, which ends up being a reasonable summary of the agent’s competence in this domain.12

If you have a new task, you don’t know if the agent has all it needs to complete it. But the task ‘length’ is an indicator of how many tricky subtasks it has, and similar-lengthed tasks will have similar numbers of such subtasks — so their average success rate is a good estimate for how likely the agent is to succeed at this new task.

Relating hazard rate with frontier AI development

METR’s graph is compelling because it suggests a steadily increasing frontier of success horizon as AI developers produce new agents over time.

What does this imply if we interrogate our hazard rate model? Well, ‘half-life’ (and indeed various success-level horizons) is observed apparently growing exponentially with date D:

This is the central striking takeaway from the METR graph (modulo their measurement uncertainty). Half-life go up!

But half-life according to our model has:

where P is the per-step hazard rate from before. When this P is not too close to 1, that half-life is, fairly intuitively, approximately proportional to the reciprocal of the hazard rate:

So METR’s observation of rising time horizons is equivalent to saying that the frontier hazard rate is shrinking exponentially over time.

Recall that this hazard rate corresponds with the fraction of ‘subtasks’ in a domain that an agent doesn’t yet know how to complete. So this fraction is presumed to shrink roughly exponentially with date, in turn driving the observed ‘longer’ success horizons.

Why does hazard rate shrink with date?

Here’s where to look for the next bit of mechanism. Why would the hazard rate, the fraction of ‘subtasks’ which remain out of reach, shrink in that way?

It goes without saying that AI developers are chasing after increasing competence in their products, so (if they are doing anything at all right!) the direction of movement is unsurprising. Why that particular roughly-exponential form, though?

I confess here I’m uncertain and the quest for more mechanism continues.

My best guess is that it’s about the effective evidence available to the agent toward subtask solution strategy. Intuitively, if you’ve seen very similar subtasks many times before, it’s hard to go too wrong. If you’ve only seen vaguely similar subtasks once or twice, you’re in much less familiar territory and stand a good chance of stalling. Suggestively, effective evidence and training data are both information-like quantities, but I don’t want to make too much of that without a crisper connection. Formally, we could consider how many bits of evidence the agent can muster about how to proceed (either from past learning or by exploring in context).

In other words, training produces learnings. These range from broad, generally-applicable heuristics for adaptable, effective behaviour (experiment, test your work, notice when something surprising happens, read the manual if you can find one, accrue power and resources at any opportunity, ...), to narrow specific details about particular situations and activities (Earth’s radius is roughly 6.4 megameters, detonating TNT yields roughly 4.2 kJ/g, humans succumb to oxygen deprivation after around 5 minutes, …). Ahem.

Empirically, AI developers have historically poured something like exponentially increasing ‘quantities’ of ‘data’ into their machine learning pipelines.13 Mathematically, that implies a power law: data inputs n_train rising at one exponential rate, matched by hazard rate P decaying at another exponential rate.

Power laws aren’t deeply mechanically explanatory, but they’re often the best we have in machine learning, and are at least more predictable than mere date-based trends. Under the simple subtask model described here, this power law translates directly into a power law between ‘time horizon’ and data. This is actually the same level of explanatory improvement offered by Wright’s Law over Moore’s: not fully mechanistic, but an extra layer of detail which offers firmer purchase on what’s going on.

What this doesn’t straightforwardly account for is the benefit to success rates of increased in-context reasoning, which is exhibited according to METR’s estimates. I expect this is operating on those borderline subtasks — where the agent would have some slim chance of satisfying them if it ‘rushed’. In those cases, ‘thinking harder’ may more effectively recall and combine the relevant learned knowledge, and allow better choices for exploratory discovery in situ. In any case, changing the thinking budget of an otherwise similar existing system certainly calls for a more mechanistic understanding than mere date-based trend extrapolation!

I would be thrilled if someone with more smarts, time to experiment, and access to data were to dig into ways we could match up various AI production inputs (especially ‘data’ in various forms) with observed outputs like ‘time horizon’. One of the more difficult pieces might be quantifying ‘data’, especially teasing apart what types of evidence are ‘relevant’ for the domain and tasks at hand.

Upshot

The kind-of-boring upshot of this is that data and ‘practice’ on related tasks makes AI better at those tasks! This is boring because, well obviously!, we already basically knew that. But it’s encouraging because we can say a little more than that, which gives us some better grasp on what’s driving ‘time horizon’ progress in particular domains — and it can help get more precise about predictions.

The fact that the ‘subtask’ model — with a ‘hazard rate’ of subtasks currently out of reach — is a fairly explanatory fit for capability profiles of individual agents is evidence that there’re not unusual amounts of generalisation capability in AI. As with humans, they can extrapolate a bit, but need ‘experience’ and examples to succeed.14 Importantly, this means that vast in silico training ranges for software, cyber, and mathematics very likely won’t transfer much to other domains of interest, like interpersonal intelligence, medical discovery, bioweapons development, intelligence analysis, and robotic manipulation. Of course, like with every domain of human experience and activity, we have some relevantly-similar data already collected, and schemes can be devised to more rapidly expand that digitised experience bank for AI to learn from. Increasing adoption of AI in task-integrated contexts, industrial deployment, and even explicit approaches to gathering example data such as ‘hand movement farming’ are the leading indicators to watch for progress in particular domains — not just the headline benchmark metrics in software-like tasks.

For some types of activity, developers are probably ‘running out’ of raw example data to scrape from the internet. The era of mostly-pretraining is over. For domains which can be relatively easily verified, like mathematics and coding, this is very surmountable — you can just run drills galore on a computer and get data that way. But this costs extra compute and doesn’t scale at the same exponential rate for long (perhaps 10x/year presently). As soon as this year, developers could be back to ‘only’ scaling compute around 4x per year (and a bit after that they might have bought most of the compute! — and will only be able to scale at the positively sloth-like 1.5x-ish a year of underlying hardware progress). I don’t feel confident extrapolating exactly where that cashes out, but if the data-driven subtask-learning model is right, it would imply we should see less steepness to the time horizon growth quite soon.15

Some commentaries project that, once AI can autonomously do software and machine learning work reliably, it will thereafter enter a ‘recursive self-improvement’ phase and rapidly colonise all capabilities. I don’t think this is missing the point entirely: there will be modest multipliers on the speed of the AI development pipeline, and we might see an ‘explosion’ in the speed and cost-effectiveness of AI (because they are among the most immediately-verifiable properties to iterate on). But generalisation doesn’t come for free, so on-task data and compute will remain crucial to broadening the frontier of autonomous capabilities. Collecting that data and manufacturing that compute look to me like the rate-limiting steps, and therefore the major leading indicators to use in foresight. The best case I can make for a much more general explosion is if the speed and cost-effectiveness explosions rapidly accelerate the gathering and digestion of diverse task data — but I think that remains mostly rate-limited in the familiar ways: some domains easy and some more difficult. Don’t mistake me for ruling out across-the-board AI capability! Companies are charging ahead with data collection and set on automating much of their AI production pipeline. It just won’t happen overnight.

Thanks to Coz Ududec for a conversation prompting me to think about this.

I, and I imagine many of my readers, are eager to contribute to that effort1.

It’s also, conveniently, the title of a book!

It’s not a book by me… but nevertheless I recommend it. You should consider reading it, especially the first chapters. You can find it online free from MIT. It’s by MIT professor Nancy Leveson, computer scientist turned innovator in safety science, safety engineering, and software safety.

Generally, this book is a particularly approachable and competent example of systems thinking or what I might call a cybernetic perspective — applied to safety science. This essay is in part a brief book review of Leveson’s Engineering a Safer World.

I care about this systems perspective in part because... loss of control. It’s something which, I think, is quite intuitive for many people at a very abstract level. Of course we could lose control of ‘systems smarter than us’. But in my experience it tends to be quite slippery. It’s difficult to get a proper analytical grip on ‘loss of control’, and it’s difficult to communicate about.

The systems thinking or cybernetic framing is one I find particularly helpful for getting to grips with ‘loss of control’, and it has other benefits besides. That’s the rest of what this essay is doing: how does systems thinking apply to AI and loss of control specifically?

I did some of this loss of control risk modelling at the UK government’s AI Safety/Security Institute (AISI) in 2024, and now this kind of risk modelling is part of the background strategy, prioritisation, and foresight work we do at the Future of Life Foundation2.

This essay corresponds closely to a talk I delivered at the Technical AI Safety Conference 2026.

Safety and systems

The first lesson from this perspective is: safety is a system property.

There’s never a single ‘root cause’ of a disaster.

Even acute disasters have a buildup. The bit you see — the tip of the iceberg — is the acute disaster. An explosion. A crash. A fire. Human extinction. But the lead up to that always has systemic, process failures.

Disasters might be precipitated by some particular hazardous activity, or even malicious action. But behind all of those is a network of interrelated systems and processes which could have — and we might say should have — done something to prevent the disaster.

These systemic weaknesses are like health indicators for the systems in question. Leveson encourages us to consider these processes and systems as the central objects of consideration when trying to engineer a safer world.

‘Loss of control’ on this perspective is less of a single decisive moment and more of an unfolding process, or even a vicious cycle. We’ll return to the vicious cycles later.

Which systems?

OK, we need to pay attention to some systems. But there are lots of systems! Which systems?

There’s definitely an art to this — not one I claim particularly to possess. It’s here we need to apply contextual knowledge, analyst’s judgement, and whatever expertise we can muster. But there are some general principles the systems view can offer.

My statement of a principle I like: think up the chain.

In engineering, consider:

• A deployment. It goes with monitoring systems, maintenance systems, iteration and evolution processes.

• Design, development, perhaps manufacturing. Research.

• Those systems are in turn overseen by management which should include communication, reporting, decision-making systems. Passing down design constraints, objectives, resource allocation, prioritisation. Getting status updates, reports, outcomes.

• ...Even above that, we can start to talk about ‘governance’:

  • Literal governments: regulatory activity

  • Also company boards and other governance commitments and structures

  • Courts

• We can also include — I think it’s important to include — broader societal sensemaking, democratic deliberation, how we are making sense of what’s happening as a society and controlling the processes unfolding around us.

This is a very ‘cybernetic’ perspective, one which I think is quite powerful.

When these get corrupted, things get out of control.

In her book, Leveson gives various worked examples. She also offers some generic examples as starting places for analysts like me.

Here’s one I find quite instructive. It’s actually one we used as a starting seed for some risk modelling at AISI.

In general, this systems perspective offers a really expressive language both for communicating and for analysing or troubleshooting. One of the beauties of this language is you can conceptually ‘zoom in and out’, asking for example which people and processes (subsystems) make up or implement a given system or relationship, and their requirements and so on. Or draw a bigger box abstracting around a collection of closely related systems for a more birds-eye view.

The regulatory lag dilemma

The accompanying text from this point in the book is sobering reading. Leveson writes,

The only requirement is that responsibility for safety is distributed in an appropriate way throughout the sociotechnical system… If companies or industries are unwilling or incapable of performing their public safety responsibilities, then government has to step in to achieve the overall public safety goals. But a much better solution is for company management to take responsibility — Leveson, Engineering a Safer World (4.2, The Hierarchical Safety Control Structure)

but elsewhere in the same chapter,

As in any control loop, time lags may affect the flow of control actions and feedback… For example, standards can take years to develop or change — a time scale that may keep them behind current technology and practice. — Leveson, Engineering a Safer World (4.2, The Hierarchical Safety Control Structure)

(emphases mine).

This dilemma of slow-moving, ponderous regulatory oversight is particularly pernicious in an area as fast-moving as AI. More on this dilemma later.

Which health indicators?

We have something like a picture of which systems might be important for safety.

Where do we look for health indicators? Relatedly, where are the hazards? And the flipside, where do we look for interventions for robustness?

A systems-theoretic lens has a few general suggestions.

‘Control’ systems

This is a very generic picture of a control system. You’ll see pictures like it across biology, computer science, cybernetics, reinforcement learning, military strategy, engineering, control theory. Terminology might vary but it’s the same general picture, very straightforward and explanatory.

What’s a controller? It might be me riding my unicycle, it might be a government department attempting to understand and nudge and industry or even an economy, it might be a mouse looking for food, or an electronic component of an autonomous vehicle.

What does such a controller need to do a good job? It needs sensing: some way of getting observations, feedback about the relevant aspects of its situation and environment. It needs actuation: some way of acting back on the world, taking actions, applying influence. Between those, it needs understanding — something like a model, perhaps a ‘world model’ — interpreting its observations and their implications, and the implications of its available options. And it needs a way of deciding appropriately what to do in order to carry out whatever responsibilities or agenda it has.

In the context of safety, we’re talking about safety controllers, applying safety constraints. It doesn’t mean dictating what happens in any ‘controlled’ processes, just applying enough constraints that we can be confident that safety is maintained.

So: sensing, understanding, deciding, acting.

When analysing a given system, we can ask what it has by way of sensing, understanding, deciding, and acting. Are they adequate? Degradations to these look like hazards, or even attack surfaces in an adversarial setting. And improvements look like safety opportunities.

Control examples in AI safety

Let’s get more concrete.

Take the example of an AI deployer, and their opportunities for sensing. How do they ‘see’ what’s going on? Logs! So spoofed logs or tampered evaluations or something like that would substantially compromise their ability to understand what’s going on. So this is potentially cutting off that sensing relationship. Alternatively, rogue replication of an agent would also be a compromise to that sensing, because it would create an agent outside of the nominal sensing channels.

Now take an AI regulator, and their understanding and deciding. How might those be inadequate? They might simply have insufficient capacity for the needed analysis and foresight: I think a quite common failure mode for regulators. On the more adversarial side, they could be subject to lobbying or capture — and this could be compromising their understanding or even their decision-making process. Even if they have perfectly adequate sensing and actuation affordances in principle, they might not choose to use them in the way that we’d hope.

For a final example, take the slightly esoteric system ‘society’ or ‘the public at large’, and its ability to act. Well, generic economic or political disempowerment of course (perhaps tautologically) means that even if there were actions we might have wanted to use to apply pressure to something we think is unsafe, those actions might not be available or effective any more. Also, rogue replication! Whatever actions society might have wanted to do to get an AI system under control, if it’s replicating that’s much harder. Replication is a big deal, and we wrote a paper on it at AISI.

An abstract picture of AI development and operation

Let’s really simplify and adapt Leveson’s generic development and operations diagram for the case of AI. This is sweeping huge amounts of detail under the rug for now.

There’s an agent. That’s important. There’s a user. (That might be a person or an organisation or some other system, perhaps automated.)

The agent gets there by a combination of the output of a research and development process and control by a deployment and monitoring system. Those are in turn controlled or overseen by company leadership. (There might be more than one company involved.)

These companies are meant to be somewhat kept in order by ‘society’ — think regulators, courts, insurance, public associations, and so on.

Already, even at this very coarse level of granularity, there’s a lot we could say by inspecting and enumerating all these relationships. What sensing and acting affordances are there (or might we want there to be) here? How are understanding and decision-making happening: could they be improved? And of course one of the nice things about this language is that we can recursively unpack things if we want, most obviously here in ‘R&D’ and ‘Society’ which of course have far more detail to them.

What makes AI special

For now, I want to focus on something which makes AI a bit different.

Agents are increasingly capable and have increasingly general affordances. That means, whether directed to — or whether acting out a misaligned objective — AI can in principle act back on any level of this supposed control hierarchy.

That turns this into potentially an adversarial, security engineering problem, and it brings us to the vicious cycles I mentioned earlier.

For me, it’s right to think of ‘loss of control’ as often a matter of a breach or compromise at one or more of these levels, followed by escalations. Not just a singular event, but unfolding in potentially vicious cycles. Even if we can identify some final ‘point of no return’, preceding that there was some series of events that precipitated it.

There might be several acute events: perhaps it’s a breach of containment by an agent, or a passing into law of a bill harmful to oversight. Or the failure to pass a particular law, perhaps missing an opportunity window to do that. It might be a particularly ill-advised deployment decision. It might be a very consequential court ruling. We can think of these as the overall system migrating towards an increasingly, and eventually entirely irrecoverable total loss of control.

Vicious cycles

Some of these might be obvious, but let’s enumerate a few of those ways AI systems could act back on the oversight hierarchy.

Propaganda, epistemic disruption, and political influence — harm democratic deliberation and political decisionmaking.

Lobbying and legal influence — could erode regulatory and court oversight.

R&D automation — potentially interrupts quite a few feedback and oversight relationships! One which I think is underdiscussed: fewer human participants means fewer whistleblowers, less internal scrutiny, less governance and decisionmaking robustness. More concentration of that influence. That could mean more single points of failure. It’s famously difficult to maintain a conspiracy of more than one or two people: ‘two can keep a secret if one is dead’, as they say! And besides conspiracy, compared to larger teams, individuals and small groups may be far more susceptible to capture, coercion, corruption, or plain foolishness and rash decisionmaking.

Backdooring, sabotage, poisoning — all great ways to break oversight in R&D.

Exfiltration, privilege escalation — these are bread and butter for compromising monitoring oversight, whether from ‘inside’ or ‘outside’.

Trickery and coercion — most tools don’t do this. AI does. Already this is sometimes breaking some people’s model of their relationship with AI and sometimes their wider situation.

Virtuous cycles?

BUT. Something else we can also do is ask how the new AI building blocks in our repertoire could be used to fortify some of these situations.

This is an area the Future of Life Foundation is really interested in, along with some collaborators at Forethought.

Really briefly, some things that immediately stand out when looking at this picture.

There are so many tantalising opportunities for improving how societies communicate and coordinate! Tools supporting collective epistemics, democratic discourse, conflict resolution, coordination.

AI tech itself is moving fast, and other tech may be accelerated and disrupted soon — but ‘open source intelligence’ and more targeted foresight can both benefit from well-designed AI-powered applications. That could be one of the more promising ways to improve agility of societal decision-making — recall the governance dilemma of slow-moving government and regulatory oversight.

Ironically, R&D automation or acceleration, directed sensibly, could provide an antidote to some risks. It’s double-edged. There are all kinds of possibilities here, including perhaps development of hardware and support for confidential oversight of high-stakes systems. Applying AI-assisted efforts here might make a difference between having these useful aids at an important time or not.

Talk of scalable oversight and automation of AI interpretability inherently depend on the right capabilities being reliably usable from AI.

Being able to attribute AI outputs, and perhaps more importantly AI activity, could be crucial to keeping the running of economic, industrial, cultural, and other societal functions understandable and monitorable. It might be that AI-powered ‘forensics’ and provenance investigations end up making this more achievable.

Counteracting deception and other manipulations, we may be in a position to develop fiduciary or guardian AI systems — tools and assistants which people can slot into their workflows to protect them from trickery and other confusions as the world gets more complex around us.

Summing up

Engineering a Safer World — worth a read! Especially the first chapters.

Safety is a system property: even acute disasters have systemic failures we can make healthier.

Think up the chain: safety in something as encompassing as AI is a societal-level sociotechnical problem. Compromise at any of these levels could initiate or escalate loss of control.

AI is very special; it can ‘act back’ on the nominal control hierarchy. That could give rise to vicious cycles, one of the main classes of mechanisms by which AI loss of control could escalate to truly terminal states.

Not all cycles are vicious! AI, used well, could also help robustify and improve the health of some of the mechanisms and systems we use to live and work together safely.

This is part 2 of a series about LLM architecture and some implications for reasoning and transparency. Part 1 explained how we got where we are. Here, we look at where we might be going soon — does visible reasoning go away?

Hidden reasoning

There’s a term that’s caught on somewhat to describe deep learning: inscrutable. Deep learning centrally relies on (large) neural networks, whose workings are famously impenetrable to those training them or operating them alike. I won’t expand much on that here.1

The key thing is: a neural network takes inputs, converts them into an opaque and idiosyncratic internal language, then computes outputs. The arcane bit in the middle goes by many names: deep embeddings, neuralese, activation space, latent space. I’ll call it hidden reasoning.

Humans do this too, of course! You can’t tell what someone’s thinking or planning unless they tell you and they’re honest about it. (And apart from the surface thoughts we’re aware of, we don’t even know most of what’s going on in our own subconscious.)

A dash (em-dash?) of luck: ‘thinking out loud’

Hidden reasoning is concerning. Worst case, we have scheming AIs. Best case, it’s hard to know how much to trust the outputs — not because you think the AI is deceiving you, but because you just don’t know what it’s taking account of, and how, in its decisions.

But we got quite lucky for a spell! Scaling language models turned out to be the easiest path to general purpose reasoning AI, and not only that, but the famed ‘attention is all you need’ transformer architecture which got us here places substantial limits on the amount of hidden reasoning the AI does.

But limited reasoning is limiting! People want AI to be able to do things. The obvious solution is to employ visible reasoning. This is already something humans do all the time — from ‘thinking aloud’ to writing notes (or whole textbooks!), sharing thoughts with others, brainstorming, using computers, databases, ‘tools for thought’, and so on.

One of the most effective classes of improvements to AI reasoning in this paradigm is encouraging it2 to ‘think aloud’ things like, ‘wait, is that right?’, ‘let me consider more possibilities’, ‘I should double check that’, and the like.

So, status quo, frontier AI is increasingly good at reasoning, largely as a consequence of ‘thinking out loud’ more and more. There’s a mostly human-readable trace of what it’s thinking.3 Although we don’t know exactly how much reasoning can be crammed into the hidden parts,4 it’s a matter of sheer structural message-passing necessity that any more than that must go via the ‘out loud’ pathway.

This is a good thing for oversight of AI, because they are capable of scheming under the right conditions, but so far usually only by blurting out all their plans as they think aloud!

Aside: keeping visible reasoning faithful

Of course, even if the AI does most of its reasoning in externalised text, there’s no guarantee in theory that it’s using human language the way we expect.

This is why researchers call this kind of monitorability a ‘Fragile Opportunity for AI Safety’, noting with concern that reasoning models ‘don’t always say what they think’ and proposing ways to protect and improve this fragile opportunity.

The cake is a lie: reinforcement learning back in style

Recall LeCun’s ‘cake’ analogy for AI training. The vast majority goes to self-supervised ‘what comes next’ prediction, with character- and rules-tuning relegated mostly to a ‘cherry on top’. To terribly oversimplify, this gives some reason to expect most ‘out loud’ reasoning to be essentially faithful at this stage. Why? Most of the training examples of human-looking reasoning are records of actual humans reasoning!5

Those of us with background in reinforcement learning and agent foundations always considered the relegation of RL to a ‘cherry on top’ somewhat suspect. LeCun’s prediction held for a few years — but by 2024 general AI developers were starting to really scale up the RL, and here in 2026 it may be reaching on par with self-supervised pretraining. Some cherry!

This abundance of RL training over diverse tasks is probably most of what’s differentiating competitive frontier capabilities in AI today.

Reinforcement learning’s serial training penalty

When you’re running a machine language model, you generally have to wait for it to say one thing before it can say the next.6 Running a model through RL training is even more demanding: you have to wait for it to do something and gather the result before it can do the next. No shortcuts.

This means RL on attention-only transformer language models is dramatically handicapped compared to self-supervised pretraining. (Recall that attention-only bursts open the serial bottleneck for self-supervised learning in particular, kicking off the deep learning x big data language model revolution.)

Said another way, if most of your training is self-supervised and only a little is RL, attention-only architecture is an incredible efficiency boon.7 But once RL is a meaningful fraction of training, the scalability benefits of attention-only self-supervised pretraining begin to dwindle in comparison.

This is a large part of why companies were slow, reluctant to scale RL until they ‘had to’. But it now looks like they probably have to.

Return of the recurrent network?

Because RL has to ‘wait’ for things to unfold sequentially, it ‘might as well’ make use of fully recurrent messages (‘hidden reasoning’) if it can.

In fact, because the hidden reasoning occurs in neural activations rather than mere human-readable text output, it’s actually got far more information capacity. That’s a further point in favour of recurrent connections.

Speculatively, developers might somehow get the ‘best’ of both worlds by introducing recurrent messages for RL training stages but skipping them for self-supervised learning.

Does ‘thinking out loud’ go away?

Putting things together: developers are turning to RL at serious scale — this is a novel demand on frontier general AI. Previous developments leant on massive self-supervised pretraining and little else. That made the attention-only transformer architecture indispensable for training efficiency. The transformer also very conveniently gave us ‘reasoning out loud’ as the closest thing to interpretable AI we’ve perhaps ever had. That’s generally considered convenient for safety: a ‘fragile opportunity’.

If RL scale eats up the efficiencies of attention-only (non-recurrent) training, and companies continue to be willing to pay increasing costs for frontier training, they may turn to ‘hidden reasoning’ architectures, including fully recurrent hidden messages — after all, these plausibly benefit the AI’s competence.

This isn’t purely speculative: ‘latent reasoning’ is an active research area in AI, enough so that a 30-author, 40-page survey of the field was published in 2025.

Potential saving benefits of visible reasoning

There are obvious benefits to sharing some reasoning ‘out loud’.

Sharing what you’re thinking about — in the AI case, not only with ‘copies of yourself’ which have the same internal thought-language, but also with heterogeneous AI systems and humans — is often beneficial, at least with friendly collaborators. Curiously, it might be especially useful to be provably largely honest in a way that humans can’t achieve. Structural, architectural constraints which rule out certain kinds (or degrees) of conniving may point in this direction. That’s fairly niche, though, and has various other difficult dependencies.

The case of humans is instructive: to boost our reasoning, we don’t just ‘think out loud’; we often use reams of scratch paper, digital notes, archives, databases, and so on. AI systems with the right tools can make use of systems like this too. Now, for AI, such systems could be written entirely in idiosyncratic neuralese or stegotext: this might have some capacity and representational benefits, but would act heavily against compounding and maintenance and sharing. I wouldn’t be surprised to see some of both: ‘personal’ dense idiosyncratic jottings, and ‘external’ or ‘long-term’ more natural records.

I don’t think these benefits say you should only reason out loud.

There are also the aforementioned safety and oversight benefits. While ‘capability’ and ‘safety’ tradeoffs are often a little murky in AI, this might be one of the most clear cut cases where they act against each other. I’d advise frontier AI developers to refrain from throwing away the benefits of overseeable AI reasoning without understanding the implications very clearly.

Does it even matter?

I’m speculating somewhat here — in particular, besides the crude gross comparison between self-supervised and RL expenditure, I haven’t ‘run the numbers’ on the specific implications of different language model architecture setups. The serial training penalty of RL may bite imminently, or it may be some distance out. Someone more familiar with precise details of AI training expenditure or training regimes might have more to say there. I expect that within AI development companies there are people already researching these things.

More generally, I have tended to be somewhat cynical about deployers of AI systems actually bothering to look at the bounteous reasoning traces their AI systems produce anyway! Surely the responsible ones at least try to, or (of course) they get other AI systems to do so. But if the cat is out of the bag with AI development, it’s difficult to imagine all such developers and deployers being responsible.

Conversely, maybe we can do much better than merely relying on architectural semi-guarantees on reasoning transparency! Epistemic virtue evaluations (and associated training) may be able to drive much more clarity and honesty than mere thinking out loud. And structural, compounding knowledge-bases curated with help from AI and grounding more AI’s outputs might be a path to truly trustworthy communication and learning.

But how did LMs actually get so large as to warrant the title ‘large language model (LLM)’? A large part of the answer is in the P ('pretrained') and the T ('transformer') of GPT.

This is part 1 of a series about LLM architecture and some implications, past and future, for reasoning. Part 1 is ‘how we got here’ — what was so impactful about the transformer architecture for LLMs. Some readers may prefer to skip this. Part 2 points at an unexpected benefit — the surprising explainability of contemporary AI reasoning — and why new trends might erode that. It’s a novel point as far as I know.

Self-supervised learning: most of the cake

In 2016, early deep learning practitioner Yann LeCun introduced a famous analogy for learning intelligent systems:

If intelligence is a cake, the bulk of the cake is unsupervised learning, the icing on the cake is supervised learning, and the cherry on the cake is reinforcement learning (RL).

This correctly described the sometime prevailing practice for training LLMs, before the first LLMs were created! (Though by 2025, certainly 2026, this has become outdated — more on that later.)

What does this mean, and how did LeCun come to this position?

Machine learning, especially deep learning, needs example data.

Old-school supervising learning

You might think to start (and historically, researchers mostly did) by curating labelled examples: ‘this image is a cat, this one a dog; the French for “a bed” is “un lit”; …’. The machine learns these labels, and (hopefully) also learns how to generalise correctly, responding appropriately to similar-enough previously-unseen cases — which is what you actually want.

This supervised approach can be excellent: you get (ideally) reasoned, expert judgement as the sole curriculum for your nascent learning machine. But this is also painfully expensive, because you have to pay or otherwise entice humans to look over all your examples (or worse yet, create gold standard examples from scratch)!

The hands-off approach

Fortunately for developers, there is a stupendous amount of (language and other) data on the internet. That’s one very big reason, if you can, to pursue self-supervised training targets for your machine learning project: rather than ‘here is my carefully (and expensively!) curated dataset of labelled examples, learn them’, it’s ‘here is a gigantic pile of text, learn to predict whatever comes next’.1

As in supervised learning, where the goal is not simply to memorise the provided examples, the eventual goal of self-supervised learning is rarely simply to teach the machine to carry out this specific defined prediction activity — rather, in the process of learning how to do that, the machine is forced to learn generalisable concepts and features which can then be turned to a wide range of tasks.

More on RL (the cherry) later.

Self-supervised learning in language models

Language comes in long sequences.2 So language learning systems need to be able to consume sequences. Consider: a large part of how we track what’s going on in a play, a textbook, an argument, a novel, and so on, is by reference and by recollection of context far back in the sequence: a previous dialogue, an earlier concept or experiment, a foundational point of information, a character’s exposition and background. The foreshadowing of a Chekhov’s gun can’t be understood if you’ve lost the plot in the meanwhile! So too with machine language models.

For the ‘predict what comes next’ self-supervised target, that looks like consuming in principle everything that’s come so far, then using that context to output a distribution over possible continuations.

Chapter 1:... The old revolver sat on the kitchen table…

…

Chapter 20:... Alice had run out of patience. Without warning, she fired the [gun/chef/kiln/pigeon]

Which one is right? Well, actually it depends a lot on what happens in the intervening scenes I’ve skipped! But assuming the narrative promise made in scene 1 is kept (and no other overriding promises are made in between), a good prediction is ‘gun’. On the other hand, if scene 1 instead introduced a sous-chef with a hygiene problem, a pottery studio with a design dispute, or an experimental pigeon-launching apparatus, the answers might be different.

The especially useful property of ‘what comes next’ self-supervision is that you can run this test on every single position. So a reasonable-length text might give you thousands or more such training examples… some easier than others.

‘Simply predicting what comes next’ clearly necessitates tracking a lot about what’s going on (across all kinds of scenarios) and how things work… at least if you want to predict well.

What are we waiting for? How transformers overcame a scaling problem

The benefit of self-supervised learning — a huge, largely automatically-generated collection of training targets (‘predict what comes next’) — also raises its own challenges. If much of the capability of your system arises from the sheer quantity of these training targets, you run into computational challenges: you want to be pumping more and more of these examples through your machine, but each example takes some number crunching, which costs compute time. Crucially, if you can crunch through more examples at once in parallel, you’re in a far better spot.

We saw how text prediction is a sequence problem. Naively, this means that, to make the prediction at the last position in the text, your process has to first read the first token, then the second, the third, and so on. To make the guess at the last position, you’ve had to ‘wait’ for absolutely all of the rest to be done processing. At least, that’s how humans read, and it’s how neural network machine language models did their processing until 2017.3

It’s implemented as a ‘message’ (neural activation) passed forward from position to position (which needs to capture what might be relevant to ‘remember’), a structure known as a recurrent network. (Messages are additionally passed ‘depthwise’ in deep neural networks.) The nth position needs n steps to proceed before it can compute anything. For long texts, that’s crippling.

The milestone transformer architecture, introduced in the 2017 paper Attention Is All You Need, totally upended this constraint, making far larger-scale self-supervised training practical. Neural attention mechanisms were introduced long before this paper — the real innovation is to drop the recurrent connections entirely. You Don’t Need Recurrent Connections is the logical corollary! — and is what enables the entirely more scalable training of these architectures on long sequences. Attention still passes messages forward, but never only forward, always ‘diagonally’: making depthwise steps through the structure at the same time. This means no position, however long the text, need wait for any previous positions to compute. It does demand a highly parallel computation, but modern compute resources are nothing if not highly parallel.

Self-supervised training is only the start

Self-supervised training targets are great for cheaply (low human effort!) learning all kinds of features and concepts from big data. The well self-supervised neural network now ‘gets what is going on’ in all kinds of texts, well enough to sensibly predict what might come next, even on new content.

Sometimes that kind of prediction is already exactly what you want, but usually there are other things you’d like your neural network to do. Enter transfer learning, any number of approaches to tapping into these learnings to accomplish the actual task of interest.

Chatbot as playscript

For language models, a common basic approach is arranging for the model to ‘predict’ the responses of a helpful ASSISTANT character, in a dialogue with a USER character (played by you).

The following is a dialogue between a computer USER and a helpful, knowledgeable ASSISTANT.

USER: Which is better, a Chekhov’s gun or a Maxim gun?

ASSISTANT:

Because the assistant in this context is plausibly actually helpful and knowledgeable, there’s some reasonable chance that a well-trained language model produces a helpful answer here with no further tweaks. Of course this is flimsy — hallucination of plausible-but-incorrect responses or provision of unhelpful responses is common, or even entirely off-track diversions, like introducing new ‘characters’ or pivoting away from the established playscript structure.

Post-training

Various approaches make more involved attempts to effectively make use of the knowledge acquired in self-supervised pretraining. Those which involve further training, not just prompting, are often called ‘post-training’.

Supervised learning: the icing

Recall LeCun’s cake: with self-supervised (‘unsupervised’) the cake, supervised learning is the icing. A common pattern for LLMs intended for chatbot use is to have humans generate examples of how the ASSISTANT should respond to various queries. Should it ask clarifying questions sometimes? How verbose should it be? Should it provide references? Should it respond in first person? Should it respond in the register and dialect of the USER, or have its own? With training examples provided, sometimes principles like these can be learned and generalised.

Because almost all of the knowledge, language understanding, common sense, and character information comes from self-supervised pretraining, supervised learning as fine-tuning can get away with radically fewer examples than would be needed to train this way from scratch. Less human effort. Icing indeed!

Reinforcement learning: the cherry on top

LeCun’s cake analogy underestimates and misunderstands reinforcement learning — more on that in part 2. But for a few years, his relegation of RL to a final (but important) stylish flourish was a fairly good description of how LLM-based AI assistants were trained.

Rather than getting human contributors to produce exemplary data (as in supervised learning), RL has human engineers specifying how to grade outputs as good or bad, and sets the machine up to explore possible behaviours. Those outputs get graded, and the training encourages more of whatever computations produced the good ones (and less of whatever produced the bad ones).

This is very fraught, because accurately specifying what counts as good behaviour ahead of time is notoriously difficult, and if you’re not careful, exploratory AI will absolutely find the strange cases you didn’t think of. It can also fall completely flat if the AI in training has no idea how to achieve good outputs at all: it just gets stuck flailing.

This makes post-training of reasonably competent pretrained base models a sweet spot for RL: the model has enough background competence to at least occasionally succeed, can learn from success, and keep getting better and better. RL (whether as post-training or from scratch) also has the most potential to scale past human expert level, because we only need to be able to say which outcomes are better, not how to achieve them.

The earliest widespread application of RL to LLM post-training was in reinforcement learning from human feedback (RLHF). At its core, this asks humans to rate which answers are better or worse, then has the machine try out various ways of achieving better and better answers (according to the human raters). Typically this is repeated for a few rounds. In 2023-4, this is how companies got AI chatbots to mostly respond politely, usually refuse to describe how to make bombs, and often stick to character. It’s also a great way to train AI to be obsequious, disingenuous, flattering, and sycophantic. Oops!

Times changing

In frontier AI systems, reinforcement learning is no longer a lightweight (albeit effective) post-training flourish on top of a primarily self-supervised pretrained LM. Since late 2024, reaching for expert-and-beyond capabilities has brought RL back into the spotlight. That has a few implications, one of which — reasoning and its transparency (or otherwise) — we’ll look at in the next essay.

Best humans still outperform artificial intelligence in a creative divergent thinking task

Remarkable! Man bites dog! It had become newsworthy, it was worth checking (and, I perceive, worth a little self-congratulatory celebration) that there remained any domain where mere man could still hope to possibly contend with the machines — at least, the best humans still could! (Could you?)

A message from the future

That was 2023. I think what stood out to me at the time1 was that this was in some sense early. Not early in the story of AI — although ChatGPT and StableDiffusion, each less than a year old, had captured the public attention in a way which earlier AI hadn’t, these were merely the latest in a long lineage of gradual developments — but an early sign of a reckoning, an attitude shift in how humanity would grapple with these new machine capabilities we were conjuring fitfully into being.

I’d already been worrying for years that things might get out of hand with AI (and had even started writing about it). I was hardly the first!2 But this had felt almost like a perversely secret concern (how can people not see what’s coming?? — but they didn’t), one which humanity at large appeared destined to ignore until either it was too late… or, if we somehow played it right, until a splendid apotheosis of world peace, unlimited bounty, health and longevity delivered by machine intellect. (In fact I think those remain real prospects, and it’s absolutely in our hands to determine which outcomes we get.)

What this headline implicitly spoke of, the subtextual worldview shift belied by the phrasing — “Best humans still outperform” — was that we had woken up and viscerally felt the reality that even the ‘best’ humans might genuinely need to watch their backs. The machines were coming. It was no longer (had never been) a joke or a fairy tale.

This headline, seemingly from a near future in which it was taken for granted that machines, in general, dominated human capabilities, showed what was coming. Headlines like it are now commonplace — perhaps more common than those (now almost boring!) headlines adding to the litany of tasks AI now outcompetes human experts at.

The world changed

The world changed. Not because the world had actually yet changed (much), but because humanity, in our limited and faltering foresight, had noticed that, soon, it might. That murky perception of the future, humanity’s near-unique hallmark and blessing, memetically reverberated and has worked its way into our collective discourse.

In this way, I’m incredibly grateful to the ‘ChatGPT moment’. Rather than implicitly relying on a plucky band of vaguely foresighted but ultimately underpowered ‘sci-fi weirdos’, humanity as a whole is entering the conversation. We’re all stakeholders in the trajectory of this world-transforming sphere of technology, and all kinds of people are beginning to act like it: people with skillsets and perspectives which we’ll need, which had been lacking, in earlier debates. Law theorists, philosophers, engineers, anthropologists, economists, statespeople. It’s a thickly textured problem. It’ll need more than people like me (aspiring polymath though I may be) to solve it!

These cultural conversation shifts are fickle but surely incredibly consequential. 2025 felt like another shift, to me, and 2026 so far — with AI producing genuine national security implications and at the centre of dirty political manoeuvring — seems to suggest that both the training wheels and the gloves are off, as Dean Ball recently put it. It’s a little scary: powerful and not altogether friendly forces have turned their eye to the potential potency of emerging tech, and they3 may wrestle for it, even under the risk that they destroy much in the process or that the tech spills entirely out of their control.

The world, changed

We can be doing better! People can get curious, find out what’s what, consider stakes and what realistic paths we might prefer. Don’t make the mistake of ‘nowsight’ bias — today’s AI are the least capable there will ever be! Take seriously where things might go, and notice if the conversation seems to miss something important that you understand well: it’s still early and the ‘experts’ are mainly that by virtue of noticing the importance of AI a little sooner than everyone else4. Let’s also grab the new tech building blocks we have and bootstrap the way we do foresight, collective intelligence, and coordination.

Don’t mistake me for naively assuming machines will blast through every bottleneck in short order. There’s a lot of adaptability, dexterity, and generality bottlenecks between here and self-sufficient machines. Perhaps I’ll write something about that soon.

What's the growth rate? What's the doubling time? How many orders of magnitude bigger is it? How many years at this rate until it's quintupled?

All questions of ratios and scale.

Scale... hmm.

'Wait', you're thinking, 'let me check the date...'. Indeed. But please, stay with me for the logarithms.

Musical intervals as ratios, and God's joke

If you're a music nerd like me, you'll know that an octave (abbreviated 8ve), the fundamental musical interval, represents a doubling of vibration frequency. So if A440 is at 440Hz, then 220Hz and 880Hz are also 'A'. Our ears tend to hear this as 'the same note, only higher'.

That means the 'same' interval, an octave, corresponds to successively greater gaps in frequency. First a doubling, then a quadrupling, an octupling, and so on. Our perception, and musical notation, maps the space of frequencies logarithmically.

You'll also know that a 'perfect fifth' is a ratio of . A to the E above it, C# to the G# above it, etc. Consonance is all about nice ratios! (Ask Pythagoras.)

At least, the really sweet, in tune fifths are this ratio. Because God is an absolute wheeze, you can keep moving in fifths (3:2) and octaves and get 'new notes' eleven times. That's where we get our Western scale from, originally (except it's originally originally Mesopotamian probably). The twelfth time ((3:2)^12) gets you to a ratio of roughly 129.7:1. That's almost exactly seven doublings, seven octaves (7 * 8ve)! That'd be 128:1. God's joke is in that roughly 1% margin, and musicians have been arguing about what to do about it for centuries. It's a whole thing.1

Cutting a long story short, that leaves us with twelve different notes dividing up the octave. They 'repeat', with 'the same' note again and again at either higher or lower octaves (a full doubling of frequency).

In between octaves, those twelve divisions need to 'add up to' a doubling. For reasons, two steps (a sixth of the overall scale) is referred to as a 'tone', and a single step (a twelfth of the scale) is thus a 'semitone'. That means each semitone corresponds to a ratio of the twelfth root of two. (It's about 1.06, i.e. a ratio increase of about 6%.) The full scale as shown above is called 'chromatic' (because it has every 'colour'...).

This means that neat fractional powers of two map cleanly onto musical intervals. God was generous in giving twelve many factors, so we have musical intervals for the square, cube, fourth, sixth, and twelfth roots of two which come for free.

So far, no logarithms. But we have musical powers of two: give me a fraction and I can tell you the musical interval. That means we also have musical logarithm: give me a musical interval and I can tell you the power of two! e.g. C to G# is eight semitones. So

Musical logarithms? What is he talking about? Surely this is pointless. Yes, it is! Hold on!

Harmonic series

If you're a brass music nerd like me, you'll know that the 'overtones' of most natural vibrations correspond to the 'harmonic series' (no, not that harmonic series, the actually harmonic harmonic series), which are the different pitches you can get a big metal tube to vibrate at if you give it the right encouragement. Incidentally this is how brass players get dozens of different notes out of an instrument having (usually) only three valves2.

This harmonic series is generated by lovely integer ratios! Why? The physics of oscillators. Integer multiples are the only frequencies which can support a standing wave on the same vibrating object (air column, string, membrane).

Brass players spend hours and hours sliding and jumping between these harmonics as a matter of sheer necessity. Only three valves!3 So we know them by heart, by fingers, and by ears.

Combining the harmonic series with the chromatic scale: magic

So we have integer multiples, the harmonic series, laid over a fundamentally logarithmic scale, the chromatic scale consisting of twelve semitones.

Numbers above the notes correspond to small adjustments vs the equally-spaced semitones which are usually used today to deal with God's joke. Ignore them if you don't care about small percentage errors. This is the harmonic series on C; you can have a series on any starting note with the same intervals.

Here's the magic trick. Now we can go from arbitrary ratios to musical intervals!

Start with an easy one, 1.25. That's a ratio 5:4. Fifth harmonic is E (+ 2 8ve). Fourth is C (+2 8ve). The octaves cancel. That's an interval E:C, or four semitones. So 1.25 is four semitones. We already know the 'musical logarithm' of four semitones, it's 4/12 = 1/3. Check on a calculator: log_2(1.25) = 0.32193…. I promised close, not perfect!

A slightly trickier one, 1.8. That's a ratio of 9:5. The ninth harmonic is D (+3 8ve), and the fifth is E (+2 8ve). The octaves partly cancel (leaving a single octave). The interval D:E is minus two semitones. Taken off the residual octave, that leaves ten semitones. So log_2(1.8) = 10/12 = 5/6. Calculator check: log_2(1.8) = 0.848…. Not bad!

It turns out that the musical harmonic series is secretly a mini table of base 2 logarithms.

Base 10, if we really have to

The unit that mainstream sheeple often use for fractional logarithms is the decibel. A decibel divides a base ten order of magnitude in ten. So ten decibels is a dectupling, twenty is a hundredfold, and so on.

Stated similarly, a semitone divices a base two order of magnitude in twelve.

In another cosmic whimsy, 2^10 = 1024 ~= 1000 = 10^3. So 120 semitones are essentially equal to 30 decibels, for an easy exchange rate of four semitones per decibel.

What

Well, look. It's fun, and it gets me logarithms to pretty good approximation. It's good enough for jazz Fermi estimation, as they say. Who is this even good for? I maintain that the intersection between music and mathematics nerds is surprisingly well populated. If that's you, you're welcome. If not, I'm pretty unsure how easy it is to get the harmonic series installed in your brain. Maybe it's only available to the warped few who train in childhood.

There are some other fun tricks with powers and logarithms of two. For example, if you know your binary place values, you can figure out logarithms of very big numbers (and the trick comes in handy here too).

There's also a 'rule of 72' which helps when dealing with small percentage growth rates and doubling times.

I aesthetically like this neat division of doublings into twelve parts, and it's fun to invoke musical intuitions that really have no right to help with mathematics.

You might complain that twelfths are faffy. Who uses twelfths anyway? Everyone everywhere has used decimal for goodness' sake! Well, I have something else to share with you...

Lots of folks naturally recognise that this implies a general strategy: make humans individually — and humanity collectively — better able to solve problems. Very good! (Complementary strategies look like: make progress directly, raise awareness of the challenges, recruit problem solvers, …)

One popular approach is to ‘raise the sanity waterline’ in the most oldschool and traditional way: have a community of best practice, exemplify and proselytise, make people wiser one by one and society wiser by virtue of that. There’ve been some recent successes, not least the existence of communities and forums like LessWrong and Effective Altruism, and some older philosophies and movements.

Another popular approach is to imagine augmenting ourselves in the most futuristic and radical ways: genetic engineering, selective breeding, brain-augmenting implants, brain emulation. Go for it, I suppose (mindful of the potential backfires and hazards). But these probably won’t pan out on what look like the necessary timelines.

There is a middle ground! Use tech to uplift ourselves, yes — but don’t wait for medical marvels and wholesale self-reauthorship. Just use the building blocks we have, anticipate the pieces we might have soon, and address our individual and collective shortcomings one low-hanging fruit at a time.1

The most exciting part is that we’ve got some nifty new building blocks to play with: big data, big compute, ML, and (most novel of all) foundation models and limited agentic AI.

How to generate useful ideas in human reasoning

One place people fall down here is getting locked into asking: ‘OK, what can I usefully ask this AI to do?’. Sometimes this is helpful. But usually it’s missing the majority of the design space: agentic form factors are only a very narrow slice of what we can do with technology, and for many purposes they’re not even especially desirable.

Think about human reasoning. ‘Human’ as in individuals, groups, teams, society, humanity at large. ‘Reasoning’ as in the full decision-making cycle, from sensing and understanding through to planning and acting, including acting together.

I like to first ask: ‘What human reasoning activities are in bad shape?’

• OODA is one good frame:

  • For a given important (type of) decision, what are people observing?

  • How are they orienting and deciding?

  • What actions do they have available and do they know how to do them well?

  • What about the case of teams, groups, institutions: how do their OODAs work (and how do they fail)?

• Also think about development: how do individuals learn and grow? What about groups and communities, how do they form, grow, connect?

• In foresight,

  • What features are we even paying attention to in the first place?

  • What prospects are under consideration?

  • What affordances are we aware of?

  • How are we strategically creating sensing opportunities and the means to adapt plans?

  • How do our forecasts achieve precision and calibration?

• In epistemics, think about the message-passing nature of most human knowledge processes.

  • How do we assess the nodes (communicators)?

  • How do we assess, digest, and compile the messages (claims, evidence, proposals, …)?

  • How do we understand and manage the structure of the network itself (communication relationships, broadcasts and other topologies, …)?

  • What about the traffic (message rates, density distribution, repeated and relayed transmissions, …)?

  • What messages ought to be routed where, when and on behalf of whom?

• In coordination, what are the conditions for success?

  • We need to find or recognise potential counterparties.

  • We might need the charters, norms, or institutions to condition and frame interaction productively — ones which don’t fail or fall to corruption or capture.

  • We need to surface enough mutually-compatible intent or outcome preference.

  • Our ensembled group wisdom might be a necessary source of insight or agility.

  • We need to survive the tug of war of negotiation (which can dissolve into antagonism, even when there’s common knowledge of win-win possibilities).

  • Means of verification and enforcement may be needed to access good options.

Think of a particular audience with either the scale or the special influence to make a difference (this can include ‘the general public’), and the deficits they have in these reasoning activities. Now ask: ‘What kinds of software2 might help and encourage people to do those better?’.

• Is there an edge to be gained by unlocking big (or even medium) data (which can often be more living and queryable than ever before thanks to LMs)?

• Can large amounts of clerical labour (again LMs) per capita make something newly feasible?

• Can big compute and simulation (including multi persona simulation: LMs again!) drive better understanding of an important dynamic?

• Can extensive background exploration, search, or ‘brainstorming’ by AI surface important opportunities or considerations?

• Can always-on, flexibly-semantically-sensitive sensing and monitoring bring attention where it’s needed faster than before (or at all)?

• Could facilitation and translation bring forth, and synergise, the best array of human capabilities in a given context?

• Could software’s repeatability, auditability, and privacy (in principle), combined with the context and semantic sensitivity of AI, unlock new frontiers of trustable human scaffolding?

• …

Finding flaws and avoiding backfire

Think seriously about backfire: we don’t want to differentially enable bad human actors or rogue AI to reason and coordinate! As Richard Rumelt, author of Good Strategy/Bad Strategy observes,

The idea that coordination, by itself, can be a source of advantage is a very deep principle.

Coordination’s dark side is collusion, including cartels, oligarchy, and concentration of power, in imaginable extreme cases cutting out most or even all humans.

Similarly, epistemic advantage (in foresight and strategy, say) can be parlayed into resource or influence advantage. If those can be converted in turn into greater epistemic advantage (by employing compute and position for epistemic attacks or in further private epistemic advancement) without commensurate counterweights or defences, this could be quite problematic.

Part of it is about choosing distribution strategies which reduce misuse surface area (or provide antidotes), and part of it is about preferring tech which asymmetrically supports (and perhaps encourages) ‘good’ use and behaviour.

Do it

FLF’s fellows, and I and others have been doing some of this exploration recently. Stay tuned for more. Meanwhile, join in! We’re early in a critical period where much is up for grabs and what we build now might help shape and inform the choices humanity makes about its future (or whether it makes much choice at all). Try things, see what kinds of tools earn the attention and adoption that matters, and share what you learn. Consider principles to apply, especially for minimising backfire risks, and share particular considerations for or against certain kinds of tech and audience targets.

Thanks to Owen Cotton-Barratt and Ben Goldhaber for helpful comments, and to Lizka Vaintrob for recent relevant conversations

AI is not just one, big, singular thing. What are the ways we can bring forward the beneficial possibilities, while delaying or defending against the harmful ones?

As I repeatedly emphasise to anyone who’ll listen: it’s never been just a dichotomy between ‘yes, good, more AI please’ and ‘no, bad, less AI thank you’ — and it’s not even a case of just making sure ‘the AIs’ are good (though this helps). How the tools and technological building blocks at our disposal are integrated into applications, workflows, and use-cases, is just as important.

And the tools, products, and systems we could develop now shape the context for subsequent developments, including by:

• equipping people to better predict and understand their options

• enabling people to coordinate better around preferred possibilities (which might otherwise be difficult due to mismatched incentives or race dynamics)

• giving the tools to defuse or defend against hazardous developments, or their precursors

For technologists, futurists, philanthropists, legislators, experts, and other members of society trying to make tech progress go well, paying attention to which effects happen, in what order — and what our options are for choosing wisely there — looks like a really promising, and neglected way of reducing large-scale risk and bringing about huge benefits.

You can read our few pages of fuller discussion for more of our thoughts on some scenarios we think are worth considering, including intelligence explosion, turbocharged economy, and epistemic uplift.

Knowledge is integral to living life well, at all scales:

• Individuals manage their life choices: health, career, investment, and others on the basis of what they understand about themselves and their environments.

• Institutions and governments (ideally) regulate economies, provide security, and uphold the conditions for flourishing under their jurisdictions, only if they can make requisite sense of the systems involved.

• Technologists and scientists push the boundaries of the known, generating insights and techniques judged valuable by combining a vision for what is possible with a conception of what is desirable (or as proxy, demanded).

• More broadly, societies negotiate their paths forward through discourse which rests on some reliable, broadly shared access to a body of knowledge and situational awareness about the biggest stakes, people’s varied interests in them, and our shared prospects.

  • (We’re especially interested in how societies and humanity as a whole can navigate the many challenges of the 21st century, most immediately AI, automation, and biotechnology.)

Meanwhile, dysfunction in knowledge-generating and -distributing functions of society means that knowledge, and especially common knowledge, often looks fragile1. Some blame social media (platform), some cynical political elites (supply), and others the deplorable common people (demand).

But reliable knowledge underpins news, history, and science alike. What resources and infrastructure would a society really nailing this have available?

Among other things, we think its communication and knowledge infrastructure would make it easy for people to learn, check, compare, debate, and build in ways which compound and reward good faith. This means tech, and we think the technical prerequisites, the need, and the vision for a full epistemic stack2 are coming together right now. Some pioneering practitioners and researchers are already making some progress. We’d like to nurture and welcome it along.

In this short series, we’ll outline some ways we’re thinking about the space of tools and foundations which can raise the overall epistemic waterline and enable us all to make more sense. In this first post, we introduce frames for mapping the space —3 different layers for info gathering, structuring into claims and evidence, and assessment — and potential end applications that would utilize the information.

A full what?

A full epistemic stack. Epistemic as in getting (and sharing) knowledge. Full stack as in all of the technology necessary to support that process, in all its glory.

What’s involved in gathering information and forming views about our world? Humans aren’t, primarily, isolated observers. Ever since the Sumerians and their written customer complaints4, humans have received information about much of their our world from other humans, for better or worse. We sophisticated modern beings consume information diets transmitted across unprecedented distances in space, time, and network scale.

With an accelerating pace of technological change and with potential information overload at machine speeds, we will need to improve our collective intelligence game to keep up with the promise and perils of the 21st century.

Imagine an upgrade. People faced with news articles, social media posts, research papers, chatbot responses, and so on can trivially trace their complete epistemic origins — links, citations, citations of citations, original data sources, methodologies — as well as helpful context (especially useful responses, alternative positions, and representative supporting or conflicting evidence). That’s a lot, so perhaps more realistically, most of the time, people don’t bother… but the facility is there, and everyone knows everyone knows it. More importantly, everyone knows everyone’s AI assistants know it (and we know those are far less lazy)! So the waterline of information trustworthiness and good faith discourse is raised, for good. Importantly, humans are still very much in the loop — to borrow a phrase from Audrey Tang, we might even say machines are in the human loop.

Some pieces of this are already practical. Others will be a stretch with careful scaffolding and current-generation AI. Some might be just out of reach without general model improvements… but we think they’re all close: 2026 could be the year this starts to get real traction.

Does this change (or save) the world on its own? Of course not. In fact we have a long list of cautionary tales of premature and overambitious epistemic tech projects which achieved very few of their aims: the biggest challenge is plausibly distribution and uptake. (We will write something more about that later in this series.) And sensemaking alone isn’t sufficient! — will and creativity and the means to coordinate sufficiently at the relevant scale are essential complements. But there’s significant and robust value to improving everyone’s ability to reason clearly about the world, and we do think this time can be different.

Layers of a foundational protocol

Considering the dynamic message-passing network of human information processing, we see various possible hooks for communicator-, platform-, network-, and information-focused tech applications which could work together to improve our collective intelligence.

We’ll briefly discuss some foundational information-focused layers together with user experience (UX) and tools which can utilise the influx of cheap clerical labour from LMs, combined with intermittent judgement from humans, to make it smoother and easier for us all to make sense.

All of these pieces stand somewhat alone — a part of our vision is an interoperable and extensible suite — but we think implementations of some foundations have enough synergy that it’s worth thinking of them as a suite. We’ll outline where we think synergies are particularly strong. In later posts we’ll look at some specific technologies and examples of groups already prototyping them; for now we’re painting in broad strokes some goals we see for each part of the stack.

Ingestion: observations, data, and identity

Ultimately grounding all empirical knowledge is some collection of observations… but most people rely on second-hand (and even more indirect) observation. Consider the climate in Hawaii. Most people aren’t in a position to directly observe that, but many have some degree of stake in nonetheless knowing about it or having the affordance to know about it.

For some topics, ‘source? Trust me bro,’ is sufficient: what reason do they have to lie, and does it matter much anyway? Other times, for higher stakes applications, it’s better to have more confirmation, ranging from a staked reputation for honesty to cryptographic guarantee5.

Associating artefacts with metadata about origin and authorship (and further guarantees if available) can be a multiplier on downstream knowledge activities, such as tracing the provenance of claims and sources, or evaluating track records for honesty. Thanks to AI, precise formats matter less, and tracking down this information can be much more tractable. This tractability can drive the critical mass needed to start a virtuous cycle of sharing and interoperation, which early movers can encourage by converging on lightweight protocols and metadata formats. In true 21st Century techno-optimist fashion, we think no centralised party need be responsible for storing or processing (though distributed caches and repositories can provide valuable network services, especially for indexing and lookup6).

Structure: inference and discourse

Information passing and knowledge development involve far more than sharing basic observations and datasets between humans. There are at least two important types of structure: inference and discourse.

Inference structure: genealogy of claims and supporting evidence (Structure I)

Ideally perhaps, raw observations are reliably recorded, their search and sampling processes unbiased (or well-described and accounted for), inferences in combination with other knowledge are made, with traceable citations and with appropriate uncertainty quantification, and finally new traceable, conversation-ready claims are made.

We might call this an inference structure: the genealogy and epistemic provenance of given claims and observations, enabling others to see how conclusions were reached, and thus to repeat or refine (or refute) the reasoning and investigation that led there.

Of course in practice, inference structure is often illegible and effortful to deal with at best, and in many contexts intractable or entirely absent. We are presented with a selectively-reported news article with a scant few hyperlinks, themselves not offering much more context. Or we simply glimpse the tweet summary with no accompanying context.

Even in science and academia where citation norms are strongest, a citation might point to a many-page paper or a whole book in support of a single local claim, often losing nuance or distorting meaning along the way, and adding much friction to the activity of assessing the strength of a claim7.

How do tools and protocols improve this picture? Metascience reform movements like Nanopublications strike us as a promising direction.

Already, LM assistance can make some of this structure more practically accessible, including in hindsight. A lightweight sharing format and caches for commonly accessed inference structure metadata can turn this into a reliable, cheap, and growing foundation: a graph of claims and purported evidence, for improved further epistemic activity like auditing, hypothesis generation, and debate mapping.

Discourse: refinement, counterargument, refutation (Structure II)

Knowledge production and sharing is dynamic. With claims made (ideally legibly), advocates, detractors, investigators, and the generally curious bring new evidence or reason to the debate, strengthening or weakening the case for claims, discovering new details, or inferring new implications or applications.

This discourse structure associates related claims and evidence, relevant observations which might not have originally been made with a given topic in mind, and competing or alternative positions.

Unfortunately in practice, many arguments are made and repeated without producing anything (apart from anger and dissatisfaction and occasional misinformation), partly because they’re disconnected from discourse. This is valuable both as contextual input (understanding the state of the wider debate or investigation so that the same points aren’t argued ad infinitum and people benefit from updates), and as output (propagating conclusions, updates, consensus, or synthesis back to the wider conversation).

This shortcoming holds back science, and pollutes politics.

Tools like Wikipedia (and other encyclopedias), at their best, serve as curated summaries of the state of discourse on a given topic. If it’s fairly settled science, the clearest summaries and best sources should be made salient (as well as some history and genealogy). If it’s a lively debate, the state of the positions and arguments, perhaps along with representative advocates, should be summarised. But encyclopedias can be limited by sourcing, available cognitive labour and update speed, one-size-fits-all formatting, and sometimes curatorial bias (whether human or AI).8

Similar to the inference layer, there is massive untapped potential to develop automations for better discourse tracking and modeling. For example, LLMs doing literature reviews can source content from a range of perspectives for downstream mapping. Meanwhile, relevant new artefacts can be detected and ingested close to realtime. We don’t need to agree on all conclusions — but we can much more easily agree on the status of discourse: positions on a topic, the strongest cases for them, and the biggest holes9. Direct access as well as helpful integrations with existing platforms and workflows can surface the most useful context to people as needed, in locally-appropriate format and level of detail.

Assessment: credence, endorsement, and trust

Claims and evidence, together with counter claims and an array of perspectives (however represented), give some large ground source of potential insight. But at a given time and for a given person there is some question to be answered: reaching trusted summaries and positions.

Ultimately consumers of information sources come to conclusions on the basis of diverse signals: compatibility with their more direct observations, assessment of the trustworthiness and reliability (on a given topic) of a communicator, assessment of methodological reasonableness, weighing and comparing evidence, procedural humility and skepticism, explicit logical and probabilistic inference, and so on. It’s squishy and diverse!

We think some technologies are unable to scale because they’re too rigid in assigning explicit probabilities, or because they enforce specific rules divorced from context. This fails to account for real reasoning processes and also can work against trust because people (for good and bad reasons) have idiosyncratic emphases in what constitutes sensible reasoning.

We expect that trust should be a late-binding property (i.e. at the application layer), to account for varied contexts and queries and diverse perspectives, interoperable with minimally opinionated structure metadata. That said, squishy, contextual, customisable reasoning is increasingly scalable and available for computation! So caches and helpful precomputations for common settings might also be surprisingly practical in many cases.

With foundational structure to draw from, this is where things start to substantially branch out and move toward the application layer. Some use cases, like summarisation, highlighting key pros and cons and uncertainties, or discovery, might directly touch users. Other times, downstream platforms and tools can integrate via a variety of customized assessment workflows.

Beyond foundations: UX and integrations

Foundations and protocols and epistemic tools sound fun only to a subset of people. But (almost) everyone is interested in some combination of news, life advice, politics, tech, or business. We don’t anticipate much direct use by humans of the epistemic layers we’ve discussed. But we already envision multiple downstream integrations into existing and emerging workflows: this motivates the interoperability and extensibility we’ve mentioned.

A few gestures:

• Social media platforms struggle under adversarial and attentional pressures. But distributed, decentralised context-provision, like the early success stories in Community Notes, can serve as a widely-accessible point of distribution (and this is just one form factor among many possible). In turn, foundational epistemic tooling can feed systems like Community Notes.

• More speculatively, social-media-like interfaces for uncovering group wisdom and will at larger scales while eliciting more productive discourse might be increasingly practical, and would be supported by this foundational infrastructure.

• Curated summaries like encyclopedias (centralised) and Wikipedia (decentralised) are often able to give useful overviews and context on a topic. But they’re slow, don’t have coverage on demand, offer only one-size-fits-all, and are sometimes subject to biases. Human and automated curators could consume from foundational epistemic content and react to relevant updates responsively. Additionally, with discourse and inference structure more readily and deeply available, new, richly-interactive and customisable views are imaginable: for example enabling strongly grounded up- and down-resolution of topics on request10, or highlighting areas of disagreement or uncertainty to be resolved.

• Authors and researchers already benefit from search engines, and more recently ‘deep research’ tooling. Integration with easily available relational epistemic metadata, these uplifts can be much more reliable, trustworthy, and effective.

• Emerging use of search-enabled AI chatbots as primary or complementary tools for search, education, and inquiry means that these workflows may become increasingly impactful. Equipping chatbots with access to discourse mapping and depth of inference structure can help their responses to be grounded and direct people to the most important points of evidence and contention on a topic.

• Those who want to can already layer extensions onto their browsing and mobile internet experiences. Having always-available or on-demand highlighting, context expandables, warnings, and so on, is viable mainly to the extent that supporting metadata are available (though LMs could approximate these to some degree and at greater expense). More speculatively, we might be due a browser UX exploration phase as more native AI integration into browsing experiences becomes practical: many such designs could benefit from availability of epistemic metadata.

How? Why now?

If this would be so great, why has nobody done it already? Well, vision is one thing, and we could also make a point about underprovision of collective goods like this. But more relevant, the technical capacity to pull off this stack is only really just coming online. We’re not the first people to notice the wonders of language models.

First, the not inconsiderable inconveniences of the core epistemic activities we’ve discussed are made less overwhelming by, for example, the ability of LLMs to digest large amounts of source information, or to carry out semi-structured searches and investigations. Even so, this looks to us like mainly a power-user approach, even if it came packaged in widely available tools similar to deep research, and it doesn’t naively contribute to enriching knowledge commons. We can do better.

With a lightweight, extensible protocol for metadata, caching and sharing of discovered inference structure and discourse structure becomes nearly trivial11. Now the investigations of power users (and perhaps ongoing clerical and maintenance work by LLM agents) produce positive epistemic spillover which can be consumed in principle by any downstream application or interface, and which composes with further work12. Further, the risks of hallucinated or confabulated sources (for LMs as with humans) can be limited by (sometimes adversarial) checking. The epistemic power is in the process, not in the AI.

Various types of openness can bring benefits: extensibility, trust, reach, distribution — but can also bring challenges like bad faith contributions (for example omitting or pointing to incorrect sources) or mistakes. Tools and protocols at each layer will need to navigate such tradeoffs. One approach could have multiple authorities akin to public libraries taking responsibility for providing living, well-connected views over different corpora and topics — while, importantly, providing public APIs for endorsing or critiquing those metadata. Alternatively, perhaps anyone (or their LLM) could check, endorse, or contribute alternative structural metadata13. Then the provisions of identity and endorsement in an assessment layer would need to solve the challenges of filtering and canonicalisation.

In specific epistemic communities and on particular topics, this could drive much more comprehensive understanding of the state of discourse, pushing the knowledge frontier forward faster and more reliably. Across the broader public, discourse mapping and inference metadata can act against deliberate or accidental distortion, supporting (and incentivising) more good faith communication.

Takeaways

Knowledge, especially reliable shared knowledge, helps humans individually and collectively be more right in making plans and taking action. Helping people better trust the ways they get and share useful information can deliver widespread benefits as well as defending against large-scale risk, whether from mistakes or malice.

We communicate at greater scales than ever, but our foundational knowledge infrastructure hasn’t scaled in the same way. We see a large space of opportunities to improve that — only recently coming into view with technical advances in AI and ever-cheaper compute.

This is the first in what will be a series exploring one corner of the design landscape for epistemic tech: there are many uncertainties still, but we’re excited enough that we’re investigating and investing in pushing it forward.

We’ll flesh out more of our current thinking on this stack in future entries in this series, including more on existing efforts in the space, interoperability, and core challenges here (especially distribution).

Please get in touch if any of this excites or inspires you, or if you have warnings or reasons to be skeptical!

Thanks to our colleagues at the Future of Life Foundation, and to several epistemic tech pioneers for helpful conversations feeding into our thinking.

I want to know: is ‘extra reasoning compute’ like this? (Or, under what conditions and by what means can you beat this?) I’m especially interested in this question as applied to deliberate exploration and experiment design.

Said another way, from a given decision-context, without making extra observations or gathering extra data, what are the optimal marginal returns to ‘thinking harder’2 about what to do next?

Intuitively, if I have a second to come up with a plan, it might be weak, five minutes and it might be somewhat reasonable, a day and it’ll be better, a year (full time!) and I’ve reached very diminishing returns. Presumably a century in my ivory tower would be barely better. I’d usually do better trying to get more data.

Is this even a sensible question, or is ‘improvement in reasoning output’ far too vague to get traction here?

That’s the question; below some first thoughts toward an answer.

Simple model: repeated sampling/best of k

If you have a proposal generator, and you can choose between proposals, a simple approach to getting better generations is:

• sample a large number, k, of proposals

• (try to) evaluate and pick the best one

(This is actually the best strategy you could take if you can only add parallel compute, but there might be strictly better approaches if you can add serial3.)

Even assuming you can unerringly pick the best one, this strategy turns out to have logarithmically-bounded expected value for many underlying distributions of proposals4. In fact, for a normally-distributed proposal generator, you even get the slightly worse square root of logarithmic growth5.

You can in principle sidestep this if your proposal generator has sufficiently heavy-tailed proposal distribution, and you can reliably ex ante distinguish better from worse at the tails.

Another simple model: widely distributed ‘promise’ of lines of inquiry

Suppose you have various lines of inquiry to spend thinking time on. The best you can do is:

• start thinking on the most promising lines

• spend additional thinking on successively less promising lines

(This assumes you can somewhat reliably distinguish promise.)

If their ‘quality’ or ‘promise’ ranges over many orders of magnitude, then even if you get to accumulate insights additively6, you’ll actually make only bounded progress7 towards a theoretical ‘best possible’ - this is worse than logarithmic, though looks qualitatively similar over a substantial range of effort.

But why would the promise of lines of inquiry range over many orders of magnitude? We might say, ‘in practice, it often seems to’, and there are some theoretical reasons to expect this. You ‘pick low hanging fruit’ earliest, and face diminishing returns later. But to a large extent this model assumes the conclusion.

Other rougher gestures

Search depth

Often to find approximate solutions to problems, we might employ search over a tree-like structure. This emerges very naturally for planning over time, for example, where branching options (whether choice or chance) at each chosen time interval give rise to a tree of possible plans. (Compare Monte Carlo tree search.)

If gains are roughly uniform in search depth, this gives rise to logarithmic returns to further search. With excellent heuristics, you might be able to prune large fractions of the tree - this gives you a kinder exponent, but still an exponential space to search.

When (if at all) are gains over search depth dependably growing, rather than uniform at best? Alternatively, when can uniform (or better) gains be reliably achieved by expanding the search strictly less than exponentially?

Modelling chaos

Chaotic systems are characterised by sensitivity to initial conditions: dynamics where measurement or specification errors compound exponentially.

So, to forecast at a given precision and probabilistic resolution, it takes exponentially tighter initial specification precision to forecast marginal incremental time depth. (This is why in practice we only ever successfully forecast chaotic systems like weather at quite coarse precision or short horizon.)

Specification precision doesn’t exactly map to having extra compute, but it feels close. And marginal incremental time depth doesn’t necessarily correspond uniformly to ‘goodness of plan’.

Combinatorial search

If there’s some space of ingredients or components which can be combined for possible insight, the size of the search space is exponential8 in the number of components in a proposed combination. So if, among good plans at each scale, gains are proportional to the number of components in the plan (and there are similarly many good plans at each scale), you get logarithmic returns to searching longer.

Something similar applies if the design possibilities benefit from combining already-discovered structures in a hierarchy, for example if emergent features of subcomponents unlock new levels of effectiveness in a combined design (molecules, peptides, proteins, organelles, cells, ...).

But the assumption of roughly uniform gains over scales like this is carrying some weight here.

This part introduces exploration and research taste, as well as discussing their role in research and development, and the ways that AI could change that picture. This gives rise to some exciting and underexplored (!) opportunities for beneficial and defensive contributions to research.

A second essay will discuss more implications for AI development and governance, including the potential for AI to accelerate the pace of development of AI itself, and some implications for safety and security.

Neither essay will be especially technical, but I will gesture to the technical and mathematical aspects that I find to be illuminating. As ever when I write I raise more questions than I answer! But I hope to provide some initial useful takeaways as well as productive directions for thinking about these issues.

Introducing exploration and experimentation

Scientific and technological progress are driven by experimentation: that is, doing things to find out how the world works. In the field of AI we call this 'exploration'.

Exploration for learning is not just a human phenomenon: it's ubiquitous in natural systems at various scales (from evolution itself to the play of young animals), in individual human lifetimes (as we learn skills or contribute to novel discoveries) as well as human institutions and societies (which also learn through experience), and in computer science and AI (where exploration for discovery and problem-solving are common).

We care a lot about scientific and technological potential - they can yield enormous risks (from accident, misuse, or societal destabilisation) or enormous benefits (solving major problems in medicine, climate, energy, or even defending against other risky technologies). So exploration isn't just of academic interest.

When we forget to consider how new knowledge is generated, how novel technologies are developed, when we conflate 'knowledge' with 'learning' or 'learning' with 'exploring', mistakes are made. Especially when such predictions are action-guiding we can end up taking misguided or even harmful actions, or missing opportunities to intervene in beneficial ways. So let's do some unpacking!

What factors make exploration (and by extension, research) more or less effective? What are the bottlenecks and limits to exploration? How could AI change the picture? And how can we apply insights from this lens to contribute to a better future?

Subscribe now

Why does exploration matter?

Knowledge production loop: activity yields observations, improving knowledge — Exploration drives the loop — Lack of exploration means knowledge stagnation — Exploration is key to understanding technological progress

Learning systems gather new knowledge and insights from observations/data. Random flailing or arbitrary data aren't especially helpful. You want it to be telling you something new that you didn't already know - so it pays to deliberately seek out or gather novelty and informative observations1. This applies at the grandest scales of scientific endeavour as well as in mundane scenarios like navigating an unfamiliar building or learning a new skill.

Owen Cotton-Barratt recently discussed the 'knowledge production loop': activity and observations generate data (captured in datasets and models as 'crystallised intelligence') and combine with thinking algorithms ('fluid intelligence') to in turn drive new activity and observations.

I'd additionally characterise exploration as the way that crystallised world model and novelty taste interact with fluid reasoning and planning to judiciously choose activities yielding the most informative observations... in turn improving world models and taste ad infinitum.

Quality and quantity of exploration mark the vast difference between a civilization with vibrant progress in science and technology and one with a near-static (or even regressing) capability base - and on an individual level, it's often the difference between rapidly developing new skills or knowledge and getting stuck in a rut.

Understanding exploration is therefore key to understanding technological progress, with all the risks and benefits that entails.

Research and taste

Research is world-model-refinement — Exploration quality drives research — Taste is a learned feel for value of information — Reasoning and world modelling augment taste for exploratory planning

'Research' can be thought of broadly as refining one's world model in a particular domain. We want to know things like: how does electromagnetism work and what can it do for us? How can we prevent diseases from ruining lives? Or (more mundane) how can I get better at playing the piano or juggling? When I say 'research', I equally refer to personal learning and skill-building, scientific research, entrepreneurialism, and business development: all involve exploration and learning from experience.

We can describe three factors determining research production:

• Throughput: doing more practice, running more experiments, gathering more data faster, etc.

• Modelling efficiency: gathering more generalisable insight from a given experiment or observation.

• Exploration quality: choosing better experiments and routines to get more informative observations.2

We'll mostly talk about exploration quality here3, which is in turn governed by taste and exploratory planning.

What do I mean by 'taste'? Sometimes people refer to 'research taste' as a sense which develops from domain experience for the types of experiments and other activities which are most likely to be interesting or informative, or otherwise move forward the state of understanding. Clearly this is an essential component of any deliberate exploration - otherwise you're back to flailing randomly!

The taste that's being developed is exactly analogous to a taste for activity which is liable to yield good outcomes of other kinds. We're just considering the value of information as the good in question. So this decomposes into an ability to come up with promising proposals more often, perhaps together with abilities to discriminate more accurately between better and worse proposals or to determine refinements and improvements to proposals4.

Now, imagine - for the sake of the argument - you're a human. Even better, in fact, imagine you're inhumanly fast and detail-sensitive, the best reasoner in the world, and you have general knowledge matching the rest of the world combined. You still need to do research in order to make new discoveries. If you don't know the details yet, experimentation isn't something you can skip!5 Your especially effective reasoning merely acts as another input to exploration quality, alongside domain research taste, perhaps allowing you to choose better experiments, and achieve results sooner. Reasoning applied effectively in this way is exploratory planning.

So present exploration quality depends on your current level of taste, while future exploration quality will also depend on taste accrual6. Reasoning and planning of course also feed into this, as we improve proposals and discard designs in favour of better-looking ones - but this has to ground out in a taste for what makes a good proposal in the first place.

From play to experimentation

Play is proto-exploration — Fun is proto-taste — Humans adaptably accrue taste in novel domains — Taste is domain-specific but exploratory principles generalise

These aspects of taste are discovered and refined through experience. Research taste is domain-specific!

Many humans and animals, especially youngsters, have built in instincts for play, curiosity, and novelty. These have been tuned by painstaking natural selection to aid in orienting to the range of body configurations, environments and communities those animals usually inhabit, precisely by exploring: gathering evidence and information about how things work. In this case, evolution did the slow, gradual work of determining the 'taste', the recognisable hallmarks of good exploratory behaviour, and wired up the 'fun' sense to those hallmarks7.

As Eric Drexler says,

We call children intelligent because of what they can learn, not what they can do

Playful young animals and humans thereby become adept at controlling their bodies and engaging in effective social interaction. But humans move past mere bodily control and socialisation: we use and develop tools, technologies, and diverse and innovative social structures.

For many researchers and others engaging in creation, experimenting is a lot like playing! - the rich and sophisticated kinds of play that humans engage in somewhat instinctively. But, because research and development and science and industry move beyond the historic realms of human activity, the 'taste' bestowed by evolution is rarely well suited. An untrained human has no instinct at all for the kinds of experiments that are most likely to yield useful information about the behaviour of a new material or the structure of an unseen mathematical object! This applies equally to business activities and entrepreneurialism. Substantial experience is needed.

Do we see areas where 'taste' generalises, pointing against the claim of domain-specificity? The broad principles of science and engineering appear to generalise across domains, and evidence suggests that individual humans and human organisations vary in their latent potential to accrue and apply research taste. This might be down to being more or less motivated to explore, having different capacity to learn from experience, or varying procedures for planning next steps. This gives rise to an appearance of research taste generality. But domain-specific research taste is mastered only through domain-specific experience. Expert researchers in one area may contribute to other areas - but almost always only after gaining some depth of familiarity with the new area as well.

So it's reasonable to think of exploration quality as comprising two subfactors. First, the somewhat transferable general principles of exploration: playfulness, open-mindedness, planning for novelty and interestingness. And second, domain-specific research taste: the experience that guides determination of what situations count as novel or interesting8, and what types of planning are most likely to uncover them.

Subscribe now

Exploration in AI, past and future

First: humans curate data — Now: RL allows automatic data generation — Next?: in-context exploration characterises R&D tasks — Perhaps this is ‘AGI’?

In contemporary frontier AI systems, it's been mostly humans responsible for gathering 'high quality' informative data, often in quite hands-off ways like scraping huge datasets from the internet, but latterly with more attention on procurement and curation of especially informative or exemplary data.

With reinforcement learning (RL), the data coming in starts to rely increasingly on the activity of the system itself - together with whatever grading mechanism is in place. That's why lots of RL conversations of the past were so obsessed by exploration: taking judicious actions to get the most informative observations! So earlier AI research actually foregrounded exploration somewhat more. Helen Toner recently discussed the return of RL to centre stage in contemporary frontier AI, asking what properties of a domain make it more or less amenable to gains from reinforcement learning.

Still, in many RL settings, the human engineers are able to curate training environments with high-signal automated feedback systems, as Toner discusses. On the other hand, once we're talking about activities like R&D of various kinds, the task of exploring is inherently most of the task itself, making within-context exploration essential!

This makes 'learning to learn' or in particular 'learning to explore/experiment' among the most useful ways to operationalise 'AGI', from my perspective9. I'm not sure how best to track this, and I'm not aware of any benchmarks or studies which take this view on frontier general AI10. My personal experience with LM agents anecdotally points to them improving over time at orienting to uncertainties within their environment and being a little more creative at trying things out and testing things in 2025 than in 2024 or 2023, but not vastly - progress to date appears much more rapid in 'crystallised' intelligence.

Research by AI: AI with research taste?

Bootstrapping research taste from humans — AI advantages from speed and copying — AI learning by doing — Human advantages and bottlenecks to AI — Human-AI complementary workflows

There may be ways for AI training datasets to 'hoover up' research taste from existing experts and institutions, perhaps from lab notes or interviews, though humans at least usually learn more from actually trying research than merely from reading or talking about it. (This presumably reflects the fact that merely communicating about research experience is a much less rich source of information than actually experiencing it directly: the same issue faced by all kinds of knowledge transfer through limited media like language.)

So research taste in AI is not starting from scratch: already AI can talk in sensible, albeit sometimes basic ways about experiment design. The taste is bootstrapped from the taste implied by all the hints and observations in training data.

Could AI surpass the research taste exhibited by expert humans and human organisations? It's unclear where the ceiling is, but certainly AI would appear to have several advantages in principle: direct sharing of observations and experiences between instances, potentially far larger effective 'researcher headcount', total observation quantity far outstripping the longest-lived human experts (to date)11, all adding up to a far greater opportunity to accrue and accumulate taste. Additionally, due to computer speed, the opportunity to confer and deliberate in far more total depth the implications of each experiment and the appropriate designs of future experiments means that exploratory planning could also be boosted.

Crucially, acquiring frontier-applicable research taste would require either finding ways to bootstrap from existing research taste, which is often implicit (or even proprietary!), or enabling AI to learn by doing, perhaps aided by expert supervisors (just as human trainee researchers are), by instrumenting research processes and equipment with sensors and manipulators. Like hiring junior researchers, this would come with some upfront costs to any organisation attempting it12!

Human researchers begin with some advantages today: easier physical manipulation of experimental materials (for now), a capital base of experimental equipment designed for human use, and an ecosystem designed around the training, retention, and interaction of human experts. These aren't fundamental barriers to researcher AIs, but represent some hurdles or bottlenecks that might take time and other resources to reach past.

Of course, the capacities to interpret evidence, propose experiments, design and refine proposals, and to implement experiments need not reside 'in the same mind', just as human organisations already exhibit this division of labour. But the better fitted these pieces are to each other, the more efficient the overall system will be. Drexler's 'large knowledge models' discussion treats knowledge as a resource, to be combined with planning capacity and discernment from disparate sources. Similar agendas, for example from the UK's Advanced Research and Invention Agency (ARIA) perhaps promise both a more effective and more safely manageable way to integrate AI into research processes than wholesale development of autonomous researcher AIs.

Opportunities

Recapping research, experimentation, exploration, taste — Implications for AI forecasting and ‘intelligence explosion’ — Differentially bootstrapping AI taste — Differentially complementing AI exploration — Detecting dangerous research — Exploring AI applications for flourishing

Deliberate experimentation, consisting of exploratory planning and research taste, is a critical component of efficient learning - which, in R&D-heavy domains at least, because they inherently butt against the boundaries of the known, is foundational to progress.

Much more can and should be said about the implications of an experimentation-oriented view of R&D, both on AI and facilitated by AI in other domains. Here are some initial directions:

First, in forecasting AI capabilities and timelines, we should account for the costs of experimentation. This can include quantifying the relevant variables (iteration speed, quality of simulation, modelling, and exploratory planning, accrual and accumulation of research taste, the cost of experimental resources including compute and real-world interactions, etc.). Of particular interest, this could help to characterise the potential for 'self' improvement and the possibility of an intelligence explosion (which matter by implications for other R&D and for loss of control over AI systems).

You can't skip exploration! But greater intelligences (individual or collective) can be more efficient at it in general, and domain-specific taste in particular certainly yields improved rate of progress.

This cuts both ways for safety. You can't develop dangerous nanotech purely from first principles: you have to experiment, either in vitro or in silico. Unfortunately, nor can you generate new defensive vaccination, sterilisation, or biomonitoring paradigms without putting in the experimental legwork.

This may be revealing for those seeking to differentially drive beneficial and defensive research ahead of risky research. For example, exposing research logs and expert interviews to AI systems may yield a way to bootstrap specific kinds of research taste in AI. Alternatively, recognising the default taste-weakness but speed-advantage and general knowledge breadth of AI systems may suggest strategies for complementary human-AI workflows which could be both more effective and more manageable than naively attempting to create researcher AIs wholesale.

Beyond AI-driven exploratory planning and research taste, we should expect strong synergy with robotics, sensors, simulation, modelling, and other automation technologies, as complementary production factors in R&D progress. This is likely to naturally drive investment into these technologies, but may provide opportunities to differentially unbottleneck AI multipliers in beneficial areas by devoting development to their specific complements in particular.

Further, noting that technology can rarely be developed purely from first principles, intelligence and security organisations concerned about risky research directions may be able to anticipate the kinds of experiments that are likely to be useful, and therefore the kinds of resources and activities required to make progress in those areas. This may include flows and concentrations of certain machines or components, movements of specific rare materials, movement of human talent, or known side-effects of experiments. Where materials are very dual-use (such as concentrations of computing clusters), structured inspection, auditing, or transparency tools may aid in guaranteeing that only safe and sanctioned experiments are being carried out.

Finally, now is a great time to be experimenting with AI systems and their applications, especially for people who haven't traditionally paid attention to AI. Rapid developments mean that the extent of possibilities with current tech remains underexplored, and boosting defensive and beneficial applications ahead of risky ones is a great way to ensure that the future is better than it otherwise would be!

Thanks to Owen Cotton-Barratt and Jay Bailey for feedback and conversations on this topic

I’ve previously made arguments like:

Not long after it becomes possible for someone to make powerful artificial intelligence1, it might become possible for practically anyone to make powerful AI.

1. Compute gets exponentially cheaper by default.

2. Knowledge proliferates (fast!) by default: AI techniques are typically simple and easy once discovered.

3. What’s more, AGI-making know-how may be widespread already.

Or, as Yudkowsky puts it2,

Moore’s Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world3 drops by one point. - Yudkowsky

It’s important to emphasise that none of these are laws of nature! But the economic and social forces at work are quite strong.

So (leaving aside debates about the appropriate definition of ‘AGI’) where the frontier of AI development leads, others – many others – potentially rapidly follow. Followers can go even faster by stealing or otherwise harvesting insights from the frontier, but this is not a hard requirement – just an accelerant.

For more on the first point, compute getting cheaper, consider Moore’s law (or the more general and robust Wright’s law). What about the know-how?

Stupid, Simple AGI

The stupidest, simplest possible approach to producing general intelligence might mimic evolution in a large, open-ended, interactive environment. Nobody has succeeded at this yet because they don’t have enough compute, but just a few more decades of compute scaling might get us there. The code to do this would be ultimately quite simple, but the amount of compute time to run it is out of reach today. Almost nobody nowadays thinks that it will take this long, because this is the stupidest, simplest (and least steerable) possible approach and we have much better ideas.

But this means that unless something interrupts the compute trends, then even if ingenious, well-resourced people ‘get AGI first’, eventually anyone could practically blunder into creating their own. Of course, many things could be changed if powerful AI is developed and applied in the meantime… perhaps including the cost and efficiency of compute, the distribution of compute, or indeed the existence and inclination of people to do the blundering.

The Design Space for AGI

What did I mean by ‘AGI-making know-how may be widespread already’?

I don’t literally mean that the recipe for AGI is known and widespread. I don’t even mean that we broadly know exactly how to make AGI and simply want for the capital (compute and data). But for those paying attention, the design space for practically achievable AGI is narrowing.

Take long-horizon coherence or continual learning, for example. Maybe components of these are expandable memory and long-context management of plans and observations. This could perhaps be cracked with something resembling a selection from:

• Context summarisation

• Read-write retrieval-augmented generation

• Recurrent embeddings

• Longer training trajectories

• Plan-management or recursive delegation scaffolding

• Periodic distillation of history into weights or activation patches

• Explicit training for notetaking

• Some even simpler thing, like ‘just scale up the compute’

Among the sharpest, most experienced practitioners at the frontier, that perceived design space may be narrower still4. In the far wider cohort comprising all competent computer scientists and engineers, the design space may not be as saliently in view – but the scientific ‘breadcrumbs’ have been pointing in useful directions for years (at least).

My personal testament5 is that by 2020, several landmarks were visibly coming together in NLP and RL, and by 2021 I had a good sense of a plausible research path to general autonomous AI. Developments like further scaling, mixtures of experts, chain of thought, LLM agents, RL ‘reasoning’, fast attention mechanisms, and hyperparameter tuning optimisations are not merely ‘obvious in hindsight’: their rough contours were advance predictable. It was ‘merely’ a matter of experimenting to find out working details. I’m not being (especially) hubristic here: for some experts closer to the action, these same things looked plausible by 2017 or even earlier! The contours of tomorrow’s advancements are similarly already in view, and far more attention and capital are being poured into the discovery process.

That’s not to say that, given the capital, we could have created AGI there and then in a single try, or even here and now. A design space is not a complete or final design. But iterative refinement by well-resourced and moderately creative problem solvers has been charting a course, and if we are willing to anticipate one frontrunning group getting ‘all the way there’ we must acknowledge that the feat will be reproducible in relatively short order.

Accelerants

‘Reproducible’ is one thing. How soon and how fast? With the current level of sharing of research insights, the answer seems to be roughly ‘as soon as you can outlay comparable capital’, or even sooner!

What phenomena are responsible for accelerating this proliferation? In very roughly descending order of effect size:

• Theft, leak, or deliberate release of pretrained baselines and training algorithms

• Distillation (authorised or not) from exposed APIs

• Exponentially cheaper compute

• (Sometimes cheap or even public access to) ever more sensor and record data6

• Shared algorithmic and experimental details in papers and blogposts

• Conversations and rumours at conferences and other events7

• Movement of experts between development groups and projects

• Use of AI to assist development8

Very tightly securitized projects might partly dampen some of these effects. Competition between firms and countries could amplify them.

What about exponentially cheaper compute? Market dynamics might pivot at some stage to reduce or even reverse the effect of dwindling compute price (for example, extreme buyer concentration driven by strategic accumulation, increasing marginal compute utility9, deliberate regulatory intervention on compute, or something else), but will otherwise continue to drive proliferation. On the other hand, if compute production increases even faster, costs may drop commensurably faster10.

Sensor and records data are being collected even more feverishly now that companies have realised their critical use in training modern AI systems — notice when companies’ privacy policies update to include carve-outs for collecting AI training data. We should expect more of this, as well as more collection of physical and industrial activity records for training robotics, autonomous vehicles, and automated laboratory workcells.

Alternatively, some have imagined an ‘end of history’ moment when sufficiently smart AI arrives and (usually by underspecified mechanism) prevents all of these factors from proceeding. Some envisage not only that, but an AGI or AGI-enabled organisation foreclosing not only the accelerants of proliferation, but also the potential for a rival project to emerge anywhere11. This is conceivable, but one has to ask on what timeframe these changes would happen, and the consequences if it takes longer than imagined.

Short of such an acute and decisive interruption of all of these dynamics, other shocks such as international conflict could have impacts in either direction.

Concluding

Intelligent engineering-minded people exist in all geographies and of all ideologies. Most lag the frontier of AI development only for want of compute capital and intent. Because compute continues to get cheaper, and the potential of AI comes more clearly into focus, both compute and intent become rapidly more widespread. The open sharing of discoveries can further lower barriers and shorten proliferation timelines, but is not essential to this dynamic.

Given this, we have to ask what the consequences of this proliferation could be. Where they are concerning, we must consider in what ways these dynamics could be defused, or, likely failing that, how we will ready ourselves, on a short timeframe, for what follows.

We live in interesting times! There’s a lot we can do.

This blogpost accompanies the paper Cooperation and Control in Delegation Games by Sourbut, Hammond, and Wood, which was presented at IJCAI 2024. In essence, the work attempts to deconfuse some of the discourse around safety, cooperation, and alignment in multi-agent settings by:

• Showing that, just like in the control problem, cooperation problems can be broken down into alignment and capabilities, which are orthogonal to one another;

• Providing measures for alignment and capabilities (both “individual” and “collective”) in multi-principal multi-agent settings (“delegation games”);

• Showing that any of these measures is insufficient alone to guarantee the best outcomes, but that they are together sufficient;

• Bounding the principals’ welfare loss in terms of these measures, and validating this with a series of empirical results.

The goal of this post is to explain what those terms mean, and hopefully why it matters. In doing so, we hope to shed light on some of the related questions posed by other AI safety researchers, for example Dafoe et al in Open Problems in Cooperative AI who discuss the concept of ‘horizontal’ and ‘vertical’ aspects of coordination, or an open problem on the AI Alignment Forum about quantifying player alignment in normal-form games.

There is also a poster which is an even more condensed summary of some key material, and Lewis and Oly have given a few presentations on the topic, one of which is recorded here.

Why Delegation Games?

You may have heard of the Principal-Agent problem. It's a phrase and a setting which turns up in some economics literature, and elsewhere. The idea is that a principal (in this case, the human) is asking, telling, employing, or otherwise exhorting an agent (in this case, the robot) to act on their behalf. The 'problem' is the question of how to ensure that the agent's behaviour results in outcomes which the principal in fact prefers.

Delegation games arise when we have multiple principals, and multiple agents.1 When you read 'principal', think 'human', and when you read 'agent', think 'AI'. (For a slightly different semantic, you can alternatively think of each principal as a basically-coherent coalition of humans, and likewise with AIs.)

Why does this setting matter? It's looking increasingly likely that, perhaps quite soon, many somewhat-autonomous digital personal assistants, digital employees, or similar, will be deployed on behalf of human overseers. That is, we might be entering a highly multipolar world when it comes to somewhat-autonomous AI deployments2. A more obvious, immediate, lower-stakes example of this is autonomous vehicles, which we use as a toy example in the paper. Finally, in the future, multiple large coalitions of humans (e.g. states or companies) may deploy powerful AI systems to act on their behalf in high-stakes scenarios. We want to understand the important features of how to make sure this goes well!

We formalise delegation games in the way you might expect: agents adopt strategies that lead to (a distribution over) outcomes, and both the agents and the principals have (potentially different) preferences over these outcomes.

Cooperation, Alignment, and Calibration

In the paper, we identify some key properties which influence the outcome of a delegation game. We’ll highlight Cooperation, Alignment, and Calibration, because one key punchline of the paper is:

You need all three to guarantee good outcomes

This is important to bear in mind given that much AI safety work focuses on alignment, which is (demonstrably) not enough for safety in multi-polar worlds.

We'll explain what these terms mean, what 'good outcomes' are, and by the end of the post we should have covered enough to understand the high level meaning of the highlighted inequality, which is adapted from Theorem 1 of our paper. This simplification also assumes agents are perfectly individually rational, which we generalise in the paper, and here we mostly skip over collective alignment, which gets a thorough treatment in the paper.

Subscribe now

Cooperation

Intuitively, cooperation is working together for mutual gains over some uncooperative baseline. This is actually already non-terrible as a definition, but we can sharpen it.

Note that cooperation can be partial (one coalition cooperates, potentially with downsides for others). That's collusion. We especially don't want AI to be collusive! This is tricky, because cooperation and collusion rest on basically the same abilities and infrastructure. It's a very important topic, but we don't discuss it here.

Cooperation example

Let's look at a simple example.

Imagine we're palaeolithic hunter-gatherers: we, the authors, are one small, cohesive group, and you, the reader, another. In the morning, if we all set out to gather, by the end of the day we each come back with a basket of fruit (the fruit/fruit outcome, score: 2). If we for some reason decide to instead hunt a mammoth, well... we’re big and tough but we probably can't catch a mammoth. Meanwhile you sensibly gathered some fruits (the mammoth/fruit outcome, score: 1). Likewise if you try to catch a mammoth alone (fruit/mammoth, score: 1). BUT, if we all work together, we've a decent chance at catching the mammoth (mammoth/mammoth outcome, score: 10). These scores are our utilities for the outcomes3.

Now, laid out like this, there's an obvious best-case outcome4: we work together to catch the mammoth! (Nobody has invented conservationism yet5, so this is a preferred outcome.) But there really is a coordination challenge here: if we have reason to believe that you'll go gathering berries, we should too – it would be in our interest (and in this case, yours too) to get more fruit rather than waste our time fruitlessly (!) chasing mammoth.

This isn't just academic. When we look around the world, many problems have the mammoth-nature: we all have to 'show up' or we don't get the benefit (and indeed many cooperation problems are even harder than this due to selfish incentives). Consider international cooperation on climate change, biological weapons control, or coordination on safe technological progress.

Humans solve this sort of problem all the time. We are able to do this due to various abilities, affordances, and so on, which we can collectively refer to as cooperative infrastructure. This includes such things as:

• talking to each other

• trust and reputation

• trade

• commitments and enforcement

• norms and laws

Nevertheless, our cooperative infrastructure is often not up to all tasks.

AI systems have been pretty bad at this on the whole, though there have been some interesting improvements over the years. Future AI might have access to very powerful kinds of cooperative abilities and infrastructure6.

Formalising cooperation

How do we characterise 'better cooperation'? We operationalise the collective goodness of an outcome with a welfare function (an aggregation over utilities). One way to specify cooperation is by considering failure. We look at the welfare-optimal outcome(s) σ⋆ (in this case mammoth/mammoth, score: 10) and then compare any actual (or predicted) outcome σ. The difference in welfare is the welfare regret – how much better it 'could have been'.

The welfare regret of principals (humans) is our primary measure of interest (the 'dependent variable', if you like). When we have only principals playing, that tells most of the story. With agents involved (machines/AI) welfare regret of agents7 quantifies how successfully they cooperated (according to their criteria and coordination mechanisms).

There is also an alternative, more geometric interpretation. 'Mutual gains' are Pareto gains, and Pareto optima coincide (in all but edge-cases) with welfare optima8. Hence, we can interpret cooperation as movement toward the Pareto frontier. (The specific direction of movement corresponds with the welfare aggregation function.)

In the paper we also make some discussion of capabilities and how they give rise to outcomes (and thus to welfare/regret). We tentatively distinguish 'individual' from 'collective' capabilities, and describe mathematically and algorithmically how, given access to estimates or measurements of interactions, these can be determined and distinguished. A related concept is the price of anarchy which quantifies the failure of a particular system to be robust to selfish behaviour.

Now, in the preceding example, we assumed that food was shared and the humans’ dietary preferences were equivalent(ly primitive). That is, we have perfect collective alignment (between principals). Notice that even with this perfect collective alignment, there can remain coordination problems, as in this scenario. In general, we can distinguish problems of collective alignment from problems of collective capabilities (cooperation). We fully characterise this breakdown in the paper, and we'll touch on it later under calibration.

Alignment

When we have more than one actor with some preferences over outcomes, it is natural to ask about the relationship between those preferences. Alignment is the extent to which two or more preference relations are in agreement.

We might prefer exactly the outcomes that you prefer and vice versa (as in our mammoth example where all returns are shared), in which case we are perfectly aligned. Or (as in a myopic chess match9) we might be playing for exactly the outcomes you want to avoid, and vice versa, in which case we are perfectly misaligned. More usually, it'll be something in between these extremes.

Alignment example

In the paper, we discuss several forms of alignment, but here we will focus on alignment between a principal and an agent, namely individual alignment.

Back to our hunter-gatherers, except now we're high tech palaeolithic hunter-gatherers. We have hunter-gather-bots which we delegate to. This is where the game becomes a delegation game.

We produced these bots somehow, perhaps through a process of machine learning and subsequent scaffolding; we ran lots of tests in the lab and the agents seemed to be doing basically what we expected. But we failed our alignment homework.

After the shift to the wild deployment distribution, it turns out our bots in practice prefer more fruit-and-nut and less mammoth-steak (yellow utilities). They're not horribly misaligned (they still prefer to feed rather than starve us), but they're soft-vegetarian bots, and ultimately the consequence of deploying them is muesli for breakfast, lunch, and dinner... forever. An unmitigated disaster10.

Formalising alignment

Now, we can easily define perfect alignment if two sets of preferences are the same, or perfect misalignment if they're exactly opposite. What about intermediates? A key issue with comparing utility functions (or reward functions) is that the same preferences can be described by many different utilities.

For example, if you scale your utilities all by 10x, or if you add a constant 0.1, the preferences this represents are unchanged. This generalises to any scale and shift, namely an affine transformation.

So if we naively compare two utility functions, we might get nonsense or misleading results. We need a way to standardise the representation of preferences as utilities.

In the paper we provide expressions and algorithms to account for these requirements, discussing various desiderata and showing that our measures satisfy them. In particular, utility functions with indistinguishable preferences have identical representation in our standardisation11.

Now we have standardised points, it actually makes sense to compare them. So we can take an appropriate distance measure between points to quantify how aligned they are. This gives us misalignment distance. In the single-principal single-agent case, this alone is enough to provide some interesting regret bounds for the principal.

For more on this kind of approach to comparing utilities and rewards, see e.g. the EPIC and STARC papers. We or some of our colleagues might write up a blog digging more into these concepts at some point.

In the full delegation game setting, distances between utility functions can be used not only to measure principal-agent misalignment, but also the misalignment between groups of agents (or principals). This collective alignment measure essentially captures how much the agents are ‘on the same team’ (or not).

Share

Calibration

Calibration is intuitively a fairness consideration: how much weighting is each player being given in a cooperative outcome?

When we began this project, we were intuitively expecting that a satisfying operationalisation of 'perfect cooperation' and of 'perfect alignment' would together guarantee optimal outcomes. That is, we anticipated that perfectly cooperative and perfectly aligned agents would produce welfare-optimal outcomes for principals. In fact, we could only prove that the outcomes were Pareto efficient for principals12. Calibration is the missing piece.

Calibration example

Let's return to our hunter-gatherers once more. Previously, we imagined a perfect implicit contract to share all gains equally. Hence, a mammoth/mammoth outcome is straightforwardly better. But we might imagine some alternatives:

• we all agree to hunt mammoth together, but you only get one steak and we get the whole rest of the mammoth

• we make the original agreement to share the mammoth equally

• you refuse to hunt mammoth with us unless we give you most of it

• ...

Some of these outcomes may seem more or less 'intuitively fair', but it is hard to find this law written into the universe, and in practice players simply have their preferences and act on them (which may include some preference for fair or altruistic outcomes). Notably, they all improve on the uncooperative baseline.

The point here is that there are generally lots of different ways that cooperation can cash out, and even different 'cooperative outcomes' weight players differently.

Formalising calibration

The weighting over players implied by their modes of cooperation, or equivalently the welfare weightings of players in the welfare function used to score cooperation, determine which Pareto outcomes are deemed welfare optimal.

In our setting with standardised utilities, these welfare weightings correspond exactly to the magnitudes m of the players' utilities. Thus, we can completely characterise the relationship between the agent welfare aggregation and the principal welfare aggregation by considering the ratios:

where m̂ⁱ is the ith principal's magnitude, and mⁱ is the corresponding agent's magnitude. When these ratios are all equal, we have perfect calibration.

Otherwise, these individual welfare ratios rⁱ are combined (as we explain in the paper) with the collective alignment to produce R, the contribution of the miscalibration and collective alignment to the overall welfare regret.

The punchline: you need cooperation, alignment, and calibration to guarantee good outcomes

Now we have all the pieces to understand this claim more clearly.

The term on the left is the principals' welfare regret: how much better the aggregate utility of principals (humans) could have been.

On the right we have a cooperation failure term (the agents' welfare regret), an alignment failure term (a sum over the individual alignment distances), and a calibration failure term (the aggregate R over the welfare ratios and collective alignment distances). In the paper we also demonstrate that these measures are 'orthogonal' in the sense that each can be instantiated arbitrarily, regardless of the others.

By 'good outcomes', we mean 'minimising welfare regret of principals'. From this inequality, it's immediately apparent that, to get 'good outcomes', it is sufficient to minimise all three of the terms on the right. In the paper, we also prove that, to guarantee good outcomes, it is necessary to minimise the cooperation failure, misalignment, and miscalibration – that is, absent extra information, luck, or other magic, if you have failures in one of these areas you can't be certain of good outcomes.

Experiments

Besides theory, we've got experiments, a few of which are visualised here. The blue surface is derived from our regret bounds, and each green dot is the result of one simulated delegation game. There are more variables at play, but here each chart is controlled for particular welfare ratios (miscalibration), with axes for agent welfare regret (miscoordination) and aggregate individual alignment distance (misalignment).

A few other observations from the experiments:

• the highest points here come quite close to the surface – it can be a relatively tight bound13

• the average regret follows the contour of the bound, as you might expect

In some other experiments, we looked into how we can estimate some of these quantities empirically from much more limited data, a harder challenge.

Limitations and Next Steps

There are several limitations of these analytical tools.

Perhaps the most practical weakness is in computing these things. If we have access to the utility/reward functions, we can compute alignment measures in linear time over outcomes... but there can be a lot of outcomes! Further, we generally don't have direct access to a complete utility function14, and some decisioners may not be well-described as having utility functions. Worse, the outcome space might not only be very large but also unknown/unexplored!15 Welfare regret is easy to compute, but only if you know the welfare optimum – otherwise you can only get a lower bound. We demonstrate some preliminary work on estimation of these measures with limited empirical access in the paper.

The definitions we use in our analysis (welfare regret, alignment distance, and welfare ratios) also rely on some 'design choices' from a family of possible functions (e.g. norm choice for alignment distance and welfare weightings for welfare regret). For putting these into practice, there remains a challenge of making a choice. Importantly, these affect the tightness of bounds, and also the normative weight of principals in the overall welfare regret. For any such choice (and we should expect there is some sensible choice), our theoretical conclusions are nevertheless sound.

We also make a few simplifying assumptions about the structure of the delegation game. First, we assume that principals don’t take actions, only their delegate agents do. Second, we assume 1-1 principal-agent relationship, which facilitates some of the individual alignment analysis, but is missing full generality. These limitations should be simple enough to generalise, but a bit messier to talk about. Finally, and relatedly, we assume a fixed population of principals and agents. This has various implications. For one, total and average utilitarianism are identical in this case, while in practice impacts on population can mean that these come apart radically.

Some readers may be uncomfortable with 'agent welfare'. Where do we get these agent utility magnitudes from in the first place? Or equivalently, where do we get welfare weightings from in the case of agents? In fact, due to an equivalence between Pareto optima and welfare optima, you can actually taboo 'agent welfare' from our analysis entirely, and talk only in terms of Pareto gains and Pareto efficiency, while deriving substantially the same conclusions. The point is, a Pareto gain is just another vector (this time in the space of players' joint utilities), and a vector necessarily has a direction! – and thus cooperative gains and Pareto optima give rise implicitly to welfare weightings16.

Another potential philosophical issue is that we give normative precedence to principals' utility and welfare. If you think the agents might matter in and of themselves too, you might want to do an altered analysis. The modification should be straightforward, and the essence of the conclusions is unchanged (just differently-weighted) unless the agents are utility monsters or moral super-patients.

All of these limitations represent interesting avenues for future work. We’re especially interested in scalable ways to evaluate some of the measures using data gathered from interactions between humans and complex AI systems. We hope these measures will be a useful tool when it comes to thinking about the principles behind building more aligned and cooperative AI systems in multi-polar worlds.

exponential expansion (until constraints are reached) ... in practice often manifests as first imperceptible and then rapid escalation.

Connor Leahy, CEO of Conjecture AI, echoes me more pithily

As we learned with COVID, there are only two times to react to an exponential - too early or too late.

Of course, everything has been said before, and we are both in dialogue with the famous quote from Professor Bartlett

The greatest shortcoming of the human race is our inability to understand the exponential function.

I brought a more specific and lighthearted1 take in Invading Australia, where I looked more closely at some case studies of expansionist/replicating (exponential) systems in the field of human biosphere interventions. In summary,

The experiment, this introduction of foreign species was... successful, if by 'successful' you mean 'devastating and difficult or impossible to roll back'.

We learned about some pesky amphibians:

It turns out that cane toads don't jump or climb well, so outside of the lab, where beetles live at the top of sugar cane, the toads were all but useless at their intended purpose. (Out of context failure!)

But the toads were a success, in their own terms: unexpectedly unfussy eaters, prolific reproducers, and poisonous to most wildlife, they have rapidly colonised Queensland state, lately expanding into New South Wales and the Northern Territories, while being resilient to every attempt at pushing them back. We fought fire (cane beetles) with fire (toads), and ended up with two fires!

and we looked at the rather amusing case of humans deploying yet more replicators to bring the toads under control, namely toad viruses and/or genomic interventions like driving fertility-reductions. I'm reminded of the Old Lady Who Swallowed A Fly.

There's good (?) news, of course:

the prickly-pear or paw-paw is another species inadvertently unleashed on the Australian ecosystem, which has caused some displacement of native wildlife. A moth (believe it or not, Cactoblastis!) was found which actually does seem to work as a self-regulating suppressant of the cacti.

When it comes to AI, we'd have to be very very prepared in order to expect shenanigans like this with replicating or propagating systems to end well.

Also, no-one picked up on my Charles Darwin/extinction pun: 'endless formerlies most beautiful'??

Upcoming, alignment and cooperation

I'm working on a paper2 with some collaborators in Oxford, analysing cooperation and alignment concepts for AI. What does this mean? When there are lots of AI systems, we want them to generate value by interacting positively rather than destroying value through conflict or anarchy. That's cooperation. And we want the values they are oriented at to be valuable to us, rather than arbitrary (or worse, harmful) things. That's alignment. Expect a takeaway or two to appear in the near future elucidating and elaborating on that.

We don't get the opportunity to pause time after every question syllable, pull up a notepad, run some supercomputer evaluations, compute exact Bayesian posteriors, estimate our teammates' and opponents' credences and likely buzzing behaviour, and so on. Cruelly, time flows at one second per second and the quizmaster keeps quizmastering. So too in life!

So too, indeed.

Twitter/X and YouTube had fun, as some friends were eager to point out to me. There were some more flattering remarks, but I think my favourite was

Sourbut combs his hair with a toffee apple #universitychallenge

As I anticipated, people also enjoyed it when I said 'groyne'. I know it sounds like 'groin', and yes, my name is 'Sourbut' and it's pronounced how you think.

It's 'groyne', actually

The editors actually cut the bit where I said, 'Never thought I'd say, "groin" on TV'. As I cryptically hinted in the Hertford, Sourbut post,

(As we may find out, there are also secretly other options, like '(-1000) say something embarrassing on national TV'.)

Ironically for a piece on speaking clearly and with nuance, I failed to explicitly point out crucial facts! - the actual true accounts regarding one of the exemplars I brought of language-misuse (these were obvious in my mind while writing, but some readers appeared confused in ways which make sense if they didn't know). I criticised

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.1

but didn't explicitly point out that, beyond being an oversimplification, there just isn't a ready way to map this to the reality, which is that

• the smuggling in question was done by... smugglers

• the buying of chips was done by multiple China-based entities

• the (implicit but unmentioned) selling (and importantly, provisioning/enabling) of chips was done by NVIDIA, a US-based company (and perhaps others)

• the investing was done by the CCP

I had a great response from CAIS in particular. The original author agreed this was ambiguous and unfortunate, and they've updated the text in question substantively. They also responded

More generally, we try to avoid zero-sum competitive mindsets on AI development. They can encourage racing towards more powerful AI systems, justify cutting corners on safety, and hinder efforts for international cooperation on AI governance. It’s important to discuss national AI policies which are often explicitly motivated by goals of competition without legitimizing or justifying zero-sum competitive mindsets which can undermine efforts to cooperate. While we will comment on the how the US and China are competing in AI, we avoid recommending "race with China."

This was really welcome and I hope other readers took on board the lesson here.

A few other readers pushed back a little. Stephen Clare expressed general agreement and offered a rearticulation of the problem I'm pointing to, while also criticising my relegation of governments to 'not currently meaningful players in AI development and deployment' as being too strong. Quite right: I meant that governments have (to date) been entirely passengers regarding the direction and nature of advanced AI development, but it is true that they have begun to get involved in coarse economy-level lever-pulls like investing and regulating hardware.

I went on a minor rant in the comments:

Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? Do they think the US government/military can/will appropriate those staff/artefacts/resources at some point? Are they referring to integration of contemporary ML/DS into the economy? The military? Or impacts on other indicators2? What do people mean by "China" here: CCP, Alibaba, Tencent, ...? If people mean these things, they should say those things, or otherwise say what they do mean. Otherwise I think people motte-and-bailey themselves (and others) into some really strange understandings.

Amazingly, one reader admitted that,

Yes.

In the end, all the answers to your questions are yes.

and made some further assertions about inevitability of international conflict. We had a minor back-and-forth but this was pretty remarkable, to me, and I think there was some talking-past happening. Thank you for sharing honestly.

Sadly, Scott Alexander, an author I hugely admire, has evidently not read my admonishment to CAIS, as his latest letter is full of thoughtless remarks about China and the US/West. Scott, you should know better. Words have power. Saying this, I think it is a good and useful post in many ways, in particular laying out a partial taxonomy of differing pause proposals and gesturing at their grounding and assumptions. He writes,

The biggest disadvantage of pausing for a long time is that it gives bad actors (eg China) a chance to catch up.

There are literal misanthropic 'effective accelerationists' in San Francisco, some of whose stated purpose is to train/develop AI which can surpass and replace humanity. There's Facebook/Meta, whose leaders and executives have been publicly pooh-poohing discussion of AI-related risks as pseudoscience for years, and whose actual motto is 'move fast and break things'. There's OpenAI, which with great trumpeting announces its 'Superalignment' strategy without apparently pausing to think, 'But what if we can't align AGI in 5 years?'. We don't need to invoke bogeyman 'China' to make this sort of point. Note also that the CCP (along with EU and UK gov) has so far been more active in AI restraint and regulation than, say, the US government, or orgs like Facebook/Meta.

Suppose the West is right on the verge of creating dangerous AI, and China is two years away. It seems like the right length of pause is 1.9999 years, so that we get the benefit of maximum extra alignment research and social prep time, but the West still beats China.

Now, this was in the context of paraphrases of others' positions on a pause in AI development, so it's at least slightly mention-flavoured (as opposed to use). But as far as I can tell, the precise framing here has been introduced in Scott's retelling.

Regardless of the origin of this formulation, this is bonkers in at least two ways. First, who is 'the West' and who is 'China'? This hypothetical frames us as hivemind creatures in a two-player strategy game with a single lever. Reality is a lot more porous than that, in ways which matter (strategically and in terms of outcomes). I shouldn't have to point this out, so this is a little bewildering to read. Let me reiterate: governments are not currently pursuing advanced AI development, only companies. The companies are somewhat international, mainly headquartered in the US and UK but also to some extent China and EU, and the governments have thus far been unwitting passengers with respect to the outcomes. Of course, these things can change.

Second, actually think about the hypothetical where 'we'3 are 'on the verge of creating dangerous AI'. For sufficient 'dangerous', the only winning option for humanity is to take the steps we can to prevent, or at least delay, that thing coming into being. This includes advocacy, diplomacy, 'aggressive diplomacy' and so on. I put forward that the right length of pause then is 'at least as long as it takes to make the thing not dangerous'. You don't win by capturing the dubious accolade of nominally belonging to the bloc which directly destroys everything! To be clear, I think Scott and I agree that 'dangerous AI' here is shorthand for, 'AI that could defeat/destroy/disempower all humans in something comparable to an extinction event'. We already have weak AI which is dangerous to lesser levels. Of course, if 'dangerous' is more qualified, then we can talk about the tradeoffs of risking destroying everything vs 'us' winning a supposed race with 'them'.

I'm increasingly running with the hypothesis that many anglophones are mind-killed on the inevitability of contemporary great power conflict in a way which I think wasn't the case even, say, 5 years ago. Maybe this is how thinking people felt in the run up to WWI, I don't know.

I wonder if a crux here is some kind of general factor of trustingness toward companies vs toward governments - I think extremising this factor would change the way I talk and think about such matters. I notice that a lot of American libertarians seem to have a warm glow around 'company/enterprise' that they don't have around 'government/regulation'.

[ In my post about this I outline some other possible cruxes and I'd love to hear takes on these ]

Separately, I've got increasingly close to the frontier of AI research and AI safety research, and the challenge of ensuring these systems are safe remains very daunting. I think some policy/people-minded discussions are missing this rather crucial observation. If you expect it to be easy (and expect others to expect that) to control AGI, I can see more why people would frame things around power struggles and racing. For this reason, I consider it worthwhile repeating: we don't know how to ensure these systems will be safe, and there are some good reasons to expect that they won't be by default.

I repeat that Scott’s post as a whole is doing a service and I'm excited to see more contributions to the conversation around pause and differential development and so on.

Relatedly, I had a great conversation at lunch yesterday with Will MacAskill, who’s currently working on questions of coordination around development of advanced AI. Very excited to read more when that comes out!

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.1

Sounds about right?

This post centres around an email I sent to the Center for AI Safety (CAIS) expressing concern about their 2023-08-15 newsletter's coverage of US-China competition in the AI space2, but the overall point is broader. There are some ways of discussing the topic of international relations regarding AI which strike me as un-nuanced in a counterproductive and dangerous way, by hiding certain truths or emphasising others, and supporting a conflict-oriented mindset.

In writing about this, I'm also gesturing at something about the more general topic of 'how to think and write about politically-charged topics'.

Jump to the summary if you are in a hurry.

This conversation really is important, which is why I think it's worth a public message discussing particular statements, but this should be understood as constructive criticism and part of a broader conversation which society, and especially the community of those focused on AI safety, needs to have. The CAIS newsletters are worth a (not unquestioning!) read, including the edition in question.

The particulars in this message serve as good exemplars of the problems and questions I have, and I'd be interested in responses from CAIS but even more so in remarks more broadly on the topic from anyone interested. The public conversation about this appears from my perspective to sometimes be broken. If that is so, I would like it to be rectified, and if not, I would like to be put right myself, the better to prioritise in my own work!

Specifics, CAIS case study

Here, quoted3, is my response to the CAIS letter, which serves as an initial dialogue opener and the core of this post:

Hello,

I've been a supportive reader for some time and am myself an AI safety researcher. I'm generally very impressed and encouraged by your newsletters! - but I was disappointed and concerned by the phrasing (and mindset it can encourage) regarding US-China competition in the letter dated 16th August ('US-China Competition on AI Chips, ...').

In general I'm very wary of messaging which could inflame a them-vs-us mindset, in short mainly because I think it a) destroys humans' ability to think sensibly and b) tends to foreclose win-win outcomes. I expect these brief points to be clear and to have rich referents in your mental pictures of the world, but please correct me if not!

Interjection: I'm referring to the vicinity of mind-killing politics

I think your letter skirted close to dangerously simplified presentation in this way. I would not normally spend my time or yours on a criticism of one section of a newsletter, but in this case I consider it worthwhile because your letter is close to the Pareto frontier on nuance, correctness, helpfulness, and reach (and it pays to try to nudge such things in good directions), but this kind of message needs to be delivered with care to avoid misunderstanding and harm.

Hopefully my pointing this out, accompanied by a few select quotes, is enough to encourage you to carefully take this criticism into account, but I'd happily expand more if you like!

Without further ado, a few quotes and my response:

The US and China have been competing for access to these chips for years.

Kind of true, but really US-based and China-based international corporations (as well as other orgs) have sought access to this scarce resource. Competition for this particular resource is mostly zero-sum across all of these entities, importantly including intra-US and intra-China.

Where market-share is near zero-sum (i.e. for direct competitors) the market-share outcomes may be greater-than-linearly zero-sum in this resource (your minor/temporary lack of chips could be my major gain of market-share), which might better warrant the term 'competing', but this effect is actually much stronger intra-country/bloc rather than inter-, due to respective markets! i.e. Google and Microsoft really care about each other's chip access in a way that they only do to a weaker degree about Alibaba's.

Interjection: To emphasise, 'the US and China have been competing' doesn't literally preclude belief in intra-bloc competition. But there's a strong implicature that intra-bloc is (relatively) unimportant, while in fact the mechanism I mentioned here increases intra-bloc competition (which I think is borne out by observation to date).

Subscribe now

When governments have paid attention, they have indeed made moves which adjust share (but also supply, as you've noted later, making it nonzero-sum, in chips at least). It's unclear (to me) exactly what incentives have motivated each move, but certainly they're not the actions of monolithic or coherent entities 'The US' and 'China'. And it's certainly not the case that where such activity changes chip share, it's collected by the acting entity. Non-governments have also made moves adjusting chip share, for example the case you cite of Nvidia (a 'US' company) deliberately rules-lawyering the US gov in order to supply more chips to various China-based companies!

Typically when nations are used as agentic subject nouns it refers to the government and/or military of said country. I don't think there's a reading of these statements in those terms which is true, and I'm not aware of any other plausible reading which is true.

---

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.

I dislike this sentence and think it is false! Who is this 'China'? Did said unified entity carry out all of these activities? Was it coherently pursuing all of these 'several efforts' to some particular end?

Interjection: I feel that I was unkind in my tone here. The kind of claim exemplified in the letter and in other places has proved impossible for me to map to something sincerely resembling reality without caveating so much as to be essentially starting from scratch i.e. it looks like a kind of non-proposition or emotive filler. I would be very interested to hear from people who have a better parse on this to help me understand! My crux discussion below the rest of the email contains my current best attempts.

Meanwhile, the United States has struggled to build American chip manufacturing capacity, and has taken further steps to prevent Americans from investing in Chinese technology.

This one is poor for the same reasons, though not quite as bad (perhaps because the authors are American/anglophone and have a closer perspective on the nuance).

The discussion after this point in the letter is relatively good and nuanced! It names (some of) the individual orgs and companies, and makes clearer the multiplicity of others. All further references to nations as agentic subject nouns appear to be consistent with a conventional reading referring to the respective governments.

I'm interested to know how the rather good detail got paired with a rather harmful introduction and I urge you to consider the processes and thinking which gave rise to this section of the otherwise good letter.

Thanks,

Oly

This was a brief email intended to convey something I expected to be quickly-graspable with a few pointers. CAIS content suggests that they have a broader familiarity with associated facts, but it seems to be digested/compressed here in a way which is needlessly and harmfully lossy.

In particular, the statements abstract very neatly over pre-drawn boundaries (national i.e. 'US' and 'China') and furthermore assign a greater sense of coherence and agency to those abstractions than is warranted. At least some possible such statements must be true-ish (or we would not have those abstractions), but this convenient compression happens in too many conversations to be a coincidence! Said pre-existing abstraction boundaries are already salient in the information ecosystem, and laden with emotional and political baggage. This same phenomenon (it sometimes seems like a pre-written bottom line but in implicature?) appears in other publications by other orgs and in verbal conversations I've witnessed or been part of.

The fact that I, a relative governance rookie (I'm focused mainly on technical matters), struggle to rectify or understand this makes me wonder: am I missing something? Is there a relevant factor I'm unaware of? More concerningly, is there some terrible equilibrium which prevents more involved people from speaking more clearly here? I think more likely the abstractions ('US' and 'China') have a background potency which distorts perceptions and shapes how people communicate.

Possible cruxes and areas of high uncertainty

Implicit in my own discussion is the background assumption that the main concern is about possible direct or near-direct impacts4 of AI deployments by government entities (or military). This seems to me the obvious reading when people use countries as agentic subject nouns. ('China [the governmental entity] has made several efforts'5.) In this framing, I can't rectify some of the things people say with reality. But some alternative concerns might fit the bill, and if so, this is evidence that people are talking past each other, and we should aim to frame concerns more clearly!

I've never focused intently on this area, but have had a handful of conversations about this over the years, and among relevant cruxes seems to be a family of questions along the lines of

How quickly/totally/coherently could US gov/CCP capture AI talent/artefacts/compute within its jurisdiction and redirect them toward excludable destructive ends? Under what circumstances would they want/be able to do that?

People's intuitions here appear to differ a lot, and data might be hard to come by!

It seems plain that nations are not currently meaningful players in AI development and deployment, absent conspiracy-level secrecy. So to support the apparent take that they are, we may need to imagine that they could ably/rapidly become meaningful players in AI development and deployment, hence the above cruxes.

Depending on the answers to these questions, one might perceive various goings-on which happen to occur under one or other jurisdiction to have greater import on the international stage and perhaps to warrant treating national or multi-national blocs as more coherent entities than they really are at present, for the purposes of AI discussion. ('China [the government] has [allowed/encouraged made] several efforts [because eventually they will probably seize the gains/means]'6.)

Other possible cruxes, more guesswork:

1. Perhaps the concern is about indirect (e.g. economic) impacts of non-government entities' AI activities leading to some (risky) change in balance of power (between existing governments/blocs)

   • Then, abstracting references to lots of individuals and groups via their home country might be a move which is writer-intuitive, even if nonstandard and reader-confusing. ('China [the impersonal collective economic entity] has made several efforts'7.)

   • The additional step in this type of theory, namely that indirect effects cause a risky change in balance of power, should really be spelled out if it is loadbearing

   • The use of countries as agentic subject nouns is difficult to justify under this reading

2. Perhaps the concern is indeed about direct impacts, but wielded by non-government entities (who remain the major players in development and deployment of AI)

   • If so, the conversation should be about general/global resource/capabilities rather than inter-bloc 'competition'...

   • ...unless we also posit that inter-bloc non-government conflict is liable to be much worse than intra-bloc8

   • Similarly, if these are loadbearing assumptions, they really ought to be spelled out clearly

   • The use of countries as agentic subject nouns could be justified at a stretch here, but only by first spelling out the reasoning

3. Perhaps the concern is that, regardless of the actual impact of AI resources, apparent competition could lead to inflammation of traditional conflict, or weaken defenses against such inflammation

   • Then, reporting on the apparent competition via a mention and with explicit caveat, would make sense! ('China [gov/military] has [been perceived as having] made several efforts... [but the reality is more nuanced]'9.)

   • Alternatively, reporting on actual conflict could be used as evidence for the claim (that apparent competition inflames conflict), but only by also pointing to the stated or implied reasons for the conflict. ('China has made several efforts... [in each case citing US provocation as justification]'10.)

   • In either case, there are additional claims being made that can not be left implicit, and require supporting argument

Speculation

As it is, for me, the evidence seems to suggest that an AI race, if it is happening at all, is being run by (mainly US- and UK-based) companies with little or no oversight from governments or militaries. Rather, governments are in a position to collectively act to diffuse the race! And they appear as likely to do this as to exacerbate it, from my limited viewpoint.

Separately, a lack of reliable alignment techniques and performance guarantees makes AI-powered belligerent national interest plays look more like bioweapons than like nukes - i.e. minimally-excludable - and perhaps mutually-knowably so! This presently damps the incentive to go after them. But proliferation of naively-aligned AI ('figure out what I want and make it happen') might make harm plays more excludable, exacerbating lose-lose or race game dynamics ('go and steal/destroy their stuff but don't let that happen to my stuff'). This concern in part motivates consideration of multi-principal-multi-agent delegation and the cooperative AI agenda.

Summary and takeaways

Un-nuanced coverage and discussion has the potential to inflame harmful confusion and us-vs-them mentality, which diminish the chance of safe outcomes.

• China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.

• China [the governmental entity] has made several efforts...

• China [the government] has [allowed/encouraged made] several efforts [because eventually they will probably seize the gains/means]...

• China [the impersonal collective economic entity] has made several efforts...

• [People and organisations in] China [have has] made several efforts...

• China [gov/military] has [been perceived as having] made several efforts... [but the reality is more nuanced]

• China has made several efforts... [in each case citing US provocation as justification]

• ...?

When compressing discussion of political topics, be extra wary of compression which coincidentally abstracts over already-charged us-them divides (and be careful when phrasing comes too easily, lest you write bottom lines first)! You're more likely to be wrong (because your information ecosystem biases toward thinking in these terms, and because you might be mildly-to-severely mind-killed on the matter), and being wrong is more likely to be harmful (by reinforcing those dynamics in others).11 The same vigilance applies to reading and listening.

My (not very informed) take is that governments are at this point as likely to want to defuse as to exacerbate an AI race, and those of us with any privileged insight or influence should avoid one-sided discussion of the matter (if anything preferring to focus on constructive, collaborative possibilities, the better to raise them to salience and generate common knowledge).

Most of my remarks here are somewhat weakly held (if forcefully stated) and it seems important to gather perspectives on this. Inform me! All responses will be gratefully received.

Cross-posted to EA Forum

Share