Best humans still outperform artificial intelligence in a creative divergent thinking task

Remarkable! Man bites dog! It had become newsworthy, it was worth checking (and, I perceive, worth a little self-congratulatory celebration) that there remained any domain where mere man could still hope to possibly contend with the machines — at least, the best humans still could! (Could you?)

A message from the future

That was 2023. I think what stood out to me at the time1 was that this was in some sense early. Not early in the story of AI — although ChatGPT and StableDiffusion, each less than a year old, had captured the public attention in a way which earlier AI hadn’t, these were merely the latest in a long lineage of gradual developments — but an early sign of a reckoning, an attitude shift in how humanity would grapple with these new machine capabilities we were conjuring fitfully into being.

I’d already been worrying for years that things might get out of hand with AI (and had even started writing about it). I was hardly the first!2 But this had felt almost like a perversely secret concern (how can people not see what’s coming?? — but they didn’t), one which humanity at large appeared destined to ignore until either it was too late… or, if we somehow played it right, until a splendid apotheosis of world peace, unlimited bounty, health and longevity delivered by machine intellect. (In fact I think those remain real prospects, and it’s absolutely in our hands to determine which outcomes we get.)

What this headline implicitly spoke of, the subtextual worldview shift belied by the phrasing — “Best humans still outperform” — was that we had woken up and viscerally felt the reality that even the ‘best’ humans might genuinely need to watch their backs. The machines were coming. It was no longer (had never been) a joke or a fairy tale.

This headline, seemingly from a near future in which it was taken for granted that machines, in general, dominated human capabilities, showed what was coming. Headlines like it are now commonplace — perhaps more common than those (now almost boring!) headlines adding to the litany of tasks AI now outcompetes human experts at.

The world changed

The world changed. Not because the world had actually yet changed (much), but because humanity, in our limited and faltering foresight, had noticed that, soon, it might. That murky perception of the future, humanity’s near-unique hallmark and blessing, memetically reverberated and has worked its way into our collective discourse.

In this way, I’m incredibly grateful to the ‘ChatGPT moment’. Rather than implicitly relying on a plucky band of vaguely foresighted but ultimately underpowered ‘sci-fi weirdos’, humanity as a whole is entering the conversation. We’re all stakeholders in the trajectory of this world-transforming sphere of technology, and all kinds of people are beginning to act like it: people with skillsets and perspectives which we’ll need, which had been lacking, in earlier debates. Law theorists, philosophers, engineers, anthropologists, economists, statespeople. It’s a thickly textured problem. It’ll need more than people like me (aspiring polymath though I may be) to solve it!

These cultural conversation shifts are fickle but surely incredibly consequential. 2025 felt like another shift, to me, and 2026 so far — with AI producing genuine national security implications and at the centre of dirty political manoeuvring — seems to suggest that both the training wheels and the gloves are off, as Dean Ball recently put it. It’s a little scary: powerful and not altogether friendly forces have turned their eye to the potential potency of emerging tech, and they3 may wrestle for it, even under the risk that they destroy much in the process or that the tech spills entirely out of their control.

The world, changed

We can be doing better! People can get curious, find out what’s what, consider stakes and what realistic paths we might prefer. Don’t make the mistake of ‘nowsight’ bias — today’s AI are the least capable there will ever be! Take seriously where things might go, and notice if the conversation seems to miss something important that you understand well: it’s still early and the ‘experts’ are mainly that by virtue of noticing the importance of AI a little sooner than everyone else4. Let’s also grab the new tech building blocks we have and bootstrap the way we do foresight, collective intelligence, and coordination.

Don’t mistake me for naively assuming machines will blast through every bottleneck in short order. There’s a lot of adaptability, dexterity, and generality bottlenecks between here and self-sufficient machines. Perhaps I’ll write something about that soon.

What's the growth rate? What's the doubling time? How many orders of magnitude bigger is it? How many years at this rate until it's quintupled?

All questions of ratios and scale.

Scale... hmm.

'Wait', you're thinking, 'let me check the date...'. Indeed. But please, stay with me for the logarithms.

Musical intervals as ratios, and God's joke

If you're a music nerd like me, you'll know that an octave (abbreviated 8ve), the fundamental musical interval, represents a doubling of vibration frequency. So if A440 is at 440Hz, then 220Hz and 880Hz are also 'A'. Our ears tend to hear this as 'the same note, only higher'.

That means the 'same' interval, an octave, corresponds to successively greater gaps in frequency. First a doubling, then a quadrupling, an octupling, and so on. Our perception, and musical notation, maps the space of frequencies logarithmically.

You'll also know that a 'perfect fifth' is a ratio of . A to the E above it, C# to the G# above it, etc. Consonance is all about nice ratios! (Ask Pythagoras.)

At least, the really sweet, in tune fifths are this ratio. Because God is an absolute wheeze, you can keep moving in fifths (3:2) and octaves and get 'new notes' eleven times. That's where we get our Western scale from, originally (except it's originally originally Mesopotamian probably). The twelfth time ((3:2)^12) gets you to a ratio of roughly 129.7:1. That's almost exactly seven doublings, seven octaves (7 * 8ve)! That'd be 128:1. God's joke is in that roughly 1% margin, and musicians have been arguing about what to do about it for centuries. It's a whole thing.1

Cutting a long story short, that leaves us with twelve different notes dividing up the octave. They 'repeat', with 'the same' note again and again at either higher or lower octaves (a full doubling of frequency).

In between octaves, those twelve divisions need to 'add up to' a doubling. For reasons, two steps (a sixth of the overall scale) is referred to as a 'tone', and a single step (a twelfth of the scale) is thus a 'semitone'. That means each semitone corresponds to a ratio of the twelfth root of two. (It's about 1.06, i.e. a ratio increase of about 6%.) The full scale as shown above is called 'chromatic' (because it has every 'colour'...).

This means that neat fractional powers of two map cleanly onto musical intervals. God was generous in giving twelve many factors, so we have musical intervals for the square, cube, fourth, sixth, and twelfth roots of two which come for free.

So far, no logarithms. But we have musical powers of two: give me a fraction and I can tell you the musical interval. That means we also have musical logarithm: give me a musical interval and I can tell you the power of two! e.g. C to G# is eight semitones. So

Musical logarithms? What is he talking about? Surely this is pointless. Yes, it is! Hold on!

Harmonic series

If you're a brass music nerd like me, you'll know that the 'overtones' of most natural vibrations correspond to the 'harmonic series' (no, not that harmonic series, the actually harmonic harmonic series), which are the different pitches you can get a big metal tube to vibrate at if you give it the right encouragement. Incidentally this is how brass players get dozens of different notes out of an instrument having (usually) only three valves2.

This harmonic series is generated by lovely integer ratios! Why? The physics of oscillators. Integer multiples are the only frequencies which can support a standing wave on the same vibrating object (air column, string, membrane).

Brass players spend hours and hours sliding and jumping between these harmonics as a matter of sheer necessity. Only three valves!3 So we know them by heart, by fingers, and by ears.

Combining the harmonic series with the chromatic scale: magic

So we have integer multiples, the harmonic series, laid over a fundamentally logarithmic scale, the chromatic scale consisting of twelve semitones.

Numbers above the notes correspond to small adjustments vs the equally-spaced semitones which are usually used today to deal with God's joke. Ignore them if you don't care about small percentage errors. This is the harmonic series on C; you can have a series on any starting note with the same intervals.

Here's the magic trick. Now we can go from arbitrary ratios to musical intervals!

Start with an easy one, 1.25. That's a ratio 5:4. Fifth harmonic is E (+ 2 8ve). Fourth is C (+2 8ve). The octaves cancel. That's an interval E:C, or four semitones. So 1.25 is four semitones. We already know the 'musical logarithm' of four semitones, it's 4/12 = 1/3. Check on a calculator: log_2(1.25) = 0.32193…. I promised close, not perfect!

A slightly trickier one, 1.8. That's a ratio of 9:5. The ninth harmonic is D (+3 8ve), and the fifth is E (+2 8ve). The octaves partly cancel (leaving a single octave). The interval D:E is minus two semitones. Taken off the residual octave, that leaves ten semitones. So log_2(1.8) = 10/12 = 5/6. Calculator check: log_2(1.8) = 0.848…. Not bad!

It turns out that the musical harmonic series is secretly a mini table of base 2 logarithms.

Base 10, if we really have to

The unit that mainstream sheeple often use for fractional logarithms is the decibel. A decibel divides a base ten order of magnitude in ten. So ten decibels is a dectupling, twenty is a hundredfold, and so on.

Stated similarly, a semitone divices a base two order of magnitude in twelve.

In another cosmic whimsy, 2^10 = 1024 ~= 1000 = 10^3. So 120 semitones are essentially equal to 30 decibels, for an easy exchange rate of four semitones per decibel.

What

Well, look. It's fun, and it gets me logarithms to pretty good approximation. It's good enough for jazz Fermi estimation, as they say. Who is this even good for? I maintain that the intersection between music and mathematics nerds is surprisingly well populated. If that's you, you're welcome. If not, I'm pretty unsure how easy it is to get the harmonic series installed in your brain. Maybe it's only available to the warped few who train in childhood.

There are some other fun tricks with powers and logarithms of two. For example, if you know your binary place values, you can figure out logarithms of very big numbers (and the trick comes in handy here too).

There's also a 'rule of 72' which helps when dealing with small percentage growth rates and doubling times.

I aesthetically like this neat division of doublings into twelve parts, and it's fun to invoke musical intuitions that really have no right to help with mathematics.

You might complain that twelfths are faffy. Who uses twelfths anyway? Everyone everywhere has used decimal for goodness' sake! Well, I have something else to share with you...

Lots of folks naturally recognise that this implies a general strategy: make humans individually — and humanity collectively — better able to solve problems. Very good! (Complementary strategies look like: make progress directly, raise awareness of the challenges, recruit problem solvers, …)

One popular approach is to ‘raise the sanity waterline’ in the most oldschool and traditional way: have a community of best practice, exemplify and proselytise, make people wiser one by one and society wiser by virtue of that. There’ve been some recent successes, not least the existence of communities and forums like LessWrong and Effective Altruism, and some older philosophies and movements.

Another popular approach is to imagine augmenting ourselves in the most futuristic and radical ways: genetic engineering, selective breeding, brain-augmenting implants, brain emulation. Go for it, I suppose (mindful of the potential backfires and hazards). But these probably won’t pan out on what look like the necessary timelines.

There is a middle ground! Use tech to uplift ourselves, yes — but don’t wait for medical marvels and wholesale self-reauthorship. Just use the building blocks we have, anticipate the pieces we might have soon, and address our individual and collective shortcomings one low-hanging fruit at a time.1

The most exciting part is that we’ve got some nifty new building blocks to play with: big data, big compute, ML, and (most novel of all) foundation models and limited agentic AI.

How to generate useful ideas in human reasoning

One place people fall down here is getting locked into asking: ‘OK, what can I usefully ask this AI to do?’. Sometimes this is helpful. But usually it’s missing the majority of the design space: agentic form factors are only a very narrow slice of what we can do with technology, and for many purposes they’re not even especially desirable.

Think about human reasoning. ‘Human’ as in individuals, groups, teams, society, humanity at large. ‘Reasoning’ as in the full decision-making cycle, from sensing and understanding through to planning and acting, including acting together.

I like to first ask: ‘What human reasoning activities are in bad shape?’

• OODA is one good frame:

  • For a given important (type of) decision, what are people observing?

  • How are they orienting and deciding?

  • What actions do they have available and do they know how to do them well?

  • What about the case of teams, groups, institutions: how do their OODAs work (and how do they fail)?

• Also think about development: how do individuals learn and grow? What about groups and communities, how do they form, grow, connect?

• In foresight,

  • What features are we even paying attention to in the first place?

  • What prospects are under consideration?

  • What affordances are we aware of?

  • How are we strategically creating sensing opportunities and the means to adapt plans?

  • How do our forecasts achieve precision and calibration?

• In epistemics, think about the message-passing nature of most human knowledge processes.

  • How do we assess the nodes (communicators)?

  • How do we assess, digest, and compile the messages (claims, evidence, proposals, …)?

  • How do we understand and manage the structure of the network itself (communication relationships, broadcasts and other topologies, …)?

  • What about the traffic (message rates, density distribution, repeated and relayed transmissions, …)?

  • What messages ought to be routed where, when and on behalf of whom?

• In coordination, what are the conditions for success?

  • We need to find or recognise potential counterparties.

  • We might need the charters, norms, or institutions to condition and frame interaction productively — ones which don’t fail or fall to corruption or capture.

  • We need to surface enough mutually-compatible intent or outcome preference.

  • Our ensembled group wisdom might be a necessary source of insight or agility.

  • We need to survive the tug of war of negotiation (which can dissolve into antagonism, even when there’s common knowledge of win-win possibilities).

  • Means of verification and enforcement may be needed to access good options.

Think of a particular audience with either the scale or the special influence to make a difference (this can include ‘the general public’), and the deficits they have in these reasoning activities. Now ask: ‘What kinds of software2 might help and encourage people to do those better?’.

• Is there an edge to be gained by unlocking big (or even medium) data (which can often be more living and queryable than ever before thanks to LMs)?

• Can large amounts of clerical labour (again LMs) per capita make something newly feasible?

• Can big compute and simulation (including multi persona simulation: LMs again!) drive better understanding of an important dynamic?

• Can extensive background exploration, search, or ‘brainstorming’ by AI surface important opportunities or considerations?

• Can always-on, flexibly-semantically-sensitive sensing and monitoring bring attention where it’s needed faster than before (or at all)?

• Could facilitation and translation bring forth, and synergise, the best array of human capabilities in a given context?

• Could software’s repeatability, auditability, and privacy (in principle), combined with the context and semantic sensitivity of AI, unlock new frontiers of trustable human scaffolding?

• …

Finding flaws and avoiding backfire

Think seriously about backfire: we don’t want to differentially enable bad human actors or rogue AI to reason and coordinate! As Richard Rumelt, author of Good Strategy/Bad Strategy observes,

The idea that coordination, by itself, can be a source of advantage is a very deep principle.

Coordination’s dark side is collusion, including cartels, oligarchy, and concentration of power, in imaginable extreme cases cutting out most or even all humans.

Similarly, epistemic advantage (in foresight and strategy, say) can be parlayed into resource or influence advantage. If those can be converted in turn into greater epistemic advantage (by employing compute and position for epistemic attacks or in further private epistemic advancement) without commensurate counterweights or defences, this could be quite problematic.

Part of it is about choosing distribution strategies which reduce misuse surface area (or provide antidotes), and part of it is about preferring tech which asymmetrically supports (and perhaps encourages) ‘good’ use and behaviour.

Do it

FLF’s fellows, and I and others have been doing some of this exploration recently. Stay tuned for more. Meanwhile, join in! We’re early in a critical period where much is up for grabs and what we build now might help shape and inform the choices humanity makes about its future (or whether it makes much choice at all). Try things, see what kinds of tools earn the attention and adoption that matters, and share what you learn. Consider principles to apply, especially for minimising backfire risks, and share particular considerations for or against certain kinds of tech and audience targets.

Thanks to Owen Cotton-Barratt and Ben Goldhaber for helpful comments, and to Lizka Vaintrob for recent relevant conversations

AI is not just one, big, singular thing. What are the ways we can bring forward the beneficial possibilities, while delaying or defending against the harmful ones?

As I repeatedly emphasise to anyone who’ll listen: it’s never been just a dichotomy between ‘yes, good, more AI please’ and ‘no, bad, less AI thank you’ — and it’s not even a case of just making sure ‘the AIs’ are good (though this helps). How the tools and technological building blocks at our disposal are integrated into applications, workflows, and use-cases, is just as important.

And the tools, products, and systems we could develop now shape the context for subsequent developments, including by:

• equipping people to better predict and understand their options

• enabling people to coordinate better around preferred possibilities (which might otherwise be difficult due to mismatched incentives or race dynamics)

• giving the tools to defuse or defend against hazardous developments, or their precursors

For technologists, futurists, philanthropists, legislators, experts, and other members of society trying to make tech progress go well, paying attention to which effects happen, in what order — and what our options are for choosing wisely there — looks like a really promising, and neglected way of reducing large-scale risk and bringing about huge benefits.

You can read our few pages of fuller discussion for more of our thoughts on some scenarios we think are worth considering, including intelligence explosion, turbocharged economy, and epistemic uplift.

Knowledge is integral to living life well, at all scales:

• Individuals manage their life choices: health, career, investment, and others on the basis of what they understand about themselves and their environments.

• Institutions and governments (ideally) regulate economies, provide security, and uphold the conditions for flourishing under their jurisdictions, only if they can make requisite sense of the systems involved.

• Technologists and scientists push the boundaries of the known, generating insights and techniques judged valuable by combining a vision for what is possible with a conception of what is desirable (or as proxy, demanded).

• More broadly, societies negotiate their paths forward through discourse which rests on some reliable, broadly shared access to a body of knowledge and situational awareness about the biggest stakes, people’s varied interests in them, and our shared prospects.

  • (We’re especially interested in how societies and humanity as a whole can navigate the many challenges of the 21st century, most immediately AI, automation, and biotechnology.)

Meanwhile, dysfunction in knowledge-generating and -distributing functions of society means that knowledge, and especially common knowledge, often looks fragile1. Some blame social media (platform), some cynical political elites (supply), and others the deplorable common people (demand).

But reliable knowledge underpins news, history, and science alike. What resources and infrastructure would a society really nailing this have available?

Among other things, we think its communication and knowledge infrastructure would make it easy for people to learn, check, compare, debate, and build in ways which compound and reward good faith. This means tech, and we think the technical prerequisites, the need, and the vision for a full epistemic stack2 are coming together right now. Some pioneering practitioners and researchers are already making some progress. We’d like to nurture and welcome it along.

In this short series, we’ll outline some ways we’re thinking about the space of tools and foundations which can raise the overall epistemic waterline and enable us all to make more sense. In this first post, we introduce frames for mapping the space —3 different layers for info gathering, structuring into claims and evidence, and assessment — and potential end applications that would utilize the information.

A full what?

A full epistemic stack. Epistemic as in getting (and sharing) knowledge. Full stack as in all of the technology necessary to support that process, in all its glory.

What’s involved in gathering information and forming views about our world? Humans aren’t, primarily, isolated observers. Ever since the Sumerians and their written customer complaints4, humans have received information about much of their our world from other humans, for better or worse. We sophisticated modern beings consume information diets transmitted across unprecedented distances in space, time, and network scale.

With an accelerating pace of technological change and with potential information overload at machine speeds, we will need to improve our collective intelligence game to keep up with the promise and perils of the 21st century.

Imagine an upgrade. People faced with news articles, social media posts, research papers, chatbot responses, and so on can trivially trace their complete epistemic origins — links, citations, citations of citations, original data sources, methodologies — as well as helpful context (especially useful responses, alternative positions, and representative supporting or conflicting evidence). That’s a lot, so perhaps more realistically, most of the time, people don’t bother… but the facility is there, and everyone knows everyone knows it. More importantly, everyone knows everyone’s AI assistants know it (and we know those are far less lazy)! So the waterline of information trustworthiness and good faith discourse is raised, for good. Importantly, humans are still very much in the loop — to borrow a phrase from Audrey Tang, we might even say machines are in the human loop.

Some pieces of this are already practical. Others will be a stretch with careful scaffolding and current-generation AI. Some might be just out of reach without general model improvements… but we think they’re all close: 2026 could be the year this starts to get real traction.

Does this change (or save) the world on its own? Of course not. In fact we have a long list of cautionary tales of premature and overambitious epistemic tech projects which achieved very few of their aims: the biggest challenge is plausibly distribution and uptake. (We will write something more about that later in this series.) And sensemaking alone isn’t sufficient! — will and creativity and the means to coordinate sufficiently at the relevant scale are essential complements. But there’s significant and robust value to improving everyone’s ability to reason clearly about the world, and we do think this time can be different.

Layers of a foundational protocol

Considering the dynamic message-passing network of human information processing, we see various possible hooks for communicator-, platform-, network-, and information-focused tech applications which could work together to improve our collective intelligence.

We’ll briefly discuss some foundational information-focused layers together with user experience (UX) and tools which can utilise the influx of cheap clerical labour from LMs, combined with intermittent judgement from humans, to make it smoother and easier for us all to make sense.

All of these pieces stand somewhat alone — a part of our vision is an interoperable and extensible suite — but we think implementations of some foundations have enough synergy that it’s worth thinking of them as a suite. We’ll outline where we think synergies are particularly strong. In later posts we’ll look at some specific technologies and examples of groups already prototyping them; for now we’re painting in broad strokes some goals we see for each part of the stack.

Ingestion: observations, data, and identity

Ultimately grounding all empirical knowledge is some collection of observations… but most people rely on second-hand (and even more indirect) observation. Consider the climate in Hawaii. Most people aren’t in a position to directly observe that, but many have some degree of stake in nonetheless knowing about it or having the affordance to know about it.

For some topics, ‘source? Trust me bro,’ is sufficient: what reason do they have to lie, and does it matter much anyway? Other times, for higher stakes applications, it’s better to have more confirmation, ranging from a staked reputation for honesty to cryptographic guarantee5.

Associating artefacts with metadata about origin and authorship (and further guarantees if available) can be a multiplier on downstream knowledge activities, such as tracing the provenance of claims and sources, or evaluating track records for honesty. Thanks to AI, precise formats matter less, and tracking down this information can be much more tractable. This tractability can drive the critical mass needed to start a virtuous cycle of sharing and interoperation, which early movers can encourage by converging on lightweight protocols and metadata formats. In true 21st Century techno-optimist fashion, we think no centralised party need be responsible for storing or processing (though distributed caches and repositories can provide valuable network services, especially for indexing and lookup6).

Structure: inference and discourse

Information passing and knowledge development involve far more than sharing basic observations and datasets between humans. There are at least two important types of structure: inference and discourse.

Inference structure: genealogy of claims and supporting evidence (Structure I)

Ideally perhaps, raw observations are reliably recorded, their search and sampling processes unbiased (or well-described and accounted for), inferences in combination with other knowledge are made, with traceable citations and with appropriate uncertainty quantification, and finally new traceable, conversation-ready claims are made.

We might call this an inference structure: the genealogy and epistemic provenance of given claims and observations, enabling others to see how conclusions were reached, and thus to repeat or refine (or refute) the reasoning and investigation that led there.

Of course in practice, inference structure is often illegible and effortful to deal with at best, and in many contexts intractable or entirely absent. We are presented with a selectively-reported news article with a scant few hyperlinks, themselves not offering much more context. Or we simply glimpse the tweet summary with no accompanying context.

Even in science and academia where citation norms are strongest, a citation might point to a many-page paper or a whole book in support of a single local claim, often losing nuance or distorting meaning along the way, and adding much friction to the activity of assessing the strength of a claim7.

How do tools and protocols improve this picture? Metascience reform movements like Nanopublications strike us as a promising direction.

Already, LM assistance can make some of this structure more practically accessible, including in hindsight. A lightweight sharing format and caches for commonly accessed inference structure metadata can turn this into a reliable, cheap, and growing foundation: a graph of claims and purported evidence, for improved further epistemic activity like auditing, hypothesis generation, and debate mapping.

Discourse: refinement, counterargument, refutation (Structure II)

Knowledge production and sharing is dynamic. With claims made (ideally legibly), advocates, detractors, investigators, and the generally curious bring new evidence or reason to the debate, strengthening or weakening the case for claims, discovering new details, or inferring new implications or applications.

This discourse structure associates related claims and evidence, relevant observations which might not have originally been made with a given topic in mind, and competing or alternative positions.

Unfortunately in practice, many arguments are made and repeated without producing anything (apart from anger and dissatisfaction and occasional misinformation), partly because they’re disconnected from discourse. This is valuable both as contextual input (understanding the state of the wider debate or investigation so that the same points aren’t argued ad infinitum and people benefit from updates), and as output (propagating conclusions, updates, consensus, or synthesis back to the wider conversation).

This shortcoming holds back science, and pollutes politics.

Tools like Wikipedia (and other encyclopedias), at their best, serve as curated summaries of the state of discourse on a given topic. If it’s fairly settled science, the clearest summaries and best sources should be made salient (as well as some history and genealogy). If it’s a lively debate, the state of the positions and arguments, perhaps along with representative advocates, should be summarised. But encyclopedias can be limited by sourcing, available cognitive labour and update speed, one-size-fits-all formatting, and sometimes curatorial bias (whether human or AI).8

Similar to the inference layer, there is massive untapped potential to develop automations for better discourse tracking and modeling. For example, LLMs doing literature reviews can source content from a range of perspectives for downstream mapping. Meanwhile, relevant new artefacts can be detected and ingested close to realtime. We don’t need to agree on all conclusions — but we can much more easily agree on the status of discourse: positions on a topic, the strongest cases for them, and the biggest holes9. Direct access as well as helpful integrations with existing platforms and workflows can surface the most useful context to people as needed, in locally-appropriate format and level of detail.

Assessment: credence, endorsement, and trust

Claims and evidence, together with counter claims and an array of perspectives (however represented), give some large ground source of potential insight. But at a given time and for a given person there is some question to be answered: reaching trusted summaries and positions.

Ultimately consumers of information sources come to conclusions on the basis of diverse signals: compatibility with their more direct observations, assessment of the trustworthiness and reliability (on a given topic) of a communicator, assessment of methodological reasonableness, weighing and comparing evidence, procedural humility and skepticism, explicit logical and probabilistic inference, and so on. It’s squishy and diverse!

We think some technologies are unable to scale because they’re too rigid in assigning explicit probabilities, or because they enforce specific rules divorced from context. This fails to account for real reasoning processes and also can work against trust because people (for good and bad reasons) have idiosyncratic emphases in what constitutes sensible reasoning.

We expect that trust should be a late-binding property (i.e. at the application layer), to account for varied contexts and queries and diverse perspectives, interoperable with minimally opinionated structure metadata. That said, squishy, contextual, customisable reasoning is increasingly scalable and available for computation! So caches and helpful precomputations for common settings might also be surprisingly practical in many cases.

With foundational structure to draw from, this is where things start to substantially branch out and move toward the application layer. Some use cases, like summarisation, highlighting key pros and cons and uncertainties, or discovery, might directly touch users. Other times, downstream platforms and tools can integrate via a variety of customized assessment workflows.

Beyond foundations: UX and integrations

Foundations and protocols and epistemic tools sound fun only to a subset of people. But (almost) everyone is interested in some combination of news, life advice, politics, tech, or business. We don’t anticipate much direct use by humans of the epistemic layers we’ve discussed. But we already envision multiple downstream integrations into existing and emerging workflows: this motivates the interoperability and extensibility we’ve mentioned.

A few gestures:

• Social media platforms struggle under adversarial and attentional pressures. But distributed, decentralised context-provision, like the early success stories in Community Notes, can serve as a widely-accessible point of distribution (and this is just one form factor among many possible). In turn, foundational epistemic tooling can feed systems like Community Notes.

• More speculatively, social-media-like interfaces for uncovering group wisdom and will at larger scales while eliciting more productive discourse might be increasingly practical, and would be supported by this foundational infrastructure.

• Curated summaries like encyclopedias (centralised) and Wikipedia (decentralised) are often able to give useful overviews and context on a topic. But they’re slow, don’t have coverage on demand, offer only one-size-fits-all, and are sometimes subject to biases. Human and automated curators could consume from foundational epistemic content and react to relevant updates responsively. Additionally, with discourse and inference structure more readily and deeply available, new, richly-interactive and customisable views are imaginable: for example enabling strongly grounded up- and down-resolution of topics on request10, or highlighting areas of disagreement or uncertainty to be resolved.

• Authors and researchers already benefit from search engines, and more recently ‘deep research’ tooling. Integration with easily available relational epistemic metadata, these uplifts can be much more reliable, trustworthy, and effective.

• Emerging use of search-enabled AI chatbots as primary or complementary tools for search, education, and inquiry means that these workflows may become increasingly impactful. Equipping chatbots with access to discourse mapping and depth of inference structure can help their responses to be grounded and direct people to the most important points of evidence and contention on a topic.

• Those who want to can already layer extensions onto their browsing and mobile internet experiences. Having always-available or on-demand highlighting, context expandables, warnings, and so on, is viable mainly to the extent that supporting metadata are available (though LMs could approximate these to some degree and at greater expense). More speculatively, we might be due a browser UX exploration phase as more native AI integration into browsing experiences becomes practical: many such designs could benefit from availability of epistemic metadata.

How? Why now?

If this would be so great, why has nobody done it already? Well, vision is one thing, and we could also make a point about underprovision of collective goods like this. But more relevant, the technical capacity to pull off this stack is only really just coming online. We’re not the first people to notice the wonders of language models.

First, the not inconsiderable inconveniences of the core epistemic activities we’ve discussed are made less overwhelming by, for example, the ability of LLMs to digest large amounts of source information, or to carry out semi-structured searches and investigations. Even so, this looks to us like mainly a power-user approach, even if it came packaged in widely available tools similar to deep research, and it doesn’t naively contribute to enriching knowledge commons. We can do better.

With a lightweight, extensible protocol for metadata, caching and sharing of discovered inference structure and discourse structure becomes nearly trivial11. Now the investigations of power users (and perhaps ongoing clerical and maintenance work by LLM agents) produce positive epistemic spillover which can be consumed in principle by any downstream application or interface, and which composes with further work12. Further, the risks of hallucinated or confabulated sources (for LMs as with humans) can be limited by (sometimes adversarial) checking. The epistemic power is in the process, not in the AI.

Various types of openness can bring benefits: extensibility, trust, reach, distribution — but can also bring challenges like bad faith contributions (for example omitting or pointing to incorrect sources) or mistakes. Tools and protocols at each layer will need to navigate such tradeoffs. One approach could have multiple authorities akin to public libraries taking responsibility for providing living, well-connected views over different corpora and topics — while, importantly, providing public APIs for endorsing or critiquing those metadata. Alternatively, perhaps anyone (or their LLM) could check, endorse, or contribute alternative structural metadata13. Then the provisions of identity and endorsement in an assessment layer would need to solve the challenges of filtering and canonicalisation.

In specific epistemic communities and on particular topics, this could drive much more comprehensive understanding of the state of discourse, pushing the knowledge frontier forward faster and more reliably. Across the broader public, discourse mapping and inference metadata can act against deliberate or accidental distortion, supporting (and incentivising) more good faith communication.

Takeaways

Knowledge, especially reliable shared knowledge, helps humans individually and collectively be more right in making plans and taking action. Helping people better trust the ways they get and share useful information can deliver widespread benefits as well as defending against large-scale risk, whether from mistakes or malice.

We communicate at greater scales than ever, but our foundational knowledge infrastructure hasn’t scaled in the same way. We see a large space of opportunities to improve that — only recently coming into view with technical advances in AI and ever-cheaper compute.

This is the first in what will be a series exploring one corner of the design landscape for epistemic tech: there are many uncertainties still, but we’re excited enough that we’re investigating and investing in pushing it forward.

We’ll flesh out more of our current thinking on this stack in future entries in this series, including more on existing efforts in the space, interoperability, and core challenges here (especially distribution).

Please get in touch if any of this excites or inspires you, or if you have warnings or reasons to be skeptical!

Thanks to our colleagues at the Future of Life Foundation, and to several epistemic tech pioneers for helpful conversations feeding into our thinking.

I want to know: is ‘extra reasoning compute’ like this? (Or, under what conditions and by what means can you beat this?) I’m especially interested in this question as applied to deliberate exploration and experiment design.

Said another way, from a given decision-context, without making extra observations or gathering extra data, what are the optimal marginal returns to ‘thinking harder’2 about what to do next?

Intuitively, if I have a second to come up with a plan, it might be weak, five minutes and it might be somewhat reasonable, a day and it’ll be better, a year (full time!) and I’ve reached very diminishing returns. Presumably a century in my ivory tower would be barely better. I’d usually do better trying to get more data.

Is this even a sensible question, or is ‘improvement in reasoning output’ far too vague to get traction here?

That’s the question; below some first thoughts toward an answer.

Simple model: repeated sampling/best of k

If you have a proposal generator, and you can choose between proposals, a simple approach to getting better generations is:

• sample a large number, k, of proposals

• (try to) evaluate and pick the best one

(This is actually the best strategy you could take if you can only add parallel compute, but there might be strictly better approaches if you can add serial3.)

Even assuming you can unerringly pick the best one, this strategy turns out to have logarithmically-bounded expected value for many underlying distributions of proposals4. In fact, for a normally-distributed proposal generator, you even get the slightly worse square root of logarithmic growth5.

You can in principle sidestep this if your proposal generator has sufficiently heavy-tailed proposal distribution, and you can reliably ex ante distinguish better from worse at the tails.

Another simple model: widely distributed ‘promise’ of lines of inquiry

Suppose you have various lines of inquiry to spend thinking time on. The best you can do is:

• start thinking on the most promising lines

• spend additional thinking on successively less promising lines

(This assumes you can somewhat reliably distinguish promise.)

If their ‘quality’ or ‘promise’ ranges over many orders of magnitude, then even if you get to accumulate insights additively6, you’ll actually make only bounded progress7 towards a theoretical ‘best possible’ - this is worse than logarithmic, though looks qualitatively similar over a substantial range of effort.

But why would the promise of lines of inquiry range over many orders of magnitude? We might say, ‘in practice, it often seems to’, and there are some theoretical reasons to expect this. You ‘pick low hanging fruit’ earliest, and face diminishing returns later. But to a large extent this model assumes the conclusion.

Other rougher gestures

Search depth

Often to find approximate solutions to problems, we might employ search over a tree-like structure. This emerges very naturally for planning over time, for example, where branching options (whether choice or chance) at each chosen time interval give rise to a tree of possible plans. (Compare Monte Carlo tree search.)

If gains are roughly uniform in search depth, this gives rise to logarithmic returns to further search. With excellent heuristics, you might be able to prune large fractions of the tree - this gives you a kinder exponent, but still an exponential space to search.

When (if at all) are gains over search depth dependably growing, rather than uniform at best? Alternatively, when can uniform (or better) gains be reliably achieved by expanding the search strictly less than exponentially?

Modelling chaos

Chaotic systems are characterised by sensitivity to initial conditions: dynamics where measurement or specification errors compound exponentially.

So, to forecast at a given precision and probabilistic resolution, it takes exponentially tighter initial specification precision to forecast marginal incremental time depth. (This is why in practice we only ever successfully forecast chaotic systems like weather at quite coarse precision or short horizon.)

Specification precision doesn’t exactly map to having extra compute, but it feels close. And marginal incremental time depth doesn’t necessarily correspond uniformly to ‘goodness of plan’.

Combinatorial search

If there’s some space of ingredients or components which can be combined for possible insight, the size of the search space is exponential8 in the number of components in a proposed combination. So if, among good plans at each scale, gains are proportional to the number of components in the plan (and there are similarly many good plans at each scale), you get logarithmic returns to searching longer.

Something similar applies if the design possibilities benefit from combining already-discovered structures in a hierarchy, for example if emergent features of subcomponents unlock new levels of effectiveness in a combined design (molecules, peptides, proteins, organelles, cells, ...).

But the assumption of roughly uniform gains over scales like this is carrying some weight here.

This part introduces exploration and research taste, as well as discussing their role in research and development, and the ways that AI could change that picture. This gives rise to some exciting and underexplored (!) opportunities for beneficial and defensive contributions to research.

A second essay will discuss more implications for AI development and governance, including the potential for AI to accelerate the pace of development of AI itself, and some implications for safety and security.

Neither essay will be especially technical, but I will gesture to the technical and mathematical aspects that I find to be illuminating. As ever when I write I raise more questions than I answer! But I hope to provide some initial useful takeaways as well as productive directions for thinking about these issues.

Introducing exploration and experimentation

Scientific and technological progress are driven by experimentation: that is, doing things to find out how the world works. In the field of AI we call this 'exploration'.

Exploration for learning is not just a human phenomenon: it's ubiquitous in natural systems at various scales (from evolution itself to the play of young animals), in individual human lifetimes (as we learn skills or contribute to novel discoveries) as well as human institutions and societies (which also learn through experience), and in computer science and AI (where exploration for discovery and problem-solving are common).

We care a lot about scientific and technological potential - they can yield enormous risks (from accident, misuse, or societal destabilisation) or enormous benefits (solving major problems in medicine, climate, energy, or even defending against other risky technologies). So exploration isn't just of academic interest.

When we forget to consider how new knowledge is generated, how novel technologies are developed, when we conflate 'knowledge' with 'learning' or 'learning' with 'exploring', mistakes are made. Especially when such predictions are action-guiding we can end up taking misguided or even harmful actions, or missing opportunities to intervene in beneficial ways. So let's do some unpacking!

What factors make exploration (and by extension, research) more or less effective? What are the bottlenecks and limits to exploration? How could AI change the picture? And how can we apply insights from this lens to contribute to a better future?

Subscribe now

Why does exploration matter?

Knowledge production loop: activity yields observations, improving knowledge — Exploration drives the loop — Lack of exploration means knowledge stagnation — Exploration is key to understanding technological progress

Learning systems gather new knowledge and insights from observations/data. Random flailing or arbitrary data aren't especially helpful. You want it to be telling you something new that you didn't already know - so it pays to deliberately seek out or gather novelty and informative observations1. This applies at the grandest scales of scientific endeavour as well as in mundane scenarios like navigating an unfamiliar building or learning a new skill.

Owen Cotton-Barratt recently discussed the 'knowledge production loop': activity and observations generate data (captured in datasets and models as 'crystallised intelligence') and combine with thinking algorithms ('fluid intelligence') to in turn drive new activity and observations.

I'd additionally characterise exploration as the way that crystallised world model and novelty taste interact with fluid reasoning and planning to judiciously choose activities yielding the most informative observations... in turn improving world models and taste ad infinitum.

Quality and quantity of exploration mark the vast difference between a civilization with vibrant progress in science and technology and one with a near-static (or even regressing) capability base - and on an individual level, it's often the difference between rapidly developing new skills or knowledge and getting stuck in a rut.

Understanding exploration is therefore key to understanding technological progress, with all the risks and benefits that entails.

Research and taste

Research is world-model-refinement — Exploration quality drives research — Taste is a learned feel for value of information — Reasoning and world modelling augment taste for exploratory planning

'Research' can be thought of broadly as refining one's world model in a particular domain. We want to know things like: how does electromagnetism work and what can it do for us? How can we prevent diseases from ruining lives? Or (more mundane) how can I get better at playing the piano or juggling? When I say 'research', I equally refer to personal learning and skill-building, scientific research, entrepreneurialism, and business development: all involve exploration and learning from experience.

We can describe three factors determining research production:

• Throughput: doing more practice, running more experiments, gathering more data faster, etc.

• Modelling efficiency: gathering more generalisable insight from a given experiment or observation.

• Exploration quality: choosing better experiments and routines to get more informative observations.2

We'll mostly talk about exploration quality here3, which is in turn governed by taste and exploratory planning.

What do I mean by 'taste'? Sometimes people refer to 'research taste' as a sense which develops from domain experience for the types of experiments and other activities which are most likely to be interesting or informative, or otherwise move forward the state of understanding. Clearly this is an essential component of any deliberate exploration - otherwise you're back to flailing randomly!

The taste that's being developed is exactly analogous to a taste for activity which is liable to yield good outcomes of other kinds. We're just considering the value of information as the good in question. So this decomposes into an ability to come up with promising proposals more often, perhaps together with abilities to discriminate more accurately between better and worse proposals or to determine refinements and improvements to proposals4.

Now, imagine - for the sake of the argument - you're a human. Even better, in fact, imagine you're inhumanly fast and detail-sensitive, the best reasoner in the world, and you have general knowledge matching the rest of the world combined. You still need to do research in order to make new discoveries. If you don't know the details yet, experimentation isn't something you can skip!5 Your especially effective reasoning merely acts as another input to exploration quality, alongside domain research taste, perhaps allowing you to choose better experiments, and achieve results sooner. Reasoning applied effectively in this way is exploratory planning.

So present exploration quality depends on your current level of taste, while future exploration quality will also depend on taste accrual6. Reasoning and planning of course also feed into this, as we improve proposals and discard designs in favour of better-looking ones - but this has to ground out in a taste for what makes a good proposal in the first place.

From play to experimentation

Play is proto-exploration — Fun is proto-taste — Humans adaptably accrue taste in novel domains — Taste is domain-specific but exploratory principles generalise

These aspects of taste are discovered and refined through experience. Research taste is domain-specific!

Many humans and animals, especially youngsters, have built in instincts for play, curiosity, and novelty. These have been tuned by painstaking natural selection to aid in orienting to the range of body configurations, environments and communities those animals usually inhabit, precisely by exploring: gathering evidence and information about how things work. In this case, evolution did the slow, gradual work of determining the 'taste', the recognisable hallmarks of good exploratory behaviour, and wired up the 'fun' sense to those hallmarks7.

As Eric Drexler says,

We call children intelligent because of what they can learn, not what they can do

Playful young animals and humans thereby become adept at controlling their bodies and engaging in effective social interaction. But humans move past mere bodily control and socialisation: we use and develop tools, technologies, and diverse and innovative social structures.

For many researchers and others engaging in creation, experimenting is a lot like playing! - the rich and sophisticated kinds of play that humans engage in somewhat instinctively. But, because research and development and science and industry move beyond the historic realms of human activity, the 'taste' bestowed by evolution is rarely well suited. An untrained human has no instinct at all for the kinds of experiments that are most likely to yield useful information about the behaviour of a new material or the structure of an unseen mathematical object! This applies equally to business activities and entrepreneurialism. Substantial experience is needed.

Do we see areas where 'taste' generalises, pointing against the claim of domain-specificity? The broad principles of science and engineering appear to generalise across domains, and evidence suggests that individual humans and human organisations vary in their latent potential to accrue and apply research taste. This might be down to being more or less motivated to explore, having different capacity to learn from experience, or varying procedures for planning next steps. This gives rise to an appearance of research taste generality. But domain-specific research taste is mastered only through domain-specific experience. Expert researchers in one area may contribute to other areas - but almost always only after gaining some depth of familiarity with the new area as well.

So it's reasonable to think of exploration quality as comprising two subfactors. First, the somewhat transferable general principles of exploration: playfulness, open-mindedness, planning for novelty and interestingness. And second, domain-specific research taste: the experience that guides determination of what situations count as novel or interesting8, and what types of planning are most likely to uncover them.

Subscribe now

Exploration in AI, past and future

First: humans curate data — Now: RL allows automatic data generation — Next?: in-context exploration characterises R&D tasks — Perhaps this is ‘AGI’?

In contemporary frontier AI systems, it's been mostly humans responsible for gathering 'high quality' informative data, often in quite hands-off ways like scraping huge datasets from the internet, but latterly with more attention on procurement and curation of especially informative or exemplary data.

With reinforcement learning (RL), the data coming in starts to rely increasingly on the activity of the system itself - together with whatever grading mechanism is in place. That's why lots of RL conversations of the past were so obsessed by exploration: taking judicious actions to get the most informative observations! So earlier AI research actually foregrounded exploration somewhat more. Helen Toner recently discussed the return of RL to centre stage in contemporary frontier AI, asking what properties of a domain make it more or less amenable to gains from reinforcement learning.

Still, in many RL settings, the human engineers are able to curate training environments with high-signal automated feedback systems, as Toner discusses. On the other hand, once we're talking about activities like R&D of various kinds, the task of exploring is inherently most of the task itself, making within-context exploration essential!

This makes 'learning to learn' or in particular 'learning to explore/experiment' among the most useful ways to operationalise 'AGI', from my perspective9. I'm not sure how best to track this, and I'm not aware of any benchmarks or studies which take this view on frontier general AI10. My personal experience with LM agents anecdotally points to them improving over time at orienting to uncertainties within their environment and being a little more creative at trying things out and testing things in 2025 than in 2024 or 2023, but not vastly - progress to date appears much more rapid in 'crystallised' intelligence.

Research by AI: AI with research taste?

Bootstrapping research taste from humans — AI advantages from speed and copying — AI learning by doing — Human advantages and bottlenecks to AI — Human-AI complementary workflows

There may be ways for AI training datasets to 'hoover up' research taste from existing experts and institutions, perhaps from lab notes or interviews, though humans at least usually learn more from actually trying research than merely from reading or talking about it. (This presumably reflects the fact that merely communicating about research experience is a much less rich source of information than actually experiencing it directly: the same issue faced by all kinds of knowledge transfer through limited media like language.)

So research taste in AI is not starting from scratch: already AI can talk in sensible, albeit sometimes basic ways about experiment design. The taste is bootstrapped from the taste implied by all the hints and observations in training data.

Could AI surpass the research taste exhibited by expert humans and human organisations? It's unclear where the ceiling is, but certainly AI would appear to have several advantages in principle: direct sharing of observations and experiences between instances, potentially far larger effective 'researcher headcount', total observation quantity far outstripping the longest-lived human experts (to date)11, all adding up to a far greater opportunity to accrue and accumulate taste. Additionally, due to computer speed, the opportunity to confer and deliberate in far more total depth the implications of each experiment and the appropriate designs of future experiments means that exploratory planning could also be boosted.

Crucially, acquiring frontier-applicable research taste would require either finding ways to bootstrap from existing research taste, which is often implicit (or even proprietary!), or enabling AI to learn by doing, perhaps aided by expert supervisors (just as human trainee researchers are), by instrumenting research processes and equipment with sensors and manipulators. Like hiring junior researchers, this would come with some upfront costs to any organisation attempting it12!

Human researchers begin with some advantages today: easier physical manipulation of experimental materials (for now), a capital base of experimental equipment designed for human use, and an ecosystem designed around the training, retention, and interaction of human experts. These aren't fundamental barriers to researcher AIs, but represent some hurdles or bottlenecks that might take time and other resources to reach past.

Of course, the capacities to interpret evidence, propose experiments, design and refine proposals, and to implement experiments need not reside 'in the same mind', just as human organisations already exhibit this division of labour. But the better fitted these pieces are to each other, the more efficient the overall system will be. Drexler's 'large knowledge models' discussion treats knowledge as a resource, to be combined with planning capacity and discernment from disparate sources. Similar agendas, for example from the UK's Advanced Research and Invention Agency (ARIA) perhaps promise both a more effective and more safely manageable way to integrate AI into research processes than wholesale development of autonomous researcher AIs.

Opportunities

Recapping research, experimentation, exploration, taste — Implications for AI forecasting and ‘intelligence explosion’ — Differentially bootstrapping AI taste — Differentially complementing AI exploration — Detecting dangerous research — Exploring AI applications for flourishing

Deliberate experimentation, consisting of exploratory planning and research taste, is a critical component of efficient learning - which, in R&D-heavy domains at least, because they inherently butt against the boundaries of the known, is foundational to progress.

Much more can and should be said about the implications of an experimentation-oriented view of R&D, both on AI and facilitated by AI in other domains. Here are some initial directions:

First, in forecasting AI capabilities and timelines, we should account for the costs of experimentation. This can include quantifying the relevant variables (iteration speed, quality of simulation, modelling, and exploratory planning, accrual and accumulation of research taste, the cost of experimental resources including compute and real-world interactions, etc.). Of particular interest, this could help to characterise the potential for 'self' improvement and the possibility of an intelligence explosion (which matter by implications for other R&D and for loss of control over AI systems).

You can't skip exploration! But greater intelligences (individual or collective) can be more efficient at it in general, and domain-specific taste in particular certainly yields improved rate of progress.

This cuts both ways for safety. You can't develop dangerous nanotech purely from first principles: you have to experiment, either in vitro or in silico. Unfortunately, nor can you generate new defensive vaccination, sterilisation, or biomonitoring paradigms without putting in the experimental legwork.

This may be revealing for those seeking to differentially drive beneficial and defensive research ahead of risky research. For example, exposing research logs and expert interviews to AI systems may yield a way to bootstrap specific kinds of research taste in AI. Alternatively, recognising the default taste-weakness but speed-advantage and general knowledge breadth of AI systems may suggest strategies for complementary human-AI workflows which could be both more effective and more manageable than naively attempting to create researcher AIs wholesale.

Beyond AI-driven exploratory planning and research taste, we should expect strong synergy with robotics, sensors, simulation, modelling, and other automation technologies, as complementary production factors in R&D progress. This is likely to naturally drive investment into these technologies, but may provide opportunities to differentially unbottleneck AI multipliers in beneficial areas by devoting development to their specific complements in particular.

Further, noting that technology can rarely be developed purely from first principles, intelligence and security organisations concerned about risky research directions may be able to anticipate the kinds of experiments that are likely to be useful, and therefore the kinds of resources and activities required to make progress in those areas. This may include flows and concentrations of certain machines or components, movements of specific rare materials, movement of human talent, or known side-effects of experiments. Where materials are very dual-use (such as concentrations of computing clusters), structured inspection, auditing, or transparency tools may aid in guaranteeing that only safe and sanctioned experiments are being carried out.

Finally, now is a great time to be experimenting with AI systems and their applications, especially for people who haven't traditionally paid attention to AI. Rapid developments mean that the extent of possibilities with current tech remains underexplored, and boosting defensive and beneficial applications ahead of risky ones is a great way to ensure that the future is better than it otherwise would be!

Thanks to Owen Cotton-Barratt and Jay Bailey for feedback and conversations on this topic

I’ve previously made arguments like:

Not long after it becomes possible for someone to make powerful artificial intelligence1, it might become possible for practically anyone to make powerful AI.

1. Compute gets exponentially cheaper by default.

2. Knowledge proliferates (fast!) by default: AI techniques are typically simple and easy once discovered.

3. What’s more, AGI-making know-how may be widespread already.

Or, as Yudkowsky puts it2,

Moore’s Law of Mad Science: Every eighteen months, the minimum IQ necessary to destroy the world3 drops by one point. - Yudkowsky

It’s important to emphasise that none of these are laws of nature! But the economic and social forces at work are quite strong.

So (leaving aside debates about the appropriate definition of ‘AGI’) where the frontier of AI development leads, others – many others – potentially rapidly follow. Followers can go even faster by stealing or otherwise harvesting insights from the frontier, but this is not a hard requirement – just an accelerant.

For more on the first point, compute getting cheaper, consider Moore’s law (or the more general and robust Wright’s law). What about the know-how?

Stupid, Simple AGI

The stupidest, simplest possible approach to producing general intelligence might mimic evolution in a large, open-ended, interactive environment. Nobody has succeeded at this yet because they don’t have enough compute, but just a few more decades of compute scaling might get us there. The code to do this would be ultimately quite simple, but the amount of compute time to run it is out of reach today. Almost nobody nowadays thinks that it will take this long, because this is the stupidest, simplest (and least steerable) possible approach and we have much better ideas.

But this means that unless something interrupts the compute trends, then even if ingenious, well-resourced people ‘get AGI first’, eventually anyone could practically blunder into creating their own. Of course, many things could be changed if powerful AI is developed and applied in the meantime… perhaps including the cost and efficiency of compute, the distribution of compute, or indeed the existence and inclination of people to do the blundering.

The Design Space for AGI

What did I mean by ‘AGI-making know-how may be widespread already’?

I don’t literally mean that the recipe for AGI is known and widespread. I don’t even mean that we broadly know exactly how to make AGI and simply want for the capital (compute and data). But for those paying attention, the design space for practically achievable AGI is narrowing.

Take long-horizon coherence or continual learning, for example. Maybe components of these are expandable memory and long-context management of plans and observations. This could perhaps be cracked with something resembling a selection from:

• Context summarisation

• Read-write retrieval-augmented generation

• Recurrent embeddings

• Longer training trajectories

• Plan-management or recursive delegation scaffolding

• Periodic distillation of history into weights or activation patches

• Explicit training for notetaking

• Some even simpler thing, like ‘just scale up the compute’

Among the sharpest, most experienced practitioners at the frontier, that perceived design space may be narrower still4. In the far wider cohort comprising all competent computer scientists and engineers, the design space may not be as saliently in view – but the scientific ‘breadcrumbs’ have been pointing in useful directions for years (at least).

My personal testament5 is that by 2020, several landmarks were visibly coming together in NLP and RL, and by 2021 I had a good sense of a plausible research path to general autonomous AI. Developments like further scaling, mixtures of experts, chain of thought, LLM agents, RL ‘reasoning’, fast attention mechanisms, and hyperparameter tuning optimisations are not merely ‘obvious in hindsight’: their rough contours were advance predictable. It was ‘merely’ a matter of experimenting to find out working details. I’m not being (especially) hubristic here: for some experts closer to the action, these same things looked plausible by 2017 or even earlier! The contours of tomorrow’s advancements are similarly already in view, and far more attention and capital are being poured into the discovery process.

That’s not to say that, given the capital, we could have created AGI there and then in a single try, or even here and now. A design space is not a complete or final design. But iterative refinement by well-resourced and moderately creative problem solvers has been charting a course, and if we are willing to anticipate one frontrunning group getting ‘all the way there’ we must acknowledge that the feat will be reproducible in relatively short order.

Accelerants

‘Reproducible’ is one thing. How soon and how fast? With the current level of sharing of research insights, the answer seems to be roughly ‘as soon as you can outlay comparable capital’, or even sooner!

What phenomena are responsible for accelerating this proliferation? In very roughly descending order of effect size:

• Theft, leak, or deliberate release of pretrained baselines and training algorithms

• Distillation (authorised or not) from exposed APIs

• Exponentially cheaper compute

• (Sometimes cheap or even public access to) ever more sensor and record data6

• Shared algorithmic and experimental details in papers and blogposts

• Conversations and rumours at conferences and other events7

• Movement of experts between development groups and projects

• Use of AI to assist development8

Very tightly securitized projects might partly dampen some of these effects. Competition between firms and countries could amplify them.

What about exponentially cheaper compute? Market dynamics might pivot at some stage to reduce or even reverse the effect of dwindling compute price (for example, extreme buyer concentration driven by strategic accumulation, increasing marginal compute utility9, deliberate regulatory intervention on compute, or something else), but will otherwise continue to drive proliferation. On the other hand, if compute production increases even faster, costs may drop commensurably faster10.

Sensor and records data are being collected even more feverishly now that companies have realised their critical use in training modern AI systems — notice when companies’ privacy policies update to include carve-outs for collecting AI training data. We should expect more of this, as well as more collection of physical and industrial activity records for training robotics, autonomous vehicles, and automated laboratory workcells.

Alternatively, some have imagined an ‘end of history’ moment when sufficiently smart AI arrives and (usually by underspecified mechanism) prevents all of these factors from proceeding. Some envisage not only that, but an AGI or AGI-enabled organisation foreclosing not only the accelerants of proliferation, but also the potential for a rival project to emerge anywhere11. This is conceivable, but one has to ask on what timeframe these changes would happen, and the consequences if it takes longer than imagined.

Short of such an acute and decisive interruption of all of these dynamics, other shocks such as international conflict could have impacts in either direction.

Concluding

Intelligent engineering-minded people exist in all geographies and of all ideologies. Most lag the frontier of AI development only for want of compute capital and intent. Because compute continues to get cheaper, and the potential of AI comes more clearly into focus, both compute and intent become rapidly more widespread. The open sharing of discoveries can further lower barriers and shorten proliferation timelines, but is not essential to this dynamic.

Given this, we have to ask what the consequences of this proliferation could be. Where they are concerning, we must consider in what ways these dynamics could be defused, or, likely failing that, how we will ready ourselves, on a short timeframe, for what follows.

We live in interesting times! There’s a lot we can do.

This blogpost accompanies the paper Cooperation and Control in Delegation Games by Sourbut, Hammond, and Wood, which was presented at IJCAI 2024. In essence, the work attempts to deconfuse some of the discourse around safety, cooperation, and alignment in multi-agent settings by:

• Showing that, just like in the control problem, cooperation problems can be broken down into alignment and capabilities, which are orthogonal to one another;

• Providing measures for alignment and capabilities (both “individual” and “collective”) in multi-principal multi-agent settings (“delegation games”);

• Showing that any of these measures is insufficient alone to guarantee the best outcomes, but that they are together sufficient;

• Bounding the principals’ welfare loss in terms of these measures, and validating this with a series of empirical results.

The goal of this post is to explain what those terms mean, and hopefully why it matters. In doing so, we hope to shed light on some of the related questions posed by other AI safety researchers, for example Dafoe et al in Open Problems in Cooperative AI who discuss the concept of ‘horizontal’ and ‘vertical’ aspects of coordination, or an open problem on the AI Alignment Forum about quantifying player alignment in normal-form games.

There is also a poster which is an even more condensed summary of some key material, and Lewis and Oly have given a few presentations on the topic, one of which is recorded here.

Why Delegation Games?

You may have heard of the Principal-Agent problem. It's a phrase and a setting which turns up in some economics literature, and elsewhere. The idea is that a principal (in this case, the human) is asking, telling, employing, or otherwise exhorting an agent (in this case, the robot) to act on their behalf. The 'problem' is the question of how to ensure that the agent's behaviour results in outcomes which the principal in fact prefers.

Delegation games arise when we have multiple principals, and multiple agents.1 When you read 'principal', think 'human', and when you read 'agent', think 'AI'. (For a slightly different semantic, you can alternatively think of each principal as a basically-coherent coalition of humans, and likewise with AIs.)

Why does this setting matter? It's looking increasingly likely that, perhaps quite soon, many somewhat-autonomous digital personal assistants, digital employees, or similar, will be deployed on behalf of human overseers. That is, we might be entering a highly multipolar world when it comes to somewhat-autonomous AI deployments2. A more obvious, immediate, lower-stakes example of this is autonomous vehicles, which we use as a toy example in the paper. Finally, in the future, multiple large coalitions of humans (e.g. states or companies) may deploy powerful AI systems to act on their behalf in high-stakes scenarios. We want to understand the important features of how to make sure this goes well!

We formalise delegation games in the way you might expect: agents adopt strategies that lead to (a distribution over) outcomes, and both the agents and the principals have (potentially different) preferences over these outcomes.

Cooperation, Alignment, and Calibration

In the paper, we identify some key properties which influence the outcome of a delegation game. We’ll highlight Cooperation, Alignment, and Calibration, because one key punchline of the paper is:

You need all three to guarantee good outcomes

This is important to bear in mind given that much AI safety work focuses on alignment, which is (demonstrably) not enough for safety in multi-polar worlds.

We'll explain what these terms mean, what 'good outcomes' are, and by the end of the post we should have covered enough to understand the high level meaning of the highlighted inequality, which is adapted from Theorem 1 of our paper. This simplification also assumes agents are perfectly individually rational, which we generalise in the paper, and here we mostly skip over collective alignment, which gets a thorough treatment in the paper.

Subscribe now

Cooperation

Intuitively, cooperation is working together for mutual gains over some uncooperative baseline. This is actually already non-terrible as a definition, but we can sharpen it.

Note that cooperation can be partial (one coalition cooperates, potentially with downsides for others). That's collusion. We especially don't want AI to be collusive! This is tricky, because cooperation and collusion rest on basically the same abilities and infrastructure. It's a very important topic, but we don't discuss it here.

Cooperation example

Let's look at a simple example.

Imagine we're palaeolithic hunter-gatherers: we, the authors, are one small, cohesive group, and you, the reader, another. In the morning, if we all set out to gather, by the end of the day we each come back with a basket of fruit (the fruit/fruit outcome, score: 2). If we for some reason decide to instead hunt a mammoth, well... we’re big and tough but we probably can't catch a mammoth. Meanwhile you sensibly gathered some fruits (the mammoth/fruit outcome, score: 1). Likewise if you try to catch a mammoth alone (fruit/mammoth, score: 1). BUT, if we all work together, we've a decent chance at catching the mammoth (mammoth/mammoth outcome, score: 10). These scores are our utilities for the outcomes3.

Now, laid out like this, there's an obvious best-case outcome4: we work together to catch the mammoth! (Nobody has invented conservationism yet5, so this is a preferred outcome.) But there really is a coordination challenge here: if we have reason to believe that you'll go gathering berries, we should too – it would be in our interest (and in this case, yours too) to get more fruit rather than waste our time fruitlessly (!) chasing mammoth.

This isn't just academic. When we look around the world, many problems have the mammoth-nature: we all have to 'show up' or we don't get the benefit (and indeed many cooperation problems are even harder than this due to selfish incentives). Consider international cooperation on climate change, biological weapons control, or coordination on safe technological progress.

Humans solve this sort of problem all the time. We are able to do this due to various abilities, affordances, and so on, which we can collectively refer to as cooperative infrastructure. This includes such things as:

• talking to each other

• trust and reputation

• trade

• commitments and enforcement

• norms and laws

Nevertheless, our cooperative infrastructure is often not up to all tasks.

AI systems have been pretty bad at this on the whole, though there have been some interesting improvements over the years. Future AI might have access to very powerful kinds of cooperative abilities and infrastructure6.

Formalising cooperation

How do we characterise 'better cooperation'? We operationalise the collective goodness of an outcome with a welfare function (an aggregation over utilities). One way to specify cooperation is by considering failure. We look at the welfare-optimal outcome(s) σ⋆ (in this case mammoth/mammoth, score: 10) and then compare any actual (or predicted) outcome σ. The difference in welfare is the welfare regret – how much better it 'could have been'.

The welfare regret of principals (humans) is our primary measure of interest (the 'dependent variable', if you like). When we have only principals playing, that tells most of the story. With agents involved (machines/AI) welfare regret of agents7 quantifies how successfully they cooperated (according to their criteria and coordination mechanisms).

There is also an alternative, more geometric interpretation. 'Mutual gains' are Pareto gains, and Pareto optima coincide (in all but edge-cases) with welfare optima8. Hence, we can interpret cooperation as movement toward the Pareto frontier. (The specific direction of movement corresponds with the welfare aggregation function.)

In the paper we also make some discussion of capabilities and how they give rise to outcomes (and thus to welfare/regret). We tentatively distinguish 'individual' from 'collective' capabilities, and describe mathematically and algorithmically how, given access to estimates or measurements of interactions, these can be determined and distinguished. A related concept is the price of anarchy which quantifies the failure of a particular system to be robust to selfish behaviour.

Now, in the preceding example, we assumed that food was shared and the humans’ dietary preferences were equivalent(ly primitive). That is, we have perfect collective alignment (between principals). Notice that even with this perfect collective alignment, there can remain coordination problems, as in this scenario. In general, we can distinguish problems of collective alignment from problems of collective capabilities (cooperation). We fully characterise this breakdown in the paper, and we'll touch on it later under calibration.

Alignment

When we have more than one actor with some preferences over outcomes, it is natural to ask about the relationship between those preferences. Alignment is the extent to which two or more preference relations are in agreement.

We might prefer exactly the outcomes that you prefer and vice versa (as in our mammoth example where all returns are shared), in which case we are perfectly aligned. Or (as in a myopic chess match9) we might be playing for exactly the outcomes you want to avoid, and vice versa, in which case we are perfectly misaligned. More usually, it'll be something in between these extremes.

Alignment example

In the paper, we discuss several forms of alignment, but here we will focus on alignment between a principal and an agent, namely individual alignment.

Back to our hunter-gatherers, except now we're high tech palaeolithic hunter-gatherers. We have hunter-gather-bots which we delegate to. This is where the game becomes a delegation game.

We produced these bots somehow, perhaps through a process of machine learning and subsequent scaffolding; we ran lots of tests in the lab and the agents seemed to be doing basically what we expected. But we failed our alignment homework.

After the shift to the wild deployment distribution, it turns out our bots in practice prefer more fruit-and-nut and less mammoth-steak (yellow utilities). They're not horribly misaligned (they still prefer to feed rather than starve us), but they're soft-vegetarian bots, and ultimately the consequence of deploying them is muesli for breakfast, lunch, and dinner... forever. An unmitigated disaster10.

Formalising alignment

Now, we can easily define perfect alignment if two sets of preferences are the same, or perfect misalignment if they're exactly opposite. What about intermediates? A key issue with comparing utility functions (or reward functions) is that the same preferences can be described by many different utilities.

For example, if you scale your utilities all by 10x, or if you add a constant 0.1, the preferences this represents are unchanged. This generalises to any scale and shift, namely an affine transformation.

So if we naively compare two utility functions, we might get nonsense or misleading results. We need a way to standardise the representation of preferences as utilities.

In the paper we provide expressions and algorithms to account for these requirements, discussing various desiderata and showing that our measures satisfy them. In particular, utility functions with indistinguishable preferences have identical representation in our standardisation11.

Now we have standardised points, it actually makes sense to compare them. So we can take an appropriate distance measure between points to quantify how aligned they are. This gives us misalignment distance. In the single-principal single-agent case, this alone is enough to provide some interesting regret bounds for the principal.

For more on this kind of approach to comparing utilities and rewards, see e.g. the EPIC and STARC papers. We or some of our colleagues might write up a blog digging more into these concepts at some point.

In the full delegation game setting, distances between utility functions can be used not only to measure principal-agent misalignment, but also the misalignment between groups of agents (or principals). This collective alignment measure essentially captures how much the agents are ‘on the same team’ (or not).

Share

Calibration

Calibration is intuitively a fairness consideration: how much weighting is each player being given in a cooperative outcome?

When we began this project, we were intuitively expecting that a satisfying operationalisation of 'perfect cooperation' and of 'perfect alignment' would together guarantee optimal outcomes. That is, we anticipated that perfectly cooperative and perfectly aligned agents would produce welfare-optimal outcomes for principals. In fact, we could only prove that the outcomes were Pareto efficient for principals12. Calibration is the missing piece.

Calibration example

Let's return to our hunter-gatherers once more. Previously, we imagined a perfect implicit contract to share all gains equally. Hence, a mammoth/mammoth outcome is straightforwardly better. But we might imagine some alternatives:

• we all agree to hunt mammoth together, but you only get one steak and we get the whole rest of the mammoth

• we make the original agreement to share the mammoth equally

• you refuse to hunt mammoth with us unless we give you most of it

• ...

Some of these outcomes may seem more or less 'intuitively fair', but it is hard to find this law written into the universe, and in practice players simply have their preferences and act on them (which may include some preference for fair or altruistic outcomes). Notably, they all improve on the uncooperative baseline.

The point here is that there are generally lots of different ways that cooperation can cash out, and even different 'cooperative outcomes' weight players differently.

Formalising calibration

The weighting over players implied by their modes of cooperation, or equivalently the welfare weightings of players in the welfare function used to score cooperation, determine which Pareto outcomes are deemed welfare optimal.

In our setting with standardised utilities, these welfare weightings correspond exactly to the magnitudes m of the players' utilities. Thus, we can completely characterise the relationship between the agent welfare aggregation and the principal welfare aggregation by considering the ratios:

where m̂ⁱ is the ith principal's magnitude, and mⁱ is the corresponding agent's magnitude. When these ratios are all equal, we have perfect calibration.

Otherwise, these individual welfare ratios rⁱ are combined (as we explain in the paper) with the collective alignment to produce R, the contribution of the miscalibration and collective alignment to the overall welfare regret.

The punchline: you need cooperation, alignment, and calibration to guarantee good outcomes

Now we have all the pieces to understand this claim more clearly.

The term on the left is the principals' welfare regret: how much better the aggregate utility of principals (humans) could have been.

On the right we have a cooperation failure term (the agents' welfare regret), an alignment failure term (a sum over the individual alignment distances), and a calibration failure term (the aggregate R over the welfare ratios and collective alignment distances). In the paper we also demonstrate that these measures are 'orthogonal' in the sense that each can be instantiated arbitrarily, regardless of the others.

By 'good outcomes', we mean 'minimising welfare regret of principals'. From this inequality, it's immediately apparent that, to get 'good outcomes', it is sufficient to minimise all three of the terms on the right. In the paper, we also prove that, to guarantee good outcomes, it is necessary to minimise the cooperation failure, misalignment, and miscalibration – that is, absent extra information, luck, or other magic, if you have failures in one of these areas you can't be certain of good outcomes.

Experiments

Besides theory, we've got experiments, a few of which are visualised here. The blue surface is derived from our regret bounds, and each green dot is the result of one simulated delegation game. There are more variables at play, but here each chart is controlled for particular welfare ratios (miscalibration), with axes for agent welfare regret (miscoordination) and aggregate individual alignment distance (misalignment).

A few other observations from the experiments:

• the highest points here come quite close to the surface – it can be a relatively tight bound13

• the average regret follows the contour of the bound, as you might expect

In some other experiments, we looked into how we can estimate some of these quantities empirically from much more limited data, a harder challenge.

Limitations and Next Steps

There are several limitations of these analytical tools.

Perhaps the most practical weakness is in computing these things. If we have access to the utility/reward functions, we can compute alignment measures in linear time over outcomes... but there can be a lot of outcomes! Further, we generally don't have direct access to a complete utility function14, and some decisioners may not be well-described as having utility functions. Worse, the outcome space might not only be very large but also unknown/unexplored!15 Welfare regret is easy to compute, but only if you know the welfare optimum – otherwise you can only get a lower bound. We demonstrate some preliminary work on estimation of these measures with limited empirical access in the paper.

The definitions we use in our analysis (welfare regret, alignment distance, and welfare ratios) also rely on some 'design choices' from a family of possible functions (e.g. norm choice for alignment distance and welfare weightings for welfare regret). For putting these into practice, there remains a challenge of making a choice. Importantly, these affect the tightness of bounds, and also the normative weight of principals in the overall welfare regret. For any such choice (and we should expect there is some sensible choice), our theoretical conclusions are nevertheless sound.

We also make a few simplifying assumptions about the structure of the delegation game. First, we assume that principals don’t take actions, only their delegate agents do. Second, we assume 1-1 principal-agent relationship, which facilitates some of the individual alignment analysis, but is missing full generality. These limitations should be simple enough to generalise, but a bit messier to talk about. Finally, and relatedly, we assume a fixed population of principals and agents. This has various implications. For one, total and average utilitarianism are identical in this case, while in practice impacts on population can mean that these come apart radically.

Some readers may be uncomfortable with 'agent welfare'. Where do we get these agent utility magnitudes from in the first place? Or equivalently, where do we get welfare weightings from in the case of agents? In fact, due to an equivalence between Pareto optima and welfare optima, you can actually taboo 'agent welfare' from our analysis entirely, and talk only in terms of Pareto gains and Pareto efficiency, while deriving substantially the same conclusions. The point is, a Pareto gain is just another vector (this time in the space of players' joint utilities), and a vector necessarily has a direction! – and thus cooperative gains and Pareto optima give rise implicitly to welfare weightings16.

Another potential philosophical issue is that we give normative precedence to principals' utility and welfare. If you think the agents might matter in and of themselves too, you might want to do an altered analysis. The modification should be straightforward, and the essence of the conclusions is unchanged (just differently-weighted) unless the agents are utility monsters or moral super-patients.

All of these limitations represent interesting avenues for future work. We’re especially interested in scalable ways to evaluate some of the measures using data gathered from interactions between humans and complex AI systems. We hope these measures will be a useful tool when it comes to thinking about the principles behind building more aligned and cooperative AI systems in multi-polar worlds.

exponential expansion (until constraints are reached) ... in practice often manifests as first imperceptible and then rapid escalation.

Connor Leahy, CEO of Conjecture AI, echoes me more pithily

As we learned with COVID, there are only two times to react to an exponential - too early or too late.

Of course, everything has been said before, and we are both in dialogue with the famous quote from Professor Bartlett

The greatest shortcoming of the human race is our inability to understand the exponential function.

I brought a more specific and lighthearted1 take in Invading Australia, where I looked more closely at some case studies of expansionist/replicating (exponential) systems in the field of human biosphere interventions. In summary,

The experiment, this introduction of foreign species was... successful, if by 'successful' you mean 'devastating and difficult or impossible to roll back'.

We learned about some pesky amphibians:

It turns out that cane toads don't jump or climb well, so outside of the lab, where beetles live at the top of sugar cane, the toads were all but useless at their intended purpose. (Out of context failure!)

But the toads were a success, in their own terms: unexpectedly unfussy eaters, prolific reproducers, and poisonous to most wildlife, they have rapidly colonised Queensland state, lately expanding into New South Wales and the Northern Territories, while being resilient to every attempt at pushing them back. We fought fire (cane beetles) with fire (toads), and ended up with two fires!

and we looked at the rather amusing case of humans deploying yet more replicators to bring the toads under control, namely toad viruses and/or genomic interventions like driving fertility-reductions. I'm reminded of the Old Lady Who Swallowed A Fly.

There's good (?) news, of course:

the prickly-pear or paw-paw is another species inadvertently unleashed on the Australian ecosystem, which has caused some displacement of native wildlife. A moth (believe it or not, Cactoblastis!) was found which actually does seem to work as a self-regulating suppressant of the cacti.

When it comes to AI, we'd have to be very very prepared in order to expect shenanigans like this with replicating or propagating systems to end well.

Also, no-one picked up on my Charles Darwin/extinction pun: 'endless formerlies most beautiful'??

Upcoming, alignment and cooperation

I'm working on a paper2 with some collaborators in Oxford, analysing cooperation and alignment concepts for AI. What does this mean? When there are lots of AI systems, we want them to generate value by interacting positively rather than destroying value through conflict or anarchy. That's cooperation. And we want the values they are oriented at to be valuable to us, rather than arbitrary (or worse, harmful) things. That's alignment. Expect a takeaway or two to appear in the near future elucidating and elaborating on that.

We don't get the opportunity to pause time after every question syllable, pull up a notepad, run some supercomputer evaluations, compute exact Bayesian posteriors, estimate our teammates' and opponents' credences and likely buzzing behaviour, and so on. Cruelly, time flows at one second per second and the quizmaster keeps quizmastering. So too in life!

So too, indeed.

Twitter/X and YouTube had fun, as some friends were eager to point out to me. There were some more flattering remarks, but I think my favourite was

Sourbut combs his hair with a toffee apple #universitychallenge

As I anticipated, people also enjoyed it when I said 'groyne'. I know it sounds like 'groin', and yes, my name is 'Sourbut' and it's pronounced how you think.

It's 'groyne', actually

The editors actually cut the bit where I said, 'Never thought I'd say, "groin" on TV'. As I cryptically hinted in the Hertford, Sourbut post,

(As we may find out, there are also secretly other options, like '(-1000) say something embarrassing on national TV'.)

Ironically for a piece on speaking clearly and with nuance, I failed to explicitly point out crucial facts! - the actual true accounts regarding one of the exemplars I brought of language-misuse (these were obvious in my mind while writing, but some readers appeared confused in ways which make sense if they didn't know). I criticised

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.1

but didn't explicitly point out that, beyond being an oversimplification, there just isn't a ready way to map this to the reality, which is that

• the smuggling in question was done by... smugglers

• the buying of chips was done by multiple China-based entities

• the (implicit but unmentioned) selling (and importantly, provisioning/enabling) of chips was done by NVIDIA, a US-based company (and perhaps others)

• the investing was done by the CCP

I had a great response from CAIS in particular. The original author agreed this was ambiguous and unfortunate, and they've updated the text in question substantively. They also responded

More generally, we try to avoid zero-sum competitive mindsets on AI development. They can encourage racing towards more powerful AI systems, justify cutting corners on safety, and hinder efforts for international cooperation on AI governance. It’s important to discuss national AI policies which are often explicitly motivated by goals of competition without legitimizing or justifying zero-sum competitive mindsets which can undermine efforts to cooperate. While we will comment on the how the US and China are competing in AI, we avoid recommending "race with China."

This was really welcome and I hope other readers took on board the lesson here.

A few other readers pushed back a little. Stephen Clare expressed general agreement and offered a rearticulation of the problem I'm pointing to, while also criticising my relegation of governments to 'not currently meaningful players in AI development and deployment' as being too strong. Quite right: I meant that governments have (to date) been entirely passengers regarding the direction and nature of advanced AI development, but it is true that they have begun to get involved in coarse economy-level lever-pulls like investing and regulating hardware.

I went on a minor rant in the comments:

Do people actually think that Google+OpenAI+Anthropic (for sake of argument) are the US? Do they think the US government/military can/will appropriate those staff/artefacts/resources at some point? Are they referring to integration of contemporary ML/DS into the economy? The military? Or impacts on other indicators2? What do people mean by "China" here: CCP, Alibaba, Tencent, ...? If people mean these things, they should say those things, or otherwise say what they do mean. Otherwise I think people motte-and-bailey themselves (and others) into some really strange understandings.

Amazingly, one reader admitted that,

Yes.

In the end, all the answers to your questions are yes.

and made some further assertions about inevitability of international conflict. We had a minor back-and-forth but this was pretty remarkable, to me, and I think there was some talking-past happening. Thank you for sharing honestly.

Sadly, Scott Alexander, an author I hugely admire, has evidently not read my admonishment to CAIS, as his latest letter is full of thoughtless remarks about China and the US/West. Scott, you should know better. Words have power. Saying this, I think it is a good and useful post in many ways, in particular laying out a partial taxonomy of differing pause proposals and gesturing at their grounding and assumptions. He writes,

The biggest disadvantage of pausing for a long time is that it gives bad actors (eg China) a chance to catch up.

There are literal misanthropic 'effective accelerationists' in San Francisco, some of whose stated purpose is to train/develop AI which can surpass and replace humanity. There's Facebook/Meta, whose leaders and executives have been publicly pooh-poohing discussion of AI-related risks as pseudoscience for years, and whose actual motto is 'move fast and break things'. There's OpenAI, which with great trumpeting announces its 'Superalignment' strategy without apparently pausing to think, 'But what if we can't align AGI in 5 years?'. We don't need to invoke bogeyman 'China' to make this sort of point. Note also that the CCP (along with EU and UK gov) has so far been more active in AI restraint and regulation than, say, the US government, or orgs like Facebook/Meta.

Suppose the West is right on the verge of creating dangerous AI, and China is two years away. It seems like the right length of pause is 1.9999 years, so that we get the benefit of maximum extra alignment research and social prep time, but the West still beats China.

Now, this was in the context of paraphrases of others' positions on a pause in AI development, so it's at least slightly mention-flavoured (as opposed to use). But as far as I can tell, the precise framing here has been introduced in Scott's retelling.

Regardless of the origin of this formulation, this is bonkers in at least two ways. First, who is 'the West' and who is 'China'? This hypothetical frames us as hivemind creatures in a two-player strategy game with a single lever. Reality is a lot more porous than that, in ways which matter (strategically and in terms of outcomes). I shouldn't have to point this out, so this is a little bewildering to read. Let me reiterate: governments are not currently pursuing advanced AI development, only companies. The companies are somewhat international, mainly headquartered in the US and UK but also to some extent China and EU, and the governments have thus far been unwitting passengers with respect to the outcomes. Of course, these things can change.

Second, actually think about the hypothetical where 'we'3 are 'on the verge of creating dangerous AI'. For sufficient 'dangerous', the only winning option for humanity is to take the steps we can to prevent, or at least delay, that thing coming into being. This includes advocacy, diplomacy, 'aggressive diplomacy' and so on. I put forward that the right length of pause then is 'at least as long as it takes to make the thing not dangerous'. You don't win by capturing the dubious accolade of nominally belonging to the bloc which directly destroys everything! To be clear, I think Scott and I agree that 'dangerous AI' here is shorthand for, 'AI that could defeat/destroy/disempower all humans in something comparable to an extinction event'. We already have weak AI which is dangerous to lesser levels. Of course, if 'dangerous' is more qualified, then we can talk about the tradeoffs of risking destroying everything vs 'us' winning a supposed race with 'them'.

I'm increasingly running with the hypothesis that many anglophones are mind-killed on the inevitability of contemporary great power conflict in a way which I think wasn't the case even, say, 5 years ago. Maybe this is how thinking people felt in the run up to WWI, I don't know.

I wonder if a crux here is some kind of general factor of trustingness toward companies vs toward governments - I think extremising this factor would change the way I talk and think about such matters. I notice that a lot of American libertarians seem to have a warm glow around 'company/enterprise' that they don't have around 'government/regulation'.

[ In my post about this I outline some other possible cruxes and I'd love to hear takes on these ]

Separately, I've got increasingly close to the frontier of AI research and AI safety research, and the challenge of ensuring these systems are safe remains very daunting. I think some policy/people-minded discussions are missing this rather crucial observation. If you expect it to be easy (and expect others to expect that) to control AGI, I can see more why people would frame things around power struggles and racing. For this reason, I consider it worthwhile repeating: we don't know how to ensure these systems will be safe, and there are some good reasons to expect that they won't be by default.

I repeat that Scott’s post as a whole is doing a service and I'm excited to see more contributions to the conversation around pause and differential development and so on.

Relatedly, I had a great conversation at lunch yesterday with Will MacAskill, who’s currently working on questions of coordination around development of advanced AI. Very excited to read more when that comes out!

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.1

Sounds about right?

This post centres around an email I sent to the Center for AI Safety (CAIS) expressing concern about their 2023-08-15 newsletter's coverage of US-China competition in the AI space2, but the overall point is broader. There are some ways of discussing the topic of international relations regarding AI which strike me as un-nuanced in a counterproductive and dangerous way, by hiding certain truths or emphasising others, and supporting a conflict-oriented mindset.

In writing about this, I'm also gesturing at something about the more general topic of 'how to think and write about politically-charged topics'.

Jump to the summary if you are in a hurry.

This conversation really is important, which is why I think it's worth a public message discussing particular statements, but this should be understood as constructive criticism and part of a broader conversation which society, and especially the community of those focused on AI safety, needs to have. The CAIS newsletters are worth a (not unquestioning!) read, including the edition in question.

The particulars in this message serve as good exemplars of the problems and questions I have, and I'd be interested in responses from CAIS but even more so in remarks more broadly on the topic from anyone interested. The public conversation about this appears from my perspective to sometimes be broken. If that is so, I would like it to be rectified, and if not, I would like to be put right myself, the better to prioritise in my own work!

Specifics, CAIS case study

Here, quoted3, is my response to the CAIS letter, which serves as an initial dialogue opener and the core of this post:

Hello,

I've been a supportive reader for some time and am myself an AI safety researcher. I'm generally very impressed and encouraged by your newsletters! - but I was disappointed and concerned by the phrasing (and mindset it can encourage) regarding US-China competition in the letter dated 16th August ('US-China Competition on AI Chips, ...').

In general I'm very wary of messaging which could inflame a them-vs-us mindset, in short mainly because I think it a) destroys humans' ability to think sensibly and b) tends to foreclose win-win outcomes. I expect these brief points to be clear and to have rich referents in your mental pictures of the world, but please correct me if not!

Interjection: I'm referring to the vicinity of mind-killing politics

I think your letter skirted close to dangerously simplified presentation in this way. I would not normally spend my time or yours on a criticism of one section of a newsletter, but in this case I consider it worthwhile because your letter is close to the Pareto frontier on nuance, correctness, helpfulness, and reach (and it pays to try to nudge such things in good directions), but this kind of message needs to be delivered with care to avoid misunderstanding and harm.

Hopefully my pointing this out, accompanied by a few select quotes, is enough to encourage you to carefully take this criticism into account, but I'd happily expand more if you like!

Without further ado, a few quotes and my response:

The US and China have been competing for access to these chips for years.

Kind of true, but really US-based and China-based international corporations (as well as other orgs) have sought access to this scarce resource. Competition for this particular resource is mostly zero-sum across all of these entities, importantly including intra-US and intra-China.

Where market-share is near zero-sum (i.e. for direct competitors) the market-share outcomes may be greater-than-linearly zero-sum in this resource (your minor/temporary lack of chips could be my major gain of market-share), which might better warrant the term 'competing', but this effect is actually much stronger intra-country/bloc rather than inter-, due to respective markets! i.e. Google and Microsoft really care about each other's chip access in a way that they only do to a weaker degree about Alibaba's.

Interjection: To emphasise, 'the US and China have been competing' doesn't literally preclude belief in intra-bloc competition. But there's a strong implicature that intra-bloc is (relatively) unimportant, while in fact the mechanism I mentioned here increases intra-bloc competition (which I think is borne out by observation to date).

Subscribe now

When governments have paid attention, they have indeed made moves which adjust share (but also supply, as you've noted later, making it nonzero-sum, in chips at least). It's unclear (to me) exactly what incentives have motivated each move, but certainly they're not the actions of monolithic or coherent entities 'The US' and 'China'. And it's certainly not the case that where such activity changes chip share, it's collected by the acting entity. Non-governments have also made moves adjusting chip share, for example the case you cite of Nvidia (a 'US' company) deliberately rules-lawyering the US gov in order to supply more chips to various China-based companies!

Typically when nations are used as agentic subject nouns it refers to the government and/or military of said country. I don't think there's a reading of these statements in those terms which is true, and I'm not aware of any other plausible reading which is true.

---

China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.

I dislike this sentence and think it is false! Who is this 'China'? Did said unified entity carry out all of these activities? Was it coherently pursuing all of these 'several efforts' to some particular end?

Interjection: I feel that I was unkind in my tone here. The kind of claim exemplified in the letter and in other places has proved impossible for me to map to something sincerely resembling reality without caveating so much as to be essentially starting from scratch i.e. it looks like a kind of non-proposition or emotive filler. I would be very interested to hear from people who have a better parse on this to help me understand! My crux discussion below the rest of the email contains my current best attempts.

Meanwhile, the United States has struggled to build American chip manufacturing capacity, and has taken further steps to prevent Americans from investing in Chinese technology.

This one is poor for the same reasons, though not quite as bad (perhaps because the authors are American/anglophone and have a closer perspective on the nuance).

The discussion after this point in the letter is relatively good and nuanced! It names (some of) the individual orgs and companies, and makes clearer the multiplicity of others. All further references to nations as agentic subject nouns appear to be consistent with a conventional reading referring to the respective governments.

I'm interested to know how the rather good detail got paired with a rather harmful introduction and I urge you to consider the processes and thinking which gave rise to this section of the otherwise good letter.

Thanks,

Oly

This was a brief email intended to convey something I expected to be quickly-graspable with a few pointers. CAIS content suggests that they have a broader familiarity with associated facts, but it seems to be digested/compressed here in a way which is needlessly and harmfully lossy.

In particular, the statements abstract very neatly over pre-drawn boundaries (national i.e. 'US' and 'China') and furthermore assign a greater sense of coherence and agency to those abstractions than is warranted. At least some possible such statements must be true-ish (or we would not have those abstractions), but this convenient compression happens in too many conversations to be a coincidence! Said pre-existing abstraction boundaries are already salient in the information ecosystem, and laden with emotional and political baggage. This same phenomenon (it sometimes seems like a pre-written bottom line but in implicature?) appears in other publications by other orgs and in verbal conversations I've witnessed or been part of.

The fact that I, a relative governance rookie (I'm focused mainly on technical matters), struggle to rectify or understand this makes me wonder: am I missing something? Is there a relevant factor I'm unaware of? More concerningly, is there some terrible equilibrium which prevents more involved people from speaking more clearly here? I think more likely the abstractions ('US' and 'China') have a background potency which distorts perceptions and shapes how people communicate.

Possible cruxes and areas of high uncertainty

Implicit in my own discussion is the background assumption that the main concern is about possible direct or near-direct impacts4 of AI deployments by government entities (or military). This seems to me the obvious reading when people use countries as agentic subject nouns. ('China [the governmental entity] has made several efforts'5.) In this framing, I can't rectify some of the things people say with reality. But some alternative concerns might fit the bill, and if so, this is evidence that people are talking past each other, and we should aim to frame concerns more clearly!

I've never focused intently on this area, but have had a handful of conversations about this over the years, and among relevant cruxes seems to be a family of questions along the lines of

How quickly/totally/coherently could US gov/CCP capture AI talent/artefacts/compute within its jurisdiction and redirect them toward excludable destructive ends? Under what circumstances would they want/be able to do that?

People's intuitions here appear to differ a lot, and data might be hard to come by!

It seems plain that nations are not currently meaningful players in AI development and deployment, absent conspiracy-level secrecy. So to support the apparent take that they are, we may need to imagine that they could ably/rapidly become meaningful players in AI development and deployment, hence the above cruxes.

Depending on the answers to these questions, one might perceive various goings-on which happen to occur under one or other jurisdiction to have greater import on the international stage and perhaps to warrant treating national or multi-national blocs as more coherent entities than they really are at present, for the purposes of AI discussion. ('China [the government] has [allowed/encouraged made] several efforts [because eventually they will probably seize the gains/means]'6.)

Other possible cruxes, more guesswork:

1. Perhaps the concern is about indirect (e.g. economic) impacts of non-government entities' AI activities leading to some (risky) change in balance of power (between existing governments/blocs)

   • Then, abstracting references to lots of individuals and groups via their home country might be a move which is writer-intuitive, even if nonstandard and reader-confusing. ('China [the impersonal collective economic entity] has made several efforts'7.)

   • The additional step in this type of theory, namely that indirect effects cause a risky change in balance of power, should really be spelled out if it is loadbearing

   • The use of countries as agentic subject nouns is difficult to justify under this reading

2. Perhaps the concern is indeed about direct impacts, but wielded by non-government entities (who remain the major players in development and deployment of AI)

   • If so, the conversation should be about general/global resource/capabilities rather than inter-bloc 'competition'...

   • ...unless we also posit that inter-bloc non-government conflict is liable to be much worse than intra-bloc8

   • Similarly, if these are loadbearing assumptions, they really ought to be spelled out clearly

   • The use of countries as agentic subject nouns could be justified at a stretch here, but only by first spelling out the reasoning

3. Perhaps the concern is that, regardless of the actual impact of AI resources, apparent competition could lead to inflammation of traditional conflict, or weaken defenses against such inflammation

   • Then, reporting on the apparent competition via a mention and with explicit caveat, would make sense! ('China [gov/military] has [been perceived as having] made several efforts... [but the reality is more nuanced]'9.)

   • Alternatively, reporting on actual conflict could be used as evidence for the claim (that apparent competition inflames conflict), but only by also pointing to the stated or implied reasons for the conflict. ('China has made several efforts... [in each case citing US provocation as justification]'10.)

   • In either case, there are additional claims being made that can not be left implicit, and require supporting argument

Speculation

As it is, for me, the evidence seems to suggest that an AI race, if it is happening at all, is being run by (mainly US- and UK-based) companies with little or no oversight from governments or militaries. Rather, governments are in a position to collectively act to diffuse the race! And they appear as likely to do this as to exacerbate it, from my limited viewpoint.

Separately, a lack of reliable alignment techniques and performance guarantees makes AI-powered belligerent national interest plays look more like bioweapons than like nukes - i.e. minimally-excludable - and perhaps mutually-knowably so! This presently damps the incentive to go after them. But proliferation of naively-aligned AI ('figure out what I want and make it happen') might make harm plays more excludable, exacerbating lose-lose or race game dynamics ('go and steal/destroy their stuff but don't let that happen to my stuff'). This concern in part motivates consideration of multi-principal-multi-agent delegation and the cooperative AI agenda.

Summary and takeaways

Un-nuanced coverage and discussion has the potential to inflame harmful confusion and us-vs-them mentality, which diminish the chance of safe outcomes.

• China has made several efforts to preserve their chip access, including smuggling, buying chips that are just under the legal limit of performance, and investing in their domestic chip industry.

• China [the governmental entity] has made several efforts...

• China [the government] has [allowed/encouraged made] several efforts [because eventually they will probably seize the gains/means]...

• China [the impersonal collective economic entity] has made several efforts...

• [People and organisations in] China [have has] made several efforts...

• China [gov/military] has [been perceived as having] made several efforts... [but the reality is more nuanced]

• China has made several efforts... [in each case citing US provocation as justification]

• ...?

When compressing discussion of political topics, be extra wary of compression which coincidentally abstracts over already-charged us-them divides (and be careful when phrasing comes too easily, lest you write bottom lines first)! You're more likely to be wrong (because your information ecosystem biases toward thinking in these terms, and because you might be mildly-to-severely mind-killed on the matter), and being wrong is more likely to be harmful (by reinforcing those dynamics in others).11 The same vigilance applies to reading and listening.

My (not very informed) take is that governments are at this point as likely to want to defuse as to exacerbate an AI race, and those of us with any privileged insight or influence should avoid one-sided discussion of the matter (if anything preferring to focus on constructive, collaborative possibilities, the better to raise them to salience and generate common knowledge).

Most of my remarks here are somewhat weakly held (if forcefully stated) and it seems important to gather perspectives on this. Inform me! All responses will be gratefully received.

Cross-posted to EA Forum

Share

A friendly local (crimson rosella) I met in the Bunya mountains

I learned a lot from hosts, guides, museums, and simply from keeping eyes and ears open while indulging my passion for wildlife and nature while walking in the bush. But the main topic I'll touch on here is the pervasive sense of barely-controlled or out-of-control ecosystem impacts, which is felt pretty viscerally there.

Oly, don't you mainly think and write about AI and computer science? This one is mostly (superficially) about nature, but don't worry, we'll get there.

In Un-unpluggability I actually had some Australian cases in mind when I wrote:

Replication and growth (with reinvestment) get special mentions as they naturally produce exponential expansion (until constraints are reached), which in practice often manifests as first imperceptible and then rapid escalation.

Replicating systems also give rise to a kind of robustness due both to redundancy and repair. They are notoriously difficult to shut down, which is why autonomous replication is rarely a deliberate part of human designs - though we see it employed under well-understood and controlled conditions in agriculture and some industry, maliciously in computer viruses and bioweapons, and sometimes accidentally in biosphere interventions. In fact, in biological, zoological, and related sciences, great care is usually taken to avoid inadvertently unleashing autonomously replicating systems, though this remains sometimes insufficient1.

Examples:

• pandemics

• invasive species (e.g. plant weeds)

• computer viruses

• wildfire

• rumours and ideologies?

But why is great care usually taken to avoid unleashing autonomously replicating systems? Well, it's been a hard-won lesson, known intimately within some disciplines and circles, while less perceptible or understood in others. And we're still making a lot of mistakes!

Australia, the big experiment

Australia, separated from the rest of the world's continents for tens of millions of years, was in many ways a natural experiment here: humans and our water-(and later air-)transport were one of the only ways in or out for the vast majority of species. Before we really knew what we were doing, sometimes accidentally and other times misguidedly deliberately, like good little experimentalists, we humans were responsible for an unprecedentedly rapid introduction of new foreign organisms, and we got the chance to watch the effects unfold in front of us. The experiment, this introduction of foreign species was... successful, if by 'successful' you mean 'devastating and difficult or impossible to roll back'.

Lessons learned?

Replication and growth! Expansionism! Within the jurisdiction of Australia, and in the limited domain of biosphere interventions, the relevant decisionmakers appear to have taken this lesson on board. Australia has the tightest border controls for bio of any nation. In my experience it wasn't all that, given the stakes, but they did scrub my boots on the way in, in case I'd brought any nasty English countryside with me.

It's unclear to me to what extent the First Nations Aboriginal people, the original (human) custodians of Australia, had learned this lesson, which would therefore merely be being remembered or rediscovered today. The popular stereotype I encountered was of a wise and learned people living in harmony with nature prior to European incursion. Certainly in recent centuries on the whole they were... more restrained than their European and Asian cousins in the destruction of the environment. Various Aboriginal cultural practices among different groups seem to have been designed or fit for the purpose of environmental protection2, but a fair share of casualties undeniably occurred under their ancestors' watch too, notably many species which are now counted only in the fossil record and on the oldest of Aboriginal artworks. There's a reason the Thylacine, whose fossils are found throughout Australia, was known colloquially as the 'Tasmanian Tiger': by the time Europeans arrived, the only remaining population lived even more remotely on the island of Tasmania! A similar story is playing out with the 'Tasmanian Devil', also once widespread on the mainland. Were the same lessons learned, once, only to be forgotten when the balance of power shifted to the European colonists? I don't think the archaeological or cultural record is intact enough that we'll ever answer that question.

A probable Thylacine depicted in ancient art at Ubirr. nettispaghetti, Wikimedia commons

Subscribe now

Characters and critters

Besides a burgeoning defensive stance against future incursions, what mitigations are pursued?

The main means of halting expansionist systems is to remove or protect the resources used for expansion, or intervene in some other way to reduce the rate. Very commonly we are forced to simply await resource exhaustion (as with some wildfires) or learn to live with it3 (as with endemic diseases or established invasive species).

My discussion of mitigation in un-unpluggability

In no particular order, here's a shortlist of some uncontrolled introduced species I encountered on my limited excursions in Australia.

• cane toads (perhaps the most famous on the list, and spotlighted below)

• various devastating tree infections

• humans4

• dingo5

• cats and foxes

• rats and mice

• deer

• rabbits

• pigs

Not one of these has yet been brought back under control despite numerous efforts. We're well into the realm of 'learn to live with it' (though try telling that to the extinct or threatened native wildlife, Aboriginal communities whose traditional food sources have dwindled, or farmers whose crops are at risk).

Cane Toad case study: Fighting fire with fire

In the early 20th Century, sugarcane farmers became plagued by cane beetles, which eat the leaves and whose larvae destroy the canes' roots. Some enterprising genius suggested that, hey, don't toads eat beetles? So they checked, and indeed, there was such a thing as toads, and a South American species that would happily eat these particular beetles if it got the chance. (Behavioural evaluation, success!) Rather than poisoning or otherwise directly attacking the beetles, why not delegate to a small population of beetle-controlling amphibians, the toads? Apparently a few people raised concerns, but they were quickly overruled and a hundred or so toads were brought in, bred, and released, becoming known as 'cane toads'.

Bufo marinus, Eli Greenbaum, Wikimedia commons

In the mid 20th Century, sugarcane farmers were still plagued by cane beetles, which eat the leaves and whose larvae destroy the canes' roots. It turns out that cane toads don't jump or climb well, so outside of the lab, where beetles live at the top of sugar cane, the toads were all but useless at their intended purpose. (Out of context failure!)

But the toads were a success, in their own terms: unexpectedly unfussy eaters, prolific reproducers, and poisonous to most wildlife, they have rapidly colonised Queensland state, lately expanding into New South Wales and the Northern Territories, while being resilient to every attempt at pushing them back. We fought fire (cane beetles) with fire (toads), and ended up with two fires! Meanwhile, pre-existing species, especially predators unfamiliar with the toads' toxins, have been pushed toward extinction6.

This is fine

In the 21st century, a few new kinds of self-replicating 'fire' are being considered for toad control. First, toad viruses! This sounds like the start of the plot of something that ends badly, at least for other wildlife and native amphibians (which are already having a hard time) - a third fire to add to the mix?

The other, more high tech approaches involve deliberate introduction of driving genes, genes which subvert the usual 'fair' sexual recombination process resulting in much greater than 50% chance of appearing in a given offspring. If the driving gene is also one which reduces fitness in the traditional sense (for example, by being always male), the population can be reduced or even eliminated by the rapid colonisation of its gene pool by the driving gene. Various other genetic shenanigans along similar lines have been suggested. My take is that, despite sounding even more sci-fi, this driving gene approach is far less liable to go awry because it recruits, as a very robust boundary, the natural incompatibility of genetic mixing between species, in contrast to pathogens, which like to mutate and hop between hosts. On the other hand, further developing this technology may be a risky prospect indeed since it's so ripe for abuse in other contexts! We'll see how successful these approaches turn out to be as and when they are rolled out.

Incidentally, Burt and Trivers' book Genes in Conflict is a fascinating and rich dive into the often surprisingly-computational world of driving genes and other intragenomic conflict.

All of this goes to exemplify that, when unleashing autonomously replicating systems, rollback is seriously difficult. Fighting replicators with replicators usually hasn't worked so far7, but there's a first time for everything.

Casualties: endless formerlies most beautiful

Darwin visited Australia (I didn't know this until I came there). In his day, whether he was aware of it or not, we'd already lost8

• giant kangaroos

• giant 'wombats'

• giant echidnas

• giant platypus

• Genyornis (a giant flightless bird)

• Megalania (a giant predatory lizard)

• Thylacoleo (a giant predatory marsupial)

• Quinkana (a giant semi-terrestrial crocodile)

• (there's a theme here; larger creatures are often fewer in number and at more precarious positions in a disrupted food chain)

• mainland Thylacine ('Tasmanian tiger')

• mainland Sarcophilus ('Tasmanian devil')

Since Darwin's time, post-Aboriginal colonisers have worked very hard at bringing new and exciting devastation to the environment. We've since lost

• Thylacine (Tasmanian populations lost, captive individuals died without offspring)

• various bandicoots9 and bilbies

• various possums

• various wallabies and kangaroo species

• several large fruit bats

• emu relatives and various fowl

• numerous pigeons and parrots

• drop bears

These are just the ones which are front of mind for me from previous reading and after visiting a few museums. There's a longer list on Wikipedia and this is not to mention plants and marine life!

I'm not an ecology preacher. It's sad, devastating perhaps, that these rather wonderful creatures can no longer be found. But for all I know they lived terrible, brutish lives of struggle, disease and misery, and they're better off absent (I'm half joking). The main point is that, mostly, these species weren't literally slaughtered to extinction by humans - instead, humans' activities, and more often than not one or another autonomous replicator that we set in motion, sealed their fate. And not one of these effects was deliberate10.

Takeaways

Replication is extremely potent! Nature actually hits us in the face with evidence for this every day, but because it all seems so normal, we don't always take the lesson on board. Because most replicators happen to exist in states of approximate equilibrium with each other, we really only witness the power of the exponential when there's some shock to the system: a new habitat unlocked, an invasive species introduced, or a novel pathogen burning through the available resources until some new equilibrium is reached.

Large groups of well-resourced people attempting to face such replicators head-on have found it to be challenging or impossible to put the cat back in the bag. I for one have learned to take risks which route via something replicating very seriously indeed - hence the special place it gets in my list of un-unpluggabilities. Covid was just another in a list of wakeup calls on this front.

Australians have learned this lesson in the limited domain of biology, perhaps more than once! And they're taking action on it, sometimes in pretty creative and ambitious ways.

Fighting fire with fire is an amusing and audacious approach that just might work for some bio interventions, but it's domain-specific, contingent, and untested: there's no law of nature that the medicine is a mirror of the poison, so we can't expect this to work for nonbiological systems without very good reason.

I write and think about AI a lot. Let's not get into a situation where AI can even semi-autonomously replicate, without thinking very hard about the consequences, including potential outcompetition of other intelligences. It only has to happen once, somewhere, by accident, and it might be hard or impossible to walk back from.

Hertford College University Challenge team 2023, with our mascot Simpkin the cat

You can tune in on 2023-09-04 at 20:30 UK time on BBC 2 or watch or catch up online if you want to see us in action.

As a relative quiz-noob1, joining an elite quizzing team (hold your applause) was an eye-opening experience in a few ways. I'm not allowed to talk about how things went on the show (on pain of getting told off by the NDA police), but actually (as with all forms of performance and competition), the vast majority of the time was spent in prep and practice, which is where most of the insights came in anyway.

I'm going to talk a bit about University Challenge, and also gesture at how the experience as a competitive quizzer relates to broader theory and practice in decision-making under uncertainty. If you just want to see some fun quiz questions and my take at answering them, you can skip the middle Real-time calibrated decision-making section, or just skip reading this entirely and watch the show.

The format and some example questions

For readers unfamiliar with University Challenge, it's a competitive quiz, where each match consists of two teams head-to-head. Importantly for this discussion, a key part of the format is buzzer rounds ('Starter for 10'): that means you don't just have to know the answer, you have to know the answer and buzz before your opponent if they also know, otherwise you get nothing. But buzz too soon with a wrong answer and you lose points2.

Here are some example questions. Maybe you know some of the answers! If you want to, imagine hearing the question word by word - when do you have a good guess or some ideas? At what point are you confident of the answer? Would you risk buzzing early and losing points if you're wrong - and on what basis?

I'll go through these examples later, and give the answers (my realtime guesses and the actual ground truth).

What single-digit number links: the element boron, the fourth root of 625, and the planet Jupiter’s position from the Sun?

Resembling a cornet but having a slightly larger bell, which instrument is a standard in British brass bands, its name being the German for ‘wing horn’?

Rayleigh-Taylor, Kelvin-Helmholtz and Rayleigh-Bénard are all types of what general physical phenomenon, characterised by the unbounded growth of small disturbances?

Following the example of the Cadbury brothers’ model at Bourneville, which manufacturer and philanthropist developed the model village of New Earswick, north east of York?

Which art gallery links How It Is by Miroslaw Balka, Shibboleth by Doris Salcedo, Embankment by Rachel Whiteread, Marsyas by Anish Kapoor and The Weather Project by Olafur Eliasson?

In 2010, which tennis player became the seventh player to win all four Grand Slam tournaments when he defeated Novak Djokovic in the US Open men’s final?

‘Chain’, ‘double treble’, ‘reverse half double’ and ‘slip stitch’ are all terms used in which handicraft, whose name is a diminutive of the French word for ‘hook’?

Subscribe now

Real-time calibrated decision-making

Uncertainty in beliefs

A lot of theory and practice point to respecting and manipulating uncertainty as being mandatory for good truth-seeking and good decision-making. There's a lot of theory here which I won't elaborate on, but Bayes' Rule features heavily in my favourite chunks of literature3.

Bayes says that (assuming certain reasonable assumptions about how you want your beliefs to work) when you see evidence which is more or less likely under different hypotheses, you should adjust the odds of your credence in each hypothesis according to how relatively likely the evidence was, in a particular way. For hypotheses A and B, and observed evidence O4:

That is, the ratio of your credence in A and B after observing O (the ratio on the left hand side) is scaled by the ratio of the likelihood of the observation under each hypothesis.

How does this apply to competitive quizzing? We're trying to hone in on an answer to a question - but we receive the question one word (indeed, syllable!) at a time. The question, word by word: that's the evidence we receive. Our ideas about what the answer might be, the hypotheses.

Calibration

Besides representation of uncertainty, another important aspect of belief-formation and decision-making is correctly handling uncertainty.

How this is often operationalised is a notion of calibration of our uncertainties. If we express (or act in such a way as to implicitly express) a confidence level of 80% in some proposition, we are calibrated if 80% of similar cases resolve positively.

This actually gets trickier and more philosophical than we might like! What are 'similar cases'? What about confidence levels of tiny orders of magnitude, like 0.00001%? - surely we'll never be able encounter or identify sufficiently many 'similar cases' to find out if we were calibrated or not! Do we have to start reaching for notions, heavens forbid, of counterfactual possible outcomes? I don't have the answers, and as far as I know these are open questions in philosophy on one hand (descriptively) and machine learning and statistics on the other (prescriptively).

When quizzing, if I feel 80% sure of the answer, I want that to correspond to my being right about 80% of the time! Otherwise I'll make a loss in expectation, either because I buzz too confidently and get it wrong (losing points), or buzz too late and lose out to someone on the opposing team.5

Logical or computational uncertainty

There's a big hole in the Bayesian literature. Actually it's a big hole in the entire statistical literature, it's just more obvious in Bayes-land because it's more explicit.

Sometimes our uncertainty is because we haven't had long enough to think yet. Consider the digits of pi. (Presumably you know them. If not, I can tell you an effective procedure for enumerating them and you can come back when you're done.) Suppose I want to know the millionth such digit. Well, I know all the facts I need to get there. There are only ten things it can be (assuming decimal). I don't need to make any more 'observations' per se to arrive at a conclusion. But still I don't know the answer yet.

One of the interesting things about being a self-reflective computer and trying to do hard things is that you start to notice when you bump up against computational constraints - especially ones like this which aren't always obvious (or which get neglected for simplicity) the first few times theorists wade in to try to disentangle things! This is just one example where an appreciation of time constraints6 as a major determinant of effective computational procedures gives rise to interesting scientific problems and insights.

It's exactly the same when quizzing. Our brains' word-association and retrieval and evaluation and updating can only run so fast - often not as fast as the quizmaster can read the question!

As a technical puzzle, understanding this aspect of uncertainty intrigues me. I'm certainly less au fait than with the standard timeless perspective on uncertainty. There's some great work by MIRI which begins to address this, for example discussing logical uncertainty and logical (or Garrabrant) induction.

Doing this stuff fast

We don't get the opportunity to pause time after every question syllable, pull up a notepad, run some supercomputer evaluations, compute exact Bayesian posteriors, estimate our teammates' and opponents' credences and likely buzzing behaviour, and so on. Cruelly, time flows at one second per second and the quizmaster keeps quizmastering. So too in life! Our decisions in modern life might not usually be as split-second as in a head-to-head quiz, but our uncertainties (including logical uncertainty) and the costs of mistakes are just as real7.

One of the main changes resulting from practising and competing in UC was that I went from 'quiz noob with broad knowledge base and slow mental retrieval' to 'quiz rookie with broad knowledge base and slightly-less-slow mental retrieval'. One of our team in particular was a much more experienced quizzer, and an absolute master of buzzer technique! Entering the world of highly-practised quizzers gave me an appreciation for the challenges involved. I don't know how well these competences generalise, but I wouldn't be surprised if competitive quizzers (more experienced than me) would on the whole be great (or have great potential) at calibrated decision-making under uncertainty, and forecasting.

Of course, maybe I'm overanalysing this: after all, maybe general knowledge quizzes mostly come down to brute knowledge-base-retrieval. You either know or you don't! But I think the head-to-head competitive aspect brings out this mandate for fast approximate calibrated estimation: you have to eke this out sometimes, or your opponent will! I didn't expect this insight going into it, but it's given me a fresh appreciation for UC the show, and for head-to-head quizzing in general.

But at what cost?

This whole discussion sidesteps the reasons for wanting to have accurate and calibrated beliefs on the basis of limited evidence. Of course, it's fun and perhaps even virtuous and whatnot to have accurate beliefs, but usually we want them because they help us to do the good things.

This raises the question of costs: if my beliefs are feeding into my actions, and I have limited computational budget (always), it matters how much difference I expect it to make. In the case of University Challenge, for each buzzer question, there are seven outcomes, where I've indicated a rough net score for each

• (-25) lose points and other team gains points

• (-20) other team gains points

• (-5) lose points and other team gets nothing

• (0) nobody gets anything

• (5) other team loses points

• (20) gain points

• (25) other team loses points and you gain points

(As we may find out, there are also secretly other options, like '(-1000) say something embarrassing on national TV'.)

Our expectation of these outcomes depends on

• our current estimate of what the answer might be

• our estimate of our teammates' state

• our estimate of our opponents' state

• (a guess at how soon the question will end)

For these reasons, when time and/or decision-making computational resources are scarce, it can pay to make fast approximations and gut checks on all of these. I felt my own system 1 slowly incorporating some of this stuff through practice, and I expect think this is a large part of what separates really good quizzers from the rest of us!

In University Challenge in particular, the cost of an interruption is a mere 5 points, which is no big deal in the scheme of things (compared to the upside of +10-25 for a correct answer, depending on bonuses). But what it really costs is the opportunity for you, or a teammate, to reduce uncertainty about the answer - either by hearing more of the clue, or by having more time to think! This is something it took a while to internalise, and you do have to internalise it to get good in this competitive setting.

To some extent, this whole thing pattern-matches a lot of work I've done in my time as a data scientist and software engineer in industry, relating to online secret auctions: our estimate of teammates and opponents' states corresponding to their 'bid' and our own estimate corresponding to our 'true evaluation' of the item. There, as in quizzing, time is limited and computational constraints reign. High throughput and rapid constrained decision-making often trumps slow and painstaking deliberation in these contexts. I even have a hunch that some of the theory I developed there might transfer over! - especially concerning how to handle optimal bidding over a time-distributed volume in an uncertain market environment. But I've not played enough with the maths, and I'd need to check what's covered by NDA, so I won't elaborate any more.

You'll spot me doing some time-saving approximations and inversions below, as well as accounting for cost, not just for belief-updating.

Just answer the questions already!

OK here goes, in dialogue format. Remember, these are honest but hypothetical (I'm not allowed to reveal any real things from the show). I've partly but not totally biased the sample toward topics I know about8.

Quizmaster (QM): What single-digit number...

Oly (O): P(anything other than [0-9]) = 0

QM: ...links: the element Boron, ...

O: Boron is B, group 3, element 5, mass 10 (right, usually?) [thinking]

QM: ... the fourth ...

O: Gotta be 5, the proton number, group number would be a weird choice and it's subject to debate anyway because of the transition elements, buzz time...?

QM: ... root of 625 ...

O: [buzz] 5

And the whole question

What single-digit number links: the element boron, the fourth root of 625, and the planet Jupiter’s position from the Sun? (5)

Review: not bad, if I'd been more confident of calibration I could have buzzed sooner. Who knows if my opponent would have been so cautious...? If I knew they were the waiting type, I could have rested a bit and double checked the fourth power of 5, just to be really sure, or waited even more and counted 'My Very Easy Method Just'.

What about a speed superintelligence? We can imagine they suffer no or limited logical uncertainty - effectively they can pause after every syllable. Well then, 'element' would be a big clue and they could precompute a mapping from the first ten chemical elements to their proton numbers and mass numbers. Because stable mass numbers aren't uniquely defined, proton number would already get most of the posterior, and once 'bor-' has been said there's really only one possible answer.

QM: Resembling a cornet...

O: Hmm, I've played a lot in brass bands, is this going to be cornet the brass instrument or something else? P(brass instrument) = 0.7. Maybe trumpet. P(trumpet) = 0.4. Should I buzz??

QM: ... but having a slightly larger bell, ...

O: 'bell'! Got to be brass! P(trumpet) = 0.7 [NB this should be lower to follow Bayes] but uhh, what are those other ones? Piccolo trumpet, soprano cornet, ...

QM: ... which instrument is a standard in British brass ...

O: ...wait, they all have smaller bells. THINK

QM: ...bands, its name being the German

O: Flugel horn! [buzz] Flugel horn

And the whole question

Resembling a cornet but having a slightly larger bell, which instrument is a standard in British brass bands, its name being the German for ‘wing horn’? (Flugelhorn)

Review: My logical uncertainty and brain speed let me down here. But I got very lucky on actually having spent a lot of time in British brass bands. Handy coincidence. The speed superintelligence would have done better in my place, but only if it had niche knowledge of brass band instruments.

QM: Rayleigh-Taylor, ...

O: [pure association] physics or maths stuff?

QM: ...Kelvin-Helmholtz and ...

O: Definitely physics! I've heard of Rayleigh scattering, won't be Taylor polynomials, Kelvin did... temperature stuff? Helmholtz did a bit of everything. Acoustics?

QM: ...Rayleigh-Bénard are all types...

O: Rayleigh again! Scattering? But I can't confidently link the other names, and I haven't heard of these name pairings at all.

QM: ... of what general physical phenomenon, characterised by the unbounded growth of small disturbances?

O: Oh, must be chaos! That's a general physical phenomenon. Question seems to be over, teammates haven't buzzed yet... [buzz] Chaos

QM (looking disappointed): I'm afraid that's the wrong answer

And the whole question

Rayleigh-Taylor, Kelvin-Helmholtz and Rayleigh-Bénard are all types of what general physical phenomenon, characterised by the unbounded growth of small disturbances? (Instability)

Ouch. If any of the other players had better physics breadth or recall, they'd have certainly got it long before I failed to. In practice there's at least one of my teammates (cough Omer Keskin cough) who I'd expect to get this question, or to have buzzed sooner with a guess than me! At least I didn't buzz early with a wrong answer.

QM: Following the example of the Cadbury brothers' model...

O: Ooh, I did my undergrad in Birmingham and I remember some history about this

QM: ... at Bourneville, which manufacturer and philanthropist...

O: OK, Bourneville was a constructed model village for confectionary workers, designed to offer better quality of life - getting distracted. It's probably another confectioner. Rowntree, Fry, ...? P(something I didn't think of) = 0.3

QM: ...developed the model village of New Earswick, north east of York?

O: My Granny lived in York and told me this! Pretty sure it was Rowntree. Confidence? Question seems over, time to [buzz]. Rowntree.

QM: Can I ask which Rowntree...?

O: Uhhhh. [I really should have read that fun Chocolate Wars book they gave us at undergrad induction. Pick a common early C20 businessman's name.] John?

QM: I'm afraid that's the wrong answer

And the whole question

Following the example of the Cadbury brothers’ model at Bourneville, which manufacturer and philanthropist developed the model village of New Earswick, north east of York? (Joseph Rowntree (1836-1926; not to be confused with his son, Seebohm Rowntree))

Review: well, that seems harsh. I'm not sure actually on the real show if they'd have given this or not. This was basically a case of fact retrieval, though it illustrates the important issue of allocating probability/credence to 'something I didn't think of yet'.

QM: Which art gallery...

O: P(art gallery) = 1. I know hardly any, thankfully my teammates might know a few more. I remember visiting the Tate and Tate Modern as a kid. P(something I've never heard of) = 0.8.

QM: ...links How It Is by Miroslaw Balka, Shibboleth by Doris Salcedo, ...

O: Sound quite modern. P(modern) = 0.8

QM: ... Embankment by Rachel Whiteread, Marsyas by Anish Kapoor ...

O: Confidently all modern

QM: ... and The Weather Project by Olafur Eliasson?

O: Wait, that sounds familiar. It might actually be Tate Modern. Question’s over. [buzz] Tate Modern

And the whole question

Which art gallery links How It Is by Miroslaw Balka, Shibboleth by Doris Salcedo, Embankment by Rachel Whiteread, Marsyas by Anish Kapoor and The Weather Project by Olafur Eliasson? (Tate Modern)

Review: I got very lucky here (as I said, I biased my sample a bit). In practice, someone else would probably have beaten me to it by a long shot, hopefully someone on my team!

QM: In 2010, which tennis player ...

O: Oh no, sport facts. Hopefully Daniel is on it. Federer? Djokovic? Williams? Nadal? Murray?

QM: ... became the seventh player to win all four Grand Slam tournaments when he defeated Novak Djokovic in the US Open men's final?

O: Yep, no idea. P(Federer or Nadal or Murray) = 0.3. Strictly, if alone, I ought to buzz and guess something now that the question has finished. But surely a teammate has a better guess than me?

QM: Anyone? Anyone going to buzz?

O: I really should buzz and guess. But besides getting answer right, my utility function also includes not putting myself forward embarrassingly wrongly on TV. P(wrong) = 0.9.

And the whole question

In 2010, which tennis player became the seventh player to win all four Grand Slam tournaments when he defeated Novak Djokovic in the US Open men’s final? (Rafael Nadal)

Review: This awkward silence actually happens, quite rarely, but sometimes, on the show. I assume this can only be when everyone involved has a similar thought process to me at the end. A superintelligence optimised exclusively to value University Challenge performance might not have similar compunctions, but there's also no knowing what instrumental strategies it might pursue. Maybe embarrassment or some analogue would actually play a part there.

QM: ‘Chain’, ‘double treble’, ‘reverse half double’...

O: Sport? 'treble' is a musical clef? Gymnastics?

QM: ... and ‘slip stitch’ ...

O: Oh, knitting or something? I remember something about this.

QM: ... are all terms used in which handicraft

O: It has to be knitting, right? Can I afford to wait? [buzz] Knitting

QM: I'm afraid that's the wrong answer and you lose five points. ...in which handicraft, whose name is a diminutive for the French word for ‘hook’?

O: [fuming] Should have waited! It's got to be crochet.

And the full question

‘Chain’, ‘double treble’, ‘reverse half double’ and ‘slip stitch’ are all terms used in which handicraft, whose name is a diminutive of the French word for ‘hook’? (Crochet (not knitting or embroidery))

Review: I'd have thought this would be an acceptable error, but then, for my sins, I don't know much about the fine distinctions between wool-based handicrafts. Alas. This is one where the competitive uncertainty bit me, my model of the opponents' lack of buzzing interpreting it as over-caution (therefore we're in a race and I'd better not waste more time thinking) rather than appropriate caution (therefore I'd better think harder to generate alternative hypotheses in case). A speed SI version of me would have generated at least 'crochet' as an alternative, and, I expect, waited until 'name is a diminutive' which would be enough evidence to be confident.

Takeaways

This post is already far too long. You can tune in on 2023-09-04 20:30 UK time on BBC 2 or watch or catch up online if you want to see me and my team in action!

Quizzing (and more importantly the practice matches and friendlies) gave me a fresh appreciation for various decision-making concerns, theoretical and practical. Plus, it was a great laugh and a chance to meet some really intriguing and friendly people, here in Oxford as well as from other teams.

Uncertainty is powerful! Calibration is a slightly elusive concept, but an important part of using uncertainty appropriately.

Logical uncertainty is a fascinating and under-studied phenomenon - especially in time- or compute-constrained settings!

As well as getting things right, we often need to accept some tradeoff for getting things wrong in order to free up decision-making and acting resources for other uses. When you're in a head-to-head time-based competition, this bites hard.

Finally, decision-making is, ultimately, about value. If my belief doesn't make a value-difference by changing my behaviour, then I might better pay attention to other beliefs which do, even at the expense of the first belief being true. Again this only bites in a constrained context: whether constrained by evidence (for factual uncertainty) or by compute (for logical uncertainty). Since there are so many things we are terribly clueless about, this actually applies in practice all the time.

Quizzing is fun, and I haven't even got round to mentioning the arcane art of quiz-writing. A peek behind the scenes at how those particular sausages are made was also illuminating. Were it possible, quiz- and puzzle-setters have gone up even further in my estimations.

If you're a quizzer or a quiz-setter (whether with more or less experience than me), I'd love to hear if any of this resonated with you, and about your own reflections on the art of quizzing and decision-making!

In Un-unpluggability I wrote

I believe there are large risks to allowing AI systems (dangerous or otherwise) to be perceived as pets, friends, or partners, despite the economic incentives.

We are making a choice to train and deploy systems which credibly portray personhood. The choice has concerning consequences. We don’t have to make that choice, and we should prepare ahead of time as we cross this particular collection of capability thresholds.

There may be a relatively narrow window of opportunity to set directions and make preparations.

This quote is describing the widespread animist instincts of behaviourally modern humans. We’re deeply, irresistibly primed to see faces, persons, and spirits everywhere we look.

Pretend persons describes what I think is new here. Consequences, Risks, Questions includes some severe consequences if we don’t navigate this right. I really don’t know what to propose here - it’s a complex issue.

Pretend persons

Our tools, toys, and products have practically forever2 been decorated with and imbued with imaginary personhood.

Left, Der Löwenmensch, the ‘lion person’ figurine, over 30,000 years old, Dagmar Hollmann / Wikimedia Commons (license: CC BY-SA 4.0); right, a similar figurine, available at Toytown

Pretend persons in AI

Naturally, this very human urge has been applied also to AI systems since the beginning of the field.

As a large language model trained by OpenAI, I...

Cutting edge AI is importantly different here, for two reasons:

1. even contemporary systems, which are probably not persons3, have the demonstrated capacity to fool some people some of the time

2. more speculatively, artificially intelligent artefacts are the first which may actually exhibit some degree of personhood

In essence, the difference arises because it is increasingly empirically hard to tell by surface detail. These are quite credible pretences at personhood, and have the potential to be more so. I won’t address the second point here, beyond saying that a mistake in either direction could be disastrous.

Perhaps a key here is the interactive modality - even ancient chatbots of the 60s, by their apparently flexible, conversational responsiveness, convinced some users of their understanding and emotional responsiveness.

Pretend?

Pretending to be a specific person is an overt fraud, of the kind that is fairly obviously malign, apart from under some relatively precise conditions e.g. in unmistakably comedic, theatrical, critical, or artistic contexts. I won’t discuss this further here.

Emulating generic personhood per se is much more covert. All the more so because the use of some language forms (especially first person pronouns) does not on its face appear to constitute an assertion at all4! - how can ‘I think…’ be a lie?? But in fact, it carries an assertion along the lines of ‘some particular entity is originating this statement and that entity is a person’, which of course can be as false as any other proposition. Contrast an ‘encyclopedia voice’, where responses might describe objects, events, and so on without ever resorting to first person. Or consider the pre-product forms of the very same language model AI systems, which, depending on conditioning and prompting5 can be coaxed into outputting text of nearly any kind, be it monologue, dialogue, descriptive text, business accounts, or even computer code.

The same applies to semblances of other kinds, like appearance in images, audio, or video, where a product which presents a coherent user experience of a ‘this is me’ image/voice/video is psychologically very different from one which can visibly produce representations of diverse things (including multiple people or non-person objects). But the coherence, which produces the entirety of the psychological effect, is a very shallow property.6

What’s the problem?

We should be clear: a subtle lie is being told each time an AI system puts on a mask of personhood. Probably! To date, the lie is primarily conveyed by the AI, and the culprits are the human organisations which design, train, and deploy them. They can and perhaps should choose to set things up otherwise, with careful deliberation around any exceptions.

Have we not been doing the same for decades? Alexa, Google Home, and others all speak in first person language, don’t they? Why worry now? Well, before, to nearly everyone, the lie was always transparent. The convincing portrayal of personhood is what is beginning to change. More people will be fooled, more of the time.

Pay attention when AI companies themselves tell you what they’re doing. OpenAI’s ChatGPT voice mode was explicitly designed to be (my emphases):

• An approachable voice that inspires trust

• A warm, engaging, confidence-inspiring, charismatic voice with rich tone

In other words, part of the overt goal is to buy (without earning) the trust and confidence of users! Quite naturally, profits incentivise companies to put substantial resources into simulating trustworthiness when that’s cheaper than the real deal.

The trouble is, when I see (or hear) a person, by default I think I should:

• Care for their wellbeing (to some extent)

• Trust and incorporate what they say (to some extent)

• Expect them to be sensitive to roughly the kinds of preferences and concerns that humans usually have (with some individual priorities and idiosyncratic particulars)

Consequences, Risks, Questions

The obvious outcome of highly credible pretend people (persuasion, rights, autonomy, human obsolescence)

The most salient concern, for me, regards un-unpluggability arising from dependence on AI systems:

In light of recent developments in AI tech, I actually expect the most immediate unpluggability impacts to come from foundationality, and for anti-unplug pressure to come perhaps as much from emotional dependence and misplaced concern7 for the welfare of AI systems as from economic dependence

The worst case is that people in positions of influence (be they developers and engineers themselves, policymakers, independent deployers and operators of autonomous systems, or otherwise) or the public at large may be moved by a misplaced concern for the welfare of the artificial ‘person’. They might feel affection or even love toward these systems. For virtuous, empathetic reasons, we might end up with the assignment of more autonomy, rights, or affordances than safely and appropriately assigned to such systems.

With inappropriate expectations of such systems’ capabilities, and a false understanding of their goals or motivations, this could be disastrous - humanity might sign its own obsolescence notice8.

Lesser (but still concerning) persuasion

Less extremely, if some group or groups of humans retain influence over the pretend people, they are granted a new and powerful means of persuasion and manipulation of the wider human population: the first truly bespoke and responsive marketing, propaganda, or persuasion devices.

And the vulnerable interacting with such systems may be driven to derangements by trusting concocted nonsense, convincing echoes of their own input, or even deliberate deceptions (implanted by the AI’s creators or emerging accidentally from the AI’s training).

The other obvious outcome of credible pretend people (hardened hearts, harms to future actual digital people)

Unfortunately, the most readily available mitigation is developing ‘social antibodies’ of skepticism about personhood. Skepticism about intentions is entirely precedented: if I receive an email from my friend in urgent need of £10k, my heart does not go out to them (and nor does my £10k) - rather, I assume it’s a scam and my £10k can be put to better use elsewhere (though for sufficiently convincing scams, I may waste some effort in verification).

Perhaps credible pretend personhood can be overcome by similar learned skepticism. But at what cost?

We are forced to ‘harden our hearts’ - no longer can I safely and responsibly intuitively treat all impressions of credible personhood as a sign that ‘here is a being worthy of dignity and care’. Sometimes it’ll instead be an AI system which shouldn’t be granted rights, autonomy, or political consideration (probably), at least not out of benevolence9.

How does this affect our relationships? Will we begin to see other people more instrumentally, like the artificial non-people they ever more closely resemble, in a kind of ‘social autoimmune’ failure?10

How do we explain this nuance to technically underprepared people, or society at large? How do we explain this to our children, who will grow up without a reference point from the before? Children and young people already suffer harmful developmental distortion from social media. I’d like to have children some day, and, if we avoid acute disaster in the meantime, I tremble at the challenge of raising a child to wholesomely and safely navigate a world full of pretend people.

Perhaps as concerningly, though speculatively, this may weaken society’s ability to respond with appropriate compassion to future non-natural moral patients - i.e. actual digital or artificial people, should they arise by other means. Once we have become used to brushing off, in law and in custom, all apparent people who lack biological plausibility, what will be the life prospects of a person who lives on an operating system somewhere?

Hard Ban? ‘Free Energy’ of credulous population

Based on a very limited conversation with Sam Altman in London a few weeks ago, and based on public statements by OpenAI and other AI developers, I expect they might reasonably point out: if we avoid (or ban) pretend personhood now, society won’t have a chance to develop social antibodies to manipulation and so on. That’s an awful lot of free energy - the hearts of a credulous population - for a potential bad actor to exploit! It’s a valid argument. I don’t know how to navigate this.

Summing up

We are surrounded by pretend people! They are all around us in fiction, toys, and other cultural artefacts. This is fine, and somewhat lovely. But, in AI, there arise the first glimmers of widely-convincing portrayal of personhood. It may even be possible to bring bona fide artificial persons with inner lives into being! - though I strongly doubt that current systems have inner experiences (at least, not corresponding to their superficial outputs).

This is confusing, because our intuitions about people’s moral status, competence, and intentions will not carry over by default.

This is hazardous, because credulous operators could be more liable to be confused and lose oversight or control of systems, credulous society may suffer deceptions and derangements or even willingly (but inappropriately) grant rights or affordances to autonomous systems, or a stone-hearted society may overadjust, harming real relationships or even future artificial people.

What to do about it is up for debate.

Can’t we just un­plug it?

Cover photo by Kelly Sikkema on Unsplash

A few weeks ago I was in­vited to the UK FCDO to dis­cuss op­por­tu­ni­ties and risks from AI. I highly ap­pre­ci­ated the open-mind­ed­ness of the peo­ple I met with, and their ea­ger­ness to be­come in­formed with­out leap­ing to con­clu­sions. One of their key ques­tions was, per­haps un­sur­pris­ingly, ‘If it gets too dan­ger­ous, can we just un­plug it?’. They were very re­cep­tive to how I framed my re­sponse, and the en­su­ing con­ver­sa­tion was, I think, pro­duc­tive and in­for­ma­tive1. I de­parted a lit­tle more op­ti­mistic about the prospects for poli­cy­mak­ers and tech­ni­cal ex­perts to col­lab­o­rate on re­duc­ing ex­is­ten­tial risks.

Here I’ll share the sub­stance of that, hop­ing that it might be helpful for oth­ers com­mu­ni­cat­ing or think­ing about ‘sys­tems be­ing hard to shut down’, hence­forth ‘un-un­plug­ga­bil­ity’. None of this is es­pe­cially novel, but per­haps it can serve as a refer­ence for my­self and oth­ers rea­son­ing about these top­ics.

This con­trasts pretty strongly with a more tech­ni­cal dis­cus­sion of ‘off switches’ and in­stru­men­tal con­ver­gence, han­dled ad­mirably by e.g. Rob Miles and MIRI, which is per­haps the re­flex fram­ing to reach for on this ques­tion (cer­tainly my mind went there briefly): ab­sent quite spe­cific and tech­ni­cally-un­solved cor­rigi­bil­ity prop­er­ties, a sys­tem will of­ten do bet­ter at an on­go­ing task/​in­tent if it pre­vents its op­er­a­tor from shut­ting it down (which gives rise to an in­cen­tive, per­haps a mo­tive, to avoid shut­down). This per­spec­tive works well for con­vey­ing un­der­stand­ing about some parts of the prob­lem, but in my case I’m pleased we dwelt more on the me­chan­ics of un-un­plug­ga­bil­ity rather than the mo­tives/​in­cen­tives (which are re­ally a sep­a­rate ques­tion).

Both per­spec­tives are in­for­ma­tive; con­sider what you are try­ing to learn or achieve, and/​or who your in­ter­locu­tors/​au­di­ence are.

Un-un­plug­ga­bil­ity factors

Broadly, I’ll dis­cuss six classes of prop­erty which can make a sys­tem less un­plug­gable2; with each, some analo­gous ex­am­ples3, a note on ap­pli­ca­bil­ity to AI/​AGI, and a ges­ture at miti­ga­tion.

In brief

• Ra­pidity and im­per­cep­ti­bil­ity are two sides of ‘didn’t see it com­ing (in time)’

• Ro­bust­ness is ‘the act it­self of un­plug­ging it is a challenge’

• Depen­dence is ‘notwith­stand­ing harms, we (some or all of us) benefit from its con­tinued op­er­a­tion’

• Defence is ‘the sys­tem may re­act (or proact) against us if we try to un­plug it’

• Ex­pan­sion­ism in­cludes repli­ca­tion, prop­a­ga­tion, and growth, and gets a spe­cial men­tion, as it is a very com­mon and nat­u­ral means to achieve all of the above

Of course this is not com­pre­hen­sive, the prop­er­ties can come in de­grees, and far from re­quiring all prop­er­ties, a sys­tem with suffi­cient of any of these prop­er­ties can be­come very un-un­plug­gable. The main point is that there are plau­si­ble paths to AI sys­tems gain­ing any or all of these prop­er­ties at least, so the most re­li­able miti­ga­tion is to work hard at avoid­ing build­ing sys­tems which are not un­plug­gable in the first place.

One an­gle that isn’t very fleshed out is the coun­terques­tion, ‘who is “we” and how do we agree to un­plug some­thing?’ - a lit­tle on this un­der Depen­dence, though much more could cer­tainly be said.

Fi­nally I’ll share a lit­tle thought on when and why we might ex­pect these fac­tors to ar­rive in AI sys­tems in un-un­plug­ga­bil­ity in­cen­tives and ex­pec­ta­tions.

Subscribe now

Ra­pidity (of gains in power)

This one is pretty straight­for­ward: if some­thing gets pow­er­ful or im­pact­ful fast enough, you can’t re­act in time to turn it off, even if you in prin­ci­ple have the nec­es­sary ac­cess and ca­pac­ity to do so.

Ex­am­ples (ob­jec­tively ‘fast’):

• ex­plo­sive de­vices mid-detonation

• flash floods

• me­te­orites?

Ex­am­ples (some­what ob­jec­tively ‘slower’ but pit­ted against slower re­ac­tion times):

• pandemics

• pos­i­tive cli­mate feed­back loop?

• mod­ern hu­man so­ciety? (from the per­spec­tive of ‘na­ture’)

The most clas­sic analogue in dis­cus­sions of AI is the hy­po­thet­i­cal re­cur­sive (self)-im­prove­ment, or ‘in­tel­li­gence ex­plo­sion’: if a sys­tem of AI(s) be­comes ca­pa­ble enough that it can con­tribute to progress in AI ca­pa­bil­ities, this may lead to a feed­back of rapidly in­creas­ing gains in in­tel­li­gence, with very un­clear (ex ante) rates of progress and no guaran­tee of de­sir­able ends. Separately, some dis­cus­sions point to the step-changes in many met­rics of in­fluence which hu­man­ity sees in its own his­tory (e.g. such rev­olu­tions as the cog­ni­tive, agri­cul­tural, sci­en­tific, or in­dus­trial) as ev­i­dence that there may be hard-to-fore­see thresh­olds of ca­pa­bil­ity which lead to com­par­a­tively very rapid gains (whether these are achieved by feed­back pro­cesses or oth­er­wise).

Leav­ing aside these plau­si­ble feed­back or thresh­old effects, hu­man en­g­ineer­ing effort alone has gen­er­ated quite fast ex­po­nen­tial growth in com­put­ing power over the last decades (com­pare Moore’s Law), and in­vest­ment in AI and ma­chine learn­ing has out­stripped even that by many mea­sures.

There is no real miti­ga­tion here be­sides an­ti­ci­pat­ing and build­ing cau­tiously—in­vest­ing in or­gani­sa­tional and so­cietal in­sight and gov­er­nance ca­pac­ity could help with this. Pro­vid­ing more rapid shut­down mechanisms (for ex­am­ple of servers or dat­a­cen­tres) may be a minor pal­li­a­tive.

Im­per­cep­ti­bil­ity (of gains in power or of harm­ful ends)

Failure to per­ceive power or im­pact­ful­ness gains un­til ‘too late’ is com­ple­men­tary to ra­pidity of those gains—if you are fore­sight­ful enough you can per­ceive things sooner and make up for lack of speed (con­sider ad­vance de­tec­tion of po­ten­tial me­te­orites), and if you are fast enough you may have time to make up for failure to per­ceive things in ad­vance (con­sider a quick-think­ing or lucky es­cape from a flash flood). In this way, ra­pidity and im­per­cep­ti­bil­ity of power gains can be con­sid­ered two sides of the same coin.

Failure to per­ceive harm­ful ends is slightly differ­ent. Whether a sys­tem is ‘de­liber­ately ori­ented at’ un­de­sired ends, or sim­ply by its na­ture con­sti­tuted to bring them about, if we fail to per­ceive the dan­ger, we may be en­tirely aware of its power and im­pact with­out be­ing moved to turn it off un­til too late.

Ex­am­ples:

• coups (mil­i­tary or oth­er­wise) in­volv­ing treachery

• la­tent pathogen reservoirs

• harms to so­cial fabric and episte­mol­ogy from ubiquitous con­tem­po­rary so­cial me­dia?

The on­go­ing dis­cus­sion and re­search top­ics around AI de­cep­tion, ‘treach­er­ous turns’, and goal mis­gen­er­al­i­sa­tion demon­strate that im­per­cep­ti­bil­ity of harm­ful ends is a live con­cern for AI. The in­scrutabil­ity of con­tem­po­rary deep learn­ing sys­tems must sep­a­rately be em­pha­sised: even for net­works with mil­lions of pa­ram­e­ters, let alone billions or trillions, our cur­rent abil­ity to un­der­stand the me­chan­ics of learned al­gorithms is in­suffi­cient to de­tect the pres­ence of ends, goals, or in­ten­tions, much less their spe­cific na­ture.

Indi­rect effects, net­work effects, and in­ter­ac­tion effects on the po­ten­tial for harm­ful ends of AI have re­ceived some at­ten­tion (es­pe­cially by anal­ogy to so­cial dilem­mas and to the ob­served effects of so­cial me­dia), though rel­a­tively less. Com­plex sys­tems, and sys­tems with feed­back, can be no­to­ri­ously difficult to study and to pre­dict, whether or not the in­di­vi­d­ual com­po­nents are rel­a­tively well-un­der­stood, hence the challenges faced in fields such as eco­nomics, con­trol the­ory, biol­ogy, so­ciol­ogy, and oth­ers. There­fore, we might ex­pect there to be, de­pend­ing on the de­ploy­ment sce­nar­ios of AI sys­tems, a sub­stan­tial fur­ther challenge be­yond un­der­stand­ing and ver­ify­ing the in­di­vi­d­ual com­po­nent(s).

An im­por­tant tech­ni­cal note here is that, with­out mechanis­tic un­der­stand­ing of an al­gorithm’s work­ings, it is math­e­mat­i­cally im­pos­si­ble to provide gen­eral guaran­tees merely by ob­serv­ing be­havi­our as with a black box (and even granted a mechanis­tic un­der­stand­ing, there are feed­back and net­work effects to ac­count for). Per­cep­ti­bil­ity is rel­a­tive to the per­ceiver. We can im­prove it by re­search­ing and de­vel­op­ing in­ter­pretabil­ity tool­ing for sys­tems which are cur­rently in­scrutable, or by limit­ing high-im­pact de­ploy­ments to sys­tems which are more ex­plain­able—both a tech­ni­cal and a gov­er­nance con­cern. Re­search into goal mis­gen­er­al­i­sa­tion and de­cep­tive AI aims to elu­ci­date and miti­gate this is­sue.

The study of com­plex sys­tems and more speci­fi­cally the field of col­lab­o­ra­tive AI may shed light on the feed­back and net­work effects.

Ro­bust­ness (re­dun­dancy)

Among en­g­ineers of soft­ware sys­tems, it is (rightly) con­sid­ered a dev­as­tat­ing crit­i­cism of a de­sign to ob­serve that it has any ‘sin­gle point of failure’ (SPOF). Depend­ing on the ap­pli­ca­tion, SPOFs may be tol­er­ated, be en­g­ineered around, or, when the cost of failure is deemed un­ac­cept­able, prompt po­ten­tially ex­pen­sive re­designs to in­cor­po­rate re­dun­dancy, fault tol­er­ance, er­ror check­ing, and so on4. We even ex­plic­itly dis­cuss how to en­sure sys­tem up­time if some­one were to liter­ally pull the power sup­ply (de­liber­ately or oth­er­wise) on some of our ma­chines! Such con­sid­er­a­tions are a large part of the re­spon­si­bil­ity of a soft­ware en­g­ineer or ar­chi­tect, and I un­der­stand the same to be true in other en­g­ineer­ing dis­ci­plines (though I can not speak from ex­pe­rience).

Ex­am­ples:

• the in­ter­net and other high-availa­bil­ity tech­nolo­gies and platforms

• mil­i­tary com­mand structures

• de­cen­tral­ised insurgencies

• biolog­i­cal tis­sues and organs

• colo­nial or­ganisms like ants

The ex­is­tence of mul­ti­ple points of failure can make a sys­tem harder to un­plug in two ways: first, the challenge of lo­cat­ing and track­ing each point of failure, and sec­ondly the com­men­su­rately in­creased effort of tar­get­ing each point.

With sys­tem re­li­a­bil­ity and up­time such a core en­g­ineer­ing con­sid­er­a­tion, we may ex­pect AI sys­tems to con­tinue to be built with such prop­er­ties in mind. Even in the ab­sence of hu­man de­sign, a mis­al­igned AGI would pre­sum­ably have no trou­ble at least iden­ti­fy­ing the use­ful­ness of such ro­bust­ness, though im­ple­men­ta­tion is an­other mat­ter. Such soft­ware so­phis­ti­ca­tion ap­pears to be out of reach for cur­rent AI sys­tems, for now.

It is un­clear how to miti­gate this.

Aside on re­pair, er­ror cor­rec­tion, course-correction

Other as­pects of ro­bust­ness are rele­vant to the con­sid­er­a­tion of AI, in­clud­ing sys­tems with re­pair, er­ror cor­rec­tion, and course cor­rec­tion, all of which are seen in sys­tems both nat­u­ral and ar­tifi­cial. Th­ese are less per­ti­nent to literal un­plug­ga­bil­ity per se, but cer­tainly re­late to the challenges of dis­rupt­ing a sys­tem in gen­eral.

Depen­dence (col­lat­eral)

When we re­fer to sys­tems or or­gani­sa­tions as ‘too big to fail’, what we usu­ally mean is, ‘too big to be per­mit­ted to fail with­out sub­stan­tial col­lat­eral dam­age’. What these sys­tems have in com­mon is that, due to their util­ity and scale, they be­come foun­da­tional to other things of value such that their re­moval would (with some de­gree of plau­si­bil­ity, or with­out ex­pen­sive miti­ga­tion) dam­age those other valuables.

Ex­am­ples:

• large banks

• sup­ply chain infrastructure

• en­ergy pro­duc­tion systems

• telecoms

• man­u­fac­tur­ing or par­tic­u­lar in­dus­trial processes

• at­mo­spheric oxy­gen from pho­to­syn­the­sis?5

• ozone?

Such foun­da­tional sys­tems are harder to un­plug for two main rea­sons. First, the col­lat­er­ally dam­aged valuables make un­plug­ging straight­for­wardly less de­sir­able. Per­haps more im­por­tantly, the col­lat­eral is valuable to some­one, whether it’s a liveli­hood or es­sen­tial, a way of life, a good, a lux­ury, or a com­fort. That per­son has a de­gree of in­cen­tive to act against any at­tempt to re­move it. This raises the ques­tion, ‘who is the “we” that in­tends to un­plug the sys­tem?’, and can bring tricky col­lec­tive ac­tion prob­lems into view. This ap­pears to bite even with rel­a­tively clearcut cases like cli­mate risk and en­ergy pro­duc­tion, and the greater the am­bi­guity (as with pan­demic risks or AI risks), the harder to gain con­sen­sus about trade­offs and ex­ter­nal­ities, which is one first step to­ward re­solv­ing such col­lec­tive ac­tion prob­lems.

It’s hard to pre­dict how sub­stan­tially and in what ways AI or AGI sys­tem(s) will be­come foun­da­tional, but in­creas­ing gen­er­al­ity and ca­pa­bil­ity has so far given rise to in­creased de­ploy­ment and per­haps de­pen­dency.

Find­ing ways to avoid de­pen­dence on in­creas­ingly ca­pa­ble and un­proven sys­tems could counter this risk—al­ter­na­tively, build­ing ca­pac­ity and plan­ning for ways to at­ten­u­ate or re­lieve col­lat­eral harms when it be­comes nec­es­sary to shut down a dan­ger­ous sys­tem.

Defence (ac­tive, re­ac­tive, de­ter­rent)

Weaponry, cy­ber ca­pa­bil­ities, pro­pa­ganda and pub­lic re­la­tions, le­gal and nor­ma­tive pro­tec­tion. Th­ese ca­pac­i­ties can be proac­tively de­ployed, or held in re­serve ready to re­act to any at­tempt to dis­able a sys­tem (in­clud­ing as a de­ter­rent).

Ex­am­ples:

• large companies

• governments

• mil­i­taries, in­sur­gents, paramilitaries

• dan­ger­ous animals

Com­pu­ta­tional sys­tems may demon­strate an ad­van­tage in re­search and de­vel­op­ment of chem­i­cal and biolog­i­cal weaponry—this is per­haps the most straight­for­ward route to a mis­al­igned AGI ac­quiring mas­sive de­struc­tive po­ten­tial. In ad­di­tion, con­ven­tional weaponry is in­creas­ingly au­to­mated or au­tomat­able: mass de­struc­tion has been available in the form of nu­clear weapons for some time, and tar­geted de­struc­tion is in­creas­ingly fea­si­ble us­ing drones or similar tech—ap­pro­pri­a­tion of these may vary in means and difficulty but broadly in­clude crack­ing cy­ber­se­cu­rity or in­fluenc­ing key hu­man op­er­a­tors. Con­tent gen­er­a­tion and per­sua­sive mes­sag­ing ap­pear to be fea­si­ble or near-fea­si­ble for to­day’s AI sys­tems, such as could be de­ployed in pro­pa­ganda. It re­mains to be seen what laws and norms will arise around use of AI, rights or re­spon­si­bil­ities of AI de­vel­op­ers and of the sys­tems them­selves—these might be able to be wielded in defence or offence by cor­po­rate-like AI, net­works of AI, or or­gani­sa­tions in­clud­ing AI sys­tems.

There are not ob­vi­ous tech­ni­cal miti­ga­tions in this area. Per­haps the most tech­ni­cal would in­clude efforts to ro­bus­tify or­gani­sa­tional- and cy­ber-se­cu­rity best prac­tices across weapons-en­abled or­gani­sa­tions, globally. Re­search and in­vest­ment in de­tec­tion and pre­ven­tion of novel pathogens may provide some defence against bioweapons. Be­sides this, de­pro­lifer­a­tion of weapon stock­piles, and of the means to pro­duce them (es­pe­cially novel biolog­i­cal and chem­i­cal sub­stances), might re­duce the scale and chance of an ap­pro­pri­a­tion. Mis­guided or oth­er­wise, as­sign­ing con­trol of weapons to AI sys­tems ob­vi­ously nul­lifies the challenge of ac­quiring con­trol of weapons for those par­tic­u­lar AI sys­tems. Whether this ends well de­pends on what those AI sys­tems do with the weapons, of course.

Whether it is over­all safer to train ma­chine learn­ing al­gorithms with ac­cess to hu­man-spe­cific data is un­clear, but cer­tainly it makes the job of cre­ative and per­sua­sive con­tent gen­er­a­tion much more straight­for­ward. Norms and laws around im­per­son­ation of hu­mans (in gen­eral, or of par­tic­u­lar hu­mans) will ev­i­dently have strong in­fluence over the pro­pa­ganda po­ten­tial for AI, as will pro­cesses for de­tect­ing and sig­nify­ing gen­er­ated con­tent, and the broader ques­tions of hu­man-AI in­ter­ac­tion modal­ities and in­ter­faces.

Ex­pan­sion­ism (repli­ca­tion, prop­a­ga­tion, growth)

Repli­ca­tion and growth (with rein­vest­ment) get spe­cial men­tions as they nat­u­rally pro­duce ex­po­nen­tial ex­pan­sion (un­til con­straints are reached), which in prac­tice of­ten man­i­fests as first im­per­cep­ti­ble and then rapid es­ca­la­tion.

Repli­cat­ing sys­tems also give rise to a kind of ro­bust­ness due both to re­dun­dancy and re­pair. They are no­to­ri­ously difficult to shut down, which is why au­tonomous repli­ca­tion is rarely a de­liber­ate part of hu­man de­signs—though we see it em­ployed un­der well-un­der­stood and con­trol­led con­di­tions in agri­cul­ture and some in­dus­try, mal­i­ciously in com­puter viruses and bioweapons, and some­times ac­ci­den­tally in bio­sphere in­ter­ven­tions. In fact, in biolog­i­cal, zo­olog­i­cal, and re­lated sci­ences, great care is usu­ally taken to avoid in­ad­ver­tently un­leash­ing au­tonomously repli­cat­ing sys­tems, though this re­mains some­times in­suffi­cient6.

Ex­am­ples:

• pandemics

• in­va­sive species (e.g. plant weeds)

• com­puter viruses

• wildfire

• ru­mours and ide­olo­gies?

The ob­vi­ous tech­ni­cal prop­erty to note about AI sys­tems is the in­her­ent copy­a­bil­ity of digi­tal soft­ware and data, and for many types of al­gorithm the in­her­ent scal­a­bil­ity of ca­pa­bil­ity with ac­cess to more/​faster com­put­ers7. From the ‘rogue AI’ per­spec­tive it is straight­for­ward to see repli­ca­tion be­ing an early strate­gic con­sid­er­a­tion. For ‘pre-rogue’ or ‘am­bigu­ously rogue’ AI, the in­her­ent copy­a­bil­ity of soft­ware also means that hu­man ac­tors are li­able to repli­cate AI sys­tems.

Beside repli­ca­tion per se, we have more gen­eral prop­a­ga­tion and growth. Com­pa­nies and in­dus­tries ex­pand, both into new and re­lated lines of busi­ness. A suffi­ciently au­tonomous AI sys­tem, or one par­ti­ci­pat­ing in a cor­po­rate-like en­tity could do the same.

The main means of halt­ing ex­pan­sion­ist sys­tems is to re­move or pro­tect the re­sources used for ex­pan­sion, or in­ter­vene in some other way to re­duce the rate. Very com­monly we are forced to sim­ply await re­source ex­haus­tion (as with some wild­fires) or learn to live with it8 (as with en­demic dis­eases or es­tab­lished in­va­sive species). Un­prece­dently so­phis­ti­cated an­tivirus pro­grams or eval­u­a­tion/​cer­tifi­ca­tion meth­ods may provide some pro­tec­tion, but for com­put­ers, the quan­tity of highly-net­worked and in prin­ci­ple ac­cessible re­sources is very large, and it is un­clear how we could in­ter­vene, as­sum­ing we de­tected an AI sys­tem au­tonomously repli­cat­ing. Li­cens­ing and mon­i­tor­ing of com­pute re­sources might offer some av­enue to con­trol this, but by the time an au­tonomous repli­ca­tor is at large, this is prob­a­bly too lit­tle too late.

Im­prove­ments to cy­ber­se­cu­rity prac­tices and or­gani­sa­tional se­cu­rity on the one hand, and changes to re­search clo­sure norms on the other, may af­fect the pro­lifer­a­tion of po­ten­tially-dan­ger­ous AI sys­tems by hu­mans.

Un-un­plug­ga­bil­ity in­cen­tives and expectations

Of course, the very fact that un-un­plug­ga­bil­ity can be in­creased by these and other prop­er­ties gives an in­cen­tive to any sys­tem (or sys­tem de­signer) to achieve these. Hence we see or­ganisms, pro­cesses, hu­man or­gani­sa­tions, and hu­man-de­signed de­vices ex­hibit­ing all of these prop­er­ties in one shape or an­other.

In the case of ro­bust­ness, there is a clear in­cen­tive for de­sign­ers and de­vel­op­ers to im­bue their sys­tems with this prop­erty, and more or less similarly for ra­pidity and de­pen­dence, at least while de­vel­op­ers are in­cen­tivised to com­pete over mar­ket share in de­ploy­ments.

In light of re­cent de­vel­op­ments in AI tech, I ac­tu­ally ex­pect the most im­me­di­ate un­plug­ga­bil­ity im­pacts to come from col­lat­eral, and for anti-un­plug pres­sure to come per­haps as much from emo­tional de­pen­dence and mis­placed con­cern9 for the welfare of AI sys­tems as from eco­nomic de­pen­dence—for this rea­son I be­lieve there are large risks to al­low­ing AI sys­tems (dan­ger­ous or oth­er­wise) to be per­ceived as pets, friends, or part­ners, de­spite the eco­nomic in­cen­tives.

For im­per­cep­ti­bil­ity, defence, and ex­pan­sion­ism, there is a definite in­cen­tive for a sys­tem to de­velop these it­self, though per­haps a more mixed in­cen­tive for the de­vel­op­ers—we might land with them any­way through mis­takes, mal­ice, or the in­her­ent in­scrutabil­ity of deep learn­ing, but oth­er­wise these ap­pear more likely to ar­rive af­ter situ­a­tion­ally-aware AGI.

Conclusion

We dis­cussed six prop­er­ties of sys­tems which can make them hard to ‘un­plug’, namely

• Rapidity

• Imperceptibility

• Robustness

• Dependence

• Defence

• Expansionism

where ex­pan­sion­ism gets a spe­cial men­tion for of­ten giv­ing rise to the oth­ers, and for be­ing es­pe­cially difficult for hu­mans to com­bat.

It is tempt­ing to rea­son about AI within a frame of sim­ple pro­grams run­ning on a lap­top, but mod­ern im­pact­ful soft­ware sys­tems are more of­ten large, com­plex and net­worked. We touched on some ways of re­lat­ing con­tem­po­rary and fu­ture AI sys­tems to the six un-un­plug­ga­bil­ity prop­er­ties men­tioned.

Co­op­er­a­tion be­tween tech­ni­cal ex­perts, policy lead­ers, de­vel­op­ers, and the pub­lic will be needed to eval­u­ate and pre­vent these prop­er­ties from aris­ing in AI; I’m cau­tiously op­ti­mistic that such co­op­er­a­tion can be achieved, but it will take sus­tained and cre­ative effort from many stake­hold­ers.

I re­mind read­ers that this should not be con­sid­ered a com­pre­hen­sive sum­mary, and that these and other po­ten­tial fac­tors are in­di­vi­d­u­ally suffi­cient to lend a sys­tem un-un­plug­ga­bil­ity, rather than be­ing re­quired all at once.

My thanks to Sam Brown for feed­back on read­abil­ity and ordering

We pre­vi­ously dis­cussed a con­cep­tual al­gorith­mic break­down of some as­pects of goal-di­rected be­havi­our with the in­ten­tion of in­spiring in­sights and clar­ify­ing thought and dis­cus­sion around these top­ics.

The ex­am­ples pre­sented here in­clude some origi­nal mo­ti­vat­ing ex­am­ples, some used to re­fine the con­cepts, and oth­ers drawn from the menagerie af­ter the con­cepts were mostly re­fined1. Each ex­am­ple is sub­jected to the anal­y­sis, in sev­eral cases draw­ing out novel in­sights as a con­se­quence.

Most of these ex­am­ples, for all their in­tri­cacy in some cases, are rel­a­tively ‘sim­ple’ as de­liber­a­tors, and I am quite con­fi­dent in the ap­pli­ca­bil­ity of the fram­ing. Anal­y­sis of more de­rived and so­phis­ti­cated de­liber­a­tive sys­tems is re­served for up­com­ing posts.

Brief frame­work summary

We de­com­pose ‘de­liber­a­tion’ into ‘pro­posal’, ‘pro­mo­tion’, and ‘ac­tion’.

• Propose:S→Δ{X}nonempty (gen­er­ate can­di­date pro­pos­als)

• Promote:S→{X}→{V} (pro­mote and de­mote pro­pos­als ac­cord­ing to some crite­rion)

• Act:{X×V}→A (take out­come of pro­mo­tion and de­mo­tion to ac­tivity in the en­vi­ron­ment)

We also iden­tify as im­por­tant whether a de­liber­a­tor’s ac­tions are fi­nal, or give rise to rele­vantly-al­gorith­mi­cally-similar sub­se­quent de­liber­a­tors (iter­a­tion and repli­ca­tion), or cre­ate or oth­er­wise con­di­tion het­ero­ge­neous de­liber­a­tors (re­cur­sive de­liber­a­tion).

Re­ac­tion examples

Chem­i­cal systems

In­nu­mer­able ba­sic chem­i­cal re­ac­tions, like ox­i­da­tion of iron, in­volve ac­tions which change the com­po­si­tion or con­figu­ra­tion of some ma­te­rial(s). For the pur­poses of this anal­y­sis these are, alone, mostly un­in­ter­est­ing, but serve to illus­trate nat­u­ral sys­tems which do not pre­serve their es­sen­tial al­gorith­mic form and thus do not con­sti­tute iter­ated sys­tems.

Some re­ac­tions, on the other hand, in­volve catal­y­sis, wherein some reagents are es­sen­tially pre­served or re­con­sti­tuted in the ac­tion, pro­duc­ing the seeds of iter­ated al­gorith­mic re­ac­tion, thus ba­sic ‘con­trol’.

De­spite push­ing in a par­tic­u­lar ‘goal’ di­rec­tion, these sys­tems are re­ac­tions rather than proper de­liber­a­tions, be­cause they oc­cur with­out com­put­ing al­ter­na­tive path­ways2.

Biolog­i­cal systems

Even very sim­ple or­ganisms can re­act (that is, act or perform some func­tion in re­sponse) to stim­uli. No proper de­liber­a­tion need be in­volved, no com­pu­ta­tion in­stan­ti­ated to con­sider al­ter­na­tives.

A re­ac­tion can take place with or with­out a brain, or even a ner­vous sys­tem: con­sider many mo­tions of sin­gle-cel­led or­ganisms3 or the snap­ping of a Venus fly­trap. The cringe of many an­i­mals from in­tense heat goes via ner­vous cir­cuitry but takes no de­liber­a­tion.

In­deed, tem­per­a­ture changes are per­va­sive in na­ture, so it is no sur­prise that we find au­to­matic heat-re­spon­sive be­havi­our at the pro­tein-ma­chin­ery level in ev­ery lineage of cel­lu­lar life. Th­ese lat­ter, along with other pro­tein ma­chin­ery and the or­gan-func­tions of mul­ti­cel­lu­lar or­ganisms, demon­strate that, in na­ture, a sin­gle or­ganism will be found to con­sist of many re­ac­tive and de­liber­a­tive sys­tems.

The class of ‘sys­tems un­der­go­ing a trans­for­ma­tion’ in chem­i­cal or phys­i­cal in­ter­ac­tions of­ten does not pre­serve the es­sen­tial al­gorith­mic char­ac­ter­is­tics of the sys­tem, but most salient biolog­i­cal ex­am­ples are iter­ated (the act es­sen­tially pre­serves the ca­pac­ity to fur­ther act in an al­gorith­mi­cally similar way). This is not sur­pris­ing given their ori­gin by nat­u­ral se­lec­tion. (Some con­text-spe­cific coun­terex­am­ples ex­ist, like some cases of one-shot au­to­tomy to dis­tract preda­tors, though this works pre­cisely be­cause in so-do­ing, an­other de­liber­a­tor, the gene com­plex cod­ing for au­to­tomy, some­times thereby pre­serves and prop­a­gates its es­sen­tial form).

Genes and sys­tems-pro­duc­ing-systems

In many bio­chem­i­cal cases, it may be most ap­pro­pri­ate to think of ac­tive genes and gene-com­plexes as highly iter­ated ac­tors, some­times liter­ally im­ple­ment­ing clas­si­cal con­trol analogues. Their prod­ucts, the pro­teins, cells, or­gans and or­ganisms, can be seen as the me­di­a­tors of the genes’ ac­tions. Some of these me­di­a­tors are them­selves, or give rise to, de­liber­a­tors or con­trol­lers.

The pri­mor­dial ge­netic repli­ca­tors, in the RNA world or even ear­lier, made lit­tle use of me­di­at­ing sys­tems, but nowa­days they pre­dom­i­nantly build cel­lu­lar and mul­ti­cel­lu­lar ve­hi­cles which are re­spon­si­ble for most of their replica­tive suc­cess. Note that ‘cre­at­ing’ is just a spe­cial case of ‘con­di­tion­ing’, and, once cre­ated, cells, tis­sues, and or­gans are available to be con­di­tioned as me­di­a­tors of other genes, cells, tis­sues, and or­gans’ ac­tivity.

Of course genes are them­selves prod­ucts and ob­jects of nat­u­ral se­lec­tion, so we have sys­tems-con­di­tion­ing-sys­tems-con­di­tion­ing-sys­tems.

Ar­tifi­cial reactions

Many ar­tifi­cial sys­tems take re­ac­tive ac­tions too. A ther­mo­stat ad­justs with­out con­sid­er­ing what it might oth­er­wise do, and similar con­trol sys­tems have been in use for cen­turies at least.

Most con­tem­po­rary pre­dic­tive sys­tems, most gen­er­a­tive sys­tems, a few game-play­ing, and many robotics sys­tems pro­duce their out­puts re­ac­tively—the al­gorithm is not eval­u­at­ing mul­ti­ple pro­pos­als, it just pro­poses some sin­gle ‘idea’ (of vary­ing qual­ity).

There are many con­tem­po­rary AI sys­tems with rudi­men­tary proper de­liber­a­tion as well, and some un­cer­tain cases, dis­cussed be­low.

Many pre­dic­tive AI sys­tems, e.g. image clas­sifiers, come clos­est to be­ing ‘non iter­ated’ be­cause their re­sults have least bear­ing on their fu­ture in­vo­ca­tions (though be­ing more or less ap­par­ently use­ful to hu­man op­er­a­tors does af­fect this).

Gra­di­ent de­scent as reaction

A sin­gle step of gra­di­ent de­scent is a re­ac­tive mo­tion. A pow­er­ful heuris­tic—com­pute (an es­ti­mate of) the gra­di­ent of some tar­get func­tion at a point—pro­duces ex­actly one pro­posed up­date (di­rec­tion). The up­date is de­gen­er­ately pro­moted and acted upon (ap­plied) in a sin­gle fused mo­tion. (In some cases minor proper de­liber­a­tion may be in­tro­duced by eval­u­at­ing mul­ti­ple can­di­date step sizes.) Vari­a­tions like mo­men­tum, and higher-or­der gra­di­ent meth­ods like New­ton-Raph­son, have the same top-level anal­y­sis (though their qual­ity as con­trol­lers may differ).

Iter­at­ing steps like this re­sults in a con­trol­ler di­rected to­ward ever lower scores4.

What makes gra­di­ent de­scent so good?

De­spite car­ry­ing out no proper de­liber­a­tion, iter­ated gra­di­ent de­scent is ca­pa­ble of some­what-re­li­able goal-di­rected con­trol. In fact, un­der the right con­di­tions, it can pro­duce re­sults just like iter­ated nat­u­ral se­lec­tion which is, in con­trast, a canon­i­cal prop­erly de­liber­a­tive pro­cess (see be­low). How so?

An es­sen­tial take­away from this anal­y­sis is that effec­tive eval­u­a­tion and pro­mo­tion are only one part of de­liber­a­tion strength. The other part is com­ing up with good pro­pos­als. Gra­di­ent de­scent hap­pens to use a strong heuris­tic5 for gen­er­at­ing good pro­pos­als—good enough to en­tirely com­pen­sate for de­gen­er­ate de­liber­a­tion, at least when com­pared to a rele­vantly-similar nat­u­ral se­lec­tion.

Ac­cu­rate con­tour ren­di­tion of a loss land­scape. For nav­i­gat­ing to the bot­tom-right basin from the marked point, a gra­di­ent step which is pushed away by the steep slope will even­tu­ally sub­se­quently route around, but bet­ter steps are pos­si­ble.

Note that in spite of this clever pro­posal heuris­tic, bet­ter-still con­trol­lers for the same do­main are con­ceiv­able (though not, per­haps, easy to find or im­ple­ment), even con­stricted to re­ac­tive de­liber­a­tion and a fixed learn­ing-rate sched­ule: lo­cal best-de­scent is only the best pos­si­ble in­tu­ition on a globally-lin­ear sur­face (which is nonex­is­tent in prac­tice). Similarly, ap­pro­pri­ately-adapt­able learn­ing-rates can pro­ceed more effi­ciently. This is part of the mo­ti­va­tion be­hind such tech­niques as mo­men­tum, trust re­gion op­ti­mi­sa­tion, and adap­tive hy­per­pa­ram­e­ter tun­ing.

Mar­ket arbitrage

At a differ­ent level of or­gani­sa­tion, ar­bi­trage be­tween suffi­ciently liquid mar­kets is an ex­am­ple of an emer­gent re­ac­tive sys­tem which serves to push to­ward price con­ver­gence, along some­thing akin to a gra­di­ent.

The es­sen­tial form of this ten­dency is not dis­turbed by its own ac­tion, so it is iter­ated: a con­trol­ler.

As a ro­bust agent-ag­nos­tic pro­cess this re­ac­tion is con­sti­tuted by the in­ter­ac­tions be­tween many in­di­vi­d­ual ac­tors, but is ro­bust to many of the par­tic­u­lars of the be­havi­our of in­di­vi­d­u­als or sub­sets of in­di­vi­d­u­als. It serves as an ex­am­ple of a (re­ac­tive) con­trol­ler at the multi-or­ganism level. Even though the in­di­vi­d­u­als en­act­ing the ar­bi­trage may be do­ing so very prop­erly de­liber­ately, this does not make the ar­bi­trage con­trol pro­cess it­self any less de­gen­er­ately re­ac­tive.

De­liber­a­tion examples

Nat­u­ral se­lec­tion, the ur-deliberator

Nat­u­ral se­lec­tion with mu­ta­tion, a weak eval­u­a­tive pro­mo­tive proper de­liber­a­tor, ‘pro­poses’ vari­a­tions on cur­rent themes, tries them out by ‘eval­u­at­ing’ their suc­cess, and moves (on av­er­age) to ‘pro­mote’ fit­ter com­bi­na­tions6. (Nat­u­ral se­lec­tion cou­ples its eval­u­a­tion and pro­mo­tion to its ac­tion, like re­ac­tive sys­tems, but other de­liber­a­tors need not do so.) It is es­sen­tial to dis­t­in­guish ‘nat­u­ral se­lec­tion’, the de­liber­a­tive mo­ment-to-mo­ment or gen­er­a­tion-to-gen­er­a­tion pro­posal, eval­u­a­tion, and pro­mo­tion/​de­mo­tion of lo­cal vari­a­tions from ‘evolu­tion by nat­u­ral se­lec­tion’, the iter­ated de­liber­a­tive pro­cess (con­trol­ler) which ac­cu­mu­lates changes over time.

The im­plicit ab­strac­tion un­der­ly­ing the pro­posal part of nat­u­ral se­lec­tion is ba­sic: some­times ran­dom mu­ta­tions will be fit­ter. The eval­u­a­tion and pro­mo­tion im­plic­itly de­pend on the ab­strac­tion that if some­thing works, this is ev­i­dence that it may work again. Th­ese ab­strac­tions are weak—the vast ma­jor­ity of lineages of biolog­i­cal or­ganisms are ex­tinct, and most ex­tant or­ganisms carry large amounts of dele­te­ri­ous genes—but we hap­pen to live in a uni­verse where they have been true enough times that life has per­sisted on Earth for billions of years and adapted to many chang­ing cir­cum­stances in that time.

Hyper­pa­ram­e­ters of nat­u­ral selection

Cer­tain par­tic­u­lars of the ma­chin­ery of life on Earth are no­table for af­fect­ing the ‘hy­per­pa­ram­e­ters’ of nat­u­ral se­lec­tion:

• Sepa­ra­tion of con­cerns into stor­age (mostly nu­cleic acids) and func­tion (mostly pro­teins)

• DNA re­pair mechanisms, their prevalence and their absence

• Sex­ual re­pro­duc­tion in many lineages

• Highly-con­served evolu­tion­ary de­vel­op­ment mechanisms

Since none of these can be said to be the pri­mor­dial state, in some sense it can be said that nat­u­ral se­lec­tion has acted on its own hy­per­pa­ram­e­ters. I think it may be ap­pro­pri­ate to rather iden­tify ‘higher level’ nat­u­ral se­lec­tions act­ing on lineages for the effec­tive­ness of their adapt­abil­ity, as de­ter­mined by the hy­per­pa­ram­e­ters of their re­spec­tive nat­u­ral se­lec­tion.

What makes nat­u­ral se­lec­tion so bad?

Note that we can turn our ques­tion about gra­di­ent de­scent around: what makes nat­u­ral se­lec­tion so bad?7 It uses far more re­sources than gra­di­ent de­scent but performs com­pa­rably.

The an­swer is the mir­ror image: the pro­posal-gen­er­at­ing and eval­u­a­tion heuris­tics of nat­u­ral se­lec­tion are about as rudi­men­tary as they could pos­si­bly be, so in spite of mas­sive par­allelism and enor­mous com­pu­ta­tional re­sources, it takes a lot of iter­a­tion to get many ca­pa­bil­ity in­no­va­tions off the ground.

Plants are sur­pris­ingly deliberate

Plants as ac­tors are of­ten con­sid­ered to be lack­ing de­liber­a­tion, but in con­trast, it is more ap­pro­pri­ate to con­sider many of their be­havi­ours to be weakly de­liber­a­tive in a fash­ion com­pa­rable to or per­haps sur­pass­ing that of nat­u­ral se­lec­tion.

For ex­am­ple, climb­ing plants have nu­mer­ous flailing, spread­ing, and oth­er­wise (lo­cal) search­ing rou­tines8 to gen­er­ate can­di­date route-pro­pos­als, which, com­posed with eval­u­a­tion mechanisms (test­ing sup­port, light availa­bil­ity, etc) and pro­mo­tion rou­tines (ex­e­cute growth, latch­ing, grasp­ing, coiling—when eval­u­ated as favourable) serve as ev­ery bit as much a heuris­tic lo­cal de­liber­a­tive search mechanism as does nat­u­ral se­lec­tion.

Th­ese pro­ce­dures are iter­ated as part of the over­all growth con­trol mechanism of the plant, to very suc­cess­ful effect (lo­cat­ing effi­cient short­cuts to re­gions of most light and other re­sources). Other plants, fungi and mostly-seden­tary or­ganisms ex­hibit similar be­havi­ours which should be in­ter­preted in similar al­gorith­mic terms (per­haps es­pe­cially com­monly in roots and hy­phae—fun­gal root-like struc­tures—which are less ob­vi­ously visi­ble).

Another no­table lo­cal-search de­liber­a­tion ex­e­cuted by seden­tary or­ganisms con­cerns their dis­per­sal of offspring and offshoots. Note that for such or­ganisms, con­di­tions as de­ter­mined by phys­i­cal lo­ca­tion are paramount! It is typ­i­cal of this lifestyle to in­vest in many can­di­date offspring and in a wide va­ri­ety of dis­per­sal mechanisms. Thus, just as genome is repli­cated with vari­a­tion, so is lo­ca­tion, and it is there­fore part of a kind of meta-genome el­i­gible to be se­lected on9. As a con­se­quence of this con­trol pro­cess, lineages of en­tirely seden­tary or­ganisms can be seen to rapidly re­li­ably pro­ceed to­ward phys­i­cal niches to which they are fit. For some species with veg­e­ta­tive prop­a­ga­tion it may be more plain that this pro­cess is car­ried out by a ‘sin­gle ac­tor’10, but it re­sults in the same kind of ‘hill climb­ing’ (or ‘shaded fer­tile valley seek­ing’ as the case may be), be­cause it is es­sen­tially the same al­gorithm.

Why is plant de­liber­a­tion ‘more effi­cient’ than nat­u­ral se­lec­tion?

Notably, these con­trol op­er­a­tions, closely al­gorith­mi­cally re­lated to iter­ated ge­netic nat­u­ral se­lec­tion, nev­er­the­less op­er­ate much faster than ge­netic nat­u­ral se­lec­tion (they may take as lit­tle as days to man­i­fest ‘large’ re­sults). Why might this be?

The ob­vi­ous way to ‘go faster’ in an iter­ated de­liber­a­tion pro­cess is to have a shorter iter­a­tion cy­cle, perform­ing more iter­a­tions per time. This can ac­count for some of the differ­ence but not all. The other ob­vi­ous way to ‘do bet­ter’ is to eval­u­ate more pro­pos­als per iter­a­tion. Notably, nat­u­ral se­lec­tion has high par­allelism at its dis­posal and of­ten eval­u­ates many more pro­pos­als per step than do climb­ing plants.

It is also im­por­tant to con­sider the di­men­sion of the search space, which has im­por­tant bear­ing on how tractable it is. The di­men­sion is straight­for­ward: ge­netic nat­u­ral se­lec­tion op­er­ates on a high-di­men­sional space, while plant climb­ing and lo­co­mo­tion op­er­ate on two or three di­men­sions, so it is sim­ply eas­ier for them to lo­cate good solu­tions by gen­er­at­ing lo­cal can­di­date pro­pos­als.

No­tice, though, that lichen or mosses climb­ing trees over time can be seen to in­stan­ti­ate the same search, with similar pa­ram­e­ters, in the same di­men­sions as larger more so­phis­ti­cated climb­ing plants, but they achieve this much more slowly (but still fast by com­par­i­son to nat­u­ral se­lec­tion). So di­men­sion and iter­a­tion speed do not ap­pear to tell the whole story.

Two other im­por­tant prop­er­ties to pay at­ten­tion to are sug­gested by the al­gorith­mic break­down de­scribed pre­vi­ously. First, re­lat­ing to pro­posal, the fit of the sam­pling heuris­tics used in the de­liber­a­tion. Se­cond, re­lat­ing to eval­u­a­tion and pro­mo­tion, the fidelity of these pro­ce­dures to track­ing the ac­tual tar­get.

The fit of the sam­pling heuris­tics is very im­por­tant for tractabil­ity: ge­netic nat­u­ral se­lec­tion gen­er­ates its pro­pos­als via some­thing similar to sym­met­ri­cal noise, lead­ing to a lot of dead ends, wasted iter­a­tions, and even re­peated or ap­prox­i­mately-re­peated com­pu­ta­tion. Moss-style mi­gra­tion up trees makes its pro­pos­als similarly nois­ily. In con­trast, ten­dril growth of climb­ing plants is both de­cid­edly bi­ased (up­ward) and has an ori­en­ta­tion, hence a ‘mem­ory’, yield­ing a ten­dency not to dou­ble back on it­self, at least lo­cally (un­less ex­e­cut­ing a coil, which is a differ­ent part of the al­gorithm). It is also able to use pho­totropism to fur­ther heuris­ti­cally bias its search.11

Th­ese sim­ple but highly-fit heuris­tic pro­posal-sam­pling bi­ases, dis­cov­ered by nat­u­ral se­lec­tion, the slow outer de­liber­a­tor, may largely ac­count for the ev­i­dent effi­ciency ad­van­tage of climb­ing plants over less so­phis­ti­cated pas­sive climbers like moss and lichen.

Colo­nial organisms

View­ing in­sect colonies (for ex­am­ple, ants) as ac­tors, we can iden­tify all of the in­gre­di­ents of de­liber­a­tion. The pat­tern is, like with climb­ing plants, struc­turally similar to nat­u­ral se­lec­tion, but once again with fit­ter, more effi­cient, sam­pling heuris­tics.

Con­sider an ant colony for­ag­ing for food. The can­di­date pro­pos­als are the paths taken by in­di­vi­d­u­als or small groups of work­ers. The eval­u­a­tions be­gin with the guesses of those in­di­vi­d­u­als (per­haps aug­mented by a mes­sage-pass­ing al­gorithm) about the qual­ity and quan­tity of any foods they come across. The prop­erly pro­mo­tive out­come in­volves lay­ing down chem­i­cal sig­nals and tac­tile in­ter­ac­tions which en­courage at­trac­tive or aver­sive be­havi­our by other work­ers (and ad­just other be­havi­oural pa­ram­e­ters). As an iter­ated con­trol pro­ce­dure, this rapidly re­sults in dense lines of work­ers fer­ry­ing the best and most plen­tiful food back to the nest. Note that no in­di­vi­d­ual ant need be at all de­liber­a­tive for this to work12.

This pro­cess, again similarly-struc­tured and with similar di­men­sion to the ten­dril-climb­ing al­gorithm of some plants, is able to pro­ceed even faster. From an al­gorith­mic point of view, this can be ex­plained by the even fit­ter sam­pling heuris­tics: for ex­am­ple, in­di­vi­d­ual ants have very strong senses of smell, which al­lows the for­ag­ing party’s search to be very heav­ily bi­ased to­ward promis­ing lo­ca­tions. This bias is more pow­er­ful even than pho­totropism ex­hibited by plants, be­cause smell is non­lo­cal. Another fac­tor is the re­source effi­ciency of the state man­age­ment: plant climb­ing al­gorithms record state and pro­mote can­di­dates by ac­tual or­ganism growth, while ant colonies ex­e­cute these parts of the al­gorithm by leav­ing much less re­source-in­ten­sive pheromone trails.

Prop­erly de­liber­a­tive ar­tifi­cial systems

The most ob­vi­ous prop­erly de­liber­a­tive ar­tifi­cial sys­tems are train­ing and search al­gorithms in­spired by nat­u­ral se­lec­tion, in­clud­ing pop­u­la­tion based train­ing (PBT) and neu­roevolu­tion for neu­ral net­works, all of which em­ploy an iter­ated proper de­liber­a­tor as a con­trol pro­cess to lo­cate and pro­mote com­plex con­figu­ra­tions which are found to score well against some tar­get.

Train­ing and search rou­tines of­ten also em­ploy similar but non iter­ated de­liber­a­tion, for ex­am­ple static hy­per­pa­ram­e­ter search and ran­dom seed search. (Some­times these them­selves end up iter­ated.)

Other times, the de­ployed arte­fact it­self is the proper de­liber­a­tor. Both value-based and policy-based RL con­trol­lers over dis­crete ac­tion spaces fall un­der the prop­erly-de­liber­a­tive Opt;Enact de­com­po­si­tion in a very ba­sic way. One could make a similar case for a dis­crete clas­sifi­ca­tion model which opts for a sin­gle cat­e­gory at the last stage.

Cherry-pick­ing from re­ac­tive gen­er­a­tive mod­els is an ex­am­ple of com­pos­ing a Promote func­tion (in this case hu­man judge­ment!) with a re­ac­tive sys­tem to boot­strap a prop­erly de­liber­a­tive sys­tem. There are also au­to­mated ex­am­ples of this type of com­po­si­tion and it is a very gen­eral for­mula for im­prov­ing de­ci­sion-mak­ing13.

Hard-coded vs learned deliberation

All of the above­men­tioned ex­am­ples of de­liber­a­tion in ar­tifi­cial sys­tems have the de­liber­a­tion struc­ture (com­pos­ing Propose, Promote and Act) coded in by the de­sign­ers.

• In the case of evolu­tion­ary al­gorithms and PBT, pro­pos­als take the form of some care­fully cho­sen lo­cal ‘mu­ta­tion’ heuris­tics and pro­mo­tion is coded around some pro­vided (per­haps ap­prox­i­mate) eval­u­a­tion or fit­ness func­tion.

• In con­tem­po­rary non-hi­er­ar­chi­cal RL with dis­crete ac­tion spaces14, the pro­pos­als con­sist iden­ti­cally of fixed hard-coded analogues of the atomic ac­tion space, the pro­mo­tion con­sists of a (usu­ally learned) Evaluate, and Opt;Enact are hand-coded.

  • Evaluate con­sists of ei­ther a (state-de­pen­dent) map­ping from the atomic-ac­tion-analogues to es­ti­mated util­ity (ex­pected re­turn, or some similar ac­tion-qual­ity es­ti­mate, as in Q learn­ing), or to (state-de­pen­dent) level-of-in­tent to take the analo­gous atomic ac­tion (prob­a­bil­ity or log­its, as in policy gra­di­ent meth­ods), both of which have V⊆R.

  • Opt trans­lates these eval­u­a­tions via hard-coded logic into a par­tic­u­lar sin­gle choice of atomic-ac­tion-analogue.

  • Enact ac­tu­ates that analogue in simu­la­tion or the real world.

Risks from Learned Op­ti­miza­tion as well as prior and sub­se­quent dis­cus­sions ask ques­tions about when we might ex­pect to find ‘op­ti­miz­ers’ aris­ing with­out hard-cod­ing them, and about what be­havi­ours and ca­pa­bil­ities we should ex­pect from them. Since we have iden­ti­fied non-ar­tifi­cial ex­am­ples of de­liber­a­tion aris­ing from the re­fine­ment or in­ter­ac­tions be­tween non-de­liber­a­tive sys­tems, it seems rea­son­able to look for analo­gies in our ar­tifi­cial sys­tems. It re­mains un­clear whether any con­tem­po­rary ar­tifi­cial sys­tems have learned de­liber­a­tion.

Fur­ther work could con­sider the ques­tion of learned op­ti­miza­tion from the de­liber­a­tion lens, and sub­se­quent posts in this se­quence will dis­cuss where de­liber­a­tion comes from, as well as its man­i­fes­ta­tion at differ­ent lev­els of so­phis­ti­ca­tion.

Parliaments

Per­haps rep­re­sent­ing the pro­to­typ­i­cal ‘de­liber­a­tion’, some mechanisms of demo­cratic par­li­a­ments illus­trate de­liber­a­tion al­gorithms at the level of mul­ti­ple in­ter­act­ing hu­mans.

Con­sider a vote on some mo­tion, pre­ceded by a de­bate. This is a proper, quasi-eval­u­a­tive, op­tive de­liber­a­tion.

Even if the mo­tion is in­di­visi­ble and sin­gu­lar, there are still at least two pro­pos­als: en­act the mo­tion, or do not en­act the mo­tion. In other cases there may be more pro­pos­als. Th­ese pro­pos­als are gen­er­ated and sur­faced by hu­mans within the pro­cess by some means or other (which pre­sum­ably in­volves its own de­liber­a­tion!).

The de­bate, and the sen­ti­ments of the in­di­vi­d­u­als par­ti­ci­pat­ing, are a pro­tracted eval­u­a­tion and pro­mo­tion mechanism over the pro­pos­als at hand.

The vote pre­cip­i­tates the choice, typ­i­cally opt­ing for some pro­posal, which is then en­acted into some de­rived re­al­i­sa­tion by one or other or­gan of the demo­cratic sys­tem. (Other pos­si­ble out­comes to de­bates are state up­dates in the form of minds changed or doc­u­ments writ­ten, and re-iter­a­tions with new or similar pro­pos­als, which is a more prop­erly pro­mo­tive than op­tive re­sult.)

Par­li­a­ments rarely per­ma­nently dis­solve them­selves (though their ac­tions can lead to their dis­solu­tion), so should be seen as iter­ated de­liber­a­tors.

Often the pa­ram­e­ters of a par­tic­u­lar par­li­a­men­tary ses­sion are set by some in­vok­ing or­gan, body, or in­di­vi­d­ual, mak­ing a com­plex of de­liber­a­tors which fit to­gether in a larger demo­cratic sys­tem.

As in pre­vi­ous cases, the power of a par­li­a­ment lies partly in pro­duc­ing ‘good’ pro­pos­als, partly in recog­nis­ing and pro­mot­ing good pro­pos­als, and partly in the suit­abil­ity of its del­e­gate or­gans to ap­pro­pri­ately en­act its pro­pos­als. In the iter­ated sense, be­havi­ours which pre­cip­i­tate im­prove­ments to these ca­pac­i­ties are in­stru­men­tal.

A note on these al­gorith­mic assignments

One may rightly opine that an as­sign­ment of a par­tic­u­lar al­gorith­mic ab­strac­tion to a sys­tem, hid­ing the full and in­tri­cate de­tail of ev­ery mov­ing piece, must in­volve some ul­ti­mately ar­bi­trary or sub­jec­tive choices. Per­haps this un­der­mines the pre­ced­ing dis­cus­sion.

In­deed, no ab­strac­tion can work well in all cir­cum­stances, but ab­strac­tions are nev­er­the­less in­escapably nec­es­sary tools to make rea­son­ing tractable in a world with bounded com­pu­ta­tional re­sources. If an ant colony were trans­ported to the sur­face of the sun, there would rapidly be no sense in which it con­tinued to carry out the pre­vi­ously-iden­ti­fied food-search­ing al­gorithm. In fact, the ab­strac­tion ‘ant’ would cease to be use­ful in the same in­stant. But if the same colony were trans­ported to many lo­ca­tions on the sur­face of the Earth, or other rele­vantly-similar hy­po­thet­i­cal places, the ab­stract al­gorithm iden­ti­fied above would con­tinue to use­fully pre­dict out­comes, in­clud­ing coun­ter­fac­tu­als (‘what if I put some food here?’, ‘what if I move some ants there?’ etc.), at least for some time, de­pend­ing on the fit­ness of the colony’s al­gorithm’s ab­strac­tions and heuris­tics to the new en­vi­ron­ment.

‘Ant’ and ‘ant colony’ cease to be use­ful de­scrip­tions in the con­text of the sur­face of the sun, but re­main defen­si­ble sum­mary de­scrip­tions on many parts of the sur­face of the Earth. On the left a pho­to­graph of a mag­nifi­ca­tion of the au­thor’s gar­den. On the right an artist’s im­pres­sion of the same ex­per­i­ment on the sun. (The au­thor was re­gret­tably un­able to at­tend for the lat­ter ex­per­i­ment.)

Conclusion

We’ve ex­plored var­i­ous de­liber­a­tive sys­tems (both proper and re­ac­tive), find­ing them at many lev­els of or­gani­sa­tion, from cells to democ­ra­cies. In sev­eral cases we un­cov­ered novel in­sights, such as com­mon­al­ities be­tween nat­u­ral se­lec­tion, climb­ing plants, and ant colonies, and the de­liber­a­tion ab­strac­tion fur­ther en­abled us to point to rele­vant dis­crep­an­cies de­ter­min­ing the ‘strength’ of some of these sys­tems. Suitabil­ity or fit­ness of Propose ap­pears at least as im­por­tant as that of Promote for de­liber­a­tion strength in many cases, ex­em­plified by the com­par­i­son of nat­u­ral se­lec­tion with gra­di­ent de­scent.

Var­i­ous de­liber­a­tive sys­tems com­prise in­ter­ac­tions be­tween mul­ti­ple de­liber­a­tors. Th­ese can be seen emerg­ing from the in­ter­ac­tion be­tween rel­a­tively ho­mo­ge­neous sub­de­liber­a­tors (as in colonies and mar­kets), or in de­liber­a­tors giv­ing rise to, or con­di­tion­ing, other het­ero­ge­neous del­e­gate de­liber­a­tors’ be­havi­our (as in cel­lu­lar and mul­ti­cel­lu­lar life and bu­reau­cra­cies).

Later posts will go into more depth about where de­liber­a­tions come from, and also ex­am­ine some de­liber­a­tors which are more so­phis­ti­cated, ei­ther in how they ar­range their del­e­ga­tion or how they perform the key com­po­nents of de­liber­a­tion. This in­cludes some ar­tifi­cial sys­tems as well as some of the more cog­ni­tively so­phis­ti­cated and pow­er­ful be­havi­ours of an­i­mals and hu­mans.

Thanks to those who probed in con­ver­sa­tion, and thanks to SERI for spon­sor­ing the re­search time and mak­ing those con­ver­sa­tions pos­si­ble.

A cluster of ques­tions fun­da­men­tal to many con­cerns around risks from ar­tifi­cial sys­tems re­gard the con­cepts of search, plan­ning, and ‘de­liber­ate­ness’. How do these arise? What can we pre­dict about their oc­cur­rence and their con­se­quences? How strong are they? What are they any­way?

Here is laid out one part of a con­cep­tual de­com­po­si­tion which maps well onto many known sys­tems and may al­low fur­ther work to­wards an­swer­ing more of those ques­tions. The am­bi­tion is to re­ally get at the heart of what is al­gorith­mi­cally hap­pen­ing in ‘op­ti­mis­ing sys­tems’, in­clud­ing hu­mans, an­i­mals, al­gorith­mic op­ti­misers like SGD, and con­tem­po­rary and fu­ture com­pu­ta­tional arte­facts. That said, I do not have any priv­ileged in­sight into the source code (or its proper in­ter­pre­ta­tion!) for the ex­am­ples dis­cussed, so while this fram­ing has already gen­er­ated new in­sights for me, it may or may not be ‘the ac­tual al­gorith­mic truth’.

We start with the anal­y­sis: a defi­ni­tion of ‘de­liber­a­tion’ and its com­po­nents, then of ‘re­ac­tions’ and ‘con­trol’. Next we con­sider, in light of these, what makes a de­liber­a­tor or con­trol­ler ‘good’. We find con­cep­tual con­nec­tions with dis­cus­sions of in­stru­men­tal con­ver­gence. Lit­tle at­ten­tion is given here to how to de­ter­mine what the goals are, which is ob­vi­ously also im­por­tant.

Th­ese con­cepts were gen­er­ated by con­tem­plat­ing var­i­ous as­pects of many differ­ent goal-di­rected sys­tems and pul­ling out com­mon­al­ities.

Some read­ers may pre­fer to start with the ex­am­ples, which in­clude an­i­mals, plants, nat­u­ral se­lec­tion, gra­di­ent de­scent, bu­reau­cra­cies, and oth­ers. Here in the con­cep­tual sec­tion I’ll foot­note par­tic­u­larly rele­vant con­crete ex­am­ples where I an­ti­ci­pate them helping to con­vey my point.

A full treat­ment is ab­sent, but two ma­jor defer­ences to em­bed­ded agency un­der­lie this anal­y­sis. A Carte­sian sep­a­ra­tion need not be as­sumed, ex­cept over ‘ac­tor-mo­ments’ rather than tem­po­rally-ex­tended ‘ac­tors’. And a ma­jor driver for this se­quence is a fun­da­men­tal recog­ni­tion that any goal-di­rected be­havi­our in­stan­ti­ated in the real world must have bounded com­pu­ta­tional ca­pac­ity per time^[1].

In­spira­tion and related

My (very brief) ‘Only One Shot’ in­tu­ition pump for em­bed­ded agency may help to con­vey some back­ground as­sump­tions (es­pe­cially re­gard­ing how time and ac­tor-mo­ments fit into this pic­ture).

Scott Garrabrant’s (A→B)→A talks about ‘agency’ and ‘do­ing things on pur­pose’. I’m try­ing to un­pack that fur­ther. The (open) ques­tion Does Agent-like Be­havi­our Im­ply Agent-like Ar­chi­tec­ture? is re­lated and I hope for the per­spec­tive here to be use­ful to­ward an­swer­ing that ques­tion.

Alex Flint’s ex­cel­lent piece The ground of op­ti­miza­tion in­forms some of the per­spec­tive here, es­pe­cially a fo­cus on scope of gen­er­al­i­sa­tion and ro­bust­ness to per­tur­ba­tion.

Daniel Filan’s Bot­tle Caps Aren’t Op­ti­misers and Abram Dem­ski’s Selec­tion vs Con­trol be­gin dis­cussing the al­gorith­mic in­ter­nals of op­ti­mis­ing sys­tems, which is the in­tent here also. Risks from Learned Op­ti­miza­tion is of course rele­vant.

John Went­worth’s dis­cus­sions of ab­strac­tion (for ex­am­ple What is Ab­strac­tion?) es­pe­cially with re­gards its pre­dic­tive prop­er­ties for a non-om­ni­scient com­puter, are cen­tral to the no­tion of ab­strac­tion em­ployed here. Re­lated are the good and gooder reg­u­la­tor the­o­rems, which touch closely com­pu­ta­tion­ally up­stream of the as­pects dis­cussed here while mak­ing fewer con­ces­sions to em­bed­ded agency.

Definitions

Deliberation

A de­liber­a­tion is any part of a de­ci­sion al­gorithm which com­poses the following

• Propose: gen­er­ate can­di­date proposals

• Promote: pro­mote and de­mote pro­pos­als ac­cord­ing to some criterion

• Act: take out­come of pro­mo­tion and de­mo­tion to ac­tivity in the environment

tak­ing place over one ‘mo­ment’ (as de­ter­mined by the par­tic­u­lar al­gorith­mic em­bod­i­ment).

Some in­stances may fuse some of these com­po­nents to­gether.

Propose

A de­liber­a­tion needs to be about some­thing. Can­di­date pro­pos­als X do not come from nowhere; some state S is mapped to a dis­tri­bu­tion over nonempty pro­posal-col­lec­tions (there must be at least one can­di­date pro­posal^[2]). Pro­pos­als cor­re­spond in some way to ac­tions, but need not be ac­tions or be one-to-one with them^[3] - more on this un­der Act.^[4]

Propose:S→Δ{X}nonempty

Note one pat­tern for achiev­ing this is to sam­ple one or more times from

ProposeOne:S→ΔX

and similarly tak­ing the union of sev­eral pro­posal-col­lec­tions could be a pro­posal-col­lec­tion in some cases.

Some in­stan­ti­a­tions may have a fixed set of pro­pos­als, while oth­ers may be flex­ible.

Pro­pose: robot con­sid­ers pro­pos­als re­lat­ing to the con­cepts of de­liv­er­ing ice cream or de­liv­er­ing grenade (NB this image con­veys a sub­stan­tial world model and pro­pos­als cor­re­spond­ing closely to plans of ac­tion, but this is merely one em­bod­i­ment of the de­liber­a­tion frame­work and not defini­tive)

Promote

Given a col­lec­tion of pro­pos­als, a de­liber­a­tive pro­cess pro­duces some pro­mo­tion/​de­mo­tion weight­ing V for each pro­posal. This may also de­pend on state.^[5]

Promote:S→{X}→{V}

In many in­stan­ti­a­tions it makes sense to treat the weight­ing as a sim­ple real scalar V=R^[6].

Many in­stan­ti­a­tions may es­sen­tially map an eval­u­a­tion over the proposals

Evaluate:S→X→V

ei­ther se­ri­ally or in par­allel. This sug­gests ‘eval­u­a­tive’ as a use­ful ad­jec­tive to de­scribe such pro­cesses.

Other in­stan­ti­a­tions may in­volve the whole col­lec­tion of pro­pos­als, for ex­am­ple by a com­par­a­tive sort­ing-like pro­ce­dure with­out any in­ter­me­di­ate eval­u­a­tion.^[7]

Pro­mote: robot eval­u­ates pro­pos­als of throw­ing ice cream or grenade re­spec­tively, pro­mot­ing ice cream and de­mot­ing grenade

Act

A de­ci­sion al­gorithm even­tu­ally acts in its en­vi­ron­ment. In this anal­y­sis, that means tak­ing pro­mo­tion-weighted pro­pos­als and trans­lat­ing them, via some ma­chin­ery, into ac­tivity A.

Act:{X×V}→A

In many cases pro­pos­als may cor­re­spond closely as pre­cur­sors to par­tic­u­lar ac­tions or plans. Pro­mo­tion may then cor­re­spond to prefer­ence over plans, in which case ac­tion may ap­prox­i­mately de­com­pose as Opt:{X×V}→X and Enact:X→A i.e. a se­lec­tion of an ac­tion, plan, or policy, which is then car­ried out^[8]. Let’s ten­ta­tively call such sys­tems ‘op­tive’. Enact may in­volve sig­nal­ling or oth­er­wise in­vok­ing other (re­ac­tive or proper) de­liber­a­tions!

Act-op­tive: robot’s eval­u­a­tion and pro­mo­tion leads to opt­ing to give ice cream, which is a pre­cur­sor to sub­se­quent mo­tor en­ac­tion, giv­ing ice cream. Mean­while the robot may or may not be perform­ing new de­liber­a­tions.

More gen­er­ally, pro­pos­als may cor­re­spond to con­figu­ra­tions, and pro­mo­tion to weight­ing-ad­just­ment be­tween those con­figu­ra­tions, which play out as ac­tivity. We might call such sys­tems ‘(prop­erly) pro­mo­tive’ in con­trast to ‘op­tive’^[9].

Act-pro­mo­tive: robot’s pro­mo­tion and de­mo­tion of plans pro­duces mostly ‘in­ter­nal’ up­dates to its plans

Reactions

A re­ac­tion is any de­gen­er­ate de­liber­a­tion with iden­ti­cally one can­di­date pro­posal. In this case the Promote step is triv­ial or ves­ti­gial, and ac­tion is effec­tively fused with the other steps.

In some sense, al­low­ing for de­gen­er­ate cases of ‘de­liber­a­tion’ means that this al­gorith­mic ab­strac­tion ap­plies to es­sen­tially ev­ery­thing (e.g. a rock is a re­ac­tive de­liber­a­tor which always pro­poses ‘do the thing a rock would’).

The im­por­tant thing is that this ab­strac­tion al­lows us to analyse and con­trast non re­ac­tive ‘proper de­liber­a­tors’ with a use­ful de­com­po­si­tion, as well as to iden­tify and rea­son about more ‘in­ter­est­ing’ and less vac­u­ous re­ac­tions^[10] (while still iden­ti­fy­ing them as such). e.g. ‘Where did this re­ac­tion come from?’, ‘What heuris­tics/​ab­strac­tions is it (im­plic­itly) us­ing for pro­pos­als?’, ‘How might this re­ac­tion com­bine with other sys­tems and could proper de­liber­a­tion emerge?’, ‘What de­liber­a­tion(s) does this re­ac­tion ap­prox­i­mate?’.

Iter­ated de­liber­a­tion is gen­er­al­ised control

A de­ci­sion al­gorithm even­tu­ally pro­duces ac­tivity in the en­vi­ron­ment. Any such ac­tivity di­rectly or in­di­rectly al­ters the rele­vant al­gorith­mic state of the pro­cess, from triv­ial ther­mal noise up to and in­clud­ing ter­mi­na­tion (or mod­ifi­ca­tion to some other al­gorith­mic form).

In many cases, the ac­tivity typ­i­cally pre­serves the es­sen­tial al­gorith­mic form and it may be ap­pro­pri­ate to analyse the al­gorithm as be­ing re­cur­rent or iter­ated, per­haps with an ap­prox­i­mate Carte­sian sep­a­ra­tion from its en­vi­ron­ment—the ac­tion has some com­po­nent play­ing out in the world (which up­dates and pro­vides new in­puts) and some com­po­nent fold­ing into a priv­ileged part of the world cor­re­spond­ing to the al­gorithm’s ‘in­ter­nal state’. In­deed, in many so­phis­ti­cated de­liber­a­tors, the most pow­er­ful effect^[11] may be on the state or con­di­tion (or ex­is­tence!) of sub­se­quent de­liber­a­tion(s) and ac­tor-mo­ments.

A sys­tem whose ac­tivity and situ­a­tion in the en­vi­ron­ment pre­serve the es­sen­tial out­line of its de­liber­a­tion al­gorithm (per­haps with state up­dates) de-facto in­vokes a re-iter­a­tion of the same (or re­lated) de­liber­a­tion pro­ce­dure. This gives rise to iter­ated de­liber­a­tion, which in this anal­y­sis is syn­ony­mous with con­trol^[12].

A ba­sic ‘re­ac­tive’ con­trol­ler is a sys­tem which performs de­gen­er­ate, re­ac­tive de­liber­a­tion, but which is (es­sen­tially) pre­served by its ac­tions, giv­ing rise to an iter­ated re­ac­tion, a rel­a­tively clas­sic con­trol sys­tem.

A (prop­erly) de­liber­a­tive con­trol­ler is a de­liber­a­tive sys­tem gen­er­at­ing mul­ti­ple pro­pos­als while be­ing (es­sen­tially) pre­served by its ac­tions, giv­ing rise to a (prop­erly) de­liber­a­tive con­trol sys­tem.

Note that in this sense, what might be con­sid­ered a sin­gle or­ganism gen­er­ally con­sists of mul­ti­ple con­trol­lers (re­ac­tions or oth­er­wise), and mul­ti­ple or­ganisms taken to­gether may com­prise one con­trol­ler. Like­wise an ar­tifi­cial ac­tor may com­prise mul­ti­ple de­liber­a­tive and re­ac­tive com­po­nents, and a sys­tem of mul­ti­ple ar­tifi­cial ac­tors may com­pose to a sin­gle de­liber­a­tive pro­cess. The rele­vant ob­ject of anal­y­sis is the al­gorithm.

Re­cur­sive de­liber­a­tion is more gen­eral still

Just as the out­come of a de­liber­a­tive ac­tor-mo­ment may re­sult in zero (ter­mi­nated) or one (iter­ated) in­vo­ca­tions of rele­vantly-similar fu­ture ac­tor-mo­ments, there is no rea­son why there should not be a vari­able num­ber some­times more than one (ho­mo­ge­neous re­cur­sive).

We ob­serve replica­tive ex­am­ples through­out na­ture, as is to be ex­pected in a world where nat­u­ral se­lec­tion some­times works.

As well as in­vok­ing mul­ti­ple copies of ‘it­self’ (or rele­vantly-similar al­gorithms) through its ac­tions, as in repli­ca­tion, a de­liber­a­tor may in­voke, cre­ate, or oth­er­wise con­di­tion other het­ero­ge­neous de­liber­a­tors, as in del­e­ga­tion (het­ero­ge­neous re­cur­sive). (In fact repli­ca­tion per se is usu­ally not atomic, and goes via other in­ter­me­di­ate pro­cesses.) In­vo­ca­tion or con­di­tion­ing of het­ero­ge­neous de­liber­a­tors is what hu­mans and an­i­mals do when lo­co­mot­ing, for ex­am­ple^[13], and what some robotics ap­pli­ca­tions do when del­e­gat­ing ac­tu­a­tion to clas­si­cal con­trol mechanisms like ser­vo­mo­tors. It’s also what we see in bu­reau­cra­cies and colonies of var­i­ous con­struc­tions and scales, from cells to ant nests to democ­ra­cies.

What fac­tors go into strong de­liber­a­tion?

Qual­ity of de­liber­a­tion de­pends on the fit­ness^[14] of the ab­strac­tions and heuris­tics which the al­gorith­mic com­po­nents de­pend on, as well as the amount of com­pute bud­get con­sumed. Th­ese are the raw fac­tors of de­liber­a­tion.

So a de­liber­a­tor whose Propose gen­er­a­tors make bet­ter sug­ges­tions (rel­a­tive to some goal or refer­ence) is ce­teris paribus stronger (for that goal or refer­ence). Like­wise a de­liber­a­tor whose Promote al­gorithm more closely tracks re­al­ity rele­vant to its goal (is bet­ter fit), or a de­liber­a­tor which can gen­er­ate and eval­u­ate more pro­pos­als with a given com­pute bud­get. We might ex­pect good de­liber­a­tors to ‘model’ the goal-rele­vant as­pects of their en­vi­ron­ment as pre­cur­sors to the state S used in pro­posal and pro­mo­tion^[15].

No model is cor­rect in all situ­a­tions but some are more effec­tive than oth­ers over a wider range of ‘nat­u­ral’ situ­a­tions. In sub­se­quent posts this fram­ing is used to ex­am­ine some real ex­am­ples, lo­cat­ing some rele­vant ways their ab­strac­tions are more or less fit, and some con­se­quences of this. Im­por­tantly, qual­ity of Propose gen­er­a­tors is em­pha­sised^[16].

Quan­tify­ing the mag­ni­tude of the above­men­tioned ce­teris paribus de­liber­a­tion im­prove­ments is im­por­tant, es­pe­cially in the con­text of mul­ti­ple com­pet­ing or col­lab­o­rat­ing de­liber­a­tors, but this is not at­tempted in de­tail here.

Con­ver­gent in­stru­men­tal goals and re­cur­sive deliberation

In light of the de­liber­a­tion fram­ing, and fo­cus­ing on the po­ten­tial for iter­ated or re­cur­sive de­liber­a­tion, we can restate and per­haps sharpen the con­ver­gence of in­stru­men­tal goals in these terms.

A de­liber­a­tor is a sin­gle ac­tor-mo­ment, finitely ca­pa­ble and nei­ther log­i­cally nor em­piri­cally om­ni­scient.

For a de­liber­a­tor ori­ented to di­rec­tions or goals which can not re­li­ably be defini­tively achieved in a sin­gle round of de­liber­a­tion, or goals more gen­er­ally for which greater suc­cess can be ex­pected by sub­se­quent re­fine­ment of ac­tion, pro­pos­als and ac­tions which pre­cip­i­tate the ex­is­tence and em­pow­er­ment of similarly-ori­ented de­liber­a­tive ac­tor-mo­ments can be ex­pected ce­teris paribus to achieve greater suc­cess.

In this fram­ing, ‘self-preser­va­tion’ and ‘goal-con­tent in­tegrity’ are spe­cial cases of ‘push­ing the fu­ture to con­tain rele­vantly-similarly-ori­ented de­liber­a­tive ac­tor-mo­ments’^[17]. Other in­stan­ti­a­tions of this in­clude ‘repli­cat­ing’. The pro­to­typ­i­cal cases from na­ture are sys­tems which do one or both of per­sist­ing in rele­vantly al­gorith­mi­cally similar form (e.g. or­ganisms hav­ing non­triv­ial lifes­pans or en­zymes and cat­a­lysts be­ing un­trans­formed), and repli­cat­ing or re­pro­duc­ing rele­vantly al­gorith­mi­cally similar forms (e.g. ge­netic el­e­ments and sim­pler au­to­cat­a­lysts di­rectly or in­di­rectly in­vok­ing copies to be made, or or­ganisms pro­duc­ing offspring). As noted pre­vi­ously we also see cases of del­e­ga­tion to het­ero­ge­neous de­liber­a­tors ei­ther by con­di­tion­ing or whole­sale cre­ation.

Now, once we recog­nise a class of de­liber­a­tor which does well by (at least some­times) in­vok­ing rele­vantly-similarly-ori­ented de­liber­a­tive fu­ture ac­tor-mo­ments (whether by per­sis­tence, repli­ca­tion, or other del­e­ga­tion), all things equal, the same de­liber­a­tor would do bet­ter to in­voke bet­ter such fu­ture ac­tor-mo­ments^[18]. So how can they be bet­ter? In all the same ways as iden­ti­fied already!

Hence if ‘self’ is the baseline for per­sis­tence or repli­ca­tion, ac­tions which in­duce fu­ture de­liber­a­tions with bet­ter ex­pected fit­ness-to-goal are bet­ter, other things equal. This can span the range of

• ‘self state up­dates’ track­ing lo­cally-rele­vant in­for­ma­tion to im­prove de­liber­a­tion (mem­ory)

• im­prov­ing Propose and Promote heuris­tics more broadly (learn­ing, ex­plor­ing, play, ex­per­i­men­ta­tion)

• more di­rect ‘self’ mod­ifi­ca­tion in­clud­ing in­vest­ing more com­pu­ta­tion into de­liber­a­tion (cog­ni­tive en­hance­ment)

• im­prov­ing Act effi­cacy by tun­ing ex­ist­ing del­e­ga­tion tem­plates or trans­form­ing or al­ign­ing re­sources into new and im­proved (in the pre­ced­ing ways) copies or del­e­gates (re­source ac­qui­si­tion and tech­nolog­i­cal im­prove­ment)

Whether to ex­pect a de­liber­a­tor to dis­cover and/​or be ca­pa­ble of act­ing on any of these kinds of im­prove­ments is a sep­a­rate mat­ter, though ev­i­dently any sys­tem able to rea­son similarly to a hu­man is ca­pa­ble of at least in prin­ci­ple ap­pre­hend­ing them.

Im­me­di­ate takeaways

We broke down ‘de­liber­a­tion’ into al­gorithms for ‘pro­posal’, ‘pro­mo­tion’ (which may be ‘eval­u­a­tive’ or ‘sortive’), and ‘ac­tion’ (which may be ‘op­tive’ or ‘(prop­erly) pro­mo­tive’). Com­bined with the ac­tor-mo­ment fram­ing, we dis­cussed cases of iter­ated or re­cur­sive de­liber­a­tion. This fram­ing al­lows us to dis­cuss com­mon­al­ities and differ­ences be­tween sys­tems and per­haps make more re­li­able pre­dic­tions about their coun­ter­fac­tual be­havi­our and emer­gence.

Point­ing at differ­ent parts of this de­com­po­si­tion, we iden­ti­fied var­i­ous axes along which a de­liber­a­tor can be im­proved, ad­di­tion­ally red­eriv­ing and re­fin­ing in­stru­men­tal con­ver­gence for iter­ated/​re­cur­sive de­liber­a­tors.

All this is quite ab­stract and sub­se­quent posts will make it more con­crete with ex­am­ples, start­ing with rel­a­tively sim­ple de­liber­a­tors, illus­trat­ing some more ap­pli­ca­tions of this fram­ing to gen­er­at­ing new in­sights.

Credit to Peter Bar­nett, Tam­era Lan­ham, Ian McKen­zie, John Went­worth, Beth Barnes, Ruby Bloom, Clau­dia Shi, and Mathieu Putz for use­ful con­ver­sa­tions prompt­ing and re­fin­ing these ideas. Thanks also to SERI for spon­sor­ing my pre­sent col­lo­ca­tion with these cre­ative and helpful peo­ple!

1. ↩︎

   So al­gorithms can not hap­pen ‘all at once’ and no pro­cess can be log­i­cally or em­piri­cally om­ni­scient, nor can it, in par­tic­u­lar, draw ‘con­clu­sions’, gen­er­ate ab­strac­tions, or oth­er­wise be im­bued with in­for­ma­tion for which there is not (yet) ev­i­dence.

2. ↩︎

   Alter­na­tively, a lack of pro­pos­als could be con­sid­ered a ter­mi­na­tion of the pro­cess.

3. ↩︎

   In prin­ci­ple, the com­pu­ta­tion perform­ing pro­posal might have side-effects—on its own pro­duce mean­ingful out­comes or ‘ac­tions’ (in­clud­ing ‘state up­dates’) - but for now this pos­si­bil­ity is elided and effects are con­sid­ered to hap­pen as part of Act.

4. ↩︎

   Type no­ta­tion.Δ can be read ‘dis­tri­bu­tion over’, that is, a gen­er­a­tive pro­cess which can be sam­pled from, not nec­es­sar­ily any­thing rep­re­sent­ing a prob­a­bil­ity dis­tri­bu­tion per se.{A} means some col­lec­tion of A. All types are in­tended to be read as pure, math­e­mat­i­cal, or func­tional—i.e. side-effect free—rather than im­per­a­tive.

5. ↩︎

   NB type no­ta­tion here uses cur­ry­ing. The re­sult­ing V col­lec­tion is also im­plic­itly de­pen­dently-shaped to match the pro­pos­als one to one. We might write some­thing like Promote:S→shape→{X}shape→{V}shape

6. ↩︎

   Pos­si­bly a cen­tral ex­am­ple of not map­ping to a real scalar might be a com­mit­tee where not only are ‘votes’ be­havi­our-rele­vant, but the par­tic­u­lars of who on the com­mit­tee voted which way.

7. ↩︎

   We might call this ‘sortive’ in con­trast to ‘eval­u­a­tive’ but I am not en­tirely sold on the use­ful­ness of this ter­minol­ogy

8. ↩︎

   Con­sider an ex­am­ple where the pro­mo­tions are one or other of value es­ti­mates, log­its, or prob­a­bil­ities over an ac­tion space in RL.

9. ↩︎

   Nat­u­ral se­lec­tion is a canon­i­cal prop­erly pro­mo­tive ex­am­ple.

   It might be most ob­vi­ous to think about the ‘prop­erly pro­mo­tive’ as a re­lax­ation of the ‘op­tive’ case. In the op­tive case, we had some pro­pos­als, and de­pend­ing on the pro­mo­tion scores for them, we se­lected ex­actly one pro­posal to carry for­ward. But more gen­er­ally we might record or ap­ply the pro­mo­tion de­ci­sions in some way, ad­just­ing be­havi­our, while re­tain­ing a rep­re­sen­ta­tion of a spec­trum of pro­posal-pro­mo­tion state go­ing for­ward to sub­se­quent de­ci­sions. Go­ing back to nat­u­ral se­lec­tion, ex­cept in de­gen­er­ate cases, there is some weighted con­figu­ra­tion space of ‘pro­pos­als’ (~genomes or gene-com­plexes) at any time, which is ad­justed by the pro­cess. But rarely does it out­right opt ex­clu­sively for a par­tic­u­lar pro­posal. It seems plau­si­ble to me that this proper pro­mo­tive ac­tion is what more so­phis­ti­cated de­liber­a­tors do in many cases, also.

   Alter­na­tively, con­sider that the dis­tinc­tion be­tween ‘opt­ing’ for one pro­posal and ‘weight­ing’ mul­ti­ple con­figu­ra­tions on pro­mo­tion is more of de­gree than kind: ‘opt­ing’ for ex­actly one pro­posal is just a de­gen­er­ate spe­cial case where all other al­ter­na­tives are effec­tively dis­carded (equiv­a­lent to some nom­i­nal zero weight­ing).

10. ↩︎

    Like SGD and bio­chem­i­cal catal­y­sis

11. ↩︎

    Pow­er­ful in the sense of coun­ter­fac­tu­ally most strongly push­ing in a goal-di­rected fash­ion. It is difficult to bot­tom-out these con­cepts!

12. ↩︎

    Ter­minol­ogy note: the con­cep­tu­ally im­por­tant thing is the iter­ated de­liber­a­tion.

    Ruby ob­jected to the use of ‘con­trol sys­tem’ be­cause it evokes clas­si­cal con­trol the­ory, which is a more con­strained no­tion. I think that’s right, but ac­tu­ally evok­ing clas­si­cal con­trol the­ory is part of the rea­son for choos­ing ‘con­trol’: clas­si­cal con­trol sys­tems are a use­ful sub­set of re­ac­tive con­trol­lers in this anal­y­sis (though not the only con­trol­lers)! Clas­si­cal con­trol liter­a­ture already uses the words ‘con­trol­ler’, ‘reg­u­la­tor’, and ‘gov­er­nor’. Alter­na­tive terms if we seek to dis­t­in­guish might be ‘nav­i­ga­tor/​nav­i­ga­tion’ or ‘di­rec­tor/​di­rec­tion’. We might pre­fer ‘gen­er­al­ised con­trol­ler/​con­trol’. I es­chew ‘agent’ as it car­ries far too many con­no­ta­tions. I try to use ‘con­trol­ler’ to evoke clas­si­cal and broader no­tions of se­quen­tial con­trol, with­out nec­es­sar­ily mean­ing a spe­cific the­o­ri­sa­tion as a clas­si­cal open- or closed-loop con­trol sys­tem.

13. ↩︎

    High level in­ten­tions con­di­tion lower-level rout­ing and foot­fall/​hand­hold (etc) place­ment, which con­di­tion micro­scopic atomic mo­tor ac­tions.

14. ↩︎

    Fit­ness in the origi­nal, non­biolog­i­cal sense of be­ing bet­ter-fit i.e. ap­pro­pri­ate­ness to situ­a­tion or con­text. For a heuris­tic this means how much it gets things right, with re­spect to a par­tic­u­lar con­text and refer­ence.

15. ↩︎

    The good reg­u­la­tor and gooder reg­u­la­tor the­o­rems make a Carte­sian sep­a­ra­tion be­tween ac­tor (‘reg­u­la­tor’) and en­vi­ron­ment (‘sys­tem’), and set aside com­pu­ta­tional and log­i­cal con­straints. Un­der these con­di­tions, those the­o­rems tell us that the best pos­si­ble reg­u­la­tor is equiv­a­lent to a sys­tem which perfectly mod­els the goal-rele­vant as­pects of its en­vi­ron­ment and acts in ac­cor­dance with that model.

    For a de­liber­a­tor to satisfy this, its Propose would need to at least once with cer­tainty gen­er­ate the best pos­si­ble pro­posal (rel­a­tive to the goal and ob­served in­for­ma­tion), and its Promote would need to re­li­ably dis­cern and pro­mote this op­ti­mal pro­posal. In the more gen­eral de­liber­a­tor set­ting, it is less clear how to trade off com­pu­ta­tional limi­ta­tions and there is (as yet) no ‘good de­liber­a­tor’ the­o­rem, but this does not pre­vent us from rea­son­ing about ce­teris paribus or Pareto im­prove­ments to those trade­offs.

16. ↩︎

    I have found many dis­cus­sions to typ­i­cally overem­pha­sise what cor­re­sponds here to Promote but it should be­come clear here and in later posts why I think this is miss­ing an im­por­tant part of the story.

17. ↩︎

    It is still use­ful to dis­t­in­guish self-preser­va­tion from goal-con­tent in­tegrity in cases when there is a good de­com­po­si­tion in­clud­ing a priv­ileged ‘goal con­tain­ing’ com­po­nent or com­po­nents of the sys­tem, rea­son­ably de­cou­pled from ca­pa­bil­ities.

18. ↩︎

    This might not be as tau­tolog­i­cal as it seems if there are fun­da­men­tally in­su­per­a­ble challenges to ro­bust del­e­ga­tion.

Be­cause most of the ac­tors that we (my hu­man read­ers and I) at­ten­tively in­ter­act fre­quently with are (pre­sum­ably) com­pu­ta­tion­ally similar (to each other and to our­selves), it is easy for our ab­strac­tions to con­flate phe­nom­ena which are in fact differ­ent^[1], or to cou­ple to­gether im­pres­sions of phe­nom­ena which are in fact sep­a­rable^[2]. On the oc­ca­sions that we at­ten­tively ob­serve dis­similar ac­tors^[3], we are of­ten-enough ‘in the wild’ (i.e. look­ing in their ‘habitat’) that con­flat­ing—an­thro­po­mor­phis­ing—is suffi­ciently use­fully pre­dic­tive to get by^[4], so these poor ab­strac­tions are in­suffi­ciently challenged.

Fur­ther, most of the peo­ple who in fact at­ten­tively ob­serve a par­tic­u­lar class of hu­man-dis­similar ac­tors (enough to per­ceive and un­der­stand the non-con­fla­tions) are fo­cused do­main ex­perts about that par­tic­u­lar class, and talk mainly to other do­main ex­perts about them^[5]. There are few venues in which it is use­ful to have un­am­bigu­ous ter­minol­ogy here. Thus, the lan­guage we use to com­mu­ni­cate our ab­strac­tions about goal-di­rected be­havi­our is prone to con­fla­tion and con­fu­sion even when some peo­ple have some of the right ab­strac­tions^[6].

Here I aim to take steps to break down ‘goal-di­rected be­havi­our’ into a con­cep­tual frame­work of com­pu­ta­tional ab­strac­tions for which I offer ten­ta­tive ter­minol­ogy, and which helps me to bet­ter un­der­stand and de­scribe analo­gies and dis­analo­gies be­tween var­i­ous goal-di­rected sys­tems. The over­ar­ch­ing mo­ti­va­tion is to bet­ter un­der­stand goal-di­rected be­havi­our, in the sense of be­ing able to bet­ter pre­dict its (es­pe­cially coun­ter­fac­tual and off-dis­tri­bu­tion) im­pli­ca­tions, its arisal, and other prop­er­ties. Hope­fully it is clear why I con­sider this worth­while.

In or­der to ground this dis­cus­sion, I re­fer to a rea­son­ably di­verse menagerie of can­di­date goal-di­rected sys­tems, in­clud­ing nat­u­ral and ar­tifi­cial sys­tems at var­i­ous lev­els of or­gani­sa­tion. Con­tem­pla­tion of this di­verse col­lec­tion was re­spon­si­ble for the ideation and re­fine­ment of the ideas and gives some con­fi­dence in the ap­pro­pri­ate­ness of the ab­strac­tions.

A col­lec­tion of a few ‘agents’ drawn from the menagerie

1. ↩︎

   Differ­ent in the sense that, even if the ob­served sur­face phe­nom­ena are similar ‘in the wild’, their be­havi­our in differ­ent con­texts might rad­i­cally come apart; that is, a con­fla­tion is a poor pre­dic­tor for out-of-dis­tri­bu­tion be­havi­our.

2. ↩︎

   Separable mean­ing that the ab­sence of one or other piece is a mean­ingful, con­ceiv­able state of af­fairs, even if in prac­tice they are al­most always found com­posed to­gether; that is, a cou­pled ab­strac­tion means if we have an im­pres­sion of the one, we (per­haps in­cor­rectly) as­sume pres­ence of the other(s).

3. ↩︎

   That is, dis­similar from hu­mans and from each other e.g. ants, genes, chess-en­g­ines, learn­ing al­gorithms, cor­po­ra­tions…

4. ↩︎

   If it is not clear why ‘in the wild’ (or ‘in the typ­i­cal set­ting’) is im­por­tant for the pre­dic­tive­ness of the an­thro­po­mor­phism-heuris­tic, hope­fully De­liber­a­tion and Reflexes and Where does De­liber­a­tion Come From? will clar­ify. In short, an ac­tor fit for a par­tic­u­lar set­ting can carry out ‘de­liber­ate-look­ing’ be­havi­ours with­out ‘de­liber­a­tive ma­chin­ery’, be­cause the pro­cess which gen­er­ated the ac­tor pro­vides enough (slow, grad­ual) ‘de­liber­a­tion’ to lo­cate such be­havi­ours and bake them into ‘re­flexes’.

5. ↩︎

   e.g. en­to­mol­o­gists, or­nithol­o­gists, AI re­searchers, busi­ness ex­ec­u­tives, economists… are not of­ten in a room to­gether, at least not mu­tu­ally-know­ingly in their ca­pac­ity as said ex­perts of their re­spec­tive fields

6. ↩︎

   I at­tempt to avoid use of the term ‘agent’ as it is a very loaded term which car­ries many con­no­ta­tions. In fact it is un­for­tu­nately a perfect ex­em­plary lin­guis­tic vic­tim of the ab­strac­tive con­fla­tion and cou­pling phe­nom­ena I have de­scribed. (I think the re­cent re­cep­tion of the Gato pa­per was con­fused in part as a con­se­quence of this.) I sub­sti­tute ‘ac­tor’ and ‘con­trol­ler’ more freely as less loaded terms.