A taxonomy of approaches to capability amplification

Summary

Q&A: One-shot question-answering

As a starting point, we can pass the task to H and return what they return:

f(x, n) = H(x)

This will only solve problems that H can solve in 15 minutes.

Iteration: A sequence of short tasks

A second idea is to increase the total time available for solving the task by calling H n times, each time providing the problem x, the number of calls remaining, and the previous partial answer. For n=3, this looks like this:

f(x, n=3) = H(x, 0, H(x, 1, H(x, 2, "init")))

This is already much more powerful. It’s question-answering with a potentially very large number of steps for thinking, sketching, and revising. Each instance of H can leave notes for its successor, as long as the notes can be written and read within the available time.

Recursion: Making sub-calls to agents that make sub-calls

As an alternative to iteration, we can allow each instance of H to make sub-calls to further instances, up to a global limit of n calls:

f(x, n) = H(x, H_{limit n})

The limiting case where n is unbounded is HCH.

However, we want to actually implement systems, and ideally get guarantees of roughly the following form: “For any task and any ‘achievable’ solution quality, there is a finite amount of computation that allows the system to solve the task at that quality.” This requires that we define how f behaves for finite n.

Pointers: Handling data abstractly

So far, the size of the data that the collection of Hs can handle has been bounded by the constraint that all information needs to be passed explicitly between different instances of H. This means that the mechanisms so far would fail even on simple tasks like sorting a list with a million elements, since we wouldn’t be able to communicate the list to the first instance of H. We can address this by introducing pointers to data.

A simple way to implement this is using nested messages (as introduced in annotated functional programming).

We’ve previously considered that H could ask the sub-question “What is 300 * 5019?” Instead, H could use square brackets to designate certain parts of the question as sub-messages:

In this case, the next instance of H (let’s call it H₂) sees a message with opaque pointers to data:

H₂ can pass these pointers to sub-questions without looking at their values:

Alternatively, H₂ can expand a pointer to reveal its value:

Using pointers, we can provide large datasets to bounded agents. For example, we could ask H to summarize a book represented as a linked list. Here H looks at the first few sentences:

Pointers don’t always have to be explicit. If we show a page in a book with arrows for flipping to the previous/next page, we can think about this as a combination of pointers and edits.

We don't have to implement pointers using nested messages. Any approach to constructing functional data structures will do. The reason to be functional is that this makes it easy to describe the intended behavior of H: "Return what is most helpful to the agent that instantiated you." If mutations can affect computations far away in the tree of agents in ways that don't flow through the parent, the intended behavior is less clear. (We'll see clean ways to allow mutation when we talk about interaction.)

A mechanism with recursion (or iteration) and pointers is computationally universal:

• Given that our evaluation and data model are close to Scheme, implementing the meta-circular evaluator from SICP on top of our system would probably be straightforward, at least the purely functional version that doesn't have definitions and assignments.
• We could also pick a simple Universal Turing Machine. Each instance of H is responsible for implementing a single step, and so only needs access to the current location of the tape. To allow for bidirectional navigation, the tape could be encoded as a doubly linked list.

Internal dialog: Pointing to agents and sending follow-up messages

We can separate the act of instantiating an agent from sending messages to it. Instantiation creates an agent that hasn’t observed any messages yet, and returns a pointer to it:

When H sends a message to @1, she receives a response and a pointer to that agent’s new state, @2:

Follow-up messages work the same way as initial messages:

We treat agents as functional data structures: @1, @2 and @3 all point to different versions of the same agent. H can send follow-up messages to any of them, as often as she likes. (The command “instantiate” doesn’t actually create an agent—it simply returns a pointer to the agent that hasn’t received any observations.)

Just like pointers to data, pointers to agents can be used in messages:

Internal dialog was introduced in Strong HCH.

Edits: Incremental changes to workspaces

So far, we have treated workspaces as append-only. For example, this is a workspace:

At this point, we’d usually append a next action (and next result) to the body:

Alternatively, our workspaces could support destructive operations (edits and deletions). One approach is to use indexed registers:

In this case, we can apply the same action as before, but we specify a target register (say A). When the call returns, it overwrites the information in register A and the workspace looks like this:

Instead of registers, we could also treat the workspace body as free-form text. In this case, the actions would be more complex, and we’d need a story for how sub-calls and their results interact with the text. (See also: Structured content)

Persistence: Iterative improvement of workspace trees

The schemes described so far are all fairly demanding for H. In any recursive question-answering scheme, if the human H at the top of the tree screws up, the overall answer will be wrong. If one of the intermediate Hs returns the wrong response and the workers consuming this result are not sufficiently careful, it can affect the correctness of the overall response as well. While each H can reduce her responsibility by only doing a tiny bit of work and passing on most of the task to her descendants, she still faces the challenge to get this reduction in responsibility right. If she takes on too much work, or otherwise chooses a bad decomposition, and doesn’t communicate well to what extent her results can be relied upon, flaws can affect the global computation. (See also: Reflection)

Edits address this issue to some extent: They reduce H’s responsibility to only making a small change to a workspace. However, once H is done editing the body and submits a response, the workspace effectively terminates, without room for improvement. This makes sense, since whoever instantiated this sub-problem can then use the response and move on. But it also means that whatever work was done before submission is now set in stone.

By contrast, consider the setting where all workspaces persist even after an initial response has been sent. In this case, we can generate an infinite sequence of tasks for each workspace by asking H to make a small improvement to a workspace we present to her (or to pass, if she doesn’t see a way to improve it). If H adds a new sub-call, we create a new workspace as usual. If there was already a response, and H changes it, we propagate this change to the parent workspace and mark it as stale, i.e. requiring editing by another instance of H. We likewise propagate changes to child workspaces when the question text of sub-calls gets edited, since these workspaces now have a persistent identity independent of the question associated with them.

In this setting, it is natural to start with a “null answer”—

—and to improve it step by step over time by iterating on notes and subcalls in the body and updating the overall response based on their outputs, thus growing and adjusting the computation tree. When parts of the tree of workspaces get edited, these edits propagate to the rest of the tree step by step as we mark dependent structures as stale.

Structured content: Organizing workspaces as documents, registers, trees, etc.

So far, we have seen a few ideas about what individual workspace bodies could look like:

• A list of H’s actions (view, instantiate, ask) and resulting observations
• Named registers, each of which contains an action and observation

This is an axis along which proposals for amplification can vary. Here are some other possibilities:

• Free-form plain text
• Rich text documents
• Tree-structured outlines (as in Workflowy)

If we want to include access to external computation:

• Computational scratchpads (think Jupyter notebook)
• Virtual machine states, along with some Desktop environment

For each choice, we want to limit the visible portion to a fixed finite size, so that we can treat the policy that H implements as finite.

Reflection: Reifying computation for inspection

As the tree of questions and answers (or workspaces) grows, one might worry that the computation will start to break down for non-local reasons. That is, each instance of H is trying to do their best to act in the context they see, and none of them take actions that locally look like mistakes, but since no one can see the big picture, some failures might go unnoticed.

To help address this, we can give H access to a command that reifies the entire tree of workspaces into a data structure:

H can then instantiate sub-agents that reason about properties of this computation.

Or:

If reified computations don’t just include sub-computations, but also information about ancestors of the workspace where “reflect” was executed, this can replace communication to some extent, as long as internal states accumulated by instances of H don’t play a big role relative to explicit content. For example, instead of asking a follow-up question directed at the parent to clarify what unit would be most useful, we can run a sub-computation with access to our ancestors:

Meta-execution: Agents made out of agents

If reflection is about the agent-collective understanding one of its computations after the fact, then meta-execution is about the collective understanding a computation ahead of time through careful planning.

We have looked at a number of mechanisms that all make calls to H. I’ll use M^H to refer to one such mechanism. Each call to H includes some context (e.g., a question) and H returns an action (e.g., a sub-question to ask) that seems good to her in this context. We don’t know how H works—she is a human, her brain is a black box to us.

In meta-execution (introduced here), we replace H with an implementation X that we will provide ourselves. Let’s call the resulting mechanism M^X. If we want to solve object-level tasks such as “Read this book and tell me why x did y.” we present them to M^X.

In the process of solving such tasks, M^X will quickly encounter sub-tasks that it would have presented to H, and will present them to our implementation X instead. Here’s how X works: We create a new cognitive task that looks like this...

...and present it to M^H. Here, #1 is a data structure with information about the current sub-task, but this context is now hidden behind pointers. For example, it might look like this:

#1 = “The question that starts with word #2 and continues with words #3”
#2 = “The word that starts with letter #4 and continues with letters #5”
#4 = “R”
#5 = “The list that starts with letter #6 and continues with letters #7”
#6 = “e”
...

Any actions that don’t look at the entire data structure will be applicable across multiple contexts (see: Caching), and if looking at the value of the data structure would have caused H to malfunction, we may be able to avoid this by analyzing it piecemeal through MH.

In short, meta-execution replaces M^H with M^MH. Our mechanism still bottoms out at H, but we have introduced a level of indirection, so that H chooses meta-level actions, and only indirectly contributes to solving the object-level task.

Previously, we could think about the collection of Hs as operating on a tree of workspaces, with each H responsible for a single change to a single workspace. Now each change itself corresponds to a tree of workspaces that generates that change.

With minor modification, we can also use meta-execution to simulate stateful agents (e.g. if we want to create an agent that maintains state over multiple actions in a workspace). In that case, the context includes the agent’s state, and the returned action includes an updated state.

Interaction: Responding to a stateful world

So far, we have assumed that the problem that our mechanisms are solving is given as a single question. Through the use of pointers, question and response can be large, but they were taken to be fully specified in some sense, with no way to query for more information. We assumed that the only interaction between the system and the real world is through the final response it returns.

How could we enable back-and-forth interaction between our mechanism and the real world?

Out of scope

There are additional mechanisms that will matter a lot for practical implementations, but that are out of scope if we are primarily concerned with what capabilities we can theoretically reach by amplifying human thinking, and don’t care about reducing the number of calls to H:

Existing systems

The table below is a rough attempt to relate the mechanisms above to some existing and proposed systems. Many have key features that are not captured by the mechanisms discussed above, especially multi-user features, so this is an extremely lossy projection.

Hover over shaded cells to reveal clarifying information.