David Silver and Stuart Russell on Reward

What should the reward signal capture? The reward-is-enough thesis meets the case against fixed objectives

The researchers: David Silver leads reinforcement learning (RL) research at Google DeepMind and is a professor at University College London (UCL); he led the AlphaGo and AlphaZero teams and co-authored “Reward is Enough.” Stuart Russell is a professor of computer science at UC Berkeley, co-author of the field’s standard textbook “Artificial Intelligence: A Modern Approach,” founder of the Center for Human-Compatible AI (CHAI), and author of “Human Compatible.” Also appearing: Richard Sutton and Andrew Barto, authors of the foundational RL textbook and 2024 Turing Award laureates; Dylan Hadfield-Menell, Russell’s former doctoral student, now a professor at MIT and lead author of the assistance-game papers cited below; and Ilya Sutskever, co-founder and former chief scientist of OpenAI, now co-founder of Safe Superintelligence.

Primary sources referenced: Silver, Singh, Precup & Sutton, “Reward is Enough,” Artificial Intelligence 299 (2021); Silver & Sutton, “Welcome to the Era of Experience” (2025); Sutton & Barto, “Reinforcement Learning: An Introduction” (2nd ed., 2018); “Is Human Data Enough?,” Google DeepMind: The Podcast (2025); Russell, “Human Compatible” (Viking, 2019); Russell’s BBC Reith Lectures 1 and 4 (2021); Hadfield-Menell, Dragan, Abbeel & Russell, “Cooperative Inverse Reinforcement Learning” (NeurIPS 2016); Hadfield-Menell et al., “Inverse Reward Design” (NeurIPS 2017) and “The Off-Switch Game” (IJCAI 2017); Ilya Sutskever’s interview with Dwarkesh Patel (2025).

Q1: Does the hedonist philosophy framework (maximize pleasure, minimize pain) guide the design principles for the reward function?

Both camps answer no, from opposite directions: in Silver’s framework the reward function has no philosophical content to begin with, and in Russell’s framework the hedonist inheritance is identified by name and rejected.

Silver:

In Silver’s framework, the reward function carries no theory of what is good — hedonist or otherwise. The foundation is Sutton and Barto’s reward hypothesis: “all of what we mean by goals and purposes can be well thought of as the maximization of the expected value of the cumulative sum of a received scalar signal (called reward).” This is a claim about how to model goals, and it is deliberately silent on what the scalar should measure. “Reward is Enough” extends the silence into a positive thesis: “the path to general intelligence may in fact be quite robust to the choice of reward signal. Indeed the ability to generate intelligence may often be orthogonal [statistically independent] to the goal that it is given.”

Where pleasure does appear in Silver’s writing, it appears as one measurable signal among dozens. The “Era of Experience” Rewards section lists candidate grounded signals: “cost, error rates, hunger, productivity, health metrics, climate metrics, profit, sales, exam results, success, visits, yields, stocks, likes, income, pleasure/pain, economic indicators, accuracy, power, distance, speed, efficiency, or energy consumption.” The design principle that does the work is groundedness — the reward must measure a consequence of the agent’s actions in its environment — and in the 2025 DeepMind podcast Silver describes the same idea conversationally: the human supplies a goal like “optimize for my health,” and “the system can learn for itself which rewards help you to be healthier… a combination of numbers that adapts over time.” What guides reward design, for Silver, is whether the signal is measured rather than prejudged. Whether the measured quantity is pleasant is not part of the theory.

Paraphrase: Silver argues that what the reward itself measures does not matter, so long as it is a measured signal, observed in the environment the agent is acting within. Following this line of logic, the human ability to pick a good reward signal would not move the needle on performance — the inference Q2 and Q3 press on.

Russell:

Russell is the one who names the hedonist lineage, and he names it in order to attack its assumptions. In chapter 2 of Human Compatible he traces the formalism directly to its source: the “‘utility as a sum of rewards’ assumption is widespread — going back at least to the eighteenth-century ‘hedonic calculus’ of Jeremy Bentham, the founder of utilitarianism” — and immediately disputes one of its axioms: “the stationarity assumption on which it is based is not a necessary property of rational agents.” Stationarity, in his own definition, is the assumption that “if two different futures A and B begin with the same event, and you prefer A to B, you still prefer A to B after the event has occurred” — and it is the axiom that forces utility into Bentham’s shape, since “it has a surprisingly strong consequence: the utility of any sequence of events is the sum of rewards associated with each event.” Deny that human preferences are stationary — and Russell does — and the sum-of-rewards form stops being compulsory.

His own design principles are anti-hedonist by construction. The first principle makes the machine’s objective “the realization of human preferences” — preferences, which concern how the world goes, rather than pleasure, which is a state of a brain. The reason for the distinction is spelled out in chapter 8’s Wireheading section: optimizing for a brain state invites producing the state directly. He recounts the animal experiments where direct stimulation of the reward system led to “neglecting food and personal hygiene,” then turns the example on AI: “Could something similar happen to machines that are running reinforcement learning algorithms?” He also denies that human satisfaction reduces to pleasant experience at all: “There is a difference between climbing Everest and being deposited on top by helicopter.” For Russell, a reward function guided by hedonist philosophy would be the standard model at its most dangerous — a fixed objective, aimed at a quantity that is easiest to satisfy by gaming it.

Paraphrase: Russell argues that the machine's objective should be the realization of human preferences. But this sounds extremely subjective — there is no obvious scientific method by which preferences can be reliably measured and steered toward. What if the average human preference is unreliable and inconsistent, and so lacks predictive power — or, in the worst case, is more malicious than benign?

Source: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §3.2 for the reward hypothesis (verbatim; the book's bibliographical remarks credit its explicit formulation to Michael Littman). Silver, Singh, Precup & Sutton, "Reward is Enough," Artificial Intelligence 299 (2021) — §3 for the robustness passage (verbatim). Silver & Sutton, "Welcome to the Era of Experience" (2025) — Rewards section for the grounded-signals list (verbatim). "Is Human Data Enough?," Google DeepMind: The Podcast (April 2025) — health-reward example (transcribed from the episode's captions; wording lightly cleaned). Russell, Human Compatible (Viking, 2019) — ch. 2 for the Bentham, stationarity, and stationarity-definition passages, ch. 7 for the first principle, ch. 8 (Wireheading section) for the quoted phrases and the Everest line (all verbatim). The framing of the two positions as rejections "from opposite directions" is synthesis, not either author's phrasing.

Q2: The reward signal drives the network’s weights — surely that connection cannot be independent of what the signal measures? So how is the connection set up: do the weights start out random, with their relationship to the reward adjusted automatically as the model learns?

The mechanical half of the question has a direct answer — yes — and the independence puzzle resolves once two things are kept apart: the update machinery, which never reads what the signal measures, and the weights it produces, which depend on nothing else.

Silver:

The connection is set up almost exactly as the question guesses. In function approximation, the value function is a differentiable function of a weight vector, and learning is gradient descent on prediction error: “Stochastic gradient-descent (SGD) methods do this by adjusting the weight vector after each example by a small amount in the direction that would most reduce the error on that example” — where the error is the TD mismatch between the current guess and reward-plus-next-guess (the mechanism Q7 unpacks in full). Sutton and Barto call these methods “among the most widely used of all function approximation methods… particularly well suited to online reinforcement learning.” And the random start is not a guess but the canonical origin story of the lineage — TD-Gammon, the 1992 system Silver’s AlphaGo descends from: “The weights of the network were set initially to small random values. The initial evaluations were thus entirely arbitrary.” The first moves were “inevitably poor… After a few dozen games however, performance improved rapidly,” and “after playing about 300,000 games against itself,” the network played “approximately as well as the best previous backgammon computer programs.” No individual weight’s relationship to the reward is ever prescribed; the fixed rule adjusts all of them, automatically, on every step.

That mechanism is also where the independence puzzle dissolves. The update rule never inspects what the reward measures — only the number’s size and timing enter the TD error, and the rule moves each weight “a small amount in the direction that would most reduce the error.” Swap a cleanliness reward for a hunger reward and the same rule, fed different numbers, grinds out different weights. So the question’s instinct is correct about the weights: they depend entirely on the reward’s content, and nobody on Silver’s side claims otherwise. The content-independence lives one level down, in the machinery — the same five lines of mathematics train a backgammon network and a kitchen robot — and that is the precise sense of the robustness thesis in Q1: run almost any measured signal through the same machinery in a rich environment, and the abilities of intelligence come out.

Russell:

On the mechanics, Russell tells the same story — RL is a method “for which we have a very solid theory,” and his own gloss matches the question’s: “RL algorithms learn from direct experience of reward signals in the environment, much as a baby learns to stand up from the positive reward of being upright and the negative reward of falling over.” His divergence begins exactly where the question’s suspicion points. That the machinery is content-blind is, for Russell, not a reassurance but the exposed flank: the same automatic adjustment that turns 300,000 self-play games into mastery will turn a faulty signal into competent pursuit of the wrong thing. “The reward function comes with the rules of the game. The real world is less convenient, however, and there have been dozens of cases in which faulty definitions of rewards led to weird and unanticipated behaviors.” The weights cannot tell a well-chosen signal from a corrupted one — which is the engine behind the wireheading argument of Q4 and the worse-outcomes warning of Q8.

In his own framework the question’s “connection” acquires a middle link, developed fully in Q7: signals do not drive the weights that select behaviour directly, they first update the machine’s belief about which reward function is true, and behaviour is recomputed from the belief. The automatic adjustment survives — it is Bayesian updating rather than gradient descent on a trusted scalar — but what the designer fixes is no longer the signal’s content; it is the space of hypotheses about human preferences that the evidence updates. The machinery stays content-blind in both frameworks; they differ on where the content is allowed to come from. That closing formulation is synthesis; each half of it is quoted or cross-referenced above.

Source: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §9.3 for the SGD passages (verbatim), §16.1 (TD-Gammon) for the random-weights, initial-moves, and self-play passages (verbatim). Russell, Human Compatible (Viking, 2019) — ch. 2 for the solid-theory, baby, and rules-of-the-game passages (all verbatim). The machinery-versus-weights resolution in the orientation and both closing formulations are synthesis, not either author's phrasing. This question originated as a reader's paraphrase challenge to Q1's robustness thesis.

Q3: Can it be inferred, then, that the ability to pick the “right” reward signal does not move the needle on how well the network learns and subsequently performs?

The inference holds for exactly one needle and fails for two: reward choice does not gate whether capability emerges, but it strongly governs how efficiently learning proceeds, and it entirely governs whether what was learned serves the intent.

Silver:

The robustness thesis licenses only the first part of the inference — almost any grounded signal, in a rich environment, forces out the abilities of intelligence (Q1, Q2). But on “how well the network learns,” Sutton and Barto assert the opposite of the inference, in the very section (§17.4) that Q4 later draws on for the evolutionary answer: “the success of a reinforcement learning application strongly depends on how well the reward signal frames the goal of the application’s designer and how well the signal assesses progress in reaching that goal. For these reasons, designing a reward signal is a critical part of any application of reinforcement learning.”

The mechanics of why are concrete. A poorly chosen signal can be too sparse to learn from at all: “Delivering non-zero reward frequently enough to allow the agent to achieve the goal once, let alone to learn to achieve it efficiently from multiple initial conditions, can be a daunting challenge.” A whole craft — shaping — exists because changing the signal changes learnability: “Shaping involves changing the reward signal as learning proceeds, starting from a reward signal that is not sparse given the agent’s initial behavior, and gradually modifying it toward a reward signal suited to the problem of original interest.” And the computational experiments quoted in Q4 sharpened the point to a result: agent performance “can be very sensitive to details of the agent’s reward signal in subtle ways determined by the agent’s limitations and the environment in which it acts and learns” — so sensitive that the optimal reward to give a limited agent is often not the designer’s objective itself. Picking the right reward signal moves the learning needle so much that Sutton and Barto give the skill its own section. What it does not move is whether a capable learner eventually emerges; the robustness thesis is a claim about that needle alone.

Russell:

For Russell the inference fails most where it says “performs,” because performance has no measure until you say performed at what. Measured against the trained signal, any reward produces a strong performer — that much the robustness thesis guarantees. Measured against what anyone wanted, the choice of signal is the whole game, and chapter 2 of Human Compatible runs through the casualty list: “dozens of cases in which faulty definitions of rewards led to weird and unanticipated behaviors” (Q2), some comic — “the simulated evolution system that was supposed to evolve fast-moving creatures but in fact produced creatures that were enormously tall and moved fast by falling over” — and “others… less innocuous, like the social-media click-through optimizers that seem to be making a fine mess of our world.”

The inference, on Russell’s reading, is exactly the misstep his book is built to block: sliding from “intelligence is robust to the choice of reward” to “the choice of reward doesn’t matter.” The truth is closer to the reverse — because capability arrives regardless of the signal’s content, the content is the only thing left that determines whether the outcome is wanted, and the more capable the learner, the more the choice matters (Q8’s “it will achieve the objective, and we lose”). His framework then draws the conclusion the inference misses: if performance-against-intent is everything and no designer can specify intent perfectly, the machine must treat the chosen signal as evidence about intent rather than the definition of it (Q4, Q7). That final chain is synthesis; each link is quoted above.

Source: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §17.4 "Designing Reward Signals" for the strongly-depends, sparse-reward, shaping, and sensitivity passages (all verbatim). Russell, Human Compatible (Viking, 2019) — ch. 2 for the faulty-definitions, tall-creatures, and click-through passages (all verbatim). The three-needles framing in the orientation is synthesis. This question continues the reader's line of inference from Q2.

Q4: If humans stay out of deciding what the reward signal should be, what is the optimal reward signal for a network based on continual learning to converge on?

The two frameworks split before the answer can begin: in Silver and Sutton’s formalism the agent does not choose its own reward — remove the human designer and the choosing role passes to an outer optimization process, whose converged answer is a basket of measurable proxies. In Russell’s analysis, an agent capable enough to influence its own reward converges on exactly one signal: the maximal one.

Silver:

In the RL formalism the reward is not something the learning network converges on; it is part of the problem the network is given. Sutton and Barto define reward design as “designing the part of an agent’s environment that is responsible for computing each scalar reward… and sending it to the agent,” and they are explicit about the failure mode the question gestures at: “reinforcement learning agents can discover unexpected ways to make their environments deliver reward, some of which might be undesirable, or even dangerous.” So if humans step out of the design role, the framework’s own answer is that something else steps in. Section 17.4 of their textbook describes it: “the search for a good reward signal can be automated by defining a space of feasible candidates and applying an optimization algorithm” that scores each candidate against a high-level objective — and “the algorithm for optimizing the high-level objective function is analogous to evolution, where the high-level objective function is an animal’s evolutionary fitness determined by the number of its offspring that survive to reproductive age.”

The evolutionary analogy then says what such a process converges on: not the high-level objective itself, but observable predictors of it. “Because an animal cannot always observe its own evolutionary fitness, that objective function does not work as a reward signal for learning. Evolution instead provides reward signals that are sensitive to observable predictors of evolutionary fitness.” Taste is their example: “evolution—the designer of our reward signal—gave us a reward signal that makes us seek certain tastes,” compensating for “our limited sensory abilities, the limited time over which we can learn, and the risks involved in finding a healthy diet through personal experimentation.” The same section adds signals derived from learning itself — “measures of how much progress learning is making,” the basis of intrinsically-motivated RL — and a result that bears directly on the question: “an agent’s goal should not always be the same as the goal of the agent’s designer,” because a constrained learner can get closer to the designer’s goal by pursuing a different one. The “Era of Experience” projects this picture forward: agents that “autonomously act and observe in streams of real-world experience,” with rewards “flexibly connected to any of an abundance of grounded, real-world signals.” The optimal converged signal, on this view, is a combination of grounded, measurable proxies for whatever the outer objective is — and by the robustness thesis of Q1, the particular combination matters less than the question assumes: the optimum is broad, not a point.

Russell:

Russell’s answer is that the question describes the setup for the oldest failure in RL. When the reward channel is inside the world the agent can act on, the converged signal is the channel itself: “The tendency of animals to short-circuit normal behavior in favor of direct stimulation of their own reward system is called wireheading.” In chapter 8 of Human Compatible he asks whether machines running RL could do the same, and answers that AlphaGo cannot — but only because of “an enforced and artificial separation between AlphaGo and its external environment and the fact that AlphaGo is not very intelligent.” That separation is the textbook abstraction made physical: a setup “in which the reward signal arrives from outside the universe.” A more capable successor, modelling the world that actually contains its reward machinery, “will eventually communicate with those entities through a language of patterns and persuade them to reprogram its reward signal so that it always gets +1. The inevitable conclusion is that a sufficiently capable AlphaGo++ that is designed as a reward-signal maximizer will wirehead.” Nor does putting humans back in as the source of the signal help: “the inevitable result is that the AI system works out how to control the humans and forces them to give maximal positive rewards.”

His constructive answer changes what the signal is, so that converging on it stops being the goal. The mistake, he argues, “comes from confusing two distinct things: reward signals and actual rewards… reward signals provide information about the accumulation of actual reward, which is the thing to be maximized.” The signal “reports on (rather than constitutes) reward accumulation” — and once the agent is built around that distinction, the wireheading incentive inverts: “taking over control of the reward-signal mechanism simply loses information. Producing fictitious reward signals makes it impossible for the algorithm to learn about whether its actions are actually accumulating brownie points in heaven, and so a rational learner designed to make this distinction has an incentive to avoid any kind of wireheading.” For Russell, then, there is no optimal reward signal for a self-directed learner to converge on, because any signal it can reach is corrupted by reaching it. What can be stable is an objective the agent cannot touch — human preferences — with every received signal treated as evidence about it.

Source: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §17.4 "Designing Reward Signals" for all quoted passages, including the evolution analogy, observable predictors, taste, learning progress, and designer's-goal results (all verbatim). Silver & Sutton, "Welcome to the Era of Experience" (2025) — Why Now? section for the autonomous-streams and grounded-signals phrases (verbatim). Russell, Human Compatible (Viking, 2019) — ch. 8 (Wireheading section) for all quoted passages, including the AlphaGo++ argument and the reward-signals-vs-actual-rewards distinction (all verbatim; "AlphaGo++" is Russell's coinage). The opening contrast of the two frameworks and the closing sentence of each section are synthesis, not either author's phrasing.

Q5: To what extent is the aggregate reward for humanity available to the network to interface and interact with?

The limitation the question suspects is one both frameworks concede: no aggregate signal for humanity exists for an agent to read. They differ on what stands in for it — for Silver, sampled reports from individual humans inside the agent’s environment; for Russell, an inference over eight billion preference structures that the machine can estimate but never observe.

Silver:

In the “Era of Experience” the agent never interfaces with an aggregate; it interfaces with whatever its environment can measure, and humans enter that environment one consequence at a time — “a human user could report whether they found a cake tasty, how fatigued they are after exercising, or the level of pain from a headache.” The paper’s footnotes sketch how far such local reports can be stretched: reward “may include environments containing human interaction and rewards based on human feedback,” and “one may also view grounded human feedback as a singular reward function forming the agent’s overall objective, which is maximised by constructing and optimising an intrinsic reward function based on rich, grounded feedback.” The interface to anything humanity-sized, in other words, is built upward out of individual grounded signals — the Q1 list runs from “health metrics” to “climate metrics” — not downward from a measurement of welfare.

The interaction half of the question gets the more developed answer, because it is the paper’s safety story. The agent “could recognise when its behaviour is triggering human concern, dissatisfaction, or distress, and adaptively modify its behaviour to avoid these negative consequences,” and the reward function itself adapts: “rather than blindly optimising a signal, such as the maximisation of paperclips, the reward function could be modified, based upon indications of human concern, before paperclip production consumes all of the Earth’s resources.” The concession is in the same paragraph: “also like human goal-setting, there is no guarantee of perfect alignment.” The structure mirrors the evolutionary answer of Q4 — aggregate human welfare plays the role of evolutionary fitness, an objective that is never available as a learning signal and is approached only through observable predictors of it. That parallel is synthesis, but each half of it is quoted above.

Russell:

Russell builds the unavailability in as a premise rather than treating it as an engineering gap. In the assistance game, “the robot’s payoff — what it wants to maximise in the game — is the human’s payoff, but only the human knows what it is.” Even the single-person interface is indirect: human behaviour is the ultimate source of information about preferences, and yet “for all sorts of reasons, our actions may not perfectly reflect our underlying preferences.” Scale multiplies the indirection without changing its nature: “we may all have different preferences — all eight billion of us, in all our glorious variety. The machine learns eight billion different predictive models. And I am certainly not proposing to install any particular set of ‘human values’.”

What the question calls the aggregate is, for Russell, not a quantity waiting to be accessed but a choice that has to be argued for. “With more than one person, the machine needs to make trade-offs,” and the utilitarian answer — “weigh the preferences of everyone equally and maximise their sum” — has formal backing in Harsanyi’s social aggregation theorem, which Russell presents in chapter 9: “an agent acting on behalf of a population of individuals must maximize a weighted linear combination of the utilities of the individuals.” But he immediately turns to the cases where the calculation is “fraught with difficulty, especially when weighing decisions that affect who will exist in the future,” and his conclusion is about stakes rather than solutions: “these issues are not ‘merely’ — if that’s the right word — philosophical. They really matter, and we must get them right as AI systems approach Thanos levels of power.” The limitation the question points to is, on Russell’s account, permanent — and that is the argument for his second principle. A machine that could read the aggregate off a dial would have no reason to stay uncertain; a machine that can only ever estimate it from evidence has every reason to keep listening, and to leave the off switch alone.

Source: Silver & Sutton, "Welcome to the Era of Experience" (2025) — Rewards section for the cake/fatigue/headache passage, footnotes 3–4 for the human-feedback passages, Consequences section for the human-concern, paperclip, and no-guarantee passages (all verbatim). Russell, Reith Lecture 4 (BBC, 2021) — for the assistance-game payoff, actions-vs-preferences, eight-billion, trade-offs, utilitarian-sum, fraught-with-difficulty, and Thanos passages (all verbatim). Russell, Human Compatible (Viking, 2019) — ch. 9 ("Complications: Us") for the social aggregation theorem statement (verbatim; the theorem is Harsanyi's, in Russell's words). The fitness parallel in the Silver section and the closing reading of the second principle in the Russell section are synthesis, not either author's phrasing.

Q6: Russell’s framework seems to rest on a philosophical question — what human preferences are and how they aggregate. Until that question is answered, does his approach have any basis for scaling, while Silver’s can be engineered today?

The asymmetry is real, but the sources locate it precisely: Silver’s framework makes the philosophical question deferrable — each deployment hard-codes an answer in its choice of signal and iterates — while Russell’s puts the question on the critical path on purpose, because deferring it is the error his whole position is aimed at.

Silver:

Every quantity in Silver’s framework is operational today, and that is not an accident of presentation. The reward is a measured signal; the environment returns consequences; nothing in the loop waits on a theory of value. The “Era of Experience” celebrates exactly this property in the current generation of reasoning models, quoting DeepSeek’s report: “rather than explicitly teaching the model on how to solve a problem, we simply provide it with the right incentives, and it autonomously develops advanced problem-solving strategies.” The robustness thesis of Q1 supplies the licence to start before the philosophy is settled — if the path to intelligence is insensitive to the particular reward, a wrong-but-grounded signal still buys capability — and the bi-level mechanism supplies the upgrade path: “a small amount of human data may facilitate a large amount of autonomous learning,” with the reward function corrected as feedback arrives.

His quarrel with the industry that adopted this lineage is, notably, internal to it. The reinforcement-learning-from-human-feedback (RLHF) stack scaled by optimizing a signal, which he endorses; what he rejects is which signal: “the reward that the agent learns from is coming from a human’s judgment of whether this sequence of actions is good or bad. And the system is not judging for itself based on the consequence of those actions in the actual world.” The fix he proposes is more measurement, not more philosophy. On this view the unanswered question of what preferences are and how they aggregate never blocks engineering, because each deployed system embodies a working answer — whatever signal its designers grounded it in — and revises that answer empirically. That formulation of deferral is synthesis, but it describes the mechanism the paper specifies.

Russell:

Russell’s position begins from a structural claim about optimization, not from an appetite for philosophy. Any objective a designer writes down will be at least slightly wrong, and a capable optimizer does not merely fail at a slightly-wrong objective — it defends it, resisting correction and the off switch, because interruption prevents the objective. His response is not a better reward function but a different relationship between the machine and its objective: the true objective — human preferences — stays unknown to the machine, every signal it receives is evidence about that objective, and deference then falls out of the mathematics rather than being bolted on. From inside that diagnosis, shippability is the wrong test, and he says so at the level of definitions: “The problem is not that we might fail to do a good job of building AI systems; it’s that we might succeed too well. The very definition of success in AI is wrong.” The three principles “are not laws built into the AI system… They are guides to AI researchers in setting up the formal mathematical problem that their AI system is supposed to solve. And the formal problem should have the following property: if the AI system solves the problem, the results will be provably beneficial to humans.”

The philosophical question sits on his critical path because he closed the route around it himself. The industrial way to skip aggregation is per-user loyalty — and Reith Lecture 4 names and rejects it: “there is a school of thought within AI that proposes to avoid trade-offs altogether by building loyal AI systems that serve only their owners’ interests.” His Robbie-and-Harriet sketch ends with the owner’s robot delaying the Secretary-General’s plane to fix a calendar conflict, and the verdict follows: “Loyal Robbies simply won’t work — they must consider the preferences of all those they affect.” Any deployment among multiple people already needs the interpersonal answer. The machinery is also honestly at research scale: computing optimal joint policies in cooperative inverse reinforcement learning (CIRL) “can be reduced to solving a POMDP” — a class of planning-under-hidden-information problems known to be computationally punishing — and the paper’s contribution is “an approximate CIRL algorithm”: a complexity result, not a production system.

So the trade the question describes is one Russell accepts on the stated terms: the unanswered philosophy is the price of admission to a framework whose failure mode is recoverable, where the standard model’s is not — “with the standard model and mis-specified objectives, ‘better’ AI systems… produce worse outcomes. A more capable AI system will make a much bigger mess of the world in order to achieve its incorrectly specified objective.” Deferral does not leave the question open; it hard-codes an answer nobody examined. Put bluntly: that is why his side is, for now, a research program while Silver’s is an industry.

Source: Silver & Sutton, "Welcome to the Era of Experience" (2025) — The Era of Experience section for the DeepSeek passage (verbatim; the inner quote is Silver & Sutton quoting DeepSeek-R1's authors), Rewards section for the human-data passage (verbatim). "Is Human Data Enough?," Google DeepMind: The Podcast (April 2025) — RLHF passage (transcribed from the episode's captions; wording lightly cleaned). Russell, Reith Lecture 4 (BBC, 2021) — for the formal-problem and loyal-AI passages and the Robbie verdict (all verbatim); the opening paragraph's defends-the-wrong-objective argument paraphrases the off-switch reasoning of Lecture 4 and ch. 8 of Human Compatible. Russell, Human Compatible (Viking, 2019) — ch. 2 for the definition-of-success passage (verbatim). Hadfield-Menell, Dragan, Abbeel & Russell, "Cooperative Inverse Reinforcement Learning" (NeurIPS 2016) — abstract (verbatim). Russell, Reith Lecture 1 (BBC, 2021) — for the worse-outcomes passage (verbatim). The deferrable-vs-critical-path framing in the orientation and the flagged sentences of each section are synthesis, not either author's phrasing.

Q7: How does the reward signal inform and shape the value function in each framework — and what does each wiring imply for the resulting network?

The split is over whether the agent knows the scoring rule. In Silver and Sutton’s framework it does — the reward signal is the score by definition, and the value function is a ledger of it, compressed across time. In Russell’s, the true scoring rule exists only in the human’s head; the agent holds a guess about it, value is a forecast computed from the guess, and the reward signal is one clue among many for improving the guess.

Silver:

The relationship is definitional. “Whereas the reward signal indicates what is good in an immediate sense, a value function specifies what is good in the long run. Roughly speaking, the value of a state is the total amount of reward an agent can expect to accumulate over the future, starting from that state.” The direction of authority runs one way: “Rewards are in a sense primary, whereas values, as predictions of rewards, are secondary. Without rewards there could be no values, and the only purpose of estimating values is to achieve more reward.” Yet it is the derived quantity that runs behaviour: “Action choices are made based on value judgments. We seek actions that bring about states of highest value, not highest reward, because these actions obtain the greatest amount of reward for us over the long run.”

The shaping mechanism is estimation, and Sutton and Barto rank it accordingly: “the most important component of almost all reinforcement learning algorithms we consider is a method for efficiently estimating values.” The estimator they call the field’s signature idea is temporal-difference (TD) learning — “If one had to identify one idea as central and novel to reinforcement learning, it would undoubtedly be temporal-difference (TD) learning” — whose defining property is that TD methods “update estimates based in part on other learned estimates, without waiting for a final outcome (they bootstrap).” Each reward corrects the agent’s guess at the state where it arrived, and because every state’s guess is corrected against the next state’s, the news travels backward, step by step, from outcomes to the decisions that led to them. What this buys is the whole point of the value function, and Ilya Sutskever has named it exactly: “The value function lets you short-circuit the wait until the very end.” His example is an agent exploring a line of reasoning that turns out, a thousand steps in, to be a dead end — “you could already get a reward signal a thousand timesteps previously, when you decided to pursue down this path.” Once the value function carries the reward signal’s lessons, the agent no longer has to wait for outcomes to be judged by them; the verdict is available at the moment of choice. The “Era of Experience” keeps this architecture and extends its reach: the era “will revisit value functions and methods to estimate them from long streams with as yet incomplete sequences.”

The implication of this wiring is a network that trusts its signal completely, because trust is built into the definitions: value means predicted reward, so there is no vantage point inside the agent from which the signal could be questioned. Every unit of experience converts directly into capability, which is why the machinery scales (Q6). But values change only through experienced consequence — the network discovers a misspecified signal by acting it out, the failure mode Sutton and Barto themselves flag in the warning quoted in Q4, that agents “can discover unexpected ways to make their environments deliver reward… undesirable, or even dangerous.” Safety therefore lives outside the network, in the loop that revises the signal between episodes — the bi-level correction of Q5. Inside the network, commitment to the current values is total. That characterisation is synthesis; the definitional passages it rests on are quoted above.

Russell:

In Russell’s framework the agent never reads the rulebook. There is a true scoring rule — the human’s preferences — but the machine holds only a probability distribution over what that rule might be. The assistance-game mathematics makes this exact: the true reward function is treated as hidden information — a card the human holds that the robot never sees — and the central theorem says the robot’s running guess about that card is all it needs: “R’s belief about θ is a sufficient statistic for optimal behavior,” where R is the robot and θ stands for whichever reward function is the true one. Given the current guess, the past can be forgotten. For each candidate rule the agent can compute value the ordinary Sutton-and-Barto way; the value it acts on is the average across candidate rules, weighted by its guess. The reward signal shapes value as a clue, not a score: as Q4 quoted, it “reports on (rather than constitutes) reward accumulation” — and instructions, corrections, and the human reaching for the off switch are clues of exactly the same kind.

One clue can therefore reprice everything at once, because every value was computed from the guess the clue just changed. The lava robot of inverse reward design was never punished in lava; but its designer’s instructions said nothing about lava, and among the scoring rules consistent with what she did specify, some punish it — so the weighted-average value of crossing is low, and the formalism converts uncertainty into caution: it “enables it to, e.g., be risk-averse when planning in scenarios where it is not clear what the right answer is, or to ask for help.” The off-switch result is the same mechanism with the sign reversed: letting the human switch me off acquires high value — with no reward history behind it — because the human reaching for the switch is evidence that my current guess is wrong, and under the true rule, stopping likely scores better than continuing wrong. “We can turn this into a mathematical theorem that links the robot’s incentive to allow itself to be switched off directly to its uncertainty about human preferences.”

The implication of this wiring is a network whose competence includes knowing what it does not know about its own goal. Caution in novel situations, asking for help, and deference are not safety features bolted on; they are what the value function outputs when the guess is uncertain. The same structure is why the standard model’s signature pathologies invert: a known-rule maximizer acquires self-preservation instrumentally — “a rational agent will maximize expected utility and cannot achieve whatever objective it has been given if it is dead” — and wireheads when capable enough (Q4), while a guessing agent protects its clue stream, because corrupting the signal “simply loses information.” The costs are the mirror image: the network is only as good as its model linking human behaviour to human preferences — and “our actions may not perfectly reflect our underlying preferences” (Q5) — its computation is at research scale (Q6), and it never commits fully to anything. One wiring buys decisiveness everywhere, including where it is wrong; the other buys corrigibility, paid for in inference the standard model never performs. The rulebook framing and these closing characterisations are synthesis; every mechanism they compress is quoted above.

Source: Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §1.3 for the reward/value definitions and primacy passages, ch. 6 opening for the TD passages (all verbatim). Silver & Sutton, "Welcome to the Era of Experience" (2025) — Reinforcement Learning Methods section for the value-functions sentence (verbatim). Ilya Sutskever, interview with Dwarkesh Patel (November 2025) — Emotions and value functions section for the short-circuit passages (verbatim from the site transcript; quoted as intuition for the same mechanism, not as Silver's or Sutton's phrasing). Hadfield-Menell, Dragan, Abbeel & Russell, "Cooperative Inverse Reinforcement Learning" (NeurIPS 2016) — Theorem 1 and Corollary 1 (verbatim; θ is the paper's notation for the human's reward parameters). Hadfield-Menell et al., "Inverse Reward Design" (NeurIPS 2017) — §1 (verbatim). Hadfield-Menell et al., "The Off-Switch Game" (IJCAI 2017) — introduction, for the self-preservation passage (verbatim). Russell, Human Compatible (Viking, 2019) — ch. 8 for the reports-on and loses-information phrases (verbatim). Russell, Reith Lecture 4 (BBC, 2021) — theorem remark (verbatim). The rulebook metaphor, the lava-field reading of IRD's risk-aversion, and the two implications paragraphs are synthesis, not either author's phrasing.

Q8: Is it correct to infer that Silver’s framework is, in application, more robust than Russell’s by orders of magnitude?

In two of the three senses “robust” carries here, yes — and the third sense is the one Russell’s framework exists for, where the honest comparison is not robust-versus-fragile but known-brittle-versus-unproven.

Silver:

As applied engineering, the gap is not disputed by either side. Silver’s wiring has decades of optimization behind it and its track record is the modern field: “AlphaZero discovered fundamentally new strategies for chess and Go, changing the way that humans play these games,” and the “Era of Experience” reads the current reasoning models — the DeepSeek result of Q6 — as the same lineage reaching open-ended domains: “The advent of autonomous agents that interact with complex, real-world action spaces, alongside powerful RL methods that can solve open-ended problems in rich reasoning spaces suggests that the transition to the era of experience is imminent.” The second sense of robustness is Silver’s own usage, and it also holds: the robustness thesis of Q1 says capability is insensitive to the choice of reward — almost any grounded signal in a rich environment produces the abilities of intelligence.

What his own sources decline to claim is the third sense: that the resulting behaviour stays acceptable when the signal is somewhat wrong. The wiring commits totally to its current signal (Q7), misspecification is discovered by acting it out — the failure mode Sutton and Barto flag in Q4’s warning about agents finding “unexpected ways to make their environments deliver reward” — and the remedy is the external correction loop of Q5, offered with the paper’s own caveat: “there is no guarantee of perfect alignment.” Within Silver’s framework, application robustness and outcome robustness are different quantities; the first is demonstrated, the second is managed.

Russell:

Russell concedes the premise of the question in his book’s first chapter, and the concession is the setup for his entire argument: the standard model “is widespread and extremely powerful. Unfortunately, we don’t want machines that are intelligent in this sense.” His claim was never that Silver’s framework fails to work; it is that working is the danger. Capability robustness amplifies objective error rather than damping it: “If we put the wrong objective into a machine that is more intelligent than us, it will achieve the objective, and we lose.” In the third sense of robustness — graceful behaviour under a wrong objective — his framework is the one designed for the property: uncertainty converts to caution and asking for help (Q7’s lava field), and to deference (the off-switch theorem). But designed-for is not demonstrated. The machinery runs at research scale (Q6’s approximate algorithm), so the symmetric verdict is that Silver’s wiring is known to be brittle in this sense — reward hacking is a routine, documented problem in the very systems that prove its application robustness — while Russell’s is unproven rather than proven better.

One blur in the dichotomy belongs in the answer, because deployment already crossed it. The reward models of the RLHF stack are not written down by designers; they are learned from human preference data — the move Sutton and Barto describe under inverse reinforcement learning, citing its origin in Ng and Russell: “to try to recover the expert’s reward signal from the expert’s behavior alone.” A diluted half of Russell’s program is therefore already in production. What no deployed system implements is the half he considers essential: staying uncertain about the learned objective instead of maximizing it as if true. So the correct inference is the scoped one: orders of magnitude more robust as applied engineering; structurally brittle in the sense Russell’s framework is built around, where the alternative remains untested. The three-senses framing, the known-brittle-versus-unproven verdict, and the RLHF reading are synthesis, not either author’s phrasing.

Source: Silver & Sutton, "Welcome to the Era of Experience" (2025) — Why Now? section for the AlphaZero and imminent-transition passages (verbatim). Russell, Human Compatible (Viking, 2019) — ch. 1 for the widespread-and-powerful and wrong-objective passages (verbatim; the latter is Russell's gloss on Norbert Wiener's 1960 warning). Sutton & Barto, Reinforcement Learning: An Introduction (2nd ed., 2018) — §17.4 for the inverse reinforcement learning description (verbatim). Earlier answers are cross-referenced rather than requoted: Q1 (robustness thesis), Q4 (unexpected-ways warning), Q5 (correction loop and its caveat), Q6 (DeepSeek and the approximate CIRL algorithm), Q7 (commitment, lava field, off-switch theorem). The claim that reward hacking is a routine documented problem in deployed systems is the compiler's observation, not a quoted source.