Response Synthesis

Strategic Direction

Research focus and pivotal questions

Two broad clusters emerged, with a real tension between them that David Reinstein flags directly.

The AI × economics/labor markets cluster drew the most convergence. Pacchiardi endorses the AI-economics interface as the most relevant focus area"AI-economics interface (eg impacts on labour market [Anthropic's work] or macro-trends à la Ord) seems the most relevant area"— Lorenzo Pacchiardi, echoed by Garavaglia, Du (labor welfare, AI literacy, inequality), Habermacher (realpolitik regulatory frameworks for labor impacts), and Tagat. This sits comfortably in UJ's wheelhouse of quantitative social science. DR pushes back on the framing: "Very much in our wheelhouse, and there is strong work. But does this really get at the GCR-relevant issues / the issues with the highest global impact?" DR also questions why Tagat's observation that there is "a lot of ongoing work" on labor impacts was framed as a warning — from UJ's curation role, a rich pipeline is a positive.

The governance / safety / risk-parameters cluster is more GCR-relevant but stretches UJ's comfort zone. Kalkar identifies a gap in regulatory interventions and risk parameter estimation"a gap in regulatory interventions, comparative international governance analysis, and estimates of risk parameters for AI safety and governance"— Uma Kalkar, with concrete RQs on what changes frontier lab behavior and how capability thresholds translate to regulatory triggers. Schwab pushes further into international relations — middle-power strategies, China cooperation lessons from arms control and CBW prohibitions. DR responses: risk-parameter estimation "seems quantitative, which is more comfortable to us"; regulatory-intervention work needs concrete examples to assess fit; middle-power/China work "would be a bit of a move from our current approach and perhaps our comfort zone" — not ruled out but requires seeing exemplars.

Habermacher's "realpolitik" framing needs unpacking per DR: what research, what methods, what decisions could it inform? Tagat suggests engaging AI catastrophic-risk funders directly to evaluate work they commission"engage with funders of AI and catastrophic risk, alignment etc...it might be useful to reach out to them to provide an open evaluation of the work that they are commissioning"— Anirudh Tagat — a promising bridge between the two clusters.

Staying timely

Convergence on lower evaluator thresholds: Manheim proposes fast-track with 1+ evaluators, inviting multiple and publishing as soon as one submits. Garavaglia suggests one evaluator plus AI-assisted briefing. Kao agrees multiple evaluators remain valuable but shouldn't block publication.

Pacchiardi's pre-booking idea drew explicit DR enthusiasm: reserve evaluator time before paper selection so timely turnaround is guaranteed"pre-booking evaluator's time before deciding what paper to review so that you can choose a paper and are sure that someone will be able to look at that in a timely manner"— Lorenzo Pacchiardi. DR: "This seems like a great idea to me…as well as giving evaluators some limited choice over which ones they would like to evaluate." Paying more for flexibility is a complementary lever.

Pacchiardi also proposes crux-targeted reviews requiring only ~1 hour of evaluator time"targeting very specific cruxes in a paper (eg highlighted from the evaluation explorer) could be more efficient, for instance requiring reviewers to be useful with 1 hour of work only"— Lorenzo Pacchiardi. Kao floats ACX/Zvi-style commentary roundups as a complement to full evaluations"something along the lines of ACX/Zvi Moshowitz style 'commentary roundups' that presents clusters of comments made by others online + light discussion"— Andrew Kao. Kalkar warns against over-automation — LLM eval + prioritization is fine but standards must hold.

Who to bring in

Concrete names and networks: GovAI current/former fellows (Manheim, Kalkar), RAND TASP Fellows (Kalkar), BlueDot (Manheim), Jonathan Prunty (Cambridge CFI) and Marko Tesic (UK DSIT) for labor-market work (Pacchiardi), and the ILO plus AI developers for labor-welfare framings (Du). Tagat endorses circulating a call through the AI Slack channel to crowdsource stakeholder ideas. Kao points to less-formal but high-effort pieces (Trammell/Patel on "Capital in the 22nd Century"; Citrini's 2028 intelligence-crisis piece) as candidates for the evaluation pipeline.

The Ord RL Scaling Evaluation

Should we still pursue it?

Initial responses split. Schwab said the moment has passed"Moment passed."— Josephine Schwab; Habermacher and Manheim were lukewarm; Pacchiardi remained on the fence. Kalkar was the strongest advocate — given messier empirical picture and Ord's 2027+ reframe, first-mover value now"Given that the empirical picture has become messier and more contested, it may actually make sense to be a 'first-mover' and evaluate it now"— Uma Kalkar.

This calculus has shifted. The author has since been contacted and is supportive of a more formal evaluation.

That answers part of the "moment passed" view (Schwab, Habermacher). An evaluation should still pin down specific quantitative claims from Ord's published posts, as distinct from the "RL has hit a limit" headline reading that some critics have attributed to him; documenting the gap between the two is itself useful.

If yes — what form?

Kalkar's recommendation stands and gains force: a long-form EA Forum / PubPub post covering the full evidence base"a long-form EA Forum/ PubPub post. It should cover the full evidence base listed above with clear separation between what's empirically established, what's contested"— Uma Kalkar, with clear separation of established / contested / open questions, and draft circulation to the GovAI community (Ord included as courtesy). The piece should center a specific, evaluable quantitative claim from Ord's published posts.

Points of apparent consensus

• Do not expand into technical AI safety as core focus — near-unanimous. DR partially concurs but notes the space may not be saturated in "rapid, expert, credible evaluations" (Alignment Journal is narrowing to alignment only).
• Lower the evaluator-count threshold for timely pieces; single-evaluator fast-track is acceptable when paired with AI-assisted briefing.
• Pre-booking evaluator time (with some evaluator choice over papers) is a promising operational reform.
• Recruit from GovAI, RAND TASP, and existing UJ AI Slack networks.

Key tensions worth discussing

• Wheelhouse vs. impact: AI × labor economics fits UJ's methods but DR questions whether it addresses the highest-GCR issues. Governance/risk-parameters work is more mission-relevant but requires either quantitative exemplars (per DR) or a stretch into IR-style qualitative analysis.
• Manheim's model-card/technical-report review is attractive for timeliness but, as DR notes, "would indeed bring us into a more 'technical AI safety' sphere" — contradicting the consensus against that expansion.
• Ord evaluation timing: evaluator availability is limited in the near term, but the author's support for a formal evaluation creates a real window. Async workflow with pre-booked evaluators may be the right operational match.
• Abstract vs. concrete framings: per DR, several proposals (Habermacher's realpolitik regulation, middle-power strategy, regulatory interventions) need concrete example works and pivotal-decision framings before UJ can assess fit.