OpenAI’s big lesson for science policy

Proving the power of Focused Research Organizations

The incredible success of Large Language Models like ChatGPT is both a scientific breakthrough and a boon for future scientific discovery. As a recent editorial in Nature explains,

…large language and vision models that can digest the literature will be used to identify gaps in knowledge, help summarize and understand unfamiliar topics, and find the most relevant references, protocols, data and experts. They will also generate and explain complex graphs and schematics, and help write and edit routine computer code as well as scientific papers, reviews, grant applications, curriculum vitae and all sorts of reports. Producing content without assistance from machine-learning applications may soon be as rare as writing snail mail.

Saving researchers’ time to focus on what matters will speed-up the scientific process and help to extract new connections from the voluminous research that already exists. Longer term, however, we may even find ourselves asking the AIs to probe deep scientific questions for us. As Nat Friedman put it, paraphrasing OpenAI’s David Dohan, imagine one day prompting a next-generation model with “A well known formula for a room temperature superconductor is…” and fully expecting it to output an answer.

More is different

Given the obvious benefits of deep learning to the scientific enterprise, it’s worth asking why it took OpenAI to push the field forward — an independent nonprofit turned capped-for-profit — and not, say, the National Science Foundation or one of America’s world-class research universities?

To be sure, most of the intellectual breakthroughs behind our current AI summer originated in academic research, in some cases many decades ago. The first multi-layer perceptron trained by stochastic gradient descent dates back to 1967. The precursor to backpropagation was articulated 5 years prior, well before the technology even existed to properly implement it. And a close cousin to transformer models was first published about in the early 1990s, along with a methodology for unsupervised pre-training.

Yet as any graying AI researcher will tell you, neural networks were a backwater of the field until relatively recently. The tide only began to turn as hardware capabilities caught up and new private research labs, such as Google Brain in 2011, began experimenting with neural networks trained with many hidden layers.

It’s hard to see how America’s scientific grant making agencies could have catalyzed this progress on their own. Grant-funded science rewards novelty, while the core architectures behind recent progress in AI are anything but. Imagine a research proposal that said something like the following:

We are requesting one billion dollars to train a really big neural network. We don’t anticipate making any major theoretical advances in algorithm design, and we don’t yet know the exact details of our plan, much less if it will work. But if there’s even a small chance that scaling existing models up will reveal a path to AGI, we think it’s worth the shot.

Such a proposal would be dead on arrival, yet it is essentially what OpenAI led with when they launched in 2015 with a billion dollars in backing. After experimenting in robotics and a few other cul de sacs, the release of the famous “Attention is all you need” paper in 2017 caused OpenAI to go all-in on transformer architectures. They released "Improving Language Understanding by Generative Pre-Training" a year later, giving birth to the GPT class of language models. The rest is history.

OpenAI as an FRO

OpenAI was founded with the specific mission to create Artificial General Intelligence. With a hefty budget, it built-up a high caliber and dedicated team to use the flexibility of an independent research organization to engineer different ideas before settling on a core technology. In the parlance of science and innovation policy, this makes OpenAI an accidental example of a Focused Research Organization.

Focused Research Organizations (FROs) are science and engineering programs addressed to “well-defined challenges that require scale and coordination but that are not immediately profitable.” The case for FROs in the context of federal funding for science and R&D was made succinctly by Sam Rodriques and Adam Marblestone:

The U.S. government is ill-equipped to fund R&D projects that require tight coordination and teamwork to create public goods. The majority of government-funded research outside of the defense sphere—including research funded through the National Institute of Health (NIH), the National Science Foundation (NSF), the Defense Advanced Research Projects Agency (DARPA), and the Advanced Research Projects Agency–Energy (ARPA-E)—is outsourced to externalized collaborations of university labs and/or commercial organizations. However, the academic reward structure favors individual credit and discourages systematic teamwork. Commercial incentives encourage teamwork but discourage the production of public goods. As a result, the United States is falling behind in key areas like microfabrication and human genomics to countries with greater abilities to centralize and accelerate focused research.

The solution is to enable the U.S. government to fund centralized research programs, termed Focused Research Organizations (FROs), to address well-defined challenges that require scale and coordination but that are not immediately profitable. FROs would be stand-alone “moonshot organizations” insulated from both academic and commercial incentive structures. FROs would be organized like startups, but they would pursue well-defined R&D goals in the public interest and would be accountable to their funding organizations rather than to shareholders. Each FRO would strive to accelerate a key R&D area via “multiplier effects” (such as dramatically reducing the cost of collecting critical scientific data), provide the United States with a decisive competitive advantage in that area, and de-risk substantial follow-on investment from the private and/or public sectors.

Yet if anything, Rodriques and Marblestone understate the challenge of moving breakthrough science through traditional funding channels, and thus the room for improvement. Funding rates for grant applications at the NSF and NIH have steadily declined since the 1970s, back when it was common for one in every two grants to be approved. Today, approval rates run as low as 10 to 20 percent, and tend to go to Principal Investigators and research teams that are older and more established, harming scientific diversity and creativity. Just 2 percent of NIH-supported institutions receive 53 percent of all research project grants, for example.

Meanwhile, PIs of federally sponsored research report spending over 40 percent of their time on administrative tasks. Yet the time cost of compliance and bureaucracy is arguably less important than the sheer inflexibility it imposes on research agendas — a fact Patrick Collison discovered after surveying a cross-section of biomedical researchers through the Fast Grants program. As he explained on the Ezra Klein podcast last year,

…we just asked them, as a general matter in your regular research, if you could spend your grant money however you want, how much would you change your research agenda?

So not an increase in the funding level, which tends to be what we discuss in as much as we’re discussing science policy across society. But much more specifically and narrowly, if you had complete autonomy in how you spend whatever grant money you’re getting, how much of your research agenda would change? And our intuition was that maybe a third of people would like to be doing something meaningfully different to what they actually are.

But of these scientists, and these are really good scientists, four out of five told us that they would change their research agendas, quote, “a lot.”

This cries out for structural innovation in how the federal government funds science and R&D. FROs are one such innovation and, while not a panacea, they come with many advantages. Three stand out in particular:

FROs are mission oriented: By orienting around a “well-defined tool or technology, a key scientific dataset, or a refined process or resource,” FROs allow researchers to flexibly allocate their time and resources toward the higher-level goal, thus avoiding the confines of a narrow project grant. To give a hypothetical example, where the NSF might fund a series of proposal with applications to battery technology, an FRO would instead set an ambitious but achievable high-level mission, something like “improve battery efficiency by 20% relative to the state of the art.” The organization would be spun up with a large initial investment and otherwise given wide autonomy in how it achieves its goal. A researcher within the FRO might decide to study the very same material, but they could also decide to abandon the idea if a more promising approach presented itself. That is, instead of a grant agency issuing new lines of inquiry, the project mission guides researchers through the most relevant search space of intermediate discoveries.

FROs are vertically integrated teams: Science is increasingly a team-led enterprise. FROs acknowledge that reality from the get-go, allowing researchers from across disciplines and institutional backgrounds to collaborate under one roof. Such collaborations are often infeasible in academic settings, as issues of financing, credit-taking, and intellectual property abound. Different teams within an FRO can further benefit by sharing human resources and equipment, or by finding new synergies at the proverbial water cooler.

FROs collapse the false dichotomy between science and engineering: Basic research and engineering often go hand in hand. Unfortunately, our modern grant making institutions tend to divided the two, as if engineering and technology were the mere “applications” of antecedent discoveries. In the real world, science and engineering feed off each other in a virtuous cycle. The Apollo Program produced myriad scientific discoveries in the quest to engineer a man on the moon. Or take solar energy costs, which continue to decline year over year not merely thanks to the work of academic scientists, but also because solar manufacturers are deploying their technology at scale. In turn, engineers and chemists are driven to find new efficiencies and process innovations through a classic case of “learning by doing.”

In short, FROs aspire to work like NASA did in the 1960s or SpaceX does today. It’s a model that can work on ideas big and small, just as long as it’s a public good. Existing FROs include:

• EvE Bio: On a 5 year mission to map the “pharmome” by building a public domain knowledge map of FDA-approved drugs’ functional protein binding partners across thousands of human gene products.

• Rejuvenome: Established to conduct the largest and most systematic study of the biological effects of putative anti-aging interventions.

• Cultivarium: Creating open-source tools for life scientists to expand access to novel microorganisms for biological research and development.

• E11 Bio: Building the foundation for a full stack, hundred-billion neuron scale mapping of the human brain.

Science as gradient descent

The tinkering culture of engineers and technologists is the experimental method by any other name. Only instead of receiving feedback from an anonymous reviewer, engineers deal with the uncompromising feedback of reality. The Wright Brothers come to mind — two bicycle mechanics who tinkered their way to the first motor-operated airplane, but only after overcoming the painful feedback of gravity.

The success of OpenAI is likewise a testament to the power of tinkering. As Sam Altman has indicated in interviews, the leap between GPT-3 and GPT-4 wasn’t due to some deep breakthrough so much as the accumulation of many small tweaks. Yet such tweaks were only discoverable because of the real-world feedback they received by deploying their models at scale.

It’s time these lessons were brought to bear in how the federal government funds science and R&D more broadly. Only a handful of FROs have been created to date, and mostly through private dollars. With robust public support, their potential use-cases could easily multiply.

After all, science advances through a process not dissimilar to gradient descent. Scientists poke at the frontiers of human knowledge through experiments and backpropagate their findings into an updated picture of reality. The errors in our model of the world steadily decrease until we’re able to generalize our understanding into all-new settings. Science at its worst, in contrast, mimics the overly symbolic approaches that led to the last AI winter, resulting in shallow models of reality that overfit the data and fail to reproduce.

This makes FROs the transformer architecture of science funding: a way to focus attention. By letting researchers work in parallel toward a common, well-defined objective, FROs are able to grok the dependencies between different disciplines. The few we have promise to blow past every normal benchmark of scientific productivity.

Scale is all they need.