Search the archive.

The most unusual thing about this week’s Senate hearing on AI was how affable it was. Industry reps — primarily OpenAI CEO Sam Altman — merrily agreed on the need to regulate new AI technologies, while politicians seemed happy to hand over responsibility for drafting rules to the companies themselves. As Senator Dick Durbin (D-IL) put it in his opening remarks: “I can’t recall when we’ve had people representing large corporations or private sector entities come before us and plead with us to regulate them.”

This sort of chumminess makes people nervous. A number of experts and industry figures say the hearing suggests we may be headed into an era of industry capture in AI. If tech giants are allowed to write the rules governing this technology, they say, it could have a number of harms, from stifling smaller firms to introducing weak regulations.

Companies are pressing ahead with building and deploying artificial intelligence applications, even as the regulatory landscape remains in flux. European lawmakers this month approved the world’s most comprehensive legislation yet on artificial intelligence, although other regions, including the U.S. and individual states, aren’t as far along.

Potential threats to customer privacy, the security of customer data, as well as the potential bias and inaccuracy of AI algorithms are all areas regulators are examining.

The advent of OpenAI's ChatGPT has directed plenty of attention and money toward startups working to improve artificial intelligence's capabilities.

Indeed, AI has been one of the few bright spots in venture-capital spending this year, accounting for nearly a fifth of the $144 billion piled into startups globally in the first half of the year, according to Crunchbase.

Containment of AI research is possible because there are only a handful of companies having success on AGI (models like PaLM, GPT-3, Bing/Sydney, or LaMDA). Of course, many companies are raising money to try to build AGI, but so far the visible successes have solely been so-called “foundation models” that cost millions to train, require experts who themselves cost millions to hire, access to massive amounts of data, and, so far, have no immediate return on investment. Did you know that DeepMind, Google’s leading AI subsidiary, had an administration budget that ballooned in 2021 from £780 million to £1,254 million? Does that sound like a technology that anyone can build? Or something restricted to a few incredibly rich tech companies?

You don’t need to get everyone on Earth to agree to stop building AI, because the only people on the entire planet capable of building Bing/Sydney level AIs are a handful of teams at Big Tech companies. It’s often said that the cost of a high-level AI researcher is equivalent to an NFL quarterback. Even outside of salaries, the cost of these companies is immense, likely around $600,000 a day to run ChatGPT, the upkeep and personnel costs are even greater (in the billions), and these companies are the only ones with access to the amount of data needed for training, as well as the only ones who can do the careful crafting of human feedback to create the most advanced versions.

@ericjang11

To come back to the title of my talk, I can now state why s-risks are the worst existential risks. S-risks are the worst existential risks because I’ll define them to have the largest possible scope and the largest possible severity. (I will qualify the claim that s-risks are the worst x-risks later.) That is, I’d like to suggest the following definition. “S-risk – One where an adverse outcome would bring about severe suffering on a cosmic scale, vastly exceeding all suffering that has existed on Earth so far.” So, s-risks are roughly as severe as factory farming, but with an even larger scope. To better understand this definition, let’s zoom in on the part of the map that shows existential risk. One subclass of risks are those that, with respect to their scope, would affect all future human generations, and, with respect to their severity, would remove everything valuable. One central example of such pan-generational, crushing risks are risks of human extinction. Risks of extinction have received the most attention so far. But, conceptually, x-risks contain another class of risks. These are risks of outcomes even worse than extinction in two respects. First, with respect to their scope, they not only threaten the future generations of humans or our successors, but all sentient life in the whole universe. Second, with respect to their severity, they not only remove everything that would be valuable but also come with a lot of disvalue – that is, features we’d like to avoid no matter what. Recall the story I told in the beginning, but think of Greta’s solitary confinement being multiplied by many orders of magnitude – for instance, because it affects a very large population of sentient uploads.

Superintelligence is related to three categories of suffering risk: suffering subroutines (Tomasik 2017), mind crime (Bostrom 2014) and flawed realization (Bostrom 2013). Humans have evolved to be capable of suffering, and while the question of which other animals are conscious or capable of suffering is controversial, pain analogues are present in a wide variety of animals. The U.S. National Research Council’s Committee on Recognition and Alleviation of Pain in Laboratory Animals (2004) argues that, based on the state of existing evidence, at least all vertebrates should be considered capable of experiencing pain. Pain seems to have evolved because it has a functional purpose in guiding behavior: evolution having found it suggests that pain might be the simplest solution for achieving its purpose.

A superintelligence which was building subagents, such as worker robots or disembodied cognitive agents, might then also construct them in such a way that they were capable of feeling pain - and thus possibly suffering (Metzinger 2015) - if that was the most efficient way of making them behave in a way that achieved the superintelligence’s goals. Humans have also evolved to experience empathy towards each other, but the evolutionary reasons which cause humans to have empathy (Singer 1981) may not be relevant for a superintelligent singleton which had no game-theoretical reason to empathize with others. In such a case, a superintelligence which had no disincentive to create suffering but did have an incentive to create whatever furthered its goals, could create vast populations of agents which sometimes suffered while carrying out the superintelligence’s goals. Because of the ruling superintelligence’s indifference towards suffering, the amount of suffering experienced by this population could be vastly higher than it would be in e.g. an advanced human civilization, where humans had an interest in helping out their fellow humans. Depending on the functional purpose of positive mental states such as happiness, the subagents might or might not be built to experience them. For example, Fredrickson (1998) suggests that positive and negative emotions have differing functions.

The existential risks posed by artificial intelligence (AI) are now widely recognized.

After hundreds of industry and science leaders warned that “mitigating the risk of extinction from AI should be a global priority alongside other societal-scale risks such as pandemics and nuclear war,” the U.N. Secretary-General recently echoed their concern. So did the prime minister of the U.K., who is also investing 100 million pounds into AI safety research that is mostly meant to prevent existential risk. Other leaders are likely to follow in recognizing AI’s ultimate threat.

Many experts believe that alongside global opportunities, AI poses global risks, which will be greater than, say, the risks of nuclear technology - which in any case have historically been underestimated. Furthermore, scientific risk analysis suggests that high potential damages should be taken very seriously even if the probability of their occurrence were low.

While the risks from domain-specific AIs appear limited in the near future, there are long-term developments to take into consideration: in the not-so-distant future, artificial intelligence could in principle pose an existential threat, similar in scope to the pandemic risks associated with biotechnology [4, 6, 17].

After years of easy money, the AI industry is facing a reckoning.

A new report from Stanford’s Institute for Human-Centered Artificial Intelligence (HAI), which studies AI trends, found that global investment in AI fell for the second year in a row in 2023.

Every year, Stanford University’s Institute for Human-Centered Artificial Intelligence releases a big report about the status of the artificial intelligence industry. This year’s report, published this week, is a whopping 502 pages and includes a wealth of insights on the trendy technology. Tucked into its section on the economics of the AI industry is the surprising revelation that global corporate investment in AI actually dropped last year for the second straight year in a row. This might come as a surprise to some since 2023 was largely considered to be a period of peak AI hype. New chatbots and companies spawned aplenty, as top executives spread hyperbolic rhetoric about the technology’s potential. And yet, the new HAI report notes that in 2023, total investment in AI…

…dropped to $189.2 billion, a decrease of approximately 20% from 2022. Despite a slight reduction in private investment, the most significant downturn occurred in mergers and acquisitions, which fell by 31.2% from the previous year. However, over the past decade, AI-related investments have increased thirteenfold.

Department of Psychology, York University) 2023. “Artificial General Intelligence, Existential Risk, and Human Risk Perception” https://arxiv.org/pdf/2311.08698 // ZX

As with any modern technology, contemporaneous societal and civilizational pressures and scientific breakthroughs continue to shape the evolution of AI. Its effects, much like those of nuclear technology, can and likely will change substantially over time. For instance, the perceived necessity of securing a decisive military advantage over the Axis powers during the Second World War motivated the US’s development of nuclear weapons, but this gave way to a military arms race once the US and Soviet Union were locked into the decades-long Cold War. Since then, there has been an expansion of nuclear-armed states and the pressures for expansion continue. While the threat of mutually-assured destruction serves as a potent first-use deterrent, there have been several close calls that raise the question of how long humans can avoid a global nuclear catastrophe (i.e., a nuclear winter) given that there appears to be no harmonious “end of history” in sight (Barrett et al., 2013; Baum, 2015). Attention to existential risk posed by AI invariably concerns possible future AI rather than the current state of AI. Although current AI carries significant risk—for example, algorithmic processes may be unexplainable (Angelov et al., 2021; Vilone & Longo, 2021) or biased and possibly unfair (Fazelpour & Danks, 2021)—and possibly biased for inexplicable reasons (Bostrom & Yudkowsky, 2018)—this is not what most experts regard as existential risk from AI. Current AI does not pose an existential threat because its development has not reached the stage where machines have parity with human intelligence across a wide range of tasks and domains.

Most AI systems today are designed to perform specific tasks, such as recognizing images or processing natural language. These models are not designed to be flexible or adaptable in the way that a human brain is and it is known as narrow AI, which is the primary type of AI that we currently have.

AI systems should be able to learn and generalize across different domains to achieve the desired form of AGI. Artificial General Intelligence (AGI) refers to a theoretical form of intelligence that can understand or learn or perform any intellectual task exactly as human beings do. It is a primary goal of some AI research.Need Breakthrough research in Machine Learning, Natural Language Processing, and Computer Vision

Generative AI tools can do all sorts of things, such as write blog posts like this one, albeit partially and with a lot of edits by a human writer. This post uses three text-to-text AI writers (anonymized for obvious reasons) to write articles about the limitations of generative AI.

What Generative AI can and can't do, according to AI writer #1

The high prices U.S. patients pay for drugs are a key driver of pharmaceutical company financial performance, generating large profits relative to many other industries and an estimated three-quarters of worldwide drug company profits. As a result, lowering U.S. drug prices would likely have an outsized impact on overall incentives to innovate. The prospect of large profits, which translate into high returns on investment (ROI), attracts the private capital that finances pharmaceutical innovation. U.S. pharmaceutical companies invested an estimated $80 billion in 2018 in drug research and development (R&D), and such spending yielded an annual average of 44 new drug approvals from 2015 to 2019, although 47 percent are “orphan drugs,” meaning that very few patients will benefit from them. The Food and Drug Administration (FDA) designated almost half of the new drugs in this period as “first in-class,” and over a quarter as “breakthrough” therapies.

Lowering high U.S. drug prices will make investing in drug development less attractive for private capital, leading to fewer new medicines coming to market. Balancing the tension between lowering U.S. drug prices while sustaining robust R&D investment poses a complex challenge for policymakers and raises the fundamental question addressed in this two-part blog post: how can the U.S. minimize the adverse effects of lower drug prices on innovation?

The high cost of drugs has been a topic of debate among many people, including me and Dr. Peter Bach, the eminent Director of the Center for Health Policy and Outcomes at Memorial Sloan-Kettering Cancer Center (MSKCC). Essentially, Dr. Bach and I disagree on the importance of pricing on sustaining innovation in pharmaceutical drug R&D. This is an important debate to have, as all can benefit by understanding the issues at hand.

But make no mistake: revenues from new drugs drive innovation. The pharmaceutical industry invests more of its top line earnings into R&D than any other industry – roughly 15%. Thus, there is a direct tie to drug sales and the amount of money available for new drug discovery and development. If it is in the public’s benefit to capitalize on the knowledge being garnered with the understanding of the genetic causes of diseases, then new patented drugs need to be priced high enough to enable companies to get a significant return on their investment (ROI).

The US Patent and Trademark Office’s newly expanded program aimed at accelerating patents for climate mitigation technology is critical to garnering the funding necessary to advance the clean energy transition, researchers say.

The PTO’s extension of its Climate Change Mitigation Pilot Program comes at a time of striking juxtaposition. While energy alternatives to fossil fuels have become more mainstream than ever—with renewable energy on track to supply 44% of American electricity by 2050—scientists warn that barriers to global adaptation persist, namely insufficient financing.

Innovation has a vital role in the fight against climate change, and new technologies will be essential in the move to "net zero." This function was recognized at last year's COP26 in Glasgow, where 40 members signed the so-called Glasgow Breakthroughs, setting targets to accelerate the deployment of clean technologies in power generation, road transport, steel, hydrogen and agriculture. The summary decision alone mentioned "technology" no fewer than 10 times.

The Intellectual Property (IP) system can support this push toward greener processes by incentivizing the development and commercialization of innovative technologies. Many sustainable inventions, such as alternative fuels, involve significant up-front investment and a degree of commercial risk. To this situation, patents bring legal certainty, providing a competitive advantage and a tool for fundraising. Other forms of IP, such as trademarks, designs and trade secrets, also have their part to play in these ventures. However, the scale of the environmental challenge is such that we need to carefully consider the function of the IP system as a whole and how it can prop up sustainable innovation. This includes ensuring that IP rights are accessible, patent information is freely available and technology transfer is as smooth as possible.

The Clean Energy Patent Growth Index reported an upwards trend in green patents applications globally since the early 2000’s, with the majority focusing on alternative and renewable energy sources, including wind, sun, geothermal, hybrid vehicles and biofuels. A clear example is related to the newly granted patent of French green chemistry company Carbios: a biotechnological process that enables PET plastic waste recycling.

IP protection enables green technology creation, adaptation and distribution. Non-renewable resources are finite, and climate change is a reality. It is necessary to adopt a change towards a more sustainable lifestyle based on green technologies, which is part of the development of new global environmental policies – stronger protection of IP rights is an integral part of this process.

Science can not solve all of our problems. While scientific understanding can help battle things like disease, hunger, and poverty when applied properly, it does not do so completely and automatically. Furthermore, there are many areas of life where science can have little impact. Let us look at some of the reasons why this is so.

First of all, there is a huge difference between knowing something and acting on it. Science is concerned with accumulating and understanding observations of the physical world. That understanding alone solves no problems. Individual people have to act on that understanding for it to help solve problems. For instance, science has found that regular exercise can lower your risk of heart disease. Knowing this fact is interesting, but it will do nothing for your personal heath unless you act on it and actually exercise. And that's the hard part. Reading an article about exercise is easy. Getting into an actual routine of regular exercise is harder. In this sense, science really solves no problems at all. Problems are only solved when people take the knowledge (or tool, or pill, or whatever) provided by science and use it. In fact, many of humanity's biggest problems are caused by lack of action, and not lack of knowledge.