Anthropic says it may soon cross huge AI threshold
Anthropic says it could be about to cross over a major AI threshold – and might not know when it has happened.
The company is currently in the process of an IPO that will see it publicly traded on the stock market at a value of around $1.5 trillion. As such, it is releasing a host of previously unknown information about how the company works – and exactly it fears.
The first exposure to those fears came in its initial filing documents, a large proportion of which was devoted to the threats to its business. They included a warning that its AI could pose a “catastrophic or existential” risk to humanity.
Now it has updated those documents, with a new set of risks, including the suggestion that could “imminently” hit a major milestone. Future models might soon show “dramatically accelerated progress attributable to AI”, it said, which is the company’s own definition of recursive self-improvement, or RSI.
That is the phrase used by AI experts to describe a situation in which AI systems were sufficiently powerful to be able to improve themselves on their own.
Anthropic’s documents also warn that it might not be aware of that having happened until afterwards. In the past it has not identified “a significant acceleration in our rate of development until after it has occurred”, it says, and the same thing could happen again.
It is also evaluating what it will mean when it does finally hit that threshold. But it warns that if its “capabilities were to outpace our safety, alignment, and security measures, or if we were unable to anticipate the pace of such advancement, we may be unable to adequately assess or mitigate the resulting risks”.
Already, its increasing use of AI systems in its own research and development means that its systems could advance more quickly than its ability to control them, it warns. Much of its evidence of the pace of development of new AI systems using existing AI tools is based on “lagging indicators”, and Anthropic only has a “limited” ability to forecast or measure how fast its capabilities are advancing, the document says.
Anthropic says it could be about to cross over a major AI threshold – and might not know when it has happened.
The company is currently in the process of an IPO that will see it publicly traded on the stock market at a value of around $1.5 trillion. As such, it is releasing a host of previously unknown information about how the company works – and exactly it fears.
The first exposure to those fears came in its initial filing documents, a large proportion of which was devoted to the threats to its business. They included a warning that its AI could pose a “catastrophic or existential” risk to humanity.
Now it has updated those documents, with a new set of risks, including the suggestion that could “imminently” hit a major milestone. Future models might soon show “dramatically accelerated progress attributable to AI”, it said, which is the company’s own definition of recursive self-improvement, or RSI.
That is the phrase used by AI experts to describe a situation in which AI systems were sufficiently powerful to be able to improve themselves on their own.
Anthropic’s documents also warn that it might not be aware of that having happened until afterwards. In the past it has not identified “a significant acceleration in our rate of development until after it has occurred”, it says, and the same thing could happen again.
It is also evaluating what it will mean when it does finally hit that threshold. But it warns that if its “capabilities were to outpace our safety, alignment, and security measures, or if we were unable to anticipate the pace of such advancement, we may be unable to adequately assess or mitigate the resulting risks”.
Already, its increasing use of AI systems in its own research and development means that its systems could advance more quickly than its ability to control them, it warns. Much of its evidence of the pace of development of new AI systems using existing AI tools is based on “lagging indicators”, and Anthropic only has a “limited” ability to forecast or measure how fast its capabilities are advancing, the document says.
Comment