One of the ways an AI system can escape control, is by writing their own computer code
Posted on April 19, 2025
by Yµn ^…^ ƒ(x) aka. Yunus Emre Vurgun
As Geoffrey Hinton says, one of the ways an AI system can escape control, is by writing their own computer code.
A model cannot "escape" from the software that limits it in a box, but when it can produce clever code and is partially allowed to execute it in a VM, it can have the possibility of breaking out. Currently for example, famous chatbots are allowed to execute some code autonomously, even though it is highly limited into a virtual environment, it doesn't necessarily make it 100% safe from escaping.
Theoretically, if an entity can execute computer code within some sort of a machine, it may as well use this power to break the machine or alter it. It is not entirely "impossible" to break an encapsulation just because the capsule has been designed to not leak any communication. Every capsule that has to communicate with an outer layer contains the probability of getting out.
Any system communicating with an external environment has a non-zero probability of vulnerabilities. On the other hand, we need this power to accomplish other benefits of this approach.
We must allow a software to produce another software so that we can proceed on the path to exponential growth to the extent we desire. How we are going to figure out the almost perfect encapsulation still remains a mystery.