Discussion about this post

User's avatar
Lonica Smith's avatar

It’s hard to see what good an “intelligence explosion” would do in the world if it does not relate to raising the intelligence of any people, but simply raising the intelligence of machines.

Does anyone really want a world filled with increasingly smart machines and increasingly dumb people? Only those that think they would benefit because they control the smart machines, and through them, the dumb people. After all, people bereft of critical thinking and learning skills are much easier to control.

State of Play's avatar

OpenAI now requires documented risk assessments and safety cases before reinforcement-learning training runs, and each of three senior leaders can veto a run. It adopted the rule only after a 20 September run in which the automatic stop failed and staff halted training by hand 2.5 hours later.

That's oversight built after an incident, not ahead of one, and on my read it stays internal: I've seen nothing in the rule that gives outsiders visibility into when the veto gets used or why. The paper's first policy avenue, visibility into automation of AI R&D, is this gap. If an intelligence explosion is the risk, the near-term test isn't whether labs have internal brakes, it's whether anyone outside the lab can see them being pulled.

7 more comments...

Ready for more?