The company that builds AGI shouldn’t be incentivized to make unlimited profits. — Ilya Sutskever

The company that builds AGI shouldn’t be incentivized to make unlimited profits.

Author: Ilya Sutskever

Insight: We usually assume that whoever invents the most powerful technology should be the one best positioned to profit from it. But building artificial general intelligence—a system potentially smarter than all humans combined—might be the one area where that logic breaks down completely. If a company's primary driver is maximizing shareholder returns, that creates a constant pressure to deploy powerful tools faster, cut corners on safety, or keep improvements proprietary rather than shared. It's like asking a surgeon to perform an operation while being paid by the number of procedures completed rather than patient outcomes. The tension here is real and often invisible. Most of us benefit from competition and profit incentives—they drive innovation and efficiency. But when the stakes are existential, the math changes. A company racing to deploy AGI while also racing to capture market share faces competing pressures that might not align with humanity's actual interests. That doesn't necessarily mean government control or open-source approaches are better, but it does suggest that however we organize this, we can't just assume the profit motive will pull us in the right direction. The uncomfortable part is that this applies more broadly too. We've already seen it with social media and recommendation algorithms—systems powerful enough to shift culture, but optimized entirely for engagement and ad revenue. The AGI version is just the extreme case of a problem we're already living with.