Aidan Gomez Unfiltered on AGI, China & AI's Biggest Risk

[00:00:00] A very small pool of these large tech players are becoming a single point of failure for the entire democratic block. That is not a resilient system, right? That is a single point of failure. And the first thing you learn as a computer scientist or engineer is don't build a single point of failure. Are we at AGI today? I mean, in many respects, yes. In reality, I think it's actually quite expensive to adopt open source models. I'm Eric Newcomer, author of the Newcomer Substack. Let's get into it.

[00:00:34] You are co-author on Attention is All You Need. Take stock of where we are in this moment with foundation models, what we're capable of, and then I'll sort of follow up with where you think we're going. But are you surprised from the moment of writing that paper to where we are today that we've gotten as far as we have in such a short time? I don't think anyone on the Transformer paper had any idea what was coming. We built it for Translate, like Google Translate.

[00:01:03] So it's a very scoped small problem to be solving. And it's a huge shock. I think an even bigger shock for me, given that it's been nearly ten years now since that paper, is the fact that we're still using the Transformer. You know, like as a scientist, you hope you build something, but then you're not married to it. You know, you want someone to come and build something on top of that, that is much better, blows it out of the water. And that is progress.

[00:01:33] There has been a ton of progress, but it's hilarious how the Transformer has been such a great artist in its ability to steal the good ideas from every other architecture. Like when SSMs took off, this was like an architecture that would let you have a massively larger context window. And everyone was saying the Transformer is dead, we're going to shift over to SSMs.

[00:01:57] And then the Transformer just copied those ideas, integrated it into the architecture, and we still call it the Transformer. So I don't know if we'll ever move on. Yeah, so how much more room does it have to run? How much does the next great leap depend on a new idea? I mean, we had, I guess, reasoning models were a big sort of leap outside of that mode. Or how do you think about what it's going to take for models to continue to make giant leaps over the next couple of years? Yeah, it's exactly that.

[00:02:26] It's like reasoning is still running inside the platform of a Transformer. And so I think people move up the stack, and that's where they invest their energy in innovating. So in the same way that like the fundamental training algorithms, the trained models haven't changed in much longer than 10 years, they're still the same thing. It might be the case that the Transformer is kind of just the platform. And we're going to continue to build on top of that for a very long time.

[00:02:53] And it would take something pretty monumental to shift us past it. But I still hope there's room for progress. Are we at AGI today? I mean, in many respects, yes. Right? Like I think definitely. It's a general artificial intelligence. And it seems like whatever problem we pointed towards, we can basically exceed human capabilities in that problem.

[00:03:21] So token maxing has become the buzzword of the moment. But it reflects a real issue, which is companies wanted to incentivize their employees to use AI and get the most out of it. So they said, run wild. We'll have leaderboards. We'll track how much you're spending. Now, you know, I think we've seen people like Uber and some other companies say, oh, maybe we should track what's going on because we're going to blow through our budgets pretty quickly. You know, you work with a bunch of enterprises.

[00:03:50] What's your read on is, you know, are the inside tech companies ahead of this trend? And there are lots of old guard companies that still need to have the token maxing failure before they pull back? Or where are we in the arc of sort of spending for spending sake on using AI? Well, I think a lot of that exuberance came out of coding and those models. You see this typically in like technology cycles.

[00:04:15] There's a exuberant adoption phase where the CFO or the CEO tells the CFO no spend caps, like just adopt, adopt, adopt. And it goes into excess. And then there's a correction period. And that comes from the enterprise itself constraining spend, but then also from like innovation and compression and efficiency within the modeling companies. So I think you'll we just did this big ramp up to a much larger tier of model.

[00:04:41] You'll now see one of these cycles of compressing back down. So costs will fall dramatically. Is that going to hurt your revenue? I mean, if their customers pull back, there's enough growth elsewhere. No. Yeah. Like the usage growth as the costs come down, it will just unlock way more applications. We're still so early in enterprise adoption of AI. Yes, coding has taken off and that might be hitting some sort of saturation point.

[00:05:10] But everything else within the enterprise is from my perspective, still essentially untouched. Like the stuff that we're doing is so simple. There's so much. There's like an iceberg, you know, underneath the surface to go do. And so there's probably another 10 or 100 X in terms of token consumption just on the enterprise side. After coding, what's two and three or what are the next big applications? It really depends on the industry.

[00:05:39] Like that's it's specific to the industry. So in finance, you'll see much more automated decision making, communication, coordination between the different business units, automation of like regulatory and compliance. There's so much work that gets done inside these organizations beyond just writing software. Actually, the overwhelming bulk of what's done is not writing software inside these legacy organizations.

[00:06:07] And all of that is going to be done by machines. I want to talk about the response or one answer to token maxing, which is open source or the idea that people move away from state of the art models to find, you know, I think there's a lot of conversation about routing to various models in a cost effective way.

[00:06:28] What is your view, I guess, putting aside the geopolitical part of it, which we'll get into, what's your view on sort of the appeal of open source models from a sort of cost structure point of view? I mean, in some ways they're cheaper if you're willing to build up all of the infrastructure around them, like all of the software stack, the harness, maintain this thing, believe that you have the best serving efficiency.

[00:06:54] In reality, I think it's actually quite expensive to adopt open source models and it looks cheap. If you're willing to use like DeepSeq's API or Alibaba's API, you see the token prices and you're like, oh, look, open source is cheap. These models release, these companies release open source models and their prices are way lower than the American equivalent. What's wrong with that thinking? What's wrong with that thinking? Well, one, you have to be comfortable sending your data to those parties.

[00:07:22] If you try to self-host, I think it's a very different equation. So I, and then there's also the maintenance burden of that, right? Like these harnesses, these pieces of software that the models power, you're going to have to keep that updated over time as capabilities evolve, new models come out. And so that's another huge expense. I think similar to databases in the same way.

[00:07:48] Yes, you could take an open source database, self-host that thing, try to maintain and manage it. Nobody does that. Like very, very few people do that. Instead, it's all managed services. There are enormous database companies because it's a hard, painful and expensive thing to operate. So part of the pitch of coming to Cohere is that you don't want to manage your own service. Models are going to keep changing. We'll be here with you, make sure it's safe.

[00:08:14] How do you think about why are your models shifting to being more open source in the first place? And then why Cohere build models versus sort of being a facilitator of your own models and other models? So on the first question, like the, yeah, I think the piece around which model you use matters a lot.

[00:08:40] And for Cohere, it's very strategic for us to build our own models because we offer more leverage to the customer. Increasingly, what we're seeing is folks care a lot about cost. They care a lot about sovereignty and off the shelf models aren't good enough or they're too expensive for what they need to do. So the ability to customize becomes an essential part of the product and offering to our customers.

[00:09:07] The other thing is that we're not trying to compete at the very frontier. We're not trying to get into like who can spend the most money competition. It's a losing race. And increasingly what we see, the model is not the bottleneck. It's actually the market that's the bottleneck. So frontier models are so far out in front of what the market is doing that even if you have a model from a year ago, 18 months ago, two years ago, the companies are still trying to catch up to that.

[00:09:35] So the model itself is largely not the bottleneck. And when it is, they can't afford these massive models. And so they're looking for a customized, bespoke model. I interviewed Ali Goetze at Databricks a week or so ago for the newsletter. He told a story about how he got in the model business. They said they spent $10 million on a training run. Really, the whole cluster cost them $20 million. And then three months later, the whole thing they'd built wasn't worth anything.

[00:10:04] And his view is basically that unlike software, which slowly loses its value, even if you have out-of-date software, your customers take a while to figure that out. Models, they switch to the next thing. It sounds like you're offering a view that older models still have a lot of value years out. Or you disagree with that point of view. What keeps customers using sort of a year-old model? Well, when was DBRX trained? It was like two years ago? It was in 2024.

[00:10:32] So he left this a long time ago. This was sort of a reflection. So when you're on this super steep part of the capability curve, and when you're in a period where models aren't very good at anything, yeah, you'll feel that, right? Because it's increasing super fast. But the reality is models have sort of gotten good enough at 90% of what an enterprise wants to do. That last 10% is still super valuable.

[00:10:55] I'm not trying to discount that, and there's like, I don't know, a decade plus of technical work to go do to solve that 10%. But the idea that models change every three months, and it's like some huge groundbreaking thing, it's just not true anymore for the vast majority of what the market wants to do. It is true at the very fringe, right? The frontier.

[00:11:19] Like in the pure sciences, yes, you can have a material breakthrough where the model from six months ago couldn't do it, and the model now can, like solving air dose problems or protein folding, whatever. But a lot of the thesis is there's a lot of value today, and people just need to figure out how to extract it out of the model. Yeah, and much more, the business is not the model itself. It's actually everything around it.

[00:11:43] This idea of sovereign AI, you know, you had this merger with Aleph Alpha, I think what they, they're like 10%, you're sort of 90% of the business, German business. Like what is the idea of sovereign AI and what countries are you sovereign to? And how do you think about sort of, yeah, nations needing to prefer companies that have some sort of affinity with them or local presence? Yeah, well there's been this over the past 30 years.

[00:12:12] The world, and in particular the part I care about, like the democratic alliance, the G7 or, you know, whatever you want to call it. It has basically consolidated to buying all of its technology from a very small set of players. And that is a really non-robust system. It's not very resilient. It's a single point of failure. And like the first thing you learn. America is the single point of failure or? Yeah.

[00:12:39] I say this is an American worry, that wants to be on team democracy and worries about it. So I'm not saying that's a crazy point of view, but just America is sort of. A very small pool of these large tech players are becoming a single point of failure for the entire democratic block. And that is not a resilient system, right? That is a single point of failure. And the first thing you learn as a computer scientist or engineer is don't build a single point of failure.

[00:13:03] So if you care about the democratic project, capability diffusion into the other G7 countries or into the broader democratic alliance is essential to resilience of democracy itself. If one goes down, you need to make sure you have backups. And I think you've started to see a much greater appreciation of how important that is, how important digital sovereignty is. It was something that was taken for granted, right?

[00:13:31] Like it will always be here, you know, I'll always be able to rely on it. But with the export ban of Anthropics models, you saw that it can get pulled away from you. And so our like Cohere's pitch has always been deploying on the infrastructure of our customers. Instead of saying, hey, send your data to me by an API, which I can switch off and then you lose access.

[00:13:58] I think that has to be the model that's pursued for critical workloads, for other stuff, for startups, whatever, like use APIs, they're great. But for critical workloads, countries need to control the infrastructure that powers their economy, like telcos, like financial institutions, healthcare, the grid, water treatment, defense. What countries does Cohere call home? Well, like with the merger with Alif Alpha, we'll have a dual HQ between Toronto and Berlin.

[00:14:26] So really, you know, once that deal is complete, it'll be Canada and Germany. I call Britain home. My mom's British. I live here. And that partnership between Canada and Germany, it was never meant to be just two countries. We want to build this alliance to help create capability beyond just one democracy. Right? Like the thing that I said at the G7 was, it is the case.

[00:14:56] We have a democracy leading in AI right now. That's fantastic. But democracies need to occupy position one, two, three, four, et cetera. And that's not true. And so I think it's essential that middle powers or whatever you want to call it, come together to develop these capabilities for the good of the democratic project. What is your gloss of, yeah, the Trump anthropic back and forth?

[00:15:22] I mean, first there was, you know, yeah, the anthropic trying to limit how much they could be used in wartime situations. And now it's this sudden blockage of fable. I don't know. Yeah. What did you take away from those two situations? Well, it wasn't super surprising. Like, this is sort of exactly the scenario that we have been warning against.

[00:15:45] And I think that anyone who was naive to the potential risk of something like this happening were woken up quite suddenly. I think that, what does it mean? I mean, for your, like a big issue here is that the way the Trump administration went about it was focused on letting anyone outside the U.S. use it.

[00:16:12] Because that sort of woke up the other democracies that there would be sort of a preference for the U.S. rather than a preference for the U.S. and its allies. Is that sort of the right depiction of your view? Yeah, the U.S. is optimizing for its interests as it should. But I think in doing so, it does present a huge risk for anyone outside of the U.S.

[00:16:39] Like, if you built your critical infrastructure on top of some service that you depend on to live or to operate or to have an economy at all, you now lose access to that. We're talking about, you know, the weakness of relying on one democracy for the rest of the democratic world. What about the other part of this, which is how do you, what's your read on how strong China is in sort of the foundation model business right now?

[00:17:08] We see DeepSeq raising this enormous round. Like, do you think there's a real risk that China leapfrogs the Western world in sort of model development? Leapfrogs? I would be very surprised. I do think the U.S. is going to continue to lead in capability. I think the dismissals of Chinese models as like just distillations or anything like that is cope.

[00:17:34] Like, I think it's, they are extraordinarily capable, extraordinarily well resourced by their government and only getting more resources. And they're just very good at building this tech.

[00:17:48] I'm sure distillation has played a part, but increasingly independent of distillation, I think they're on a very strong trajectory, which just drives more urgency to make sure, you know, players like Cohere, alliances form around them, get supported so that we have more than one democracy competing against China for the global market. Because very quickly you could see the world split into two.

[00:18:16] But what you really want is a democratic alliance of two, three, four, five against an autocratic singular player. Give us, for the last question, a sort of view of what you think the next 12 to 18 months looks like in terms of technological capability. Is it about agents? Yeah, I guess that's my answer.

[00:18:43] But if you agree, then like sort of shape that up a little bit because we hear that word so often. What's that mean? Or what do you think is going to be possible in the next 12 to 18 months? The shift is it's going to be entirely in the enterprise, the way we do work, who we work with, who we communicate with. Increasingly, it's going to shift towards working with models. And it's already happened in this kind of microcosm of developers. Like they spend most of their day interacting with a model to do their work.

[00:19:12] That's their collaborator that they're partnering with. And they, you know, manage it. They call it to go do some work, comes back with a bunch of code, they give you some feedback. No, I actually meant like this. Can you fix this? Increasingly, we work as colleagues with agents in software development. That will start to permeate out into many more fields. And if you look at, I mean, these are early numbers, but if you look at employment numbers inside of software engineering, it's actually still very healthy.

[00:19:42] And so the big thing that I'm watching, concerned about, but I do remain optimistic about is how this impacts employment and how that shapes public sentiment towards the technology. I'm hopeful that, you know, whatever it's called, Jevons paradox, where a massively efficiency augmenting technology emerges, you would think that would displace tons of people. But instead, demand rises. Way more productive, we need more of them. Demand rises so quickly that you actually need more people to manage these machines.

[00:20:13] But your view is you don't know which way it'll go yet. I don't know because the counter argument to Jevons paradox is that with previous technological revolutions, those tools, those machines that came out, they were so far from, like a loom, they're so far from a human, they're so different than us. They were so specific and narrow in terms of what they could be applied to.

[00:20:36] We have something now that looks so much more like us that maybe it is more displacing. You know, I'm hopeful that's not the case, but I do think we need to plan for that scenario and come up with the tools to address that. Things like maybe UBI, but more likely retraining programs, potentially even right to work or, you know, jobs guarantee programs.

[00:21:02] Aiden, we're at time. Thank you so much for standing up for the democratic world. I appreciate it. That's our episode. This is the Newcomer Pod where we chop it up on all things Silicon Valley. Thanks for watching. Please like, comment, subscribe, go read the Newcomer sub stack if you want to get really in the weeds and understand the real business of venture capital and startups at newcomer.co. If you still yearn for podcasts, I've got this Rewal Valley show with Max Stroud and James Wilsterman.

[00:21:29] Otherwise, probably see you in about a week. Thanks for all your support.