Is generative AI a bubble or is it going to change the world?

Advice to my fellow humans: Never stop learning.

Today’s date is July 19, 2024. It’s been roughly ~20 months since ChatGPT launched in late November 2022. For a technology that already feels so mainstream, it’s crazy to think of it as barely a toddler. All of the headlines about ChatGPT’s remarkable growth as the fastest new product category ever are true and impressive. Fastest to 100 million users. Fastest to generate over $1 billion in revenue. In just 20 months, however, LLMs have demonstrated real-world product market fit with tangible benefits for users as diverse as K-12 students, customer support teams, and software engineers. Is it going to kill Google? Probably not. Is Google rightfully concerned that ChatGPT already has a multi-billion dollar annual run rate? Yes.

As is the nature of any transformational technology, the hype cycle eventually gets out over its skis. In today’s post we’ll review why I think we’ve reached the apex of the AI bubble. I expect the road from here to be a long and gentle plateau of small improvements. I do not expect the bubble to burst. We will surely see steady and continued utilization of LLMs over the years to come. Fun fact: smartphone penetration only hit 70% in the USA in 2018, 11 years after the launch of the iPhone. But as each year ticks by, we’ll have fewer and fewer reasons to freak out over Elon vs. Sam Altman or whether OpenAI stole Scarlett Johansson’s voice. As a “knowledge worker” myself, that spends far more hours behind a screen than swinging a hammer, I can wholeheartedly affirm that I’m not worried about job security.

Hot take #1: The pace of improvement has slowed. 

How can we know if the pace of innovation is accelerating or decelerating? Such a question is inherently subjective. Since OpenAI’s ChatGPT is the unquestioned front runner in the space, we’ll focus our gaze there, while also noting that competition remains fierce amongst OpenAI, Google, Facebook, Anthropic, and others. The best public estimates about ChatGPT usage are based on SimilarWeb browser data, which should always be taken with a grain of salt. They suggest that usage plateaued starting April 2023 and has held roughly steady over the past 15 months. 

We can also look at the simple timeline of model and feature releases for ChatGPT, and the narrative is consistent. Since the launch of GPT 4.0 in March 2023, not much has changed. Accordingly, while growth from 0 to 100m users in just 3 months is incredible, the lack of any press announcing additional milestones (e.g. exceeding 200m users or let alone billions of user) in the ~17 months since is consistent with the fact that growth has stalled.

Hot take #2: GPT5 will not be a game-changer.

In any maturing market, product adoption eventually reaches its terminal value as the opportunities for innovation are harvested. As such, if LLMs are no longer getting better, we would naturally expect to see growth taper off.  So why is it that LLMs are no longer getting better? We’ll explore the three most important reasons below.

But first, a simple ELI5 story for LLMs: 

In the early 2000s, Google crawled and indexed the entire internet, organizing the world’s information and making it universally accessible and useful. Google delivered the 10 best links to any query, for free. The internet was much smaller back then, but was growing rampantly. Since Google was hands-down the best at crawling, indexing, and then ranking resulting links, they won the market. Fast forward to ~2020, and the invention of GPTs (Generative Pretrained Transformers) meant that software could now generate coherent text in close to real-time. So rather than just returning a bunch of blue links, software can now generate actual answers. So with GPTs at the helm, OpenAI basically followed Google’s original playbook:

1. Crawl the entire internet.

2. Train a massive model (large language model) on all of the data from the internet.

3. Use that well-trained model to generate answers to any user's queries, in close to real-time.

But the internet today is quite large. And so training a model with millions or billions or trillions of parameters is expensive. GPT1 contained 117 million parameters, which might sound like a lot, but in reality meant it only trained on ~0.01% of the internet. GPT2 got more than 10x bigger and better with 1.5 billion parameters, but that still meant it trained on less than 1% of the internet. GPT3 grew another two orders of magnitude with 175 billion parameters, and was now scratching the surface of being useful. But it had still only trained on less than 10% of the internet. It also included training data up until 2021, so was fairly outdated by the time users got to play with it in late 2022. But with the release of ChatGPT, the arms race to train bigger and better models was on. Nvidia has skyrocketed into (one of) the world’s most valuable companies because they build the most powerful GPU chips. If you haven’t heard of the A100 or H100 or the recently announced Blackwell chip, you’re not missing out. Just know that most powerful chips mean models can train on more data, faster and more affordably. And so every company wanting to do cool things with LLMs needed to buy lots of Nvidia chips. And so they did -- to the tune of $61 billion worth in the past year. With this newfound brute force capability, LLMs were able to be trained on basically the entire public internet. GPT4 was reported to have 1.75 trillion parameters. Unsurprisingly, it was this latest jump in scale that produced the most dramatic step change in the functionality of LLMs. 

Which brings us to today. Why has the pace of progress now slowed? We’ll explore three reasons.

1, We’re running out of data, and the incrementally useful data is private.

In short, now that we’ve trained models on basically the entire public internet, there’s simply not all that much more data remaining. While Nvidia’s chips will get more powerful and the internet will never become static, we should not expect another step change in model performance without a massive step change in the available data.

Yes, it is true that the internet is still growing, just like the universe. Supposedly there are 330 million terabytes of data created each day. However, most of those tweets, baby photos, and TikTok reels have zero incremental value to GPT5. Additionally, the majority of the information created each day is private. It resides within proprietary databases, private emails, Slacks, or slide decks presented in board meetings. This private data is incredibly valuable. There are already third-party firms like PitchBook, which sell access to proprietary data for hefty sums. PitchBook alone generates an estimated $290 million in annual revenue from its 74k unique customers, who are willing to pay for access to its curated and verified private data. The world of private data is incredibly valuable. It includes trade secrets, experimental results, specific algorithms, and massive troves of personally identifiable information (PII). If LLMs could get access to train on all of this data, it would certainly unlock the next step-change of generative AI utility. 

But don’t hold your breath. It’s already becoming harder for LLMs to ingest public data. In the past two quarters, OpenAI has inked $100m+ content licensing deals with news publications like Vox Media and The Atlantic. But those deals are more akin to the cost of maintaining access to training data on news and current events, which might be otherwise hidden behind a paywall. Actual private data, like your email inbox and your company’s CRM database, remain fully inaccessible, and will likely never be for sale. While synthetic data might be a solution and is certainly already helpful, the jury is still out. But without access to more data, LLMs cannot continue to learn.

2. Free Interns have never built a billion-dollar business.

Even if we cannot produce bigger and better models, is there still room for generative AI to become more useful? Possibly. But not likely. The best analogy I’ve heard about generative AI has been to liken the experience to having an unlimited source of free interns at your disposal, all for just $20/mo. In the real world, having an intern is amazing. They can fix the printer, organize your calendar, and go pick up coffee. Give them any small task, and they will run off and come back with a solution. Similarly, whereas we’ve all had Google and its ten links at our fingertips for a few decades now, ChatGPT actually brings you solutions. It can save massive amounts of time, just like an intern. 

• Help me write an email to my vendor asking for a discount. 

• Summarize this lengthy email thread and tell me the action items.

• Give me 10 ideas for a new subject line for our weekly marketing email.

However, just like human interns, generative AI is not infinitely useful. While one or two interns are great, interns also require supervision, training, and management. They make mistakes. While LLMs can assist with seemingly countless types of tasks, they also cannot operate autonomously. They hallucinate. A billion-dollar business cannot be built on free labor, and the same principle applies to generative AI. Especially if your business exists in the real world, where humans like to interact with other humans, there is only so much time to be saved by utilizing LLMs. As such, if you’re not worried about being replaced by the college intern, you likewise need not worry about being replaced by ChatGPT.

3. Critical thinking skills are far more important than total knowledge alone.

Last and most importantly, the upside potential remaining for generative AI is limited by the nature of the models themselves. At the most fundamental level, LLMs produce answers by predicting the next token (word or phrase or concept) in a chain, based on the previously observed patterns in the training data. But an LLM does not truly “understand” those concepts in the way we do as humans. 

In the realm of K-12 education, a good math teacher cares not just that their students arrive at the correct answer, but that they understand the process of getting there. This emphasis on understanding over rote memorization is crucial, and it highlights a significant challenge with generative AI. Just like a math teacher might still fail a student that doesn’t "show their work," LLMs today are sorely lacking in their ability to explain themselves (e.g. even simply citing sources, where Perplexity shines best). Users are often left in the dark about how and why an AI model arrived at a particular conclusion. This opacity hinders trust and limits the practical application of AI in critical areas where transparency is essential, such as healthcare, finance, and legal services. Because a model does not think like we do as humans, it cannot question its own judgment and reconsider alternatives. 

While K-12 school teachers universally praise the value of teaching critical thinking skills, they’ll also admit that such skills are very hard to measure, especially with any sort of standardized testing. The same is true for LLMs. It may yet be possible to train a future LLM to think critically like we do as humans. Certainly those that believe in artificial general intelligence (AGI) expect it to happen. But with the current state of LLMs on the market today, critical thinking remains exclusively in the human domain. As humans we also have access to each other, and to the private data that other humans have trusted in us. With that, we can always keep learning.

In conclusion, Generative AI has undoubtedly revolutionized the landscape for knowledge workers, offering unprecedented capabilities and ways to save time or improve our productivity. However, progress has hit a plateau, primarily due to the scarcity of incrementally useful data. It is unlikely that private data will ever be made public or available to public LLMs. Until then, generative AI will remain a powerful yet limited tool, akin to a free intern: definitely helpful, but with limitations. You can trust the interns with fixing your printer or debugging some code. But they probably shouldn’t sit on your executive leadership team. Like many other technologies humans have invented, AI is an incredibly valuable tool to be used, but not feared. Will we ever go back to using typewriters or flip phones? Of course not. Enjoy the plateau.