By now, most of you have probably heard about the recently leaked document by a Google Researcher commenting on the state of the AI Space and the Large Language Models Arms Race. The sentiments expressed in the article document paint a very dark picture of the so-called “Age of Foundation Models”- in stark contrast to the hype being propagated by VCs, AI Companies, Social Media Influencers, and even AI doomers. In this article, I will be going over the document and its implications in more detail. Stay tuned, because you definitely don’t want to miss out on this (we’ll follow up on this with a piece on how businesses can use open-source to thrive tomorrow).
We’ve done a lot of looking over our shoulders at OpenAI. Who will cross the next milestone? What will the next move be?But the uncomfortable truth is, we aren’t positioned to win this arms race and neither is OpenAI. While we’ve been squabbling, a third faction has been quietly eating our lunch.
Why Open Source is Winning- Firstly it’s free. Having been a creator for a while, I can tell you firsthand that a price tag, no matter how small immediately causes many people to turn away. The free and open nature of open source enables contributions from everyone, which allows for solutions to scale much quicker. However, that is not all. After all, this would not explain the rapid development in the open-source community and how they have been able to beat Big Tech companies with Million Dollar Budgets and multiple years of building Large Language Models-
- Open-Source companies are doing things in weeks with $100 and 13B params that we struggle with at $10M and 540B.
There is one other thing that has allowed the open-source community to stack them Ws and pull off feats that Big Tech hasn’t been able to.
Open Source is more diverse- Think back to how Big Tech Hires people. And how it evaluates them. And how regimented practices are. All of these are great for efficiency, but it comes with a huge drawback- they kill innovation. The unfortunate side-effect of implementing systems to solve any problem is that your operations become more inflexible. There is a very strong selection bias in the kinds of people who will thrive in any regimented system. People who don’t conform to those ideals are filtered out- either in the hiring stages, through layoffs, or through employee churn (employees quitting because they don’t like the work/culture).
To a bird born in a cage, flying is a sickness
- We become blind to the flaws of the circumstances surrounding us.
Here’s an example- research positions in these companies generally require a Masters/Ph.D. degree. If most people in your research teams are from research team are from this background, they will replicate many of the flaws found in academia and upper-level education. And these will be invisible to the people in the system (and very hard to change because of the status quo bias). Once again, the Open Source community has a much lower barrier to entry- allowing more people to contribute. And when we put lots of different people together, and let them cook, great things happen. This is a clear example of that.
The document also made a claim that’s got a lot of AI Companies Sweating- “People will not pay for a restricted model when free, unrestricted alternatives are comparable in quality. We should consider where our value add really is.” This had a lot of people shaken but aligns well with what I’ve been saying for a while. Most recently, we had a discussion around AI Models and Moats on April 14th in a group chat I’m in. In it, I said the following about AI Moats-
I want to draw attention to the fact that this document specifically says “We Have No Moat, And Neither Does OpenAI”. I’ve been calling this for months.
This has now been backed up by the Google Researcher themselves. So what’s next for AI Models and companies looking to profit from these models? Just look at the last line- monetize the products around AI not AI. There is a world of difference between AI Models and AI Solutions, and that difference is where money is made. Unfortunately, this is a lot trickier than it seems. If you’d like more input on that side, make sure you come in tomorrow for the piece on how you can use open source to help your business, or reach out to me to discuss your needs.
Around this time last year, Meta did something that no one saw coming- they decided to open-source their research. The first thing they did was to release its OPT Model- its equivalent to GPT. Even though I covered it (and even called it the biggest development in AI), most people didn’t pay too much attention to the release. However, this has been the case. Take a look at two quotes from my analysis of the situation-
Firstly, since the model is open, it is possible for people to find and discover areas for improvement. This facet of the open-source culture is what is responsible for the explosive growth of tech over the last 2 decades. This gives them access to potentially millions of hours of free debugging/testing done by the community.
and
And the model being released for free also means that people are now much less likely to use paid models from their competitors.
These are both quotes from my article- Implications of Meta’s challenge to GPT-3 and Open AI, written on May 12th of last year. And both of these events played out exactly as I said. And while companies were investing billions into the LLM wars, Meta stayed out of the whole thing. Their competitors have burned billions, there is no profitability in sight (after spending 540 Million Dollars on building ChatGPT, Open AI is considering raising money through more funding), and now open source is lapping everyone up using resources that were made available from Meta. Here is what the document described as the problem with Google and OpenAI and why they were being beaten by OpenSource.
Wild Conspiracy theory- Meta leaked LLaMa themselves to break push open source community even further.
Notice how the Google researcher’s assessment of the situation aligned almost exactly with what I said would happen. Between this, my calls on Crypto and LLMs being a bubble, my call on AI Emergence being BS, and my assessment that making ML Models bigger wasn’t useful- I have a much better track record on predicting tech than most of these so-called experts. Maybe I should quit Engineering and become an investor.
Overall, this document didn’t say much that should be shocking. And the fact that people are losing their minds over these claims shows how disconnected people in AI have become from common sense. If you’re someone/know someone who needs a person to tell shocking insights such as-
- Retraining models from scratch is the hard path
- Large models aren’t more capable in the long run if we can iterate faster on small models
- Data quality scales better than data size
then I have a secret 50,000 USD investing tip to sell you.
That is it for this piece. I appreciate your time. As always, if you’re interested in working with me or checking out my other work, my links will be at the end of this email/post. If you like my writing, I would really appreciate an anonymous testimonial. You can drop it here. And if you found value in this write-up, I would appreciate you sharing it with more people. It is word-of-mouth referrals like yours that help me grow.
Upgrade your tech career with my newsletter ‘Tech Made Simple’! Stay ahead of the curve in AI, software engineering, and tech industry with expert insights, tips, and resources. 20% off for new subscribers by clicking this link. Subscribe now and simplify your tech journey!
Using this discount will drop the prices-
800 INR (10 USD) → 640 INR (8 USD) per Month
8000 INR (100 USD) → 6400INR (80 USD) per year (533 INR /month)
Use the links below to check out my other content, learn more about tutoring, reach out to me about projects, or just to say hi.
Small Snippets about Tech, AI and Machine Learning over here
If you like my writing, I would really appreciate an anonymous testimonial. You can drop it here.
To help me understand you fill out this survey (anonymous)
Check out my other articles on Medium. : https://rb.gy/zn1aiu
My YouTube: https://rb.gy/88iwdd
Reach out to me on LinkedIn. Let’s connect: https://rb.gy/m5ok2y
My Instagram: https://rb.gy/gmvuy9
My Twitter: https://twitter.com/Machine01776819
