- The Deep View
- Posts
- ⚙️ Meta developed new techniques to train Llama 4. The difference is minimal
⚙️ Meta developed new techniques to train Llama 4. The difference is minimal

Good morning. It was hard to look away from the CNBC homepage yesterday. I’ve never seen such wide or wild minute-by-minute swings before.
Neither has anyone else, for that matter; the Dow set a record for its largest intraday point swing. Ever.
The tariffs were announced less than a week ago.
Oh boy.
— Ian Krietzberg, Editor-in-Chief, The Deep View
In today’s newsletter:
⚕️ AI for Good: Enhanced anesthesiology
🏛️ A California bill that would have stymied OpenAI gets mysteriously gutted
👁️🗨️ Meta leveraged new techniques to train Llama 4. The difference is minimal
AI for Good: Enhanced anesthesiology

Source: Unsplash
Though anesthesia has advanced at a remarkable rate, complications related to anesthesia can occur during surgery, a risk that is compounded with older patients.
For Predictheon, “you can’t prevent what you can’t predict.”
The details: Spinning out of the Hospital CLINIC de Barcelona, the company is on a mission to develop and deploy predictive software — powered by machine learning algorithms — specifically designed to aid anesthesiologists during surgical operations.
At the core of the software — which is designed to integrate into medical devices — is advanced data analysis and mathematical modeling. Predictheon gathers millions of data points from thousands of real patients to construct their algorithms and models, then validates those models against real patient data.
In use, the system processes signals from a patient undergoing surgery in real time, dislaying predictions concerning potential complications that might arise.
In action: Eva Gubern, the CCO of Predictheon, has said that the company’s tech enables the detection of “over 80% of anesthetic-related adverse events up to 10 minutes in advance, enabling clinicians to take preventive action.”
In places where anesthesiologists are plentiful, the solution could enable better outcomes for patients and hospitals alike, according to Gubern. And in places suffering from massive shortages of anesthesiologists, the software can be “life-saving.”

How AI enhances hyperautomation
Camunda’s latest guide, The Ultimate Guide to Building Your Hyperautomation Tech Stack, breaks down everything you need to streamline operations, boost productivity, and drive innovation with AI and automation.
From orchestrating workflows to integrating AI-driven decision-making, this guide provides a step-by-step approach to selecting and implementing the right technologies for your business.
Whether you're just starting your automation journey or refining an existing strategy, you'll gain the insights needed to stay ahead in an increasingly automated world.
Download the guide today and take the first step toward a smarter, more agile enterprise.
A California bill that would have stymied OpenAI gets mysteriously gutted

Source: Unsplash
In February, California State Assembly member Diane Papan introduced AB 501, a bill that, if it passed, would have prevented OpenAI’s pending transition to a for-profit corporation.
It has been completely gutted.
The details: While it didn’t mention OpenAI by name, the bill dealt with “startup venture capital nonprofits” that earn more than $100 million in revenue over a five-year period. The bill would have prohibited non-profit startups that meet those criteria from converting to any “wholly or partially for-profit entity.”
Shortly after it was introduced, a coalition of scientists published an open letter urging the California State Legislature to pass the bill, writing: “nonprofits should fulfill their charitable missions, not generate private wealth.”
But on April 3, the entirety of the text of the bill was struck, replaced with a paragraph regarding aircraft liens.
The open letter added a postscript on April 7, writing: “our bill author mysteriously changed the bill language from preventing non-profit abuse to aircraft liens. We are shocked.”
You can track the bill changes here.
Papan’s office did not respond to a request for comment concerning the changes, and whether Altman spoke with her before the changes were made.
The landscape: OpenAI just secured a $40 billion funding round that is partially contingent upon the company completing a transition to a for-profit organization by the end of this year. Elon Musk’s lawsuit against OpenAI and Altman, part of which aims to prevent the transition, is set to go to a jury trial in 2026, after OpenAI’s funding-related conversion deadline.
And in November, OpenAI CEO Sam Altman was named to freshly elected San Francisco Mayor Daniel Lurie’s transition team, where Lurie said he would help bring AI into San Francisco’s government.
Details concerning that relationship and Altman’s work since are unclear.

From design to live in seconds.
Focus on what it looks like live, we’ll do the rest.
Publish your design as a live site in seconds. Hit publish, and your site is instantly live worldwide with Framer's state-of-the-art hosting.


Wild, wild swings: Stocks flirted with disasters and comebacks Monday, coming off of the two consecutive, enormous stock market routs that closed out last week. The Nasdaq was down as much as 4%, but closed the day slightly in the green, mainly thanks to a surge from Nvidia. The S&P fell as much as 5% during the day, but finished down just .2%. The Dow set a new record for the largest intraday point swing.
The content battle: Hundreds of news organizations — including some that have signed licensing agreements with OpenAI and the like — launched a campaign Monday urging lawmakers to “make Big Tech pay for the content it takes to run its AI products.” This somewhat shortly follows policy recommendations from Big Tech players, including OpenAI and Google, for the Trump Administration to axe copyright protections in the face of AI training.

iPhones were already losing ground in China. Then came Trump’s tariffs (Rest of World).
DOGE ditching tape storage could put data at risk, say experts (New Scientist).
Google AI Search shift leaves website makers feeling betrayed (Bloomberg).
OpenAI has discussed buying Sam Altman’s AI device startup (The Information).
Google’s AI Mode search can now answer questions about images (Ars Technica).
Meta leveraged new techniques to train Llama 4. The difference is minimal

Source: Meta
Over the weekend, Meta dropped Llama 4, a new “herd” of multimodal language models that purportedly boast greater speed, efficiency and overall performance when compared to Llama’s previous incarnations. This follows reports that the new family of models faced consistent setbacks and delays, internally.
Before we get into it, the release — once again — came bundled up in a (remarkably lengthy) blog post, and hasn’t been independently verified or peer-reviewed. You can read the system cards here.
The details: Meta started by building Llama 4 “behemoth,” a two-trillion-parameter language model that represents Meta’s “most powerful (model) yet.” Though the giant model is not yet available, Meta researchers leveraged it as a ‘teacher’ model to distil the two models it did release: Llama 4 Scout and Llama 4 Maverick (who comes up with these names?)
Distillation has become a popular technique, especially in the wake left by DeepSeek earlier in the year.
Both models were released with open weights, a far cry from open-source that is, nonetheless, much more accessible than, for instance, any of OpenAI’s models.
Meta employed a new mixture of experts (MoE) architecture to develop the models, something that, according to Meta, means that “a single token activates only a fraction of the total parameters,” resulting in models that are far more efficient to both train and operate (though how much more efficient it is remains unclear, especially for model operation).
The Maverick model has 17 billion active parameters and 400 billion total parameters, according to Meta. The Scout model also has 17 billion active parameters, with total parameters of 109 billion. This model can be run on a single Nvidia H100 GPU, according to Meta.
The training process for just Maverick and Scout resulted in 2,000 tons of carbon emissions. Meta did not share the carbon emissions associated with Behemoth, or the cost in electricity of running the models, though it did say that Maverick would likely cost between 19 and 49 cents per million input/output tokens, roughly in line with its competition.
The models were trained on more than 30 trillion tokens of data, “more than double the Llama 3 pre-training mixture,” according to Meta. This data included licensed data, posts that Meta scraped from its social media services and content Meta scraped from the internet.
Meta is currently facing a lawsuit regarding its training practices. The action has thus far unearthed internal documentation indicating that Meta knowingly trained its AI models on datasets of pirated books.
Beyond this, Meta said that it figured out a way — involving interleaved attention layers — to dramatically expand the models’ context length. Though trained with only a 256,000-token context length, Meta said the model can support windows of up to 10 million tokens.
Context length is a big deal since it represents the maximum length of a given piece of content or code that a model can process at once.
AI PhD and author Andriy Burkov said that the 10 million context length is “virtual because no model was trained on prompts longer than 256k tokens. This means that if you send more than 256k tokens to it, you will get low-quality output most of the time.”
Meta did not return a request for comment regarding this.
Both Maverick and Scout slightly outperform equivalent models on benchmarks presented by Meta.

Source: Meta
The release represents a break from popular approaches — o3, Claude hybrid, DeepSeek, Gemini, etc. — in that none of the Llama models on display are ‘reasoning’ models.
Meta added that it aimed to address issues with model bias, specifically, that “they historically have leaned left when it comes to debated political and social topics.” Meta said that now, Llama refuses to respond to fewer than 2% of contentious social and political topics.
“If today's disappointing release of Llama 4 tells us something, it's that even 30 trillion training tokens and 2 trillion parameters don't make your non-reasoning model better than smaller reasoning models,” Burkov said. “Model and data size scaling are over.”
Burkov is not alone in his disappointment over the release.

As I’ve discussed often here, benchmarks are practically useless. They rarely test what they purport to test, and when training data and processes are unknown, they don’t actually indicate anything about model capability.
That said, considering the scale of the new approach Meta took to build this model family, minimal improvements over competitors on its benchmarks doesn’t seem like a huge win. It’s a very high cost for something that’s marginally better than its competitors and predecessors. On the MMLU reasoning benchmark, for instance, the 70 billion parameter Llama 3 model (one year ago) scored an 82; Maverick achieved an 84.6 on the same benchmark.
I would add that Meta’s addendum regarding model bias misses the point completely; erasing algorithmic bias is impossible, since models are reliant on data — whose quantities are necessarily skewed — that is produced by humans.
An MIT professor focusing on algorithmic fairness told me last year that the focus should instead be on enhancing interpretability and transparency, so that users know when to trust a model and when to not.


Which image is real? |



🤔 Your thought process:
Selected Image 1 (Left):
“Vein on hand on fake image was mostly right, but why was there a valve on the vein at the end closest to the finger. Lighting seemed odd. Was a tough one.”
Selected Image 1 (Left):
“The tools looked just a little bit ‘off’ in the second image, like they were drawn by an artist who had never actually used them.”
💭 A poll before you go
Thanks for reading today’s edition of The Deep View!
We’ll see you in the next one.
P.S. Enjoyed reading? You can listen, too! We’ve got exclusive, in-depth interviews for you on “The Deep View: Conversations” podcast every Tuesday morning. Subscribe here!
Here’s your view on NJ’s deepfake law:
70% of you think it’s a great move; a little more than half of those would love to see a similar law enacted at the federal level.
15% don’t love it.
Don’t love it:
“How to enforce it is the problem for any new law. With so much AI coming at you from around the world, it's not clear how they could enforce it. However, I agree with the intent. We'll have to see what happens in the first few cases.”
Was needed:
“It's definitely needed, but including political deep-fakes is a little worrisome. There's a lot of satire out there being distributed by all parties. Who decides what's a harmful political deep-fake?”
Have you used Llama 4? How do you like it? |
If you want to get in front of an audience of 450,000+ developers, business leaders and tech enthusiasts, get in touch with us here.