ThursdAI - The top AI news from the past week From Weights & Biases, Join AI Evangelist Alex Volkov and a panel of experts to cover everything important that happened in the world of AI from the past week

Hey 👋 (show notes and links a bit below)
This week has been a great AI week, however, it does feel like a bit "quiet before the storm" with Google I/O on Tuesday next week (which I'll be covering from the ground in Shoreline!) and rumors that OpenAI is not just going to let Google have all the spotlight!
Early this week, we got 2 new models on LMsys, im-a-good-gpt2-chatbot and im-also-a-good-gpt2-chatbot, and we've now confirmed that they are from OpenAI, and folks have been testing them with logic puzzles, role play and have been saying great things, so maybe that's what we'll get from OpenAI soon?
Also on the show today, we had a BUNCH of guests, and as you know, I love chatting with the folks who make the news, so we've been honored to host Xingyao Wang and Graham Neubig core maintainers of Open Devin (which just broke SOTA on Swe-Bench this week!) and then we had friends of the pod Tanishq Abraham and Parmita Mishra dive deep into AlphaFold 3 from Google (both are medical / bio experts).
Also this week, OpenUI from Chris Van Pelt (Co-founder & CIO at Weights & Biases) has been blowing up, taking #1 Github trending spot, and I had the pleasure to invite Chris and chat about it on the show!
Let's delve into this (yes, this is I, Alex the human, using Delve as a joke, don't get triggered 😉)
TL;DR of all topics covered (trying something new, my Raw notes with all the links and bulletpoints are at the end of the newsletter)
* Open Source LLMs
* OpenDevin getting SOTA on Swe-Bench with 21% (X, Blog)
* DeepSeek V2 - 236B (21B Active) MoE (X, Try It)
* Weights & Biases OpenUI blows over 11K stars (X, Github, Try It)
* LLama-3 120B Chonker Merge from Maxime Labonne (X, HF)
* Alignment Lab open sources Buzz - 31M rows training dataset (X, HF)
* xLSTM - new transformer alternative (X, Paper, Critique)
* Benchmarks & Eval updates
* LLama-3 still in 6th place (LMsys analysis)
* Reka Core gets awesome 7th place and Qwen-Max breaks top 10 (X)
* No upsets in LLM leaderboard
* Big CO LLMs + APIs
* Google DeepMind announces AlphaFold-3 (Paper, Announcement)
* OpenAI publishes their Model Spec (Spec)
* OpenAI tests 2 models on LMsys (im-also-a-good-gpt2-chatbot & im-a-good-gpt2-chatbot)
* OpenAI joins Coalition for Content Provenance and Authenticity (Blog)
* Voice & Audio
* Udio adds in-painting - change parts of songs (X)
* 11Labs joins the AI Audio race (X)
* AI Art & Diffusion & 3D
* ByteDance PuLID - new high quality ID customization (Demo, Github, Paper)
* Tools & Hardware
* Went to the Museum with Rabbit R1 (My Thread)
* Co-Hosts and Guests
* Graham Neubig (@gneubig) & Xingyao Wang (@xingyaow_) from Open Devin
* Chris Van Pelt (@vanpelt) from Weights & Biases
* Nisten Tahiraj (@nisten) - Cohost
* Tanishq Abraham (@iScienceLuvr)
* Parmita Mishra (@prmshra)
* Wolfram Ravenwolf (@WolframRvnwlf)
* Ryan Carson (@ryancarson)
Open Source LLMs
Open Devin getting a whopping 21% on SWE-Bench (X, Blog)
Open Devin started as a tweet from our friend Junyang Lin (on the Qwen team at Alibaba) to get an open source alternative to the very popular Devin code agent from Cognition Lab (recently valued at $2B 🤯) and 8 weeks later, with tons of open source contributions, >100 contributors, they have almost 25K stars on Github, and now claim a State of the Art score on the very hard Swe-Bench Lite benchmark beating Devin and Swe-Agent (with 18%)
They have done so by using the CodeAct framework developed by Xingyao, and it's honestly incredible to see how an open source can catch up and beat a very well funded AI lab, within 8 weeks! Kudos to the OpenDevin folks for the organization, and amazing results!
DeepSeek v2 - huge MoE with 236B (21B active) parameters (X, Try It)
The folks at DeepSeek is releasing this huge MoE (the biggest we've seen in terms of experts) with 160 experts, and 6 experts activated per forward pass. A similar trend from the Snowflake team, just extended even longer. They also introduce a lot of technical details and

Hey 👋 Look it May or May not be the first AI newsletter you get in May, but it's for sure going to be a very information dense one. As we had an amazing conversation on the live recording today, over 1K folks joined to listen to the first May updates from ThursdAI.
As you May know by now, I just love giving the stage to folks who are the creators of the actual news I get to cover from week to week, and this week, we had again, 2 of those conversations.
First we chatted with Piotr Padlewski from Reka, the author on the new Vibe-Eval paper & Dataset which they published this week. We've had Yi and Max from Reka on the show before, but it was Piotr's first time and he was super super knowledgeable, and was really fun to chat with.
Specifically, as we at Weights & Biases launch a new product called Weave (which you should check out at https://wandb.me/weave) I'm getting more a LOT more interested in Evaluations and LLM scoring, and in fact, we started the whole show today with a full segment on Evals, Vibe checks and covered a new paper from Scale about overfitting.
The second deep dive was with my friend Idan Gazit, from GithubNext, about the new iteration of Github Copilot, called Copilot Workspace. It was a great one, and you should definitely give that one a listen as well

TL;DR of all topics covered + show notes
* Scores and Evals
* No notable changes, LLama-3 is still #6 on LMsys
* gpt2-chat came and went (in depth chan writeup)
* Scale checked for Data Contamination on GSM8K using GSM-1K (Announcement, Paper)
* Vibes-Eval from Reka - a set of multimodal evals (Announcement, Paper, HF dataset)
* Open Source LLMs
* Gradient releases 1M context window LLama-3 finetune (X)
* MaziyarPanahi/Llama-3-70B-Instruct-DPO-v0.4 (X, HF)
* Nous Research - Hermes Pro 2 - LLama 3 8B (X, HF)
* AI Town is running on Macs thanks to Pinokio (X)
* LMStudio releases their CLI - LMS (X, Github)
* Big CO LLMs + APIs
* Github releases Copilot Workspace (Announcement)
* AI21 - releases Jamba Instruct w/ 256K context (Announcement)
* Google shows Med-Gemini with some great results (Announcement)
* Claude releases IOS app and Team accounts (X)
* This weeks Buzz
* We're heading to SF to sponsor the biggest LLama-3 hackathon ever with Cerebral Valley (X)
* Check out my video for Weave our new product, it's just 3 minutes (Youtube)
* Vision & Video
* Intern LM open sourced a bunch of LLama-3 and Phi based VLMs (HUB)
* And they are MLXd by the "The Bloke" of MLX, Prince Canuma (X)
* AI Art & Diffusion & 3D
* ByteDance releases Hyper-SD - Stable Diffusion in a single inference step (Demo)
* Tools & Hardware
* Still haven't open the AI Pin, and Rabbit R1 just arrived, will open later today
* Co-Hosts and Guests
* Piotr Padlewski (@PiotrPadlewski) from Reka AI
* Idan Gazit (@idangazit) from Github Next
* Wing Lian (@winglian)
* Nisten Tahiraj (@nisten)
* Yam Peleg (@yampeleg)
* LDJ (@ldjconfirmed)
* Wolfram Ravenwolf (@WolframRvnwlf)
* Ryan Carson (@ryancarson)
Scores and Evaluations
New corner in today's pod and newsletter given the focus this week on new models and comparing them to existing models.
What is GPT2-chat and who put it on LMSys? (and how do we even know it's good?)
For a very brief period this week, a new mysterious model appeared on LMSys, and was called gpt2-chat. It only appeared on the Arena, and did not show up on the leaderboard, and yet, tons of sleuths from 4chan to reddit to X started trying to figure out what this model was and wasn't.
Folks started analyzing the tokenizer, the output schema, tried to get the system prompt and gauge the context length. Many folks were hoping that this is an early example of GPT4.5 or something else entirely.
It did NOT help that uncle SAMA first posted the first tweet and then edited it to remove the - and it was unclear if he's trolling again or foreshadowing a completely new release or an old GPT-2 but retrained on newer data or something.
The model was really surprisingly good, s

Hey hey folks, happy ThursdAI 🎉
Not a lot of house-keeping here, just a reminder that if you're listening or reading from Europe, our European fullyconnected.com conference is happening in May 15 in London, and you're more than welcome to join us there. I will have quite a few event updates in the upcoming show as well.
Besides this, this week has been a very exciting one for smaller models, as Microsoft teased and than released Phi-3 with MIT license, a tiny model that can run on most macs with just 3.8B parameters, and is really punching above it's weights. To a surprising and even eyebrow raising degree! Let's get into it 👇
ThursdAI - Recaps of the most high signal AI weekly spaces is a reader-supported publication. To receive new posts and support my work, consider becoming a free or paid subscriber.

TL;DR of all topics covered:
* Open Source LLMs
* Microsoft open sources Phi-3 (X, HF)
* LLama3 70B top5 (no top 6) on LMsys (LMsys Arena)
* Snowflake open sources Arctic - A massive hybrid MoE (X, Try it, HF)
* Evolutionary Model merges support in MergeKit (Blog)
* Llama-3 8B finetunes roundup - Longer Context (128K) and Dolphin & Bagel Finetunes
* HuggingFace FINEWEB - a massive 45TB (the GPT4 of datasets) and 15T tokens high quality web data dataset (HF)
* Cohere open sourced their chat interface (X)
* Apple open sources OpenElm 4 models + training library called corenet (HF, Github, Paper)
* Big CO LLMs + APIs
* Google Gemini 1.5 pro is #2 on LMsys arena
* Devin is now worth 2BN and Perplexity is also a Unicorn
* A new comer called Augment (backed by Eric Schmidt) is now coming out of stealth (X)
* Vision & Video
* Adobe releases VideoGigaGAN - high quality upscaler with temporal consistency (paper)
* TLDraw autocomplete UI demo (X)
* This Weeks Buzz - What I learned in WandB this week
* Joe Spisak talk about Llama3 on Stage at WandB Fully connected (Full Talk, TLDR)
* Voice & Audio
* Play.ai (previously play.ht) releases conversational Voice AI platform (X)
* AI Art & Diffusion & 3D
* IMGsys.org- like LMsys but for image generation model + leaderboard from FAL (try it)
* Tools & Hardware
* Rabbit R1 release party & no shipping update in sight
* I'm disillusioned about my AI Pin and will return it
Open Source LLMs
Llama-3 1 week-aversary 🎂 - Leaderboard ranking + finetunes
Well, it's exactly 1 week since we got Llama-3 from Meta and as expected, the rankings show a very very good story. (also it was downloaded over 1.2M times and already has 600 derivatives on HuggingFace)
Just on Monday, Llama-3 70B (the bigger version) took the incredible 5th place (now down to 6th) on LMSys, and more surprising, given that the Arena now has category filters (you can filter by English only, Longer chats, Coding etc) if you switch to English Only, this model shows up 2nd and was number 1 for a brief period of time.
So just to sum up, an open weights model that you can run on most current consumer hardware is taking over GPT-4-04-94, Claude Opus etc'
This seems dubious, because well, while it's amazing, it's clearly not at the level of Opus/Latest GPT-4 if you've used it, in fact it fails some basic logic questions in my tests, but it's a good reminder that it's really hard to know which model outperforms which and that the arena ALSO has a bias, of which people are using it for example and that evals are not a perfect way to explain which models are better.
However, LMsys is a big component of the overall vibes based eval in our community and Llama-3 is definitely a significant drop and it's really really good (even the smaller one)
One not so surprising thing about it, is that the Instruct version is also really really good, so much so, that the first finetunes of Eric Hartfords Dolphin (Dolphin-2.8-LLama3-70B) is improving just a little bit over Meta's own instruct version, which is done very well.
Per Joe Spisak (Program Manager @ Meta AI) chat at the Weights & Biases conference last week (which you

Happy LLama 3 day folks! After a lot of rumors, speculations, and apparently pressure from the big Zuck himself, we finally can call April 18th, 2024, LLaMa 3 day!
I am writing this, from a lobby of the Mariott hotel in SF, where our annual conference is happening called Fully Connected, and I recorded today's episode from my hotel room. I really wanna shout out how awesome it was to meet folks who are listeners of the ThursdAI pod and newsletter subscribers, participate in the events, and give high fives.
During our conference, we had the pleasure to have Joe Spisak, the Product Director of LLaMa at Meta, to actually announce LLaMa3 on stage! It was so exhilarating, I was sitting in the front row, and then had a good chat with Joe outside of the show 🙌
The first part of the show was of course, LLaMa 3 focused, we had such a great time chatting about the amazing new 8B and 70B models we got, and salivating after the announced but not yet released 400B model of LLaMa 3 😮
We also covered a BUNCH of other news from this week, that was already packed with tons of releases, AI news and I was happy to share my experiences running a workshop a day before our conference, with focus on LLM evaluations. (If there's an interest, I can share my notebooks and maybe even record a video walkthrough, let me know in the comments)
Ok let's dive in 👇
Happy LLama 3 day 🔥
The technical details
Meta has finally given us what we're all waiting for, an incredibly expensive (2 clusters of 24K H100s over 15 Trillion tokens) open weights models, the smaller 8B one and the larger 70B one.
We got both instruction fine tune and base models, which are great for finetuners, and worth mentioning that this is a dense model (not a mixture of experts, all the parameters are accessible for the model during inference)
It is REALLY good at benchmarks, with the 7B model beating the previous (LLaMa 2 70B) on pretty much all benchmarks, and the new 70B is inching on the bigger releases from the past month or two, like Claude Haiku and even Sonnet!
The only downsides are the 8K context window + non multimodality, but both are coming according to Joe Spisak who announced LLama3 on stage at our show Fully Connected 🔥
I was sitting in the front row and was very excited to ask him questions later!
By the way, Joe did go into details they haven't yet talked about pulblicly (see? I told you to come to our conference! and some of you did!) and I've been live-tweeting his whole talk + the chat outside with the "extra" spicy questions and Joes winks haha, you can read that thread here
The additional info
Meta has also partnered with both Google and Bing (take that OpenAI) and inserted LLama 3 into the search boxes of Facebook, Instagram, Messenger and Whatsapp plus deployed it to a new product called meta.ai (you can try it there now) and is now serving LLama 3 to more than 4 Billion people across all of those apps, talk about compute cost!
Llama 3 also has a new Tokenizer (that Joe encouraged us to "not sleep on") and a bunch of new security tools like Purple LLama and LLama Guard. PyTorch team recently released finetuning library called TorchTune is now supporting LLama3 finetuning natively out of the box as well (and integrates Wandb as it's first party experiment tracking tool)
If you'd like more details, directly from Joe, I was live tweeting his whole talk, and am working at getting the slides from our team. We'll likely have a recording as well, will post it as soon as we have it.
Here's a TL;DR (with my notes for the first time) of everything else we talked about, but given today is LLaMa day, and I still have to do fully connected demos, I will "open source" my notes and refer you to the podcast episode to hear more detail about everything else that happened today 🫡
TL;DR of all topics covered:
* Meta releases LLama 3 -8B, 70B and later 400B (Announcement, Models, Try it, Run Locally)
* Open Source LLMs
* Meta LLama 3

this week was absolutely bonkers. For starters, for the first time ever, we got an Open Weights model (Command R+) to jump over GPT-4 in human rankings on LMsys, this is huge!
Then on Tuesday, it seems that all the companies just wanted to one up one another, first Gemini 1.5 released with updates, made it available in 180 countries, added audio mode + tons of API improvements and system prompts, then less than an hour later, OpenAI has given us a "majorly improved" GPT-4 Turbo version (2024-04-09) that is now back to being the BEST LLM IN THE WORLD and to cap that day off, Mistral did the thing again, the thing being, dropping a torrent link in a tweet with no explanations.
What was in that torrent is a Mixtral 8x22B MoE (which we started calling Bixtral) which comes with an Apache2 license and seems to be VERY good!
We also saw the first finetune from HuggingFace/KAIST folks less than 48 hours later (the authors of said finetune actually came on the show 🎉 )
Fully Connected is a week from today! If you haven't yet signed up, use THURSDAI promo code and come hear from Richard Socher (You.com), Jerry Liu (Ilamaindex CEO), Karoly (TwoMinutePapers), Joe Spisak (Meta) and and leaders from NVIDIA, Snowflake, Microsoft, Coatue, Adobe, Siemens, Lambda and tons more 👇
TL;DR of all topics covered:
* Open Source LLMs
* 🔥 Mistral releases Mixtral 8x22 Apache 2 licensed MoE model (Torrent, TRY IT)
* Cohere CMDR+ jumps to no 6 on LMSys and beats GPT4 (X)
* CodeGemma, RecurrentGemma & Gemma Instruct 1.1 (Announcement)
* Auto-code-rover gets 22% on SWE bench (Announcement)
* HuggingFace - Zephyr 141B-A35B - First Bixtral Finetune (Announcement)
* Mistral 22B - 1 single expert extracted from MoE (Announcement, HF)
* This weeks Buzz - Weights & Biases updates
* FullyConnected is in 1 week! (Come meet us)
* Big CO LLMs + APIs
* 🔥 GPT-4 turbo is back to being number 1 AI with 88.2% Human Eval score (X)
* Gemini 1.5 Pro now understands audio, uses unlimited files, acts on your commands, and lets devs build incredible things with JSON mode (X)
* LLama 3 coming out in less than a month (confirmed by Meta folks)
* XAI Grok now powers news summaries on X (Example)
* Cohere new Rerank 3 (X)
* Voice & Audio
* HuggingFace trained Parler-TTS (Announcement, Github)
* Udio finally launched it's service (Announcement, Leak, Try It)
* Suno has added explore mode (suno.ai/explore)
* Hardware
* Humane AI pin has started shipping - reviews are not amazing

Open Source LLMs
Command R+ first open weights model that beats last year GPT4 versions
This is massive, really a milestone to be discussed, and even though tons of other news happened, the first time an open weights model is beating GPT-4 not on a narrow case (coding, medical) but on a general human evaluation on the arena.
This happened just a year after GPT-4 first came out, and is really really impressive.
Command R+ has been getting a lot of great attention from the community as well, folks were really surprised by the overall quality, not to mention the multilingual abilities of CommandR+
Mixtral 8x22B MoE with 65K context and Apache 2 license (Bigstral)
Despite the above, Cohere time in the sun (ie top open weights model on lmsys) may not be that long if the folks at Mistral have anything to say about it!
Mistral decided to cap the crazy Tuesday release day with another groundbreaking tweet of theirs which includes a torrent link and nothing else (since then they of course uploaded the model to the hub) giving us what potentially will unseat Command R from the rankings.
The previous Mixtral (8x7B) signaled the age of MoEs and each expert in that was activated from Mistral 7B, but for this new affectionally named Bixtral model, each expert is a 22B sized massive model.
We only got a base version of it, which is incredible on it's own right, but it's not instruction finetuned yet, and the finetuner community is already cooking really hard! Though it's hard because this model requi

Happy first ThursdAI of April folks, did you have fun on April Fools? 👀 I hope you did, I made a poll on my feed and 70% did not participate in April Fools, which makes me a bit sad!
Well all-right, time to dive into the news of this week, and of course there are TONS of news, but I want to start with our own breaking news! That's right, we at Weights & Biases have breaking new of our own today, we've launched our new product today called Weave!
Weave is our new toolkit to track, version and evaluate LLM apps, so from now on, we have Models (what you probably know as Weights & Biases) and Weave. So if you're writing any kind RAG system, anything that uses Claude or OpenAI, Weave is for you!
I'll be focusing on Weave and I'll be sharing more on the topic, but today I encourage you to listen to the launch conversation I had with Tim & Scott from the Weave team here at WandB, as they and the rest of the team worked their ass off for this release and we want to celebrate the launch 🎉
TL;DR of all topics covered:
* Open Source LLMs
* Cohere - CommandR PLUS - 104B RAG optimized Sonnet competitor (Announcement, HF)
* Princeton SWE-agent - OSS Devin - gets 12.29% on SWE-bench (Announcement, Github)
* Jamba paper is out (Paper)
* Mozilla LLamaFile now goes 5x faster on CPUs (Announcement, Blog)
* Deepmind - Mixture of Depth paper (Thread, ArXiv)
* Big CO LLMs + APIs
* Cloudflare AI updates (Blog)
* Anthropic adds function calling support (Announcement, Docs)
* Groq lands function calling (Announcement, Docs)
* OpenAI is now open to customers without login requirements
* Replit Code Repair - 7B finetune of deep-seek that outperforms Opus (X)
* Google announced Gemini Prices + Logan joins (X)קרמ
* This weeks Buzz - oh so much BUZZ!
* Weave lunch! Check weave out! (Weave Docs, Github)
* Sign up with Promo Code THURSDAI at fullyconnected.com
* Voice & Audio
* OpenAI Voice Engine will not be released to developers (Blog)
* Stable Audio v2 dropped (Announcement, Try here)
* Lightning Whisper MLX - 10x faster than whisper.cpp (Announcement, Github)
* AI Art & Diffusion & 3D
* Dall-e now has in-painting (Announcement)
* Deep dive
* Jamba deep dive with Roi Cohen from AI21 and Maxime Labonne
Open Source LLMs
Cohere releases Command R+, 104B RAG focused model (Blog)
Cohere surprised us, and just 2.5 weeks after releasing Command-R (which became very popular and is No 10 on Lmsys arena) gave us it's big brother, Command R PLUS
With 128K tokens in the context window, this model is multilingual as well, supporting 10 languages and is even beneficial on tokenization for those languages (a first!)
The main focus from Cohere is advanced function calling / tool use, and RAG of course, and this model specializes in those tasks, beating even GPT-4 turbo.
It's clear that Cohere is positioning themselves as RAG leaders as evident by this accompanying tutorial on starting with RAG apps and this model further solidifies their place as the experts in this field. Congrats folks, and thanks for the open weights 🫡
SWE-Agent from Princeton
Folks remember Devin? The super cracked team born agent with a nice UI that got 13% on the SWE-bench a very hard (for LLMs) benchmark that requires solving real world issues?
Well now we have an open source agent that comes very very close to that called SWE-Agent
SWE agent has a dedicated terminal and tools, and utilizes something called ACI (Agent Computer Interface) allowing the agent to navigate, search, and edit code.
The dedicated terminal in a docker environment really helps as evident by a massive 12.3% score on SWE-bench where GPT-4 gets only 1.4%!
Worth mentioning that SWE-bench is a very hard benchmark that was created by the folks who released SWE-agent, and here's some videos of them showing the agent off, this is truly an impressive achievement!
Deepmind publishes Mixture of Depth (arXiv)
Thanks to Hassan who read the paper and wrote a deep dive, this paper by Deepmind shows thei