Hey hey everyone, how are you this fine ThursdAI? 👋 I’m gud thanks for asking! I’m continuing my experiment of spilling the beans, and telling you about everything we talked about in advance, both on the pod and in the newsletter, so let me know if this is the right way to go or not, for the busy ones it seems that it is. If you don’t have an hour 15, here’s a short video recap of everything we chatted about: ThursdAI - Jan 11 2024 TL;DR TL;DR of all topics covered + Show notes * Open Source LLMs * 🔥 Donut from Jon Durbin is now top of the LLM leaderboard (X, HF, Wolframs deep dive and scoring) * OpenChat January Update - Best open source 7B LLM (X, Hugging Face) * Our friends at NousResearch announce a seed round of 5.2M as their models pass 1.2 million downloads (X) * Argilla improved (Distillabeled?) the DPO enhanced Neural Hermes with higher quality DPO pairs (X) * New MoEs are coming out like hotcakes - PhixTral and DeepSeek MoE (X, Omar Thread, Phixtral Thread) * Microsoft makes Phi MIT licensed 👏 * Big CO LLMs + APIs * OpenAI adds personalization & team tiers (Teams announcement) * OpenAI launches GPT store (Store announcement, Store link) * Mixtral medium tops the LMsys human evaluation arena, is the best LLM overall after GPT4 👏 (X) * Hardware * Rabbit R1 is announced, $200/mo without a subscription, everybody has a take (X) * This weeks Buzz from Weights & Biases * Hackathon with Together, Langchain and WandB (and ME!) this weekend in AGI house (X, Signup) * Video * Bytedance releases MagicVideo-V2 video gen that looks great and passes Pika labs in human tests (X) * AI Art & Diffusion & 3D * Luma launched their online version of Genie and it's coming to the API (X) * Show notes and links mentioned * MergeKit (github) * Jon Durbins Contextual DPO dataset (HuggingFace) * Phixtral from Maxime Lebonne (X, HuggingFace) * WandGPT - out custom Weights & Biases GPT (GPT store) * Visual Weather GPT by me - https://chatg.pt/artweather * Ask OpenAI to not train on your chats - https://privacy.openai.com/policies AI Hardware It seems that the X conversation had a new thing this week, the AI hardware startup Rabbit, showcased their new $200 device (no subscriptions!) at CES and everyone and their mom had an opinion! We had quite a long conversation about that with (his first time on ThursdAI 👏) as we both pre-ordered one, however there were quite a few red flags, like for example, GPUs are costly, so how would an AI device that has AI in the cloud just cost a 1 time 200 bucks?? There were other interesting things they showed during the demo, and I’ll let you watch the full 30 minutes and if you want to read more, here’s a great deeper dive into this from . UPDATE: Ss I’m writing this, the CEO of Rabbit (who’s also on the board of Teenage Engineering, the amazing company that designed this device) tweeted that they sold out the initial first AND second batch of 10K unites, netting a nice $2M in hardware sales in 48 hours! Open Source LLMs Mixtral paper dropped (ArXiv, Morgans take) Mistral finally published the paper on Mixtral of experts, the MoE that's the absolutel best open source model right now, and it's quite the paper. Nisten did a full paper reading with explanations on X space, which I co-hosted and we had almost 3K people tune in to listen. Here's the link to the live reading X space by Nisten. And here's some notes courtecy Morgan McGuire (who's my boss at WandB btw 🙌) Strong retrieval across the entire context window Mixtral achieves a 100% retrieval accuracy regardless of the context length or the position of passkey in the sequence. Experts don't seem to activate based on topic Surprisingly, we do not observe obvious patterns in the assignment of experts based on the topic. For instance, at all layers, the distribution of expert assignment is very similar for ArXiv papers (written in Latex), for biology (PubMed Abstracts), and for Philosophy (PhilPapers) documents. Ho