Marin 535B Model Starts Open Pretraining on 18.75T Tokens
Researchers begin transparent pretraining of a 535B-parameter model using public data and configs.
TLDR
Posts from researchers and engineers report that the Marin project has started pretraining its 535B-parameter model on 18.75 trillion tokens. The run uses 11 NVIDIA GB200 NVL72 systems and is projected to last roughly three months. Participants describe the effort as fully open, with public pretraining mixtures, sampled documents, live loss curves, and training configs. Replies highlight the scale and transparency while comparing it to prior open initiatives. No independent verification of completion or final results appears in the packet.
Combined views
109K
17 Sources, first seen ago
Sources
EL 
elie@eliebakouchmarin is the most open model training initiative in the world you can visualize the exact pre training mixture per domain, look at sampled documents, follow the live training loss, dig into configs, scaling laws https://twitter.com/percyliang/status/2090918065634684997
- likes: 860
- replies: 12
- bookmarks: 491
- reposts: 59
RI rishi@RishiBommasani5 years ago, almost to the day, @siddkaramcheti @laurel_orr1 @percyliang put out their first public writing on training open models. https://crfm.stanford.edu/2021/08/26/mistral.html Even then, model training was not for the faint of heart. Mistral (before there was @MistralAI)…
- likes: 43
- replies: 0
- bookmarks: 22
- reposts: 7
JA Julius Adebayo@juliusadmlpre-registering your pretraining loss forecast is the ultimate confidence move. 535B moe training happening all in the open. https://twitter.com/percyliang/status/2090918067987689486
- likes: 15
- replies: 0
- bookmarks: 6
- reposts: 0
AD Alex Dimakis@AlexGDimakisIn medieval times, technical knowledge was kept hidden : guilds guarded recipes, processes, and tools and alchemists often wrote in deliberately obscure language. Knowledge was kept secret because it gives economic, professional, or political advantage. Enlightenment and the…
- likes: 405
- replies: 10
- bookmarks: 123
- reposts: 44
ZW Zhaoran Wang@zhaoran_wang- likes: 2
- replies: 0
- bookmarks: 0
- reposts: 0
AK Amin Karbasi@aminkarbasiOpen science AI in its finest form. https://twitter.com/percyliang/status/2090918065634684997
- likes: 13
- replies: 0
- bookmarks: 2
- reposts: 1