XAI Grok-1 is now open-source, claiming the title of the largest open-source LLM ever built!

Grok-1 🧠 With a whopping 314-billion parameters, Grok-1 leverages a Mixture of Experts (MoE) model, actively using 86 billion parameters to supercharge its processing power. 💪 🔍 Quick Look at Grok-1 Specs: - 🧮 Parameters: 314 billion, with 25% active per token. - 🏗️ Architecture: 8 Expert MoE, 2 experts per token. - 📚 Layers: 64 transformer layers with cutting-edge attention and dense blocks. - 🔤 Tokenization: SentencePiece with a vast 131,072 vocab size. - 📐 Embedding & Positioning: 6,144-size embeddings with matching rotary positional embeddings. - 🧐 Attention: 48 heads for queries, 8 for keys/values, each head sized at 128. - 📈 Context Length: Handles an impressive 8,192 tokens, utilizing bf16 precision.