Create one slide based on the following information. The slide should use simple graphical techniques without looking AI-generated: The DeepSeekMoE architecture (used in models like DeepSeek-V3) handles token routing completely differently from traditional MoE architectures like Mixtral or Google's GShard. Instead of choosing 1 or 2 large experts from a small pool, DeepSeek routes tokens using a specialized two-tier architecture built on two main concepts: Shared Expert Isolation and Fine-Grained Expert Segmentation. Before routing, a standard Feed-Forward Network (FFN) layer is structurally split into a large pool of much smaller expert networks. • Shared Experts: 1 or 2 experts are permanently isolated. • Routed Experts: The remaining FFN capacity is segmented into a large pool of tiny experts (e.g., 256 distinct experts per layer). When a token arrives at a DeepSeekMoE layer, • Every single token is automatically sent to the Shared Experts. • Purpose: These experts are always active and capture "common knowledge" (like basic grammar, punctuation, and general language structures). By handling the generic capabilities universally, it prevents the remaining experts from overlapping or wasting parameters on redundant, basic functions. Simultaneously, the token hidden vector passes through a Gating Network (Router) to choose its specialized experts: Relevance Scoring: The router calculates the dot product between the token’s vector and the linear routing weights of all 256 routed experts. This produces a score (logit) for every expert, indicating how well that expert fits the token. Top-K Selection: DeepSeek selects the Top-8 highest-scoring experts out of the 256. Because the 256 experts are incredibly small (fine-grained), activating 8 of them provides highly flexible, specialized algorithmic combinations without spiking compute costs. Softmax Normalization: The scores of the 8 selected experts are normalized using a Softmax function so they sum to 1. The non-selected 248 experts are zeroed out (deactivated).
Create one slide based on the following information. The slide should use simple graphical techniques without looking AI-generated: The DeepSeekMoE architecture (used in models like DeepSeek-V3) handles token routing completely differently from traditional MoE architectures like Mixtral or Google's GShard.
Instead of choosing 1 or 2 large experts from a small pool, DeepSeek routes tokens using a specialized two-tier architecture built on two main concepts: Shared Expert Isolation and Fine-Grained Expert Segmentation. Before routing, a standard Feed-Forward Network (FFN) layer is structurally split into a large pool of much smaller expert networks.
• Shared Experts: 1 or 2 experts are permanently isolated.
• Routed Experts: The remaining FFN capacity is segmented into a large pool of tiny experts (e.g., 256 distinct experts per layer).
When a token arrives at a DeepSeekMoE layer,
• Every single token is automatically sent to the Shared Experts.
• Purpose: These experts are always active and capture "common knowledge" (like basic grammar, punctuation, and general language structures). By handling the generic capabilities universally, it prevents the remaining experts from overlapping or wasting parameters on redundant, basic functions.
Simultaneously, the token hidden vector passes through a Gating Network (Router) to choose its specialized experts:
Relevance Scoring: The router calculates the dot product between the token’s vector and the linear routing weights of all 256 routed experts. This produces a score (logit) for every expert, indicating how well that expert fits the token.
Top-K Selection: DeepSeek selects the Top-8 highest-scoring experts out of the 256. Because the 256 experts are incredibly small (fine-grained), activating 8 of them provides highly flexible, specialized algorithmic combinations without spiking compute costs.
Softmax Normalization: The scores of the 8 selected experts are normalized using a Softmax function so they sum to 1. The non-selected 248 experts are zeroed out (deactivated).
Created using ChatSlide
DeepSeekMoE introduces an innovative routing model that divides feedforward network capacity into smaller experts while isolating 1-2 shared experts. Tokens are processed through a two-tier system, utilising shared experts and scoring 256 routed specialists to select the top 8, weighted by softmax. This fine-grained routing approach activates only a fraction of specialists, allowing for the capture of both common and specialised knowledge, demonstrating a flexible routing mechanism with fixed...