AI & Machine Learning

Understand the machines rewriting everything

Novice

Mastery0%

Novice

Note. AI tools change fast and confidently get things wrong. Verify anything high-stakes against primary sources, and never paste secrets/PII into untrusted endpoints.

Concept 11 of 19

framework

Mixture of Experts (MoE)

(Mixture of Experts)

Many specialist sub-models. Only a few wake up per token. You get a huge brain at a small compute bill.

Definition

An architecture where a large model contains many specialised sub-networks (experts) and a router decides which small subset of experts processes each token, so most parameters stay inactive on any one forward pass.

Step 1

Predict first

In one or two sentences, what do you think this concept means? Wrong is fine; the small struggle is what makes the answer stick.