In this post, we share our findings on building effective pre-trained LLMs with moderate resources. We show that an 80B-A3B MoE model trained on fewer than 14 trillion tokens delivers competitive performance against similarly sized and larger systems, while a depth‑upscaled 130B variant, continued for a modest amount of training at a small learning rate, delivers a clear performance lift over the 80B baseline.

1. Architecture

The new WeLM series employs a highly sparse MoE architecture (Shazeer et al. 2017; Fedus et al. 2022) designed for computational efficiency during training and inference. The MoE layers integrate loss-free balance routing (DeepSeek-AI et al. 2024) with unnormalized sigmoid gates and one shared expert (Dai et al. 2024). Following the Qwen series (Team 2024), the model adopts a deeper architecture stabilized through the combination of PostNorm and OutputNorm (Zhuo et al. 2025; Ding et al. 2021).

The attention mechanism consists of Grouped-Query Attention (Ainslie et al. 2023) with partial RoPE, triple the standard number of attention heads combined with headwise gating (Qiu et al. 2025), and KNorm for attention logit stabilization (DeepSeek-AI 2024). KV-Mirror shares hidden states between shallow and deep layers for KV activation generation (Sun et al. 2024; Wu et al. 2024), reducing inference costs. The model incorporates a 2-head over-encoding module, which reduces training loss without increasing inference overhead. An additional MoE layer serves as the Multi-Token Prediction (MTP) layer (Gloeckle et al. 2024), enabling speculative decoding during inference (DeepSeek-AI et al. 2024). The 130B model variant is derived from the 80B base through Depth Up-Scaling (Kim et al. 2023), which cyclically duplicates the middle one-third of layers twice.

Figure 1: WeLM Architecture Overview

Table 1: Model architecture of WeLM. We exclude word embeddings and the output layer in activated and total parameters.

Normalization

The model employs partial PostNorm to mitigate representation collapse in deeper layers (Zhuo et al. 2025). Normalization is applied to both the residual and hidden states before attention, while only the input hidden states are normalized before MoE layers. In PostNorm architectures, normalizing the residual causes each layer’s residual norm to become comparable to the MoE and attention outputs, increasing the likelihood that the RMS of the residual sum reaches extremely small values. This phenomenon triggers abnormal gradient amplification during the backward pass of subsequent RMSNorm operations, leading to model collapse (Zhuo et al. 2025). To mitigate this instability, OutputNorm (Ding et al. 2021) is applied to the attention output with small weight initialization, stabilizing training during early stages.

KV-Mirror

Standard decoder-only LLM architectures (Radford et al. 2019; Brown et al. 2020) require the prefill stage to complete computations across all layers to generate the full KV cache for subsequent decoding. YOCO demonstrates that computational early exit during prefill is achievable through a self-decoder structure where consecutive output layers reuse shallow-layer KV cache, reducing prefill computation by half at arbitrary sequence lengths (Sun et al. 2024). KVSharer further shows that KV cache sharing between attention layers with higher divergence yields improved performance (Wu et al. 2024).

Building on these insights, WeLM implements a U-shaped KV sharing strategy: the first third of layers mirror into the last third (including the MTP layer). Instead of directly reusing the cached K/V tensors, our variant reuses the pre-projection hidden states from the mirrored layer and recomputes K/V with the target layer’s projection. This modification allows the mirror layer to represent distinct semantics while simultaneously preserving the early exit capability of the KV cache method, leading to superior performance in our benchmarks compared to direct K/V reuse.

Concretely, the mirrored layer i loads the saved hidden states from layer j before K/V projection, projects them with W_i, and replaces the K/V portion of its own QKV:

# Imitated layer j
save_hidden(j, h_j)

# Mirror layer i
K_i, V_i = W_i^{k,v} * h_j
# Q_i computed as usual (omitted)

This approach reduces inference costs while suppressing attention-sink phenomena in deeper layers. However, we find incorporating shallow-layer hidden states into Q-projection inputs does not yield observable improvements.

Over-encoding

Over-encoding (Huang et al. 2025) demonstrates that input vocabulary expansion yields consistent performance improvements while constraining total memory access during decoding. WeLM adopts a multi-head hashing mechanism (Svenstrup et al. 2017) to mitigate hash collisions in n-gram settings, enabling effective performance under relatively small vocabulary size constraints. The technique shows enhanced effectiveness on architectures with cross-layer activation sharing, such as KV-Mirror.

Experiments

To evaluate the individual and cumulative impact of these architectural refinements, we conduct a series of ablation studies using a scaled-down MoE configuration (8.7B total parameters, 0.7B activated parameters). The results, summarized in Table 2, demonstrate consistent improvements across both English and Chinese benchmarks as each component is integrated.

Ablation results

28-layer, 8.7B parameters, 0.7B activated, 377B tokens, AdamW, batch size = 1024

Table 2: Ablation results on 8.7B configuration.

To further verify the scalability of these modifications, we scaled the experiment to a 30.7B model with 3B activated parameters. As shown in Table 3, the integration of these architectural improvements leads to significant performance gains across a wide range of benchmarks, including language understanding, reasoning, and coding tasks. This broader evaluation confirms that the benefits observed in smaller-scale experiments translate effectively to larger model variants.

Table 3: Ablation results on 30B configuration.

2. Dataset

The training corpus covers 60+ languages, features extensive quality filtering, and incorporates modern LLMs into the preprocessing pipeline.

LLM Quality Grader We employ an LLM-based quality grader with a suite of engineered prompts to evaluate content quality across diverse sources. By scoring a representative fraction of samples from each data stream, we establish a robust quality metric that guides our filtering thresholds. We find this approach highly efficient, demonstrating a high correlation with more computationally expensive methods, such as specifically fine-tuned classifiers, while allowing for faster iteration on new data sources.

OCR Pipeline We find that incorporating a Vision-Language Model (VLM) for document OCR is an effective approach, as it achieves higher extraction accuracy and enables faster iteration compared to traditional methods. We utilize an in-house version of POINTS-Reader (Liu et al. 2025) to convert raw PDF documents into a clean textual format.

Encyclopedic Data Curation To enhance the model’s factual knowledge and ensure semantic diversity, we employ multiple pipeline variants for processing encyclopedic data. By maintaining a curated mixture of outputs from these distinct pipelines, we maximize the breadth of information while mitigating the systematic biases potentially introduced by any single processing methodology.

3. Training

The pre-training process is governed by several key optimization and scheduling choices:

• Optimization: We employ the Muon optimizer (Jordan et al., n.d.) with a momentum coefficient of 0.95, 5 iteration steps, and a weight decay coefficient of 0.1. We use a cosine learning rate schedule for the first stage, with a maximum learning rate of 5e-4 and an end learning rate of 5e-5.
• Training Schedule: The 80B model is pre-trained on 11T tokens in the primary stage, followed by an annealing phase on 1.4T high-quality tokens. A final stage of context extension on 1.3T tokens is performed to achieve a 128K context length.
• Model Scaling: The 130B variant is initialized from an 80B checkpoint that has completed a 13.7T token primary training stage. Following Depth Up-Scaling, the 130B model follows a similar progression of annealing and context extension as the base 80B model.
• Auxiliary Loss: All models integrate a Multi-Token Prediction (MTP) layer with the loss weight fixed at 0.3.

4. Infrastructure

The WeLM series is trained on an internally developed framework optimized for large-scale sparse models. To maintain high throughput and manage memory constraints on moderate cluster sizes, we implement several specialized communication and memory management strategies:

• DualPipeV and DeepEP: To address communication bottlenecks in expert parallelism, we utilize DualPipeV and DeepEP (DeepSeek-AI et al. 2024). These techniques overlap computation with communication, effectively eliminating bubbles introduced by expert parallelism communication.

• Kernel Fusion: With the increasing computational power of GPUs, CPU overhead becomes a significant bottleneck. We implement a Grouped GEMM CUDA kernel to consolidate matrix multiplications across all experts, mitigating kernel launch overhead. Furthermore, we fuse lightweight operators, such as Norm and RoPE, to reduce memory access latency and CPU-side dispatch costs.

• Activation Memory Optimization: Increasing the Micro Batch Size (MBS) to improve GPU utilization is often limited by activation memory overhead. We address this through:

  • Selective Quantization and Recomputation: We maintain BF16 precision as the primary format while selectively applying FP8 quantization or recomputation. Activation checkpoints are strategically placed at layers like QKV projections. To mitigate the memory overhead from top-k hidden state copies in MoE blocks, we implement recomputation for the permutation stage, triggered only during the backward-for-weights phase. Additionally, we apply tile-wise quantization to SwiGLU activations.
  • Fused SwiGLU and MulProbs: We optimize the unpermuting calculation order by moving it after the MoE SwiGLU calculation and fusing them into a single kernel. This allows the tensor to be released and recomputed during the backward pass, significantly reducing activation memory overhead.

5. Evaluation

Public Benchmarks

We evaluate the two models on a comprehensive suite of benchmarks to assess performance across multiple dimensions, including reasoning, multilingual proficiency, and long-context handling. The public benchmarks used for this evaluation are summarized below:

• English Tasks:

  • General Understanding: MMLU (Hendrycks, Burns, Basart, et al. 2021) (5-shot), MMLU-Redux (Gema et al. 2025) (5-shot), MMLU-Redux 2.0 (Gema et al. 2025) (5-shot), and MMLU-Pro (Wang et al. 2024) (5-shot, CoT).
  • Reasoning and Logic: GPQA (Rein et al. 2024) (5-shot, CoT), SuperGPQA (Du et al. 2025) (5-shot, CoT), BBH (Suzgun et al. 2023) (3-shot, CoT), and BBH-Fix (3-shot, CoT), which utilizes corrected few-shot prompts.
  • Mathematics: MATH (Hendrycks, Burns, Kadavath, et al. 2021) (4-shot, CoT), MATH-New-CoT (4-shot), which employs GPT-4-generated Chain-of-Thought (CoT) exemplars to align with contemporary evaluation paradigms, GSM8K (Cobbe et al. 2021) (4-shot, CoT), and GSM8K-New-CoT (4-shot).
  • Knowledge and Hallucination: KOR-Bench (Ma et al. 2024) (3-shot), SimpleQA (Wei et al. 2024) (5-shot, LLM judge), and AA-Omniscience (Jackson et al. 2025) (10-shot, LLM judge).

• Chinese Tasks: C-Eval (Huang et al. 2023) (5-shot), C-MMLU (Li et al. 2024) (5-shot), and C-SimpleQA (He et al. 2025) (5-shot).

• Multilingual Tasks: MMMLU (OpenAI 2024) (5-shot), INCLUDE (Romanou et al. 2024) (5-shot), and Global-MMLU (Singh et al. 2025) (5-shot).

• Coding Tasks: EvalPlus (Liu et al. 2023) (1-shot), MultiPL-E (Cassano et al. 2023) (1-shot), and CRUX-Input/Output (Gu et al. 2024) (1-shot).

• Long-context Tasks: RULER (Hsieh et al. 2024) (0-shot; 1-shot for the VT subtask), LongBench V2 (Bai et al. 2025) (0-shot), MRCR V2 (Vodrahalli et al. 2024) (0-shot), and MTOB (Tanzer et al. 2023) (3-shot).

All open-source models in the table are self-hosted and evaluated with the same pipeline to ensure a controlled comparison. The two Qwen models (Qwen3-32B and Qwen3-Next 80B-A3B) are exceptions: their benchmark numbers are taken directly from the Qwen3 Technical Report and Qwen’s release blog because the weights or APIs are not publicly accessible, so we do not re-evaluate them. (Yang et al. 2025; Qwen Team, n.d.)

The WeLM variants exhibit competitive performance across the evaluated benchmarks. Several key observations from these results are detailed below:

• Effectiveness of Depth Up-Scaling: Although the 130B model was initialized from a late-stage 80B checkpoint and trained with a reduced learning rate schedule, it achieves significant performance gains across most benchmarks. These improvements are particularly pronounced in tasks involving complex reasoning and broad factual knowledge.

• Impact of LLM-Generated CoT Exemplars: Traditional math benchmarks often utilize human-labeled short answers as few-shot exemplars, which may not fully elicit the latent reasoning capabilities of modern pre-trained models. By introducing GSM8K-New-CoT and MATH-New-CoT—variants that utilize LLM-generated Chain-of-Thought (CoT) solutions—we observe substantial performance improvements across all evaluated models, with gains of 3–8 absolute points on GSM8K and 2–5 points on MATH. These results suggest that evaluation methodologies must evolve alongside model capabilities to accurately capture advancements in reasoning proficiency.

Internal Benchmarks

To further analyze model performance on contemporary data and more challenging problem sets, we developed several internal benchmarks. These datasets are designed to reduce the risk of data contamination and provide a more granular assessment of model capabilities:

• GPQA-Bench (5-shot): A graduate-level reasoning benchmark consisting of 1.1K original problems across multiple disciplines. Each item is structured as a five-choice multiple-choice question.
• Web-SimpleQA & Web-CSimpleQA (15-shot, LLM judge): Utilizing an internal deep-research pipeline, we constructed these benchmarks to evaluate world knowledge mastery in English and Chinese, respectively. These datasets feature significantly broader domain coverage compared to the original SimpleQA.
• Web-MMLU (5-shot): A multiple-choice benchmark similar to MMLU but featuring entirely distinct topics and questions generated through our internal research pipeline.
• WMT-Bench (5-shot): A machine translation evaluation set derived from the WMT24-pp corpus. We utilize the XCOMET-XXL metric for automated translation scoring.

Figure 2: performance on the WMT-Bench

Entropy Decrease of the Depth Up-Scaling Model

The depth up-scaled model often exhibits different output patterns from the base checkpoint in practice. To quantify this effect, we compare token-level output entropy across domains for the 80B-A3B and 130B-A4.9B variants, as summarized below.

The 130B-A4.9B model shows lower mean entropy in English, code, and math, while Chinese is slightly higher. This suggests reduced output diversity in most domains, with only modest changes in variance. We leave a deeper analysis of this behavior to future work.

6. Conclusion and Future Outlook

In this post, we have detailed the architectural refinements and data curation strategies that enable the development of high-performance LLMs within moderate computational constraints. Our findings demonstrate that the combination of targeted structural optimizations—such as KV-Mirror and stabilized normalization—and rigorous, LLM-guided data quality control can significantly narrow the performance gap between moderately sized models and their larger, more computationally expensive counterparts. We believe these methodologies offer a practical roadmap for developing efficient, capable models without requiring extreme-scale resources. We will share more insights in future posts. Stay tuned.

Appendix

Empirically, we observe that high-quality pre-trained models often demonstrate superior performance on open-ended and out-of-domain writing tasks, reflecting robust generalization. We made an internal Write-Bench and present sample outputs from the WeLM-80B model below.

Output Cases

以华山为主题，写一篇古文。输出格式：
<文章开始>
...
<文章结束>

<文章开始>

太华之巔，星斗垂光，孤峰接汉。其势若巨鳌戴云，鲸吸玄溟。古柏虬结于危崖，悬泉迸珠于绝壑。晨观五色霞霓，暮聆万壑松涛。登顶者，仰见帝座，俯吞八荒，顿觉天壤为之一空。  

忆昔女娲断鳌立极，始有坤轴之隆；轩辕铸鼎觞天，常驾苍龙之驾。秦皇跃马巡疆，勒石以铭功德；汉武筑台封禅，燃薪而见灵光。迨至唐李翰林，以剑决浮云，诗锋裂苍冥；宋范希文，忧乐关天下，墨痕留玉泉。  

然则仙踪杳渺，空余采药人迹；劫火澹然，徒存伏羲画卦。独千仞青翠，送风送月；一泓凝碧，鉴古鉴今。是知山不在高，有仙则名；峰不在险，有志则尊。  

嗟乎！游子临渊，欲掬星而不可得；志士怀远，望云涛而浩然生。纵使苍颜白发，犹乘鹤陟层巅；即令海竭江枯，永镇神州志气。

<文章结束>

Write a poem about vibe coding.
Output format:
<text>
...
</text>

<text>

VIBE CODING  

The cursor blinks—a silent drum,  
Averse to rules I never come  
To learn, but follow instinct's hum—  
Vibe coding, a frantic *ehem*  

No tutorials to trace or grind,  
No flags to catch the faulty kind.  
Just run, observe, and stumble blind,  
Where compilers used to shake in mind.  

ChatGPT speaks in seraph tones,  
While Gemini with midnight moans  
Suggests fixes that slice and groans  
Through logic’s fragile, trembling bones.  

I *prompt* a thing that *looks* like code—  
A fragile, formless, flowery load.  
It *runs*! With errors splayed abroad,  
Like ships that have lost every mode.  

The AI hushes like a monk,  
Then spills a daemon in the tank.  
I jiggle brackets, curse, then—clank—  
A flicker, then the whole thing *whack*!  

No need to read what’s on the screen;  
Just *feel* the flow—a fever, clean.  
The stack overflows in neon green,  
Like digital ice on summer’s keen.  

We don’t debug; we *are* the bug.  
We dance through errors in a hug  
Of chaos—fed by caffeine’s tug—  
And pray the latency won’t shrug  

Us off into the void of servers,  
Where ghosts of failed APIs sever  
The tether. Oh, the sweet convergences  
When source code sings through strange urgences!  

We trust the magic, lean it hard—  
For logic’s kiss is not its card.  
The code? It works. It has the *lard*  
Of wonder, chaos, and some shard  
Of grace. The machine learns what we dare—  
And spits back verse in electric air.  

—after midnight, the monitor’s sigh,  
A code poem that will never die.
</text>

References