Overlooking his laptop. Maybe that's how he knew.
Artificial Intelligence thread
- Thread starter 9dashline
- Start date
Beijing with new public data center for smart computation. Another 10 EFLOPS computation. Had 22 EFLOPS as of January, so this brings it to 32 EFLOPS
first Chinese AI IDE Trade is launched by ByteDance inside China. It had been already available globally.
The size of data centers for some of these Chinese automakers seem pretty unbelievable.
Geely has a 23.5EFLOPS data center for various tech initiatives.
Geely has a 23.5EFLOPS data center for various tech initiatives.
Microsoft making DeepSeek R1 7B and 14B distilled models for CoPilot and PCs available in its Azure AI foundry.
Chain-of-Experts: Unlocking the Communication Power of MoEs
Introduction
We propose Chain-of-Experts (CoE), which fundamentally changes sparse Large Language Model (LLM) processing by implementing sequential communication between intra-layer experts within Mixture-of-Experts (MoE) models.
Mixture-of-Experts (MoE) models process information independently in parallel between experts and have high memory requirements. CoE introduces an iterative mechanism enabling experts to "communicate" by processing tokens on top of outputs from other experts.
Experiments show that CoE significantly outperforms previous MoE models in multiple aspects:
- Performance: CoE with 2x iterations reduces Math validation loss from 1.20 to 1.12
- Scaling: 2x iterations matches performance of 3x expert selections, outperforming layer scaling
- Efficiency: 17.6-42% lower memory usage with equivalent performance
- Flexibility:823x increase in expert combinations, improving utilization, communication, and specialization
English Blog:
Chinese Blog:
This was made by former DeepSeek intern (now PhD student at Northwestern) Wang Zihan.
Huawei's anticipation for token generation out of China