What is most impressive to me is that with a 70B model trained on 18T tokens they reached the performance of the 405B Llama model trained on 15T tokens.
For (dense) Transformer models computation per token is proportional to the number of parameters, this means that Qwen 70B reached the same performance of Llama 405B with
a 5 times smaller computing budget!
This is even more impressive considering that Llama 3 is only few months old and that the Meta team behind Llama is world class, they are state-of-the-art.
Since about 2 years there is a new trend toward better training techniques applied to smaller models: this indirectly greatly helps China to workaround GPU limitations for the time being.
The model range 7B-70B is here to stay, these models will be the workhorses of future applications, even more when the new breed of
reasoning models (that is actually 90% new training techniques) will spread, so that even a small 7B model will gain "reasoning" capabilities.
BTW in the last 18 months the papers on eliciting reasoning in LLM sprung up like mushrooms. Many
on internet say that OpenAI o1 is based
...the other big players are months away from OpenAI, not years away.
