PanGu-Σ goals mainly is to serve as proof of concept that training trillion parameters is possible. Data training quality and model quality is secondary.
Artificial Intelligence thread
- Thread starter 9dashline
- Start date
It is proof of concept on heterogenous training of large models - they offload optimizer states to CPU to reduce memory footprint (RAM is used instead of GPU memory) and use the sparse architecture of Pangu to do selective updates because device-host communication is slow. That way they trained the model on a smaller cluster of 512 Ascend 910 accelerators while "normal" training process would require at least x2 of those. Of course such training is slower because of said communication inefficiency and smaller pool of computational power but overall it is a viable approach towards training large models when you are restricted on resources.
There were trillion parameter models before Pangu-Sigma came out.
interesting, i think moffett AI also used a sparse architecture for their model and were able to get pretty good results.
anyways, more on the developer toolkit
This to me just seems like a great idea. You have a 8 TOPS INT8 hardware that have your usual ports, supports linux OS and MindStuio developer IDE. And you can use it to validate your own AI program for real world applications. 1080P 30FPS is a little old for gaming GPU, but it should be sufficient for your regular desktop application
also this Omni-Perception Pre-Trainer by CAS is quite interesting. One of the 30+ lage models with hundreds of billions parameter
More on AI and Huawei. So one thing always said about Nvidia advantage is just how many developers use Cuda and how that is hard to beat.
So I did a look yesterday. As of August 2020, Nvidia announced 2 million registered developers with Cuda
and said they were adding 39k a month.
Now, let's say they at least doubled again in the past 32 months, they will be somewhere between 4 and 5 million registered developers with a good number of them in China.
Now, if we compare that to Huawei & Ascend, we know from earlier that Kunpeng & Ascend have over 3.5 million developers in total
big number, but what is the breakdown, well it says in there that over 2 million developers & 4500 JV partners are in Kunpeng.
So based on the total of 3.5 million developers bw two and 5600 JV partners, then there are at least 1.5 million Ascend developers & 1100 JV partners.
Now, that's what it says here
So I think they need to find ways to get more as many Chinese AI developers off Cuda and onto Ascend as possible. And we need to get the BAT franchises, China mobile & China telecom to also use more Ascend GPUs
Now, the other interesting part is MindSpore
So I did a look yesterday. As of August 2020, Nvidia announced 2 million registered developers with Cuda
and said they were adding 39k a month.
Now, let's say they at least doubled again in the past 32 months, they will be somewhere between 4 and 5 million registered developers with a good number of them in China.
Now, if we compare that to Huawei & Ascend, we know from earlier that Kunpeng & Ascend have over 3.5 million developers in total
big number, but what is the breakdown, well it says in there that over 2 million developers & 4500 JV partners are in Kunpeng.
So based on the total of 3.5 million developers bw two and 5600 JV partners, then there are at least 1.5 million Ascend developers & 1100 JV partners.
Now, that's what it says here
I wonder if the actual number is more than this, because the 2 should have some overlap. But even these numbers are pretty respectable compared to Nvidia/Cuda developer size. I think it just shows that outside of America and China, there aren't that many AI developers. If Ascend can even capture all the AI developers in China, it would likely be close in size to Cuda. Even now, it's probably anywhere from 1/3 to 1/2 the number of non-Chinese Cuda developers.
So I think they need to find ways to get more as many Chinese AI developers off Cuda and onto Ascend as possible. And we need to get the BAT franchises, China mobile & China telecom to also use more Ascend GPUs
Now, the other interesting part is MindSpore
this is just for higher level AI developers. According to this 4.51 million downloads and over 13,000 developers have contributed code to MindSpore. Not such how many people using MindSpore, but probably good number. 55000 companies have built applications. But it's good to see that the AI stack Huawei built is not just a Huawei thing. It involves a whole bunch of enterprises in China
They claim MindSpore 2.0 is world's first fusion AI framework. Combining AI and HPC
importance of AI to a country's soft/hard power can be seen here
Using it to attract rest of global south
Using it to attract rest of global south
Reposting some related material I put on semiconductor thread
Huawei talking about Pengcheng 3 project, a 16 EFLOPS FP16 datacenter aimed to be the leading AI data decenter in the world using more next gen Huawei Ascend and Kunpeng chips, to start construction next to Shenzhen and start running by end of the year. Will be fully complete by 2025.
More on AI and the great expansion of domestic AI
Research on Pangu AI platform of Huawei
Performance of Ascend 910 vs A100 and H100. It's very comparable to A100 (at least the current version of 910)
Huawei talking about Pengcheng 3 project, a 16 EFLOPS FP16 datacenter aimed to be the leading AI data decenter in the world using more next gen Huawei Ascend and Kunpeng chips, to start construction next to Shenzhen and start running by end of the year. Will be fully complete by 2025.
More on AI and the great expansion of domestic AI
大数据时代，算力与数据增长齐头并进。根据《中国算力白皮书（2022 年）》的 数据，2021 年我国算力总规模达到 140Eflops（每秒一万四千亿亿次浮点运算，包含通用算力、智能算力、超算算力，边缘算力暂未纳入统计范围），全球占比约 为 27%，近五年年均增速超 30%。展望未来，工信部印发的《“十四五”信息通 信行业发展规划》指出 2025 年我国数据中心算力总规模将增长到 300EFLOPS，CAGR 达 22%；另一方面，伴随 5G、人工智能、物联网等技术的应用普及，数据流量增长速率也在不断加快。根据 IDC 的预测，全球数据总量在 2020 年 将达 50ZB，而这一数据到 2025 年有望达到 175ZB，CAGR 达 28%。与此同 时，根据中央网信办的数据统计，2019 年度中国移动互联网数据接入量为 1,655.50 亿 GB，预计 2024 年将达到 5,680.90GB，CAGR 也高达 28%。因 此，我们认为在当前大时代下，适度超前建设以数据中心为首的新型基 建具有明确的战略意义。
“东数西算”正式启动，预计每年带动 4000 亿投资。今年二月，国家发展改革 委、中央网信办、工业和信息化部、国家能源局联合印发通知，同意在京津冀、、粤港澳大湾区、成渝、内蒙古、贵州、甘肃、宁夏等 8 地启动建设国家 算力枢纽节点，并规划了 10 个国家数据中心集群，标志着“东数西算”工程正 式全面启动。与“西气东输”“西电东送”“南水北调”等工程相似，“东数西 算”是一个国家级算力资源跨域调配战略工程，针对我国东西部算力资源分布总 体呈现出“东部不足、西部过剩”的不平衡局面，引导中西部利用能源优势建设 算力基础设施，“数据向西，算力向东”，服务东部沿海等算力紧缺区域，解决 我国东西部算力资源供需不均衡的现状。“东数西算”工程对我国的长远发展有 重要的意义：
有利于提升国家整体算力水平：通过全国一体化的数据中心布局建设，扩大 算力设施规模，提高算力使用效率，实现全国算力规模化集约化发展。
有利于促进绿色节能：数据中心属于高耗能行业，又被称为“不冒烟的钢厂”。将数据中心从能耗指标紧张的东部地区迁往清洁能源富集、电力更加廉价的 西部，将大幅提升使用比例，降低数据中心运营的成本。
有利于扩大有效投资：数据中心产业链条长、投资规模大，带动效应强。发 改委预计“十四五”期间大数据中心投资还将以每年超过 20%的速度增长，每年预计各方面投资额达 4,000 亿元。
有利于推动区域协调发展：通过算力设施由东向西布局，将带动相关产业有 效转移，促进东西部数据流通、价值传递，延展东部发展空间，推进西部大 开发形成新格局
Research on Pangu AI platform of Huawei
Performance of Ascend 910 vs A100 and H100. It's very comparable to A100 (at least the current version of 910)
AssassinsMace
Lieutenant General
Look at the West's criticism of TikTok pointing to the Chinese original version, Douyin. Douyin, they charge, is educational while TikTok is a reflection of Western culture. They charge it's a conspiracy by China to destroy Western society. No, TikTok is a refection of Western values just like US artificial intelligence is a refection of American alt-right incel male rage that wants to see the world burn. It's a Rorschach test of summing up how they see the world.
The West is upset that China wants its own AI when they expect China to embrace theirs. China ain't forcing its AI onto them yet they're upset that an alternate AI existing when it should be only theirs like God. There can be only one true AI. And like intelligence they hate technology. Technology exists to make life easier which also means independence from them. And of course the hypocrisy of is all it they can have AI and technology and independence but you can't because they're the only ones that are responsible enough to have them...
Edit: Removing offensive anti-Christian comment.
The West is upset that China wants its own AI when they expect China to embrace theirs. China ain't forcing its AI onto them yet they're upset that an alternate AI existing when it should be only theirs like God. There can be only one true AI. And like intelligence they hate technology. Technology exists to make life easier which also means independence from them. And of course the hypocrisy of is all it they can have AI and technology and independence but you can't because they're the only ones that are responsible enough to have them...
Edit: Removing offensive anti-Christian comment.
Last edited by a moderator:
Aside from this example of 东数西算, there are many other projects like under way to push China's total computation to 300 EFLOPS by 2024.
8 regions build computation forming 10 national data center clusters. A national strategic priority with 400B RMB investment a year
You need 3 things in AI: data, computation power & algo/model
Alibaba has a 12 EFLOPS data center building in 张北 + another 3 EFLOPS data center in 乌兰察布
Baidu has a 4 EFLOPS computation center that came up in YangQuan 阳泉 that's been used for ErnieBot
Tencent uses a distributed computational platform to provide 16 EFLOPS of computation in total