Peking University, has made significant progress in multimodal sensing fusion research.
Peking University has achieved a major breakthrough in multimodal sensing fusion by addressing the limitations of current AI systems that rely on energy-intensive "sensing-first" algorithms. Inspired by the human brain's ability to integrate vision and hearing with low power consumption through neural principles like superadditivity, Professor He Ming's team at the School of Integrated Circuits developed a novel approach driven by physical computation rather than traditional software processing. This research challenges existing paradigms that lack adaptive, nonlinear coupling characteristics, aiming instead to replicate biological efficiency directly within hardware architecture to overcome latency and power constraints in edge intelligence applications.
The core innovation involves the creation of in-situ acousto-optic fusion schemes using strain-engineered ferroelectric semiconductor transistors (FeS-FETs). By manipulating lattice symmetry in two-dimensional Bi₂O₂Se material, the team induced intrinsic ferroelectric properties while maintaining semiconducting behavior, allowing sound and light signals to be coupled at a physical device level. Experimental results demonstrate that this single-device architecture achieves an impressive 2800% fusion enhancement factor with extremely low power consumption of just 15 pJ per operation. Furthermore, when scaled into arrays with pulse generation circuits, the system exhibits biomimetic synaptic plasticity and high frequency selectivity, effectively converting analog multi-physics signals into time-coded sequences that mimic neuronal behavior.
To realize end-to-end multimodal perception, the researchers integrated the FeS-FET array with a TaOx-based resistive random-access memory (RRAM) chip to form a hierarchical neuromorphic system capable of autonomous recognition. This architecture leverages the parallel computing advantages of RRAM for accurate classification of complex targets, such as fuzzy vehicles, achieving up to 98.2% accuracy even under extreme Gaussian noise interference. The findings, published in Nature Communications, highlight the potential of combining ferroelectric-semiconductor physics with neuromorphic engineering to create robust, energy-efficient perception systems that surpass traditional algorithmic methods while maintaining high resilience in noisy environments.
