The 2024 ACM Multimedia (ACMMM) conference held in Melbourne, Australia, from October 28 to November 1, witnessed the outstanding achievement of Mininglamp Technology’s Multimodal team and their collaborators from Peking University. Their research paper, titled “Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding,” garnered a prestigious Best Paper nomination. This accomplishment stands testament to their innovative approach and significant contribution to the field of multi-modal AI.

Mininglamp Technology’s team, led by founder, chairman, and CEO Wu Minghui, along with Zhao Chenxu, head of the Multimodal Large Model department, and Su Anyang, head of the Mingjing Algorithm department, were invited to attend the conference in Melbourne.

The ACM Multimedia conference is a premier venue for researchers and practitioners in the field of multimedia and artificial intelligence. This year’s event saw a total of 4,385 submissions, with 1,149 papers accepted for presentation. Among those, 174 were selected for oral presentations, with only 26 receiving Best Paper nominations.

What is the ACMMM Conference?

The ACMMM Conference is a top international academic conference in the field of multimedia, sponsored by the Association for Computing Machinery (ACM). It is also a Class A international academic conference recommended by the China Computer Federation (CCF-A). This year marks the 32nd conference since its inception in 1993.

The conference covers various aspects of multimedia computing, such as multimedia content analysis, multimedia retrieval, multimedia security, human-computer interaction, and computer vision.

Mininglamp Technology’s Multimodal Team Achieves “Best Paper Nomination” at the ACMMM Global Conference

Addressing the limitations of current AI in video content understanding, which mainly focuses on objective aspects and lacks subjective measurement methods, as well as the development of effective methods for simulating human subjective responses, Mininglamp Technology’s latest research integrates non-standard modalities such as EEG and eye movement data to build a novel multimodal language model paradigm. This represents a significant step forward in the research direction of machine understanding and simulation of human subjective responses.

Title: Hypergraph Multi-modal Large Language Model: Exploiting EEG and Eye-tracking Modalities to Evaluate Heterogeneous Responses for Video Understanding

Authors: Minghui Wu, Chenxu Zhao, Anyang Su, Donglin Di, Tianyu Fu, Da An, Min He, Ya Gao, Meng Ma, Kun Yan, Ping Wang

Abstract: Understanding of video creativity and content often varies among individuals, with differences in focal points and cognitive levels across different ages, experiences, and genders. There is currently a lack of research in this area, and most existing benchmarks suffer from several drawbacks: 1) a limited number of modalities and answers with restrictive length; 2) the content and scenarios within the videos are excessively monotonous, transmitting allegories and emotions that are overly simplistic. To bridge the gap to real-world applications, we introduce a large-scale Video Subjective Multi-modal Evaluation dataset, namely Video-SME. Specifically, we collected real changes in Electroencephalographic (EEG) and eye-tracking regions from different demographics while they viewed identical video content. Utilizing this multi-modal dataset, we developed tasks and protocols to analyze and evaluate the extent of cognitive understanding of video content among different users. Along with the dataset, we designed a Hypergraph Multi-modal Large Language Model (HMLLM) to explore the associations among different demographics, video elements, EEG and eye-tracking indicators. HMLLM could bridge semantic gaps across rich modalities and integrate information beyond different modalities to perform logical reasoning. Extensive experimental evaluations on Video-SME and other additional video-based generative performance benchmarks demonstrate the effectiveness of our method.

What are human subjective feelings? What is the significance of measuring subjective responses of different groups to watching advertising videos?

When people watch advertising videos, their cognitive level of understanding of the material elements, emotional highs and lows, and eye gaze intensity are all subjective feelings. These will vary based on different genders, ages, occupations, and identities.

If machines can simulate the different subjective feelings of different groups of people watching advertising videos, then it is equivalent to being able to effectively measure the content, creativity, etc. of advertising videos, guiding the process of creating advertisement films and saving advertising costs.

The following video demonstrates the analysis of a classic advertisement film using the methods (HMLLM) in the paper, from both subjective and objective dimensions:

The following video demonstrates the unterschied subjective responses of a general audience and a specific audience to the same advertising video using the method (HMLLM) in the paper:

Enabling machines to learn, understand, and simulate human subjective feelings could be the beginning of giving machines subjective consciousness. The new baseline Video-SME proposed by Mininglamp Technology is expected to become a new starting point in the field, marking a shift in machines’ understanding of videos from objective to subjective dimensions.

As a brand-new paradigm, the development of Mininglamp Technology’s multimodal large model HMLLM is committed to providing researchers in the field with valuable experience and inspiration to solve non-standard modality issues, thus promoting the field of large models toward a bright future of human-machine collaboration.

This research project is supported by the Ministry of Science and Technology of China’s “New Generation Artificial Intelligence (2030)” major project.

More Resources

• Paper Link: https://dl.acm.org/doi/10.1145/3664647.3680810
• GitHub Address: GitHub – mininglamp-MLLM/HMLLM: [ACM MM2024] The code for HMLLM.