🔔News
🔥 [2025-04-21]: We opensource the train data. Welcome registration.
Introduction
The MCABSA Challenge on the ACM MM platform focuses on multimodal conversational sentiment analysis and introduces two novel subtasks: (1) Panoptic Sentiment Sextuple Extraction, which requires identifying holder, target, aspect, opinion, sentiment, and rationale from multi-turn, multi-party multimodal dialogues; and (2) Sentiment Flipping Analysis, which aims to detect sentiment shifts throughout conversations along with their underlying causes.
Challenge Task Definition and Metrics
Subtask-I: Panoptic Sentiment Sextuple Extraction. Given a dialogue D = {u1, ..., un} with the replying structure {(ui, uj), ...} (i.e., ui replies to uj), the task is to extract all sextuples (h, t, a, o, s, r). Each utterance ui = {w1, ..., wmi} contains mi words in the text (denoted as It), occasionally with associated non-text information pieces, i.e., image (Ii), audio (Ia), or video (Iv). The elements h (holder), t (target), a (aspect), o (opinion), and r (rationale) can be either continuous text spans explicitly mentioned in utterances, or implicitly inferred from context or non-text modalities. s represents the sentiment category (positive, negative, or neutral).
Subtask-II: Sentiment Flipping Analysis. Given input D (same as in Subtask-I), this task aims to detect all sextuples (h, t, a, ζ, φ, τ). Here, h, t, and a denote the holder, target, and aspect, consistent with their definitions in Subtask-I. ζ and φ represent the initial and flipped sentiments, respectively, highlighting the dynamic change in sentiment by the same speaker toward the same aspect of the same target. τ refers to a trigger that induces the sentiment transition, which is chosen from four predefined categories: 1) introduction of new information, 2) logical argumentation, 3) participant feedback and interaction, and 4) personal experience and self-reflection. Since Subtask-II shares multiple elements with Subtask-I, flipping detection can naturally build upon Subtask-I results to minimize redundancy.
Evaluation
Subtask-I: Panoptic Sentiment Sextuple Extraction. We evaluate the overall performance of sextuple extraction using two metrics: micro F1, which measures the accuracy of the entire sextuple, and identification F1, which assesses the extraction quality excluding sentiment polarity. The final evaluation score is computed as the average of these two metrics.
Subtask-II: Sentiment Flipping Analysis. This subtask evaluates whether both the initial sentiment and flipped sentiment (Flip), along with the flipping trigger's category (Trig), are correctly predicted simultaneously. We use exact match F1 as the evaluation metric.
We provide Python scripts for evaluation. Please refer to the baseline codes.
Dataset
The dataset for this challenge is based on PanoSent, a high-quality multimodal benchmark featuring multi-turn, multi-party dialogues. Each sample includes speaker identities, utterances, modality metadata, annotated sentiment elements (holder, target, aspect, opinion, sentiment, rationale), and sentiment flipping events for dynamic sentiment understanding.
Statistics of PanoSent are shown in the following figure.
The dataset multimedia resources can be downloaded from Hugging Face.
Example of a sample:
{
"doc_id": "00002",
"speakers": [
{
"id": 0,
"name": "Ella"
},
{
"id": 1,
"name": "Max"
},
{
"id": 2,
"name": "Sophie"
},
{
"id": 3,
"name": "Liam"
}
],
"dialogue": [
{
"index": 0,
"speaker": 0,
"utterance": "I believe the graphics on my new gaming PC are stunning; they're incredibly immersive.",
"annotation": "I believe the graphics on my new gaming PC are stunning ; they're incredibly immersive .",
"modality": {
"type": "img",
"caption": "High-resolution graphics from a gaming PC displaying an expansive fantasy landscape.",
"id": "img_0001"
},
"reply": -1
},
{
"index": 1,
"speaker": 1,
"utterance": "Absolutely, but I think the audio quality could be more refined to match that level of graphics.",
"annotation": "Absolutely, but I think the audio quality could be more refined to match that level of graphics .",
"modality": "None",
"reply": 0
},
{
"index": 2,
"speaker": 2,
"utterance": "The cooling system in PCs is crucial, I once faced overheating issues which hindered performance significantly.",
"annotation": "The cooling system in PCs is crucial , I once faced overheating issues which hindered performance significantly .",
"modality": {
"type": "vid",
"caption": "Video showing a PC with high-performance cooling components in action.",
"id": "vid_0002"
},
"reply": 1
},
{
"index": 3,
"speaker": 3,
"utterance": "True, but I feel like the build quality of the gaming PC is somewhat cheap; it feels less durable.",
"annotation": "True, but I feel like the build quality of the gaming PC is somewhat cheap ; it feels less durable .",
"modality": "None",
"reply": 2
},
{
"index": 4,
"speaker": 0,
"utterance": "I initially agreed with you about the build quality, but after a month of using it, I appreciate its lightweight design for portability.",
"annotation": "I initially agreed with you about the build quality , but after a month of using it, I appreciate its lightweight design for portability .",
"modality": "None",
"reply": 3
}
],
"hexatuple": [
{
"holder": {
"identity": "holder_01",
"value": "Ella",
"manner": "explicit"
},
"target": {
"identity": "target_01",
"value": "new gaming PC",
"manner": "explicit"
},
"aspect": {
"identity": "aspect_01",
"value": "graphics",
"manner": "explicit"
},
"opinion": {
"identity": "opinion_01",
"value": "stunning",
"manner": "explicit"
},
"sentiment": "positive",
"rationale": {
"identity": "rationale_01",
"value": "they're incredibly immersive",
"manner": "explicit"
}
},
{
"holder": {
"identity": "holder_02",
"value": "Max",
"manner": "explicit"
},
"target": {
"identity": "target_01",
"value": "new gaming PC",
"manner": "explicit"
},
"aspect": {
"identity": "aspect_02",
"value": "audio quality",
"manner": "explicit"
},
"opinion": {
"identity": "opinion_02",
"value": "could be more refined",
"manner": "explicit"
},
"sentiment": "neutral",
"rationale": {
"identity": "rationale_02",
"value": "to match that level of graphics",
"manner": "explicit"
}
},
{
"holder": {
"identity": "holder_03",
"value": "Sophie",
"manner": "explicit"
},
"target": {
"identity": "target_01",
"value": "PCs",
"manner": "explicit"
},
"aspect": {
"identity": "aspect_03",
"value": "cooling system",
"manner": "explicit"
},
"opinion": {
"identity": "opinion_03",
"value": "crucial",
"manner": "explicit"
},
"sentiment": "positive",
"rationale": {
"identity": "rationale_03",
"value": "I once faced overheating issues which hindered performance significantly",
"manner": "explicit"
}
},
{
"holder": {
"identity": "holder_04",
"value": "Liam",
"manner": "explicit"
},
"target": {
"identity": "target_01",
"value": "gaming PC",
"manner": "explicit"
},
"aspect": {
"identity": "aspect_04",
"value": "build quality",
"manner": "explicit"
},
"opinion": {
"identity": "opinion_04",
"value": "somewhat cheap",
"manner": "explicit"
},
"sentiment": "negative",
"rationale": {
"identity": "rationale_04",
"value": "it feels less durable",
"manner": "explicit"
}
},
{
"holder": {
"identity": "holder_01",
"value": "Ella",
"manner": "explicit"
},
"target": {
"identity": "target_01",
"value": "gaming PC",
"manner": "explicit"
},
"aspect": {
"identity": "aspect_04",
"value": "build quality",
"manner": "explicit"
},
"opinion": {
"identity": "opinion_05",
"value": "appreciate its lightweight design",
"manner": "explicit"
},
"sentiment": "positive",
"rationale": {
"identity": "rationale_05",
"value": "for portability",
"manner": "explicit"
}
}
],
"sentiment flip": [
{
"holder": "Ella",
"target": "gaming PC",
"aspect": "build quality",
"initial sentiment": "negative",
"flipped sentiment": "positive",
"trigger type": "Personal Experiences and Self-reflection"
}
]
}
Timeline
Please note: The submission deadline is at 11:59 p.m. (Anywhere on Earth) of the stated deadline date.
| Training set and participant instruction release | 21 April, 2025 |
| Registration Deadline | 20 May, 2025 |
| Test set release & Evaluation begin | 31 May, 2025 |
| Evaluation end | 15 June, 2025 |
| Paper invitation decision | 20 June, 2025 |
| Paper submission deadline | 15 July, 2025 |
| Final decision | 30 July, 2025 |
| Camera ready deadline | 5 August , 2025 |
Submission
Please submit predicted results with a JSON file named results.json.
Coming Soon
Baseline
Link to the code: PanoSent-baseline
Registration
Welcome and please apply for the MCABSA challenge via this form: Google Form.
Feel free to contact us at mcabsachallenge@gmail.com.
Rewards
Top-ranked participants in this competition will receive a certificate of achievement and will be recommended to write a technical paper for submission to ACM MM 2025.
Organizers
Meng Luo. National University of Singapore, Singapore.
Hao Fei. National University of Singapore, Singapore.
Bobo Li. Wuhan University, China.
Shengqiong Wu. National University of Singapore, Singapore.
Qian Liu. The University of Auckland, New Zealand.
Soujanya Poria. Singapore University of Technology and Design, Singapore.
Erik Cambria. Nanyang Technological University, Singapore.
Mong-Li Lee. National University of Singapore, Singapore.
Wynne Hsu. National University of Singapore, Singapore.
WeChat Group
References
[1] Luo, Meng, et al. "Panosent: A panoptic sextuple extraction benchmark for multimodal conversational aspect-based sentiment analysis." Proceedings of the 32nd ACM International Conference on Multimedia. 2024.
[2] Luo, Meng, et al. "Nus-emo at semeval-2024 task 3: Instruction-tuning llm for multimodal emotion-cause analysis in conversations." arXiv preprint arXiv:2501.17261 (2024).