Important

The overview paper of the first CASTLE Grand Challenge at ACM Multimedia 2025 has been published in the proceedings of ACM Multimedia 2025: Overview of the First CASTLE Grand Challenge at ACM Multimedia 2025.

Congratulations to the multiXview team multiXview team for winning the interactive track!

1st CASTLE Challenge at ACM Multimedia 2025

The first CASTLE multimodal analytics challenge will be held at ACM Multimedia 2025 in Dublin, Ireland.

To express your interest in participating to the challenge, please fill this form.

To submit your paper and your results (for participants of the automatic track), please use submission form on OpenReview.

Timeline

• 09 March 2025: Challenge Announcement & Website Launch
• 09 March 2025: Registration Opens
• 24 March 2025: Dataset & Query Release
• 27 July 2025: Fully-Automated Submission & Paper Deadline
• 11 August 2025: Notification to Authors
• 26 August 2025: Camera-Ready Deadline

Guidelines for Participants

Participants will be required to register and agree to the dataset usage policy. For details on the submission format, please see below.

Tasks

The inaugural edition of the CASTLE Challenge features a diverse set of tasks, including event detection, retrieval, and question answering. Future editions will expand the scope, but for this edition, the tasks include:

🔍 Event Instance Search

Given a textual description (in English), participants must identify all timeframes where a specific event occurs. Events should be reported with both a time range and a video ID.

📦 Object Instance Search

Given a textual (in English) or visual (i.e., using an image) example of a physical object, participants must find all occurrences of that object across any of the video streams.

💬 Question Answering

Given a question in natural language (in English), participants must provide an answer. The response should be formulated in natural language and include references to relevant sensor streams and time intervals as supporting evidence.

Evaluation

The challenge will operate across two tracks: fully-automatic and interactive.

⚙️ Fully-Automatic Track

Participants receive queries in advance and generate results using any method they choose. These results are then submitted to the challenge organizers for evaluation. Please see the list of queries below.

🎮 Interactive Track

This track will be evaluated live during the conference. Participants must solve tasks synchronously and interactively within a limited timeframe. This format follows established competitions such as the Video Browser Showdown and the Lifelog Search Challenge.

Queries

🔍 Event Instance Search

• MM25-EIS01: Find instances of somebody using a portable electric kitchen gadget.
• MM25-EIS02: Find instances of somebody unwrapping a sweet snack and eating it.
• MM25-EIS03: Find instances of somebody telling a joke or making a pun.
• MM25-EIS04: Find winning hands of poker.
• MM25-EIS05: Find instances pouring the last amount of liquid from a beverage container.
• MM25-EIS06: Find instances of somebody complimenting the food.
• MM25-EIS07: Find instances of people mispronouncing the German name of a food item.
• MM25-EIS08: Find instances of people singing.
• MM25-EIS09: Find instances of starting the dishwasher.
• MM25-EIS10: Find instances of someone using the pepper shaker that actually contains salt.

📦 Object Instance Search

• MM25-OIS01: Find the bird-shaped cookie cutter.
• MM25-OIS02: Find the thermal image camera.
• MM25-OIS03: Find the ace of spades.
• MM25-OIS04: Find 'A Christmas Carol' (the book).
• MM25-OIS05: Find the set of colored pens.
• MM25-OIS06: Find a paper airplane made from a white sheet of paper.
• MM25-OIS07: Find a partially eaten apple.
• MM25-OIS08: Find the yellow octopus toy.
• MM25-OIS09: Find the squirrel-shaped tree ornament.
• MM25-OIS10: Find a kitchen scale.

💬 Question Answering

• MM25-QA01: How many ‘Zimtstern’ cookies were made in total?
• MM25-QA02: Who drew a picture of a ‘Connect Four’ board?
• MM25-QA03: What was the category in the first round of ‘Happy Quiz’?
• MM25-QA04: Who won at Mikado?
• MM25-QA05: What was the ingredient not available (i.e. also not replaced) for the pumpkin risotto?
• MM25-QA06: What open-source movie did Klaus want to watch?
• MM25-QA07: What K-pop band is Cathal into?
• MM25-QA08: Whom did Werner give a small bottle of hand sanitizer?
• MM25-QA09: Which was the name or brand of the wine, of which the empty bottle was used as a rolling pin?
• MM25-QA10: What book did Florian spend the longest time reading?

Submission Format

⚙️ Fully-Automatic Track

For the fully-automatic track, please submit your results together with your paper on OpenReview. Results should be submitted in CSV format with one file per task type (i.e., three files, one each for the 🔍 Event Instance Search, 📦 Object Instance Search, and 💬 Question Answering tasks) in a ZIP file. The name of each file should indicate the task type. The columns for the files are listed below.

🔍 Event Instance Search

• task: the Id of the task
• day: the day on which the event occured as integer from 1 to 4
• starttime: the start time of the event in HH:MM:SS format
• endtime: the end time of the event in HH:MM:SS format
• source: the perspective/camera the event was recorded from (e.g., Allie, Kitchen, etc.)

📦 Object Instance Search

• task: the Id of the task
• day: the day on which the object can be seen as integer from 1 to 4
• starttime: the start time from when the object can be seen in HH:MM:SS format
• endtime: the end time until when the object can be seen in HH:MM:SS format
• source: the perspective/camera the object can be seen from (e.g., Allie, Kitchen, etc.)

💬 Question Answering

• task: the Id of the task
• answer: a natural language answer to the question in English

🎮 Interactive Track

The interactive track will be evaluated during a dedicated session at ACM Multimedia 2025 in Dublin. Participants are expected to be on-site for the interactive evaluation. Systems are supposed to submit their task solutions to the Distributed Retrieval Evaluation Server via its API.