The competition, titled IEEE CIS Technical Challenge on Domain-Specific Question Answering with LLMs, is organized by the Andalusian Inter-university Institute in Data Science and Computational Intelligence (DaSCI) at the University of Granada in collaboration with the IFMIF-DONES Spain Consortium. It is funded by the IEEE Computational Intelligence Society (IEEE CIS) and offers up to $20,000 in prizes.
The challenge, which began on September 28, will run for two months. Teams can register until November 1 and submit their solutions by November 30. In its initial days, over twenty participants have already registered on the competition platform.
Current large language models (LLMs), while proficient in general tasks, have limitations when faced with highly specialized technical knowledge. The most powerful solutions often rely on commercial models running in the cloud, an option that is unfeasible for critical scientific infrastructure or companies with confidential documentation. Furthermore, access to large computing centers is not always available.
«We have fantastic models, but when they need to answer very technical and domain-specific questions, they can fall short, especially if computational resources are limited, which is common in many real-world scenarios,» explains Isaac Triguero, director of the DaSCI Institute and head of the competition. «You can't always turn to a cloud provider, either because you lack the means or the money, or simply because you don't want your data to be accessible to those providers».
To replicate this real-world scenario, the competition requires teams to work with the open-source model OLMo, developed by the Allen Institute for AI, which fits on a single graphics card (GPU). Solutions will run on servers at the University of Granada, offline and with limited computation time, ensuring that neither the questions nor the reference documents leave the evaluation environment at any point. The goal is to adapt the model intelligently, using techniques such as efficient fine-tuning, prompt engineering, knowledge graphs, or fuzzy modeling, rather than large-scale retraining.
Solutions will be evaluated using a benchmark of 100 questions and answers developed by experts in two domains: the IFMIF-DONES facility, a particle accelerator under construction in Escúzar (Granada) for testing materials for future fusion reactors, and specialized scientific literature in artificial intelligence. The questions range from retrieving specific data to comparative, causal, or predictive reasoning. Participants have access to a sample of 25 examples, and the complete test bank has been prepared and labeled over nearly a year by expert personnel from IFMIF-DONES and DaSCI Institute researchers.
The collaboration with IFMIF-DONES is part of the DONES-AXIA project, a multi-agent system aimed at improving the operation, safety, and energy efficiency of the facility. It is funded by the CDTI with FEDER co-financing and led by Plain Concepts. Within this project, the UGR is developing specialized, lightweight, and secure language models for IFMIF-DONES, aligning with the competition's objectives. The initiative is also linked to the DONITOS program at the University of Granada, focused on doctoral training in collaboration with IFMIF-DONES. Representing the consortium are its director, Moisés Weber, and Patricia Martín, head of the Operation and Maintenance Department.
The competition consists of two phases. The first, automated, executes each team's code on hidden questions, displaying scores on an online leaderboard. «Optimizing just one metric isn't enough; we're looking for the science behind it,» notes Triguero. «We don't just want the best numerical result, but also the most innovative and highest quality.» Consequently, the top proposals, around twenty, will advance to a second phase where domain expert panels will assess the quality of the answers, and a scientific committee will evaluate technical innovation. The final ranking will be announced on December 14, with prizes of $9,000 for first place, $5,000 for second, $3,000 for third, and $1,500 each for fourth and fifth place.
The organizing team from the UGR includes Isaac Triguero, researchers Jaime Uclés and Carlos Peláez González, and Computer Science PhD Ignacio Aguilera Martos. The challenge committee features Xingyu Wu (Hong Kong Polytechnic University), Christian Wagner (University of Nottingham), and Katherine Malan (University of South Africa). The scientific committee also includes Javier Del Ser (University of the Basque Country) and Daniel Molina (UGR). The University of Granada is providing the computational infrastructure for the evaluation.
