Publications

Unit-test rewards train coding agents for functional correctness, yet provide no signal on patch quality or the agent's workflow. Rubric rewards address this gap by having an external judge inspect and evaluate candidate solutions against explicit criteria. Existing work adopts disparate rubric-construction strategies and evaluates them primarily on targeted in-domain behaviors, leaving it unclear how rubric specification effort affects learned agent behavior. We study this question systematically, comparing universal rubrics written once per quality dimension, rubric-anchored pairwise comparisons against a stronger reference agent, and agent-generated task-specific rubrics across both patch and trajectory rewards. Our proposed approaches preserve pass@1 while substantially improving patch and trajectory quality over unit-test RL alone (e.g., reducing leftover scratch artifacts on SWE-bench Verified from 29.5% to 5.4% and matching the gold patch size). We further show how well each rubric design, specifically targeted at SWE-style tasks, generalizes across in- and out-of-distribution agentic benchmarks: when grading final submissions, we observe that universal rubrics already match the more expensive designs on SWE-bench, while pairwise and task-specific rubrics carry more of the learned behavior to out-of-distribution benchmarks such as TerminalBench; when grading the agent's trajectory, however, universal rubrics prove effective enough to steer agent behavior.

@misc{scalena2026rubricrewards,
  title={What Makes a Good Rubric Reward? Reinforcement Learning for Coding Agents Beyond Correctness},
  author={Daniel Scalena and Leon Engländer and Sarra Habchi and Patrick Lewis and Ahmet Üstün and Sophia Althammer},
  year={2026},
  note={Preprint},
  url={https://danielsc4.it/assets/pdf/rubric-rewards-for-coding-agents.pdf}
}

Entropy-based pruning has been proposed as an effective method for compressing Chain-of-Thought (CoT) reasoning with negligible accuracy loss. We test the robustness of low- and high-entropy CoT step selection methods across various models and reasoning tasks, showing that entropy offers no advantage over random pruning in any evaluated setting. Moving from sentences to tokens, we then show that retaining low-entropy tokens seems effective only on mathematical benchmarks. We find this is due to the inherently low-entropy nature of numeric tokens, which also convey semantic content in such problems. Finally, we demonstrate that patching a subset of a few CoT tokens with their original activations recovers near-perfect full-trace performance, providing causal evidence that task information is not concentrated in a small set of CoT tokens identifiable by heuristics, but rather distributed across the full reasoning chain.

@misc{candussio2026demystifying,
  title={Demystifying Entropy-based Selection for Chain-of-Thought Compression in Large Reasoning Models},
  author={Sara Candussio* and Daniel Scalena* and Luca Bortolussi and Elisabetta Fersini and Malvina Nissim and Gabriele Sarti},
  year={2026},
  eprint={2607.28707},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2607.28707}
}

The game of Taboo requires describing a target word without using a set of forbidden words, so that other players can guess it. This deceptively simple task combines strict lexical constraints with the need for communicatively effective descriptions, making it a compelling playground for examining how LLMs navigate competing demands at inference time. We evaluate two open-weight models under conditions that intervene at progressively deeper levels of the generative process, from prompting to generation-time constraints to internal representations manipulations. We assess their outputs through forbidden word violation detection, LLM-as-a-judge measuring the degree to which generated descriptions successfully evoke the target concept for both human and machine guessers, and examining whether the strategies models adopt under constraint align with those of human players. Our results show that compliance with the rules of the game and communicative effectiveness trade off differently across conditions, and that models remain substantially weaker than humans as guessers, suggesting that lexical grounding under constraint is an open challenge for current language models.

@inproceedings{candussio2026taboo,
  title={“Don’t Say It!”: Constraints, Compliance, and Communication when Language Models Play Taboo},
  author={Sara Candussio* and Francesca Padovani* and Daniel Scalena* and Malvina Nissim},
  booktitle={Proceedings of the Twelfth Italian Conference on Computational Linguistics (CLiC-it 2026)},
  year={2026},
  eprint={2607.00601},
  archivePrefix={arXiv},
  url={https://arxiv.org/abs/2607.00601}
}

Chain-of-thought (CoT) reasoning is the dominant paradigm for inference-time scaling in language models, yet the causal influence of individual steps on the final answer poorly understood. We estimate each step's causal importance via early exit and use this measure to study how answers form across the reasoning traces of several model families. Across diverse tasks, we find that reasoning typically crosses a commitment boundary – a sharp transition from transient intermediate guesses to a stable, high-confidence answer. This transition often happens in a single step, well before the model's reasoning block ends, and is followed by epiphenomenal CoT steps that leave the final answer probability unaltered. Using attention probes, we show that answer-formation stages can be linearly decoded from intermediate reasoning steps with high accuracy and generalize robustly to unseen reasoning tasks. We exploit this signal to early-exit reasoning blocks at the commitment boundary, reducing the length of CoTs up to 55% on average with negligible impact on model performance.

@misc{scalena2026commitmentboundary,
  title={Beyond the Commitment Boundary: Probing Epiphenomenal Chain-of-Thought in Large Reasoning Models},
  author={Daniel Scalena* and Sara Candussio* and Luca Bortolussi and Elisabetta Fersini and Malvina Nissim and Gabriele Sarti},
  year={2026},
  eprint={2606.13603},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://arxiv.org/abs/2606.13603}
}

With the rise of reasoning language models and test-time scaling methods as a paradigm for improving model performance, substantial computation is often required to generate multiple candidate sequences from the same prompt. This enables exploration of different reasoning paths toward the correct solution, however, allocates the same compute budget for each prompt. Grounded on the assumption that different prompts carry different degrees of complexity, and thus different computation needs, we propose EAGer, a training-free generation method that leverages model uncertainty through token-wise entropy distribution to reduce redundant computation and concurrently improve overall performance. EAGer allows branching to multiple reasoning paths only in the presence of high-entropy tokens, and then reallocates the saved compute budget to the instances where exploration of alternative paths is most needed. We find that across multiple open-source models on complex reasoning benchmarks such as AIME 2025, EAGer can reallocate the budget without accessing target labels, achieving the best efficiency-performance trade-off in terms of reasoning length and Pass@k. When target labels are accessible, EAGer generates up to 65

@misc{scalena2025eager,
  title={EAGer: Entropy-Aware GEneRation for Adaptive Inference-Time Scaling},
  author={Daniel Scalena and Leonidas Zotos and Elisabetta Fersini and Malvina Nissim and Ahmet Üstün},
  year={2026},
  eprint={2510.11170},
  archivePrefix={arXiv},
  primaryClass={cs.LG},
  url={https://openreview.net/forum?id=G0Mw5MaPcR},
  note={Proceedings of the 43rd International Conference on Machine Learning, (ICML 2026)}
}

The rapid progress of Large Language Models (LLMs) has transformed natural language processing and broadened its impact across research and society. Yet, systematic evaluation of these models, especially for languages beyond English, remains limited. "Challenging the Abilities of LAnguage Models in ITAlian" (CALAMITA) is a large-scale collaborative benchmarking initiative for Italian, coordinated under the Italian Association for Computational Linguistics. Unlike existing efforts that focus on leaderboards, CALAMITA foregrounds methodology: it federates more than 80 contributors from academia, industry, and the public sector to design, document, and evaluate a diverse collection of tasks, covering linguistic competence, commonsense reasoning, factual consistency, fairness, summarization, translation, and code generation. Through this process, we assembled a benchmark of over 20 tasks and almost 100 subtasks, and established a centralized evaluation pipeline that supports heterogeneous datasets and metrics. CALAMITA is conceived as a rolling benchmark, enabling continuous integration of new tasks and models, offering a blueprint for other languages and communities seeking inclusive and rigorous LLM evaluation practices.

@article{nissim2026challenging,
  title={Challenging the Abilities of Large Language Models in Italian: a Community Initiative},
  author={Malvina Nissim and Danilo Croce and Viviana Patti and Pierpaolo Basile and Giuseppe Attanasio and Elio Musacchio and Matteo Rinaldi and Federico Borazio and Maria Francis and Jacopo Gili and Daniel Scalena and others},
  journal={Italian Journal of Computational Linguistics},
  volume={12},
  number={1},
  year={2026},
  doi={10.17454/ijcol121.02},
  eprint={2512.04759},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://journals.openedition.org/ijcol/1813}
}

High-quality machine translation systems based on large language models (LLMs) have simplified the production of personalized translations reflecting specific stylistic constraints. However, these systems still struggle in settings where stylistic requirements are less explicit and might be harder to convey via prompting. We explore various strategies for personalizing LLM-generated translations in low-resource settings, focusing on the challenging literary translation domain. We explore prompting strategies and inference-time interventions for steering model generations towards a personalized style, and propose a contrastive framework exploiting latent concepts extracted from sparse autoencoders to identify salient personalization properties. Our results show that steering achieves strong personalization while preserving translation quality. We further examine the impact of steering on LLM representations, finding model layers with a relevant impact for personalization are impacted similarly by multi-shot prompting and our steering method, suggesting similar mechanism at play.

@misc{scalena2025steeringlargelanguagemodels,
  title={Steering Large Language Models for Machine Translation Personalization},
  author={Daniel Scalena* and Gabriele Sarti* and Arianna Bisazza and Elisabetta Fersini and Malvina Nissim},
  year={2026},
  eprint={2505.16612},
  archivePrefix={arXiv},
  primaryClass={cs.CL},
  url={https://aclanthology.org/2026.eacl-long.217/},
  note={Proceedings of the 19th Conference of the European Chapter of the Association for CL, (EACL 2026)}
}

Large Language Models (LLMs) are increasingly deployed in real-world applications, raising urgent concerns around their safety, reliability, and ethical behavior. While existing safety evaluations have primarily focused on English, low- and mid-resource languages such as Italian remain critically underexplored. In this paper, we present the first comprehensive and multidimensional evaluation of LLM safety in the Italian language. We assess seven state-of-the-art LLMs across key safety dimensions using several automatic moderators tailored to cover the Italian settings. Furthermore, we analyze the challenges of adapting English-centric safety benchmarks to Italian via machine translation, highlighting limitations and proposing best practices for developing culturally and linguistically grounded evaluation frameworks.

@inproceedings{rizzi-etal-2025-uncovering,
  title = "Uncovering Unsafety Traits in Italian Language Models",
  author = "Rizzi, Giulia  and Magazz{\`u}, Giuseppe  and Sormani, Alberto  and Puler{\`a}, Francesca  and Scalena, Daniel  and Fersini, Elisabetta",
  booktitle = "Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)",
  month = sep,
  year = "2025",
  address = "Cagliari, Italy",
  publisher = "CEUR Workshop Proceedings",
  url = "https://aclanthology.org/2025.clicit-1.91/",
  pages = "974--982",
  ISBN = "979-12-243-0587-3"
}

Large Language Models (LLMs) have achieved remarkable success in generating human-like text and are increasingly integrated into real-world applications. However, their deployment raises significant safety concerns, including the risk of generating harmful, biased, or culturally inappropriate content. While several safety benchmarks exist for English, non-English contexts—such as Italian—remain critically underexplored, despite the growing demand for localized and culturally sensitive AI technologies. In this paper, we introduce BeaverTails-IT, the first Italian safety benchmark for LLMs, created through the machine translation of the original English BeaverTails dataset. We employ five state-of-the-art translation models, evaluate translation quality using automated metrics and human judgments, and provide guidelines for selecting high-quality safety prompts. Our benchmark enables the preliminary evaluation of Italian LLMs across key safety dimensions such as toxicity, bias, and ethical compliance.

@inproceedings{magazzu-etal-2025-beavertails,
  title = "{B}eaver{T}ails-{IT}: Towards a Safety Benchmark for Evaluating {I}talian Large Language Models",
  author = "Magazz{\`u}, Giuseppe  and Sormani, Alberto  and Rizzi, Giulia  and Puler{\`a}, Francesca  and Scalena, Daniel  and Cariddi, Stefano  and Michielon, Edoardo  and Pasqualini, Marco  and Stamile, Claudio  and Fersini, Elisabetta",
  booktitle = "Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)",
  month = sep,
  year = "2025",
  address = "Cagliari, Italy",
  publisher = "CEUR Workshop Proceedings",
  url = "https://aclanthology.org/2025.clicit-1.60/",
  pages = "625--635",
  ISBN = "979-12-243-0587-3"
}

This paper investigates the application of various prompting strategies and Italian-language large language models (LLMs) to extract salient characteristics of gender-based crimes from judicial courtroom decisions. Recognizing the complex linguistic and legal structures inherent in such documents, we evaluate several types of prompting across multiple LLMs fine-tuned or pretrained on Italian corpora. Our approach focuses on identifying key elements such as crime typology, victim-perpetrator relationships, modus operandi, and main motivations behind the crimes against women. We present a comparative analysis of LLM performance on a small set of judicial courtrooms, highlighting the impact of prompt design on the extraction of legally and socially relevant information. The findings demonstrate the potential of prompt engineering to enhance the ability of LLMs to support socio-legal research and policy development in the context of gender-based violence.

@inproceedings{rizzi-etal-2025-gender,
  title = "Gender Violence in Numbers: Prompting {I}talian {LLM}s to Characterize Crimes against Women",
  author = "Rizzi, Giulia  and Scalena, Daniel  and Fersini, Elisabetta",
  booktitle = "Proceedings of the Eleventh Italian Conference on Computational Linguistics (CLiC-it 2025)",
  month = sep,
  year = "2025",
  address = "Cagliari, Italy",
  publisher = "CEUR Workshop Proceedings",
  url = "https://aclanthology.org/2025.clicit-1.90/",
  pages = "965--973",
  ISBN = "979-12-243-0587-3"
}

Adapting models to a language that was only partially present in the pre-training data requires fine-tuning, which is expensive in terms of both data and computational resources. As an alternative to fine-tuning, we explore the potential of activation steering-based techniques to enhance model performance on Italian tasks. Through our experiments we show that Italian steering (i) can be successfully applied to different models, (ii) achieves performances comparable to, or even better than, fine-tuned models for Italian, and (iii) yields higher quality and consistency in Italian generations. We also discuss the utility of steering and fine-tuning in the contemporary LLM landscape where models are anyway getting high Italian performances even if not explicitly trained in this language.

@inproceedings{scalena-etal-2024-gentle,
  title = "A Gentle Push Funziona Benissimo: Making Instructed Models in {I}talian via Contrastive Activation Steering",
  author = "Scalena, Daniel  and Fersini, Elisabetta  and Nissim, Malvina",
  editor = "Dell'Orletta, Felice  and Lenci, Alessandro  and Montemagni, Simonetta  and Sprugnoli, Rachele",
  booktitle = "Proceedings of the 10th Italian Conference on Computational Linguistics (CLiC-it 2024)",
  month = dec,
  year = "2024",
  address = "Pisa, Italy",
  publisher = "CEUR Workshop Proceedings",
  url = "https://aclanthology.org/2024.clicit-1.98/",
  pages = "909--920",
  ISBN = "979-12-210-7060-6"
}

Activation steering methods were shown to be effective in conditioning language model generation by additively intervening over models' intermediate representations. However, the evaluation of these techniques has so far been limited to single conditioning properties and synthetic settings. In this work, we conduct a comprehensive evaluation of various activation steering strategies, highlighting the property-dependent nature of optimal parameters to ensure a robust effect throughout generation. To address this issue, we propose Dynamic Activation Composition, an information-theoretic approach to modulate the steering intensity of one or more properties throughout generation. Our experiments on multi-property steering show that our method successfully maintains high conditioning while minimizing the impact of conditioning on generation fluency.

@inproceedings{scalena-etal-2024-multi,
  title = "Multi-property Steering of Large Language Models with Dynamic Activation Composition",
  author = "Scalena, Daniel  and Sarti, Gabriele  and Nissim, Malvina",
  editor = "Belinkov, Yonatan  and Kim, Najoung  and Jumelet, Jaap  and Mohebbi, Hosein  and Mueller, Aaron  and Chen, Hanjie",
  booktitle = "Proceedings of the 7th BlackboxNLP Workshop: Analyzing and Interpreting Neural Networks for NLP",
  month = nov,
  year = "2024",
  address = "Miami, Florida, US",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2024.blackboxnlp-1.34",
  pages = "577--603"
}

The rapid development of Large Language Models (LLMs) has called for robust benchmarks to assess their abilities, track progress, and compare iterations. While existing benchmarks provide extensive evaluations across diverse tasks, they predominantly focus on English, leaving other languages underserved. For Italian, the EVALITA campaigns have provided a long-standing tradition of classification-focused shared tasks. However, their scope does not fully align with the nuanced evaluation required for modern LLMs. To address this gap, we introduce “Challenge the Abilities of LAnguage Models in ITAlian” (CALAMITA), a collaborative effort to create a dynamic and growing benchmark tailored to Italian. CALAMITA emphasizes diversity in task design to test a wide range of LLM capabilities through resources natively developed in Italian by the community. This initiative includes a shared platform, live leaderboard, and centralized evaluation framework. This paper outlines the collaborative process, initial challenges, and evaluation framework of CALAMITA.

@inproceedings{attanasio-etal-2024-calamita,
  title = "{CALAMITA}: Challenge the Abilities of {LA}nguage Models in {ITA}lian",
  author = "Attanasio, Giuseppe  and Basile, Pierpaolo  and Borazio, Federico  and Croce, Danilo  and Francis, Maria  and Gili, Jacopo  and Musacchio, Elio  and Nissim, Malvina  and Patti, Viviana  and Rinaldi, Matteo  and Scalena, Daniel",
  booktitle = "Proceedings of the Tenth Italian Conference on Computational Linguistics (CLiC-it 2024)",
  month = dec,
  year = "2024",
  address = "Pisa, Italy",
  publisher = "CEUR Workshop Proceedings",
  url = "https://aclanthology.org/2024.clicit-1.116/",
  pages = "1054--1063",
  ISBN = "979-12-210-7060-6"
}

Due to language models' propensity to generate toxic or hateful responses, several techniques were developed to align model generations with users' preferences. Despite the effectiveness of such methods in improving the safety of model interactions, their impact on models' internal processes is still poorly understood. In this work, we apply popular detoxification approaches to several language models and quantify their impact on the resulting models' prompt dependence using feature attribution methods. We evaluate the effectiveness of counter-narrative fine-tuning and compare it with reinforcement learning-driven detoxification, observing differences in prompt reliance between the two methods despite their similar detoxification performances.

@misc{scalena2023let,
  title={Let the Models Respond: Interpreting Language Model Detoxification Through the Lens of Prompt Dependence},
  author={Daniel Scalena and Gabriele Sarti and Malvina Nissim and Elisabetta Fersini},
  year={2023},
  eprint={2309.00751},
  url={https://arxiv.org/abs/2309.00751},
  archivePrefix={arXiv},
  primaryClass={cs.CL}
}

This paper describes the participation of the research laboratory MIND, at the University of Milano-Bicocca, in the SemEval 2023 task related to Learning With Disagreements (Le-Wi-Di). The main goal is to identify the level of agreement/disagreement from a collection of textual datasets with different characteristics in terms of style, language and task.The proposed approach is grounded on the hypothesis that the disagreement between annotators could be grasped by the uncertainty that a model, based on several linguistic characteristics, could have on the prediction of a given gold label.

@inproceedings{rizzi-etal-2023-mind,
  title = "{MIND} at {S}em{E}val-2023 Task 11: From Uncertain Predictions to Subjective Disagreement",
  author = "Rizzi, Giulia  and Astorino, Alessandro  and Scalena, Daniel  and Rosso, Paolo  and Fersini, Elisabetta",
  booktitle = "Proceedings of the The 17th International Workshop on Semantic Evaluation (SemEval-2023)",
  month = jul,
  year = "2023",
  address = "Toronto, Canada",
  publisher = "Association for Computational Linguistics",
  url = "https://aclanthology.org/2023.semeval-1.77",
  pages = "556--564"
}