Cookies
O website necessita de alguns cookies e outros recursos semelhantes para funcionar. Caso o permita, o INESC TEC irá utilizar cookies para recolher dados sobre as suas visitas, contribuindo, assim, para estatísticas agregadas que permitem melhorar o nosso serviço. Ver mais
Aceitar Rejeitar
  • Menu
Publicações

Publicações por HumanISE

2025

Mitigating false negatives in imbalanced datasets: An ensemble approach

Autores
Vasconcelos, M; Cavique, L;

Publicação
EXPERT SYSTEMS WITH APPLICATIONS

Abstract
Imbalanced datasets present a challenge in machine learning, especially in binary classification scenarios where one class significantly outweighs the other. This imbalance often leads to models favoring the majority class, resulting in inadequate predictions for the minority class, specifically in false negatives. In response to this issue, this work introduces the MinFNR ensemble algorithm, designed to minimize False Negative Rates (FNR) in imbalanced datasets. The new approach strategically combines data-level, algorithmic-level, and hybrid-level approaches to enhance overall predictive capabilities while minimizing computational resources using the Set Covering Problem (SCP) formulation. Through a comprehensive evaluation of diverse datasets, MinFNR consistently outperforms individual algorithms, showing its potential for applications where the cost of false negatives is substantial, such as fraud detection and medical diagnosis. This work also contributes to ongoing efforts to improve the reliability and effectiveness of machine learning algorithms in real imbalanced scenarios.

2025

Large Language Model for Querying Databases in Portuguese

Autores
Figueiredo, L; Pinheiro, P; Cavique, L; Marques, N;

Publicação
Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)

Abstract
This study introduces a system that helps non-expert users find information easily without knowing database languages or asking technicians for help. A specific domain is explored, focusing on a subscrip- tion-based sports facility, which serves as an open-source version of a real case study. Utilizing the star schema, the available data in the database is structured to provide accessibility through Portuguese Natural Language queries. Using a Large Language Model (LLM), SQL queries are generated based on the question and the provided star schema. We created a dataset with 115 highly challenging questions drawn from real-world usage scenarios to validate the correctness of the system. Challenges found during testing, like attribute value interpretation, out-of-scope questions, and temporal interval adequacy issues, highlight the insufficiency of the star schema alone in providing the needed context for generating accurate SQL queries by the LLM. Addressing these challenges through enhanced contextual information shows significant improvement in query correctness, with validation results increasing from 57.76% to 88.79%. This study shows the potential and limitations of LLMs in generating SQL queries from Portuguese Natural Language queries. © The Author(s), under exclusive license to Springer Nature Switzerland AG 2025.

2025

Exploring ChatGPT Efficiency in Automatic Test Generation for Python: A Comparative Analysis

Autores
Guerino, LR; Rizzo Vincenzi, AM;

Publicação
SBQS

Abstract
Context: Large language models (LLMs) like ChatGPT have gained attention in automated software testing. This study evaluates ChatGPT-3.5-turbo’s ability to generate test sets for Python programs, comparing it with Pynguin and pre-existing test sets. Problem: Automated testing remains challenging for dynamically typed languages like Python, requiring adaptable tools for diverse code structures. Solution: We assessed ChatGPT-3.5-turbo’s test generation using different prompt configurations and temperature settings. Method: Using 40 Python programs, we generated Pytestcompliant tests via the OpenAI API, varying temperature settings (0.0 to 1.0). Tests were validated using Pytest, with coverage and mutation scores measured via Coverage, MutPy, and Cosmic-Ray. Pynguin-generated and pre-existing test sets served as baselines. Summary of Results: ChatGPT-3.5-turbo successfully generated valid tests for simpler programs, but averaged below 28% overall, with a low cost. Higher temperatures (0.5–1.0) improved results, but combining test cases from all temperatures introduces diversity in the LLM-generated test sets, making it possible to overcome both Pynguin and pre-existing test sets in terms of decision coverage and mutation score.

2025

Automated Generation of End-to-End Web Test Cases via a Generic AI Agent: A Comparative Study of DeepSeek V3 and Claude Sonnet 5

Autores
Monteiro, CEO; Guerino, LR; Fernandes, GF; Pereira, MH; Souza-Zinader, JPd; Braga, RD; Pocivi, VCB; Vincenzi, AMR;

Publicação
Proceedings of the 31st Brazilian Symposium on Multimedia and the Web (WebMedia 2025)

Abstract
Web applications are widespread and can be accessed from anywhere, in theory, using aweb browser on a computer or smartphone. Primarily due to the diversity of web browsers and frameworks available for developing web application interfaces, testing such applications is a challenging task. With the advent of large language models, several works are utilizing them to automate software engineering tasks, including test case generation. This use of LLMs for test case generation prioritizes unit testing. More recently, we have seen the advent of Generic Artificial Intelligence Agents, which are tools that utilize LLMs and also possess the ability to run additional tools, such as cloning repositories, navigating websites, and compiling programs. In this work, which is part of a research and development project, we evaluate a specific Generic AI Agent Assistant regarding its capability to navigate web applications and create fully automated end-to-end test cases, utilizing Selenium WebDriver and JUnit 5 framework. Results show that, considering a set of nine websites, in overall end-to-end test case generation, Suna configured with DeepSeek V3 produced 165 successful test cases out of 481 generated tests, a success rate of 34.3%. On the other hand, Suna configured with Claude Sonnet 4 produced 336 successful test cases out of 479 generated tests, a success rate of 70.1%, which is very impressive, mainly due to the complexity of creating end-to-end testing. In terms of cost, we used a free and a paid LLM model. The paid model generates successful test cases at an average price of $ 0.15 per test case.

2025

Exploring Documentation Strategies for NFR in Agile Software Development

Autores
Moreira, I; Adolfo, LB; Melegati, J; Choma, J; Guerra, E; Zaina, L;

Publicação
XP

Abstract
Abstract Companies adopt agile methodologies for various reasons, primarily due to their adaptability to change and evolving business demands. In this context, addressing non-functional requirements (NFRs) may not always be a priority and can present challenges for agile teams. The focus on User Stories present in agile methods and tools often does not offer explicit alternatives for documenting NFRs. In this research, we perform a survey to explore five different strategies for documenting NFRs, to identify which fits better for different types of quality attributes and to understand the strengths and drawbacks of each one. As a result, the participants considered certain strategies as being more or less suitable for specifying different types of quality attributes. For instance, while Story Labeling was rarely recommended for security requirements, using Story Sub-sections or Verification Rules were highly recommended for this kind of quality attribute. Our results also evaluated the strategies considering several factors, such as the level of detail and requirement duplication. As a practical implication, the results of this work can provide guidance to agile development teams in choosing the most suitable alternative for each NFR documentation.

2025

Exploratory Test-Driven Development Study with ChatGPT in Different Scenarios

Autores
Pancher, JC; Melegati, J; Guerra, EM;

Publicação
XP

Abstract
Abstract Generative AI has been rapidly adopted by the software development industry in various ways, offering innovative approaches to transforming requirements into working software. Combining Generative AI with Test-Driven Development (TDD) presents a creative method to accelerate this transformation. However, questions remain about ChatGPT’s readiness for this challenge, including the techniques and best practices required for success and the scenarios where this approach can consistently deliver results. To explore these questions, we designed a study where a group of master’s students performed programming assignments using TDD, first independently and then with the support of ChatGPT. The three assignments represent distinct scenarios: mathematical calculations (function), text processing (class), and system integration (class with dependencies). We performed a qualitative analysis of the submitted code and reports identifying key strategies that significantly influence success rates, such as providing contextual information, separating instructions in prompts following an iterative process, and assisting AI in fixing errors. Among the scenarios, the integration task achieved the highest performance. This study highlights the potential of leveraging Generative AI in TDD for software development and presents a list of effective strategies to maximize its impact. By applying these positive strategies and avoiding identified pitfalls, this research marks a step toward establishing best practices for integrating Generative AI with TDD in software engineering.

  • 62
  • 739