Evaluating Software Quality of Research Software
Prof. Dr. Andreas Vogelsang
Computer Science - Software and System Engineering
Idea
Software becomes more and more important to research. (…) We are interested in inspecting which software development best practices are applied and which relevance code quality has in research software. The results will give insight into current code quality and provide possibilities for improvement of the quality and correctness of the programs used. More specifically, we want to answer the following research questions:
- RQ1: Which role does code quality currently play in research software?
- RQ2: How “good” is research software in comparison to other open-source (and commercial) software?
- RQ3: How can we improve productivity and code quality in research software?
By answering these three research question, we will be able to identify further topics that will investigate further as part of KPA initiatives or offer as training for researchers.
Result
Within this project, interviews were conducted and analysed with scientists from our institutes associated with the KPA. The focus is on the quality requirements for software that is created and used by scientists themselves as part of their scientific work. Since this research software (RSE) is often accompanied by complex issues (e.g. model simulations and data analysis), it is created by scientists who are experts in the respective field, while the actual processes of software development tend to run in the background. Since modern science is increasingly dependent on the use of software and the results are based on this software, the question of the quality of this research software must also be given higher priority.
The main questions in the interview are:
- RQ1: How do researchers elicit, document, and analyze requirements for research software?
- RQ2: What processes do researchers follow and which parts relate to requirements?
- RQ3: What are requirements-related challenges that researchers face?
The interview analysis shows that the primary quality feature of the research software named by the researchers is the functional correctness of the research software, followed by performance. Other quality characteristics are consistency, portability and installability. Reproducibility was named as a decisive quality feature across almost all subject areas.
Furthermore, it turns out that the requirements for the software to be created are often not defined at all. As a result, the process of validation and verification based on previously created requirements hardly exists. The management from the requirements to the test is not clearly defined. One problem is that the processes of structured software development are simply not known in most cases because they are not learnt or taught. Given the often small size of the teams or working groups, this is seen as too much time spent for too little benefit. There is no reward for high-quality software.
Andreas Vogelsang and his colleagues conducted and analysed these interviews and documented the results in detail in a publication together with the future challenges and visions for research software.
Bajraktari, M. Binder, A. Vogelsang: "Requirements Engineering for Research Software: A Vision“, In: 32nd IEEE International Requirements Engineering (RE) 2024 conference, https://arxiv.org/abs/2405.07781