Key Performance Indicators and Metrics

1. Introduction

In software development, a metric is the result of measuring something or even the

set of measurements of the same variable that allows the monitoring of its

evolution [Prikladnicki et. al. 2014]. Among the most common metrics used in

software development with agile methodologies can be mentioned: Burn Down Chart,

Cumulative Flow Diagram, Total Stories Delivered, Velocity, Time x Ratio

points, Lead Time, Cycle Time, number of commits or lines changed in the code,

number of tests, code coverage with unit tests, number of points remaining in tasks not yet completed, total hours used for development in pairs, and others that may interest the agile team. Metrics can be used in different ways as long as they serve to improve team communication and collaboration in their deliverables, this naturally makes metrics become key performance indicators [Davis 2015].

Key performance indicators are commonly used to measure and

evaluate the quality and competence of teams in the performance of their activities and

Processes. Such indicators are also present in software development

with agile methodologies, which bring with them a range of concepts and tools that

can help in this metering. Just as there are different types of metrics that can be used, they can also represent different aspects such as: project management; architecture, analysis and design; development practices; software testing; and deliveries. In this way, it helps to measure different perspectives of software development. They help managers to quantify, analyze, and optimize processes, making decision-making easier. And they can be further enhanced when using specific software to collect such metrics [Rossberg, 2019].

There are, therefore, some problems faced in defining, collecting, and analyzing metrics:

a) the definitions of measurement are not direct, clear, objective or well defined, on the contrary they are usually subjective and unclear aspects for some team members.

b) agile development focuses on the product rather than the project, a project is a defined set of time within which something is developed and tracked while a product is a living thing that continues to change to meet consumer needs [Davis 2015].

c) the data can be in different places and different tools and each member of the team can be dedicating himself to different subjects or problems.

Speed and capacity are aspects that can be evidenced, speed in developing a certain functionality and ability to develop a number of functionalities in a given period of time. Metrics bring maturity to the software development process and provide greater efficiency and quality in deliveries. Accurate collection of metrics can be a great opportunity to observe how the team works, but it can also serve to answer other questions such as: how accurate are the requirements writings?

Is the system becoming better, more secure, more scalable, more resilient, and performant? Is the team delivering value to the end customer?

An agile activity directly related to metrics is the estimation of tasks that can be defined as an approximation, calculated or not, of an outcome that can be used as information, even if it is incomplete or uncertain [Prikladnicki et. al. 2014].

With this activity, it is sought to anticipate the time, effort and cost for the development of each new feature or maintenance of the product.

Therefore, it is evident that the collection of metrics does not depend exclusively on the estimation and scoring of features, but on the other hand, it relies on a large scale in this activity. Estimating the complexity of a new feature is not an easy task and requires maturity, experience, and team synergy. Estimates may eventually change over time, but it is not a rule.

There are different methods used for the estimation activity, among them: Story Points, Planning Poker, ideal days, T-Shirt Size, pomodoro method.

This article proposes to discuss how to effectively measure the speed and capacity of a software development team that uses agile methodology and relies on the proper collection of metrics. To this end, it intends to answer the research questions presented in Session 3 and is divided as follows:

1. Session 2: Contains the theoretical framework with the description of different types of metrics and estimates.

Theoretical Framework Effort estimation and monitoring of metrics are not exclusive characteristics of agile methodologies, as they are also present in traditional or predictive methodologies. Despite the massive adoption and gain in popularity of agile methodologies since the Agile Manifesto and its 12 principles published in 2001, companies and development teams encounter several problems and difficulties in their execution. One of the problems faced is the estimation of effort, which becomes a great challenge due to its high complexity and subjectivity [Beh et. al. 2022]. Software metrics are process attributes that can be measured or used to evaluate and analyze the process, monitor risks, calculate cost, measure quality, determine reliability, verify customer satisfaction, observe time to market, and observe other aspects determined by management [Maddox and Walker 2021]. In their study, Maddox and Walker (2021) analyzed 5 articles from a total of 142 publications on metrics in software development with agile methodologies and pointed out 135 different metrics used in different aspects and biases. The goal then is to find the north to identify the metrics that are really useful for evaluating the quality of product development and its result

2. Session 3: Presents the method used to search for references; Session 4 comprises the discussion of the results found;

Lead Time Represents the time between the beginning and end of a given story or activity. The faster a team moves tasks between columns on a Kanban board, for example, the better the Lead Time collected. Sometimes this metric is confused with speed, but that is not exactly what it represents, because Lead Time represents how long it takes for a demand to be started and completed, regardless of the speed of an individual delivery [Davis, 2015]. Such a metric is very valuable when it is possible to measure different stages of the delivery process and thus realize what are the points of attention and improvement.

Cycle Time Similar to Lead Time also represents the time elapsed between two states of an activity, however Cycle Time covers a larger scope because it considers the beginning of the process from the creation of the demand, refinements and the time that the activity was waiting in the Back Log before being started. In this way, it is possible to say that Lead Time is contained in Cycle Time because it is the first minor part of the second [Haidabrus et. al. 2023].

Method: This article proposes a literature review on the topic of using metrics in software development teams that work with agile development. The task estimation activity that is a related subject and necessary for discussion has also been revised, therefore bringing more arguments and propriety to the discussion.

1. Research questions The following research questions were proposed:

• Question 1: What are the key indicators of success and how can they be measured effectively?

This question cannot be answered objectively, since each project may have its success indicators, more or less adequate. The most used indicators are CFD, Lead Time and speed, the first is able to quickly point out the bottlenecks faced by the team and thus establish action plans to solve problems. Lead Time and speed are directly related to delivery time and should not be confused, the former evaluates the time occupied by a demand from the beginning to its delivery and the latter relates the estimate to the actual time of an individual demand [Haidabrus et. al. 2023]. In the case of indicators that depend directly on the estimation stage, it is important to invest in knowledge and training so that the team can correctly estimate the stories and thus the metrics will serve to celebrate success and correct any problems.

• Question 2: What is the impact of performance metrics on the motivation and engagement of development teams?

Only one article [Kurnia et. al. 2018] points out that the collection of metrics such as Earned Value Management (EVM) and Job Satisfaction - job satisfaction of the development team - has a great impact when analyzing the success of the cycle just ended. Measuring job satisfaction has a great effect on productivity and the quality of deliveries. Therefore, it is not possible to determine the direct impact that can bring more or less motivation to the team, but it is an area that should be better explored.

• Question 3: How do performance metrics impact decision-making and project success?

As long as they are collected correctly, they are used by leaders and managers for decision-making regarding the project. In addition to identifying bottlenecks, problems, and points for improvement, the correct use of performance metrics helps to improve workflow, providing greater efficiency and effectiveness in delivering results and assisting in decision-making and eventual course corrections [Haidabrus et. al., 2023]. By correctly identifying the most appropriate metrics for the project, they serve to support decision-making [Maddox and Walker, 2021]

2. Databases and search methods: The IEEE Xplore platform was used to search for scientific articles that deal with the desired topic. The searches in the IEEE Xplore database were carried out with and without restriction. Table 2 represents the phrases, parameters, constraints and results obtained

3. Session 5: Presents the conclusion.

This article presented a literature review on effort estimates and performance metrics in software development using agile methodologies. We identified the direct relationship between these two practices and the importance of having an integrated and cohesive team, as well as well-established standards and references for the most accurate performance of effort estimates. We point out the difficulty for less experienced people in relating subjective points to the complexity of tasks and point out possible solutions such as discussions, training and repetition of methods. The estimation stage is not an exclusively technical activity and depends on the participation of all team members. Metrics collection is more fluid when the estimation step has been well executed, but not all metrics depend on this step. There are at least 135 different metrics that can be used in different aspects of the project. The most used are speed and Lead Time. By adopting consistent estimation and metrics approaches, teams can continuously improve their efficiency and quality, thereby ensuring the success of their software projects in agile environments.

References:

Prikladnicki, R., Willi, R., & Milani, F. (2014). Agile methods for software development. Bookman Publisher.

Davis, C. W. (2015). Agile metrics in action: measuring and improving the performance of agile teams. Publications, Manning Co.

Rossberg, J. (2019). Agile project management with Azure DevOps: concepts, models, and metrics. Apress

Beh, H. C., Jusoh, Y. Y., & Abdullah, R. (2022). Dimensions in the measurement of the performance of agile software development projects: a literature review. In 2022, International Conference on Applied Informatics (AiIC) (pp. 83-87). IEEE.

Haidabrus, B., Grabis, J., Ivanov, V., Druzhinin, E., & Psarov, O. (2023). Improving the effectiveness of agile teams through metrics. In 2023, IEEE 64th International Scientific Conference on Information Technology and Management Science of the Technical University of Riga (ITMS) (pp. 1-5). IEEE.

Kurnia, R., Ferdiana, R., & Wibirama, S. (2018). Classification of software metrics for the agile scrum process: a literature review. In 2018, International Research Seminar on Information Technology and Intelligent Systems (ISRITI) (pp. 174-179). IEEE.

Maddox, M., & Walker, S. (2021). Agile software quality metrics. In 2021 IEEE MetroCon (pp. 1-3). IEEE.

Mallidi, R. K., & Sharma, M. (2021). Study on agile techniques and challenges of story point estimation. Int. J. Comput. Appl, 174(13), 9-14.