Approach the analysis phase
Approach the analysis phase: method and template
The method and template proposed supports the implementation of a generic Measurement&Analysis process as specified by quality frameworks such as CMMI for Dev or ISO 9001-2008.The main analytical activities address the following topics:
- measurement goals identification,
- KPIs or metrics relative to measurement goals specification,
- criteria for the interpretation of the metrics values and representation,
- measure attributes collection,
- source of measurement identification,
- Kpis or metrics visualization and communication criteria,
- security criteria.
The following template may be used to collect, formalize and share with all stakeholders the information collected during analytical activities for each measurement objective.
SECTION 1 - Establish measurement objectives
This first section establishes measurements objectives and metrics interpretation. The GQM method is applied to align metrics with measurement goals of the organization.
| Item | Description |
|---|---|
| Goal | describes measurement goal |
| Question | the question specifies in a quantifiable way if the goal is measurable |
| Metric | describes the metric that satisfies the question. |
| Metric Interpretation | provides guidance to help the users in interpretation of metrics results and representation |
Guidelines
- All the stakeholders must be involved in these activities to refine results.
- GQM is useful to involve persons of all the organizational levels in order to assure that measurement objectives and identified metrics are aligned.
- Maintain the history of all the deliverables produced in this activities, they will be useful to implement action for process improvement.
- Focus the most important goals for the organization. Also a low number of metrics could require a collection of a high number of measure attributes.
- Measurement objectives could be constrained by existent process. If the measurement objective is an important goal, consider how to modify the process, procedures or upgrade tools.
- Evaluate benefits in comparison with the necessary efforts and costs to obtain the metrics related to the measurement objective.
SECTION 2 - Measurements The second section provides specification for the metrics (algorithm, thresholds, measure attributes) and relationship with process and activities linked with measurement objective.
| Item | Description |
|---|---|
| Metric algorithm | specifies the algorithm to calculate the metric. |
| Threshold | a value to compare the current value of the metric. |
| Measure attributes | specifies each attribute needed to calculate the metric (from a logical point of view). |
| Relationship to others metrics | specifies other metrics eventually used in the algorithm. |
| Relationship to processes | specifies process and activity linked to the measurement objective. For example, the metric could be linked to CMMI process area and specific practice or to a specific IT service task. |
Guidelines
- It is recommended to specify the type of measure collection:
- “On-line” collection gathers measures during the process execution. In this way monitoring and forecast of the results are possible. It is appropriate to provide the points of measurement into the process.
- “Off-line” collection gathers measures at the end of the process. In this way monitoring and forecast of the results are not possible.
- Thresholds specification: trustworthy references about threshold value are not always available in literature, so the adoption of a pragmatic approach is appropriate. A first threshold value could be defined based on “organization experience” then refined using the results of data analysis.
- The quality of data collected from a real environment will be evaluated. All the organizational and technical needs to improve the quality will be implemented. Data collection procedures must discharge data, on the basis of defined rules, and then provide mechanism to correct and recollect them.
- The algorithm specification could require compromises to prefer the industrial point of view against the academic point of view. A simplified result may guarantee better readability and homogeneity across the projects under evaluation.
- Metrics and relative algorithms should be based on objective not subjective measure attributes. Try to avoid measure attributes classification assigned by persons (e.g.: size of project: “small”, “medium”, “large”). In this case it’s useful to associate the labels with a range of values as a guidance to allow an objective choice.
SECTION 3 - Data Collection
The third section specifies sources and methods to collect measure attributes. Source of measure attributes | specifies tools, repositories and others sources containing measure attributes to collect.
| Item | Description |
|---|---|
| Attributes | specifies for each source the attributes to collect |
| Frequency | specifies when and how often the measure attributes are collected. |
| Procedures | specifies how the measure attributes will be gathered from sources: - automatically; - partially automatic (manual procedure is needed to prepare or move data before starting automatic procedures; - manual input, measure attributes are collected using a GUI. |
| Automatic collection rules | specifies rules and upgrade organizational procedures in order to facilitate the automatic collection. |
| Filtering and decoding | specifies criteria for filtering measure attributes (for example: attributes will be gathered for specific values of other attributes as “status” or “category”). Specifies decoding attributes to have a unique value across the organization’s projects. |
- Check how long the collected measure attributes will be stored in source repositories.
- Check if the measure already collected attributes may be updated.
- Provide data collection procedures well integrated with the work process, optimizing data input and reducing people work load.
The forth section specifies how to report and distribute the results of data analysis related to the measurement objective defined in section 1.
| Item | Description |
|---|---|
| Relevant stakeholders | specifies who (role in the organization) will receive periodically the results of data analysis. |
| Security | specifies security requirements to access reported results. |
| Results presentation | specifies how the results are represented (graphics, report, …) and provides examples. |
| Feedback | specifies how relevant stakeholders may notify comments, improvements or modification of results and processes. |
Guidelines
- Provide a tool to visualize the results based on user’s role and contextual guidance in the interpretation.
- Specific functionalities could be useful to store measurement objectives and track feedback and improvements.
- Metrics representation must be clear, concise and appropriate to the stakeholder’s role.
- Prefer a graphical view of the results.
- The results must be available in time according to the needs of the stakeholders.