There are no formal quantitative figures of the benefits of using a configuration management system and tools, this is partly due to the fact that most organisations that do not have a formal configuration management system do not have the required figures to compare the cost of development before and after a configuration management system. Therefore it follows that justifying the implementation of a configuration management system on the basis of tangible benefits is quite difficult.
Whilst there are no published figures that demonstrate the tangible benefits of a configuration management system, there are many intangible benefits that make a configuration management system essential. Major intangible benefits are outlined below:
Configuration management will assist in stabilising the development environment through the formal release and change processes. A stable development environment is vital for effective development.
Implementing change to an IT system can be a complex process. Change control provides a mechanism where the impact of each change is assessed to ensure that it is implemented correctly and with minimal side effects.
Effective control of environments and effective build management will assist in the generation of repeatable builds. This enables one to re-generate builds and reproduce problems with relative ease.
Recovering from problems following a series of changes can be a complex and time-consuming problem. The audit trail generated by the various configuration management processes will help in identifying the audit trail for all changes and hence facilitate problem resolutions.
Configuration management formalises the many essential processes and provides management information relating to the volume of changes, defects, releases, ... etc.
Configuration management forms an essential part of many international standards for quality management. Implementing a configuration management system will therefore facilitate the development and roll out of quality management systems.
Source and version control tools provide an effective mechanism for preserving and securing configuration items. Such tools are also able to provide an effective process for identifying changes and recovering previous copies of items that have undergone change.
The costs associated with the implementation of a configuration management system are proportional to the level of automation required. The cost of an implementation that automates the clerical aspects of configuration management using a first generation tool can be very minimal. On the other hand the cost of implementing a configuration management system using a second-generation tool can be considerably higher.
Cost elements to take into account include the following:
· Evaluation of current processes.
· Evaluation of current technical architecture.
· Defining the configuration management process.
· Tool evaluation.
· Tool procurement and implementation.
· User training.
· Tool usage.
· Tool support and maintenance.
The cost associated with the first three activities is independent of the final solution and represents a minimal investment when compared with the cost of the complete solution. For most organisations this overall cost of performing the first three tasks should be of the order of 40 to 60 man-days. Whilst there may be temptation to minimise the investment in the first three activities this should be avoided. Understanding the weaknesses and strengths of the current development and configuration management procedures is essential and this should enable one to provide clear and firm requirements for the required system.
Tool evaluation can be a long and expensive process. Expenditure of the order of several hundred man days is not uncommon. An organisation that is prepared to delay tool evaluation until after the configuration management process has been implemented is likely to make considerable savings when evaluating the available tools. Organisations that have invested effort in defining their current technical architecture and development procedures may be able to minimise the cost of tool evaluations by inviting vendors to respond to the detailed requirements rather than try to define the requirements by reviewing the tools.
The cost of tool procurement and implementation can vary from a few hundred pounds per user up to several thousand. As a rule, second generation tools are much more expensive than first generation tools. Implementing a first generation tool is generally much cheaper than implementing a second-generation tool. Following are some cost areas to consider:
· User licence costs: the cost per user for a second generation tool can be up 10 times more than the cost of a first generation tool. When considering the returns on investment that this will provide it is important to identify the level of coverage that the second-generation tool can provide. Can the tool work across the complete technical platform? Will it interface with all the development tools? Will it support all the development and configuration management processes?
· Installation and configuration costs: Most first generation tools can be installed and configured with minimal effort. A number of tools can be up and running in a matter of minutes. The installation and configuration of second-generation tools is a different issue. One vendor quoted a price of £80,000 for the software and an additional £40,000 to install and configure the tool. The user opted for a first generation tool instead, at the cost of £6000 for the software and £4000 to install and configure the tool and provide all the necessary user training.
· Software requirements: Most of the second-generation tools make use of a repository that is implemented using a relational database. In certain cases the cost of the tool does not include the cost of installing the required relational database. It is important to take this cost into account.
· Hardware requirements: Both first and second-generation tools require additional disc storage and archiving requirements. Whilst both classes of tools may require additional hardware to act as a central server for the tool, second generation tools will generally require considerable more memory and processor resource. This is needed to support the extensive features provided by these tools when compared with the features supported by the first generation tools.
The cost of implementing a configuration management process using manual processes without tool support can be considerable. Employing a first generation tool will reduce the required work by automating the more mundane and repetitive processes, maintaining an audit trail for example. Employing a second-generation tool provides additional benefits by providing closer integration between the tools and the process being followed. Using a configuration management tool will impose additional workload on its users, however this will always be significantly less than the overhead required if the configuration management process is implemented manually.
First generation tools require much less support resource than second generation tools. This is mainly due to the inherent simplicity of such tools. A large financial organisation employed a single part-time administrator to support a user base of 180 users. Support of second-generation tools is much more demanding due to the complex technology the supports the tools. There is also a constant need for database administration support to manage the relational database supporting the second-generation repository based tool.
The evaluation of the technical platform, defining and piloting the configuration management process will enable one to identify the requirements for the configuration management tool. This will highlight the tools required and the functionality that will be needed.
Whilst all the configuration management processes are essential and must be implemented to enable one to provide an effective configuration management system, it is not mandatory to automate all the processes using configuration management tools. In certain situations, automating the process may be impossible or not cost effective.
Once the required tools are identified, one should review the process that these tools are expected to support and then establish the available tools. This can be performed by evaluating different tools or may be performed by inviting tool vendors to submit proposals based on the documented requirements and processes.
When requesting a proposal from a vendor it is important to confirm that the vendor has the necessary configuration management know how to analyse the requirements and propose an approach to automate them. A key criterion is the ability of the vendor to provide an effective implementation consultancy service. The vendor must be able to provide suggestions on how the tool can be used to satisfy the configuration management process and also what changes will be required to the process if their tool is selected. Some vendors provide an implementation consultancy service that is restricted to helping the client to install and administer the tool. What is required is a vendor who is able to provide guidance on how the tool will operate with the technical platform, the development tools and the development methods being employed. This vendor must also be able to suggest how one may be able to interconnect their configuration management tool with the development tools being used.
Where possible, vendors should be asked to provide presentations that demonstrate how their tool can be used within the technical platform and processes. A vendor who is unable to demonstrate support for the client’s development tools or life cycle is acknowledging:
· The tool’s inability to support the specified requirements; cannot support development within a 4GL environment for example.
· The vendor’s inability to provide an effective implementation service.
· The complexity of the customisation process that is required to enable the tool to support the client’s processes.
It is essential that the tool vendor is involved in analysing the requirements of the configuration management system and it is also essential that vendor is made to provide tangible proof that his proposed solution can support these requirements.
A tool that cannot support all the technical requirements will obviously have limited benefits and may not provide sufficient return on investment. It is important that the vendor is able to provide evidence that the tool will manage the items created by all the development tools being used. Where there are limitations it is essential that these are identified at the evaluation stage rather than after the tool has been procured.
It is important that the vendor is able to provide guidance on how to interface with other development tools and it is also essential that the vendor is able to provide estimates of how much this service is likely to cost. A vendor who is able to provide reference sites that match your technical platform is more is likely to satisfy your requirements than one who cannot provide any reference sites.
Customising some tools is a simple process, however some tools requires considerable effort to customise. This customisation may be very expensive, especially if this has to be performed by the vendor. It is also important to establish the impact of this customisation on future upgrade plans. Will the upgrade maintain the customisation or will the customisation need to be repeated each time a major upgrade is implemented?
One of the major problems that will be encountered when analysing the requirements with the vendor is the vendor’s inability to provide skilled resource with sufficient experience with the tools being used by the customer. Whilst most vendors can demonstrate considerable experience in proving how their tool can be used with a 3GL, many have considerable problems providing any form of guidance when 4GL tools are being used.
The analysis of requirements should identify the following:
Platform coverage: it is important to identify the platforms that the tool will support and those that the tool cannot.
Platform connectivity: whilst some tools will only support a limited number of platforms, the vendor may be able to provide guidance on how objects on one platform can be managed using a tool on a different platform.
Development tool coverage: it is important to identify the development tools that can be managed by the configuration management tool. Where coverage is not complete and one is forced to select a number of configuration tools from different vendors then it is important to establish the work necessary to interface the various tools together.
Process support: it is important to identify the processes that the configuration management tool will support. Where a tool requires customisation to support a process then it is important to establish the likely cost and the ease with which the tool can be customised. An important point to consider is that the configuration management process is likely to evolve and change over the years. It is essential that the tool is able to provide sufficient flexibility to support the process.
Vendor support: it is important to identify the vendor’s ability to provide an effective implementation consultancy service. All vendors are able to provide good pre-sale service, but not all vendors are able to provide an effective implementation service.
Analysing the requirements for the configuration management system and the available tools will highlight the following:
Configuration management tools can be grouped into two discrete groups. First generation tools that provide features to manage configuration items only and second generation tools that can provide features to control and define the development life cycle.
The majority of first generation tools are sold as modular stand-alone tools. One vendor provides stand alone source, build and change management tools. These tools are designed to provide minimal life cycle support and therefore impose minimal constraints on the development and configuration management processes. In essence, these tools are designed to manage the physical objects with minimal management control facilities. First generation tools also provide minimal integration. Whilst this may be viewed as a problem, one may not be able to integrate change control with source control for example, this restriction has a major benefit in that a user is able to utilise different tools from different vendors with minimal problems. Different vendors may provide the source control, build management and change control tools.
The majority of second-generation tools provide closer integration between the various tools, source control and build management for example. Such tools may provide considerable life cycle support and therefore may impose restrictions on the configuration and development processes. Whilst some tools can be customised to support different processes, this customisation is invariably subject to some constraints. The major strength of second-generation tools is derived for the level of integration that they provide between the various modules that make up the tools. This integration has one major disadvantage, inter-connectivity between second-generation tools is very limited and one is generally forced to accept a single vendor solution.
Selecting configuration management tools requires special attention for the following reasons:
· One will most probably need to procure several tools rather than one single tool.
· The tools will be required to automate several different processes. Some are predominantly technical and some are predominantly management.
· The tools will need to support development on a number of platforms.
· The tools will have to integrate with development procedures and development and support tools.
Configuration management tools will have to be employed into a dynamic environment. Everything from the platform, to the development tools, methods, and organisation are likely to change during the life span of the configuration management tool.
The above places special demands on the configuration management tool
Configuration management tools can be classified into first or second-generation tools. First generation tools are designed to automate manual and mechanical activities without constraining the process being used. Second generation tools are designed to model the development process and provide a framework within which the process and the workflow is managed.
First generation tools are very easy to implement. Most first generation tools only automate the most manual processes and therefore they can be employed into any development environment without any complications. Also such tools do not require any customisation or re-configuration if the development process changes. First generation tools are excellent in an environment where development processes are not stable or where there is a diversity of processes.
First generation tools are interchangeable. Most tools provide similar features and one is generally able to swap tools without any major impact on the project or procedures. Apart from the effort required to migrate items from one tool to another, mostly automated, minimal effort is required to re-train the developers. First generation tools are excellent when one is still unsure about what is required.
Whilst different first generation tools may provide slightly different features, the core features that they provide are very similar. It is very easy to define a single process that can be used across several first generation tools. This is particularly useful in an organisation that develops across several platforms and is forced to employ different tools from different vendors.
Second generation tools are radically different. They may be platform specific and they certainly require a well-defined process. Most tools can be configured to provide different workflow management features, however most cannot provide different workflow models within a single installation.
Second generation tools should not be deployed into an environment that does not have a stable development process. Re-engineering the development process may require re-configuration of the configuration management tools. This can often be an expensive process.
An investment into a second-generation tool is a long-term investment. An organisation should only contemplate this route if one is certain that the tool is fit for purpose and is confident that the service and support provided by the vendor is acceptable. Swapping tools can be very expensive, not to mention the adverse impact that this is likely to have on the development process.
There are additional considerations when selecting a configuration management tool. These are listed below:
· Reliability. A configuration management tool must be very reliable. A major failure in a source control tool can result in the loss of many months of development work. Also developers will not use a tool that is not reliable.
· Performance. Some tools provide dismal performance. It is important that the tool is able to support the relevant development and support activities. If a tool is going to be used to manage large projects, then it must provide adequate performance.
· Effective support. Unlike other development tools, configuration management tools require considerable support. It is important that the vendor is able to provide adequate support to ensure the effective implementation of the tool.
· Skill shortage. It is important that the organisation is able to recruit suitably skilled staff to support the selected tool. There is a remarkable skill shortage in the area of configuration management. This makes the recruitment process very difficult and expensive.
Like most process engineering and quality improvement programmes, the implementation of configuration management must be gradual and must be supported by formal training and education. Many organisations provide the necessary training but fail to provide the essential education.
A comprehensive education programme must be defined to support the roll out of the configuration management system. This must span the complete implementation process to ensure that developers, managers and users understand and appreciate the basic concepts of configuration management and why the process is required. It is also important that the expectations of all are managed to ensure that they have realised appreciation of what their role is likely to be and the level of activity that will be required to operate the configuration management system.
It is essential that all have the correct expectation and it is important that it is clear to all that the employment of configuration tools will not automate the complete process and that all are likely to spend more time and effort on configuration management. Once all staff have a good general appreciation of the process, more targeted education should follow to explain the main processes and the roles that are required within a configuration management system.
It is essential that all staff have a good appreciation of the various configuration management processes. It is not necessary for all to know how the processes will be implemented but they must appreciate the basic principles and the relationships between each process. This stage of education should be tool independent, as most staff will assume a limited set of roles within the configuration management process.
Within a configuration management system, staff will assume one or more configuration management roles. These include:
· Developer.
· Tester.
· Change manager.
· Configuration manager.
· Maintenance manager.
· Support manager.
· Librarian.
· Build manager.
· Release manager.
The second stage of education should address the rules and responsibilities of each of the user roles within the configuration management system. It is not essential to specify and define the individual tasks that each role will perform, however it is important that all staff have a good understanding of each role and what their responsibilities will be. Providing outline information on the required roles has several benefits:
· Enables all staff to appreciate the relationship between the various roles.
· Helps staff to appreciate the levels of control and their accountability.
· Provides staff with an opportunity to identify the various opportunities that will arise as the configuration management system is rolled out.
The final point is very important as many configuration management implementations are hindered due to lack of suitably educated and motivated staff. Identifying the available roles in the early stages of implementation may increase the likelihood of having the required staff as the process is implemented.
The education programme should span the complete implementation process. It is important not to cram all the education process into a short period of time, as this is likely to disrupt the development process.
In contrast to education, training aims to provide a detailed definition of how the various configuration management tasks should be performed. The training should be specific to the user role, the configuration management and development tools being used.
Generic tool training should be avoided. It is essential that the training is aimed at demonstrating how the tool should be used to satisfy the requirements of the configuration management processes being implemented.
Training should also be targeted at the development tool being used. Whilst this may complicate the training programme, it is vital that developers are taught how to use the selected configuration management tool to manage the deliverables that are produced by the development tools that they employ. Providing a 4GL developer with training that demonstrates how to use the configuration management tool with a 3GL language is totally unacceptable.
The above two requirements may impose a significant burden on the training provider, since he/she will be expected to provide training that demonstrates how the tools should be used within the frameworks defined by the configuration management process and development tools being used.
The need to provide customised training may delay the implementation process, however this is often well worth the investment and wait.
It is important to pilot the various processes on selected projects before the complete process is rolled out. When selecting pilot projects the following should be taken into consideration:
· Avoid high profile projects. Apart from increasing the risk of impeding the development process, a high profile project is more likely to be exempt from following all the required processes than any other project. This reduces the benefits of the pilot and also may act as a precedent for other lower profile projects.
· Any project at any stage of its life cycle can be used in the pilot process. There is a common misconception that a pilot project must be in its early stages. It is more beneficial to pilot the release management process within a project that is in the later stage of system testing than it is with a project that has just started the requirement stage.
· Pilot projects need not pilot all the processes within the configuration management system. It is perfectly acceptable to pilot change control on one project and source control on another.
· At the later stages of the pilot implementation, select a single small project to pilot the complete process. A small project with a short life cycle will exercise all the configuration management processes within a short period. A pilot project that spans 6 or more months is not suitable, as this would unduly delay the implementation of the configuration management system.
External consultancy may be essential in progressing the implementation of a configuration management system. There are several areas where consultancy can provide valuable benefits:
Selection of tools: Having produced the requirements for the configuration management system, an external consultancy with suitable experience may be able to provide a short list of suitable tools without the need to spend an extended period of time evaluating individual tools. Limitations of configuration management tools are rarely discovered during formal evaluations.
Appraisal of vendor responses: An independent review, of vendor responses, by an experienced consultancy can be of great benefit, especially if the organisation does not have sufficient know how to perform a formal appraisal. A detailed technical review of a vendor response may highlight certain discrepancies that may not be obvious to those with limited experience.
Training: Whilst many tool vendors are able to provide customised training, many do not have the experience to customise their training material to address the requirements of the development tools being used. Training should provide a clear statement of how the configuration management tool should be used in conjunction with the development tool, it therefore follows that the trainer must have good understanding of the development tool being employed. This is an essential requirement for organisations that employ 4GLs.
Development tool integration: This represents a major technical challenge in many organisations. How does one integrate a given 4GL with a source and version control tool? Whilst many organisations may be able to investigate and resolve this problem, an experienced consultancy may be able to provide the required answers in a single training session.
There are many professional organisations and user groups who may be able to provide much of the information required above.