Notes on Test estimation
|
Why estimate
In the wider scheme, estimating can be used to reduce risk. The rationale is that if insufficient resources or the wrong kind of resources, major defects may not be found. Leading to a potential failure of the project. When to estimate
The danger is that the estimate document is done as part of some "box ticking" exercise and is then put on the shelf and forgotten about. In this instance, the exercise may never actually have taken place. The development methodology can make a difference in the times that the estimations take place. Additionally the depth to which the estimation takes place. The traditional Waterfall method would demand that the process takes place months, even years before the testing is due to take place. There are a number of risks with this, firstly the estimate can become a document that is never revisited to take account of changes in the project. Secondly as the project progresses, the time allotted for requirements capture and development etc increase. However updating the estimate is not updated. As a consequence the time allotted for testing constantly declines. Iterative development methodologies with the constant, build a little test a little require more constant estimation. The difficulty is with trying to set the right amount of estimation for each iteration. Procuring the correct amount of resources, to be in situ at the time they are needed takes care. The most common iterative development environment is that of the Rational Unified Process (RUP). A risk driven process where the aim is to mitigate risk at the earliest possible opportunity. Iterations, within phases (Inception, Elaboration, Construtction and Transition) are the control mechanism to ensure that this happens. Testing in the RUP is present in all phases. However the degree to which it takes place, and the type of testing, change, depending on the phase. At the start of the project in Inception, a coarse estimate can be made of resources required as part of the business case. In planning each phase or iteration, a finer estimation can be made. This constant estimating over different phases can demand a large amount of historical data about previous projects. Who to estimate
Does the organisation use a top-down or a bottom-up approach? In the top down approach, estimation is undertaking by senior management at the project level. Time and other resources are allocated from this high level estimate. In the bottom up approach, testers at the lower level indicate how long they estimate their area will take, an aggregate is then taken. Both approaches have their advantages and disadvantages. In most cases a compromise is made between the two. For example a project manager will allocate a month for testing. This is then split down to finer elements. Testing staff can then feed back up their comments, after estimates and allocations may be amended. Influencing Factors
Tester Skill Level Technical skill of the tester. An inexperienced tester without training, will take longer to find all the defects in code than an experienced tester. A test team where experience of testers are at the extremes will skew the estimate greatly. Test Tool Knowledge Does everyone on the team know how to use the tools available to them? Otherwise time and training resources will be required. Maturity of Organization A good measure of the maturity of an organisation is against the Capability Maturity Model for Integration (CMMI). At the lower levels of maturity testing is either not at all or is merely repeated, no metrics are collected. Estimates in this scenario, will have a wide variability. It is highly likely that the actual time to test will not be near the estimate. And At the other extreme highly optimised organisations, quality awareness is pervasive throughout the organisation. Defects are found at the documentation stage through inspections. Metrics are collected. For these highly optimiezed organisations, estimation should be very accurate. Test Requirements Scope What kind of testing is required. Manual, automated or both? Is a large amount regression testing to be done? Are non-functional tests (load, performance or usability) to be undertaken? Domain Knowledge How much the Test Team knows about the subject matter of the application. E.g. Ex-brokers will have excellent domain or business knowledge if they are testing an insurance application. None if they are testing a satellite communication system. Test Team Organisation At the team level organisation can have an effect. For example if their is a hierachial, linear structure or more like a matrix. Additionally, are the testers encouraged to interact with analysts or developers, also involved in the project. Number of builds planned If a number have to be ran each time a build is generated, the test effort will increase dramatically. Automation could reduce this, only if the build number is high enough. Process Does the team/organization have documented or defined processes. If everyone is allowed to do their own thing, it has to be translated for their colleagues. Junior testers will face a very long and steep learning curve. Risk What are the risk levels for failure? A failure in aircraft control software is going to be much greater than for a failure in a game. Consequently the testing for aviation is much more rigourous. Regulatory Control Does the organisation have to conform to a particular industry or customer standard? Examples include IEEE 829:1998 Software Test Documentation. The UK Ministry of Defence and the US Defense Department enforce their own standards on suppliers. Physical location Are the testers in close proximity to each other? Is the Test Team close to the developers? Efficient communication is very important, and location can be a large barrier to be factored in. Language Is everyone speaking the same language? As ever more development tasks are being sent offshore, language difficulties can affect the estimate. How to estimate
Guess Of course this is the easiest and quickest way. However invariably you leaving the project open to many nasty surprises. A huge contingency will be needed. Lastly any one making a decision based on a guessed estimate, are themselves on very shaky ground.Organisations in this position would be considered very immature. They are also most likely to run into difficulties when deadlines approach. Tester/Developer Ratio The number of testers required for a project is set based on a ratio. Thus for every 3 developers we shall have 1 tester. The ratio can be defined at many levels, e.g. the same for all projects, split into project types or at the individual leve. The higher up the project chain the ratio is set, the less accurate it will be at the individual level. To refine the process, the complexity, type and risk factors of the project/s will need to be examined. More risk and complexity indicates a need for a higher tester/developer ratio. Allegorically Microsoft reckon on two testers for every one developer. This would be considered high for most organisations. The life cycle will also play a part. In the Waterfall approach, the organisation will need a sudden increase in the ratio when it comes to the integration phase. The RUP on the other hand requires a much more steady increase of the ratio as the project passses through the phases. Lastly the demarcation lines between Unit Testing, System Testing and Integration Testing need to be set. E.g. is the unit testing to be done by the "developers"? If so the tester/developer ratio can be lower. Project Staff Ratio The ratio is between the tester and all other types of project member, not just the developers. The greater complexity of calculation, i.e involving analysts and project management staff, requires higher levels of metrics to be factored in. In addition, more information is required about configuration management, reviews and project administration. Test Procedure Method The number of test procedures required to guarantee full coverage of the project. To employ this technique, the organisation would need to be structured enough to collect accurate metrics about its developments. Size of developments can be calculated from function points, lines of code or number of test procedures. Also needed are the personnel hours for the project and the indivual areas. To estimate the next project's required resources. Calculate how long on average previous tasks took. Count the number of test procedures and multiply by the average. This will give total hours e.g 4,800. Divide by the time allocated e.g. 8 months or 1,600 hour. Thus 3 full time testers. Task Planning Method Again Task Planning requires a very structured organisation, as it requires a history of accurately recorded metrics. In this case, how long it takes to undertake a particular task. Instead of the procedures, the base is that of testing tasks. Such as automation of testing, start up procedures, planning etc. A total number of hours for the project is calculated. (Possibly using the Test Procedures method.) The hours are then apportioned according to the historical percentage for a project type of the current one. This gives a number of hours for each task. The next stage is to adjust the figures to take account of project specific details. Tasks that are not to be done, are removed, and the total hours re-calculated. The hours are then divided to show how many testers are required.
|
Similar Articles Selected Books
|