Testing Procedure

Testing Process

1. Introduction

Testing is an important stage of s/w life cycle. There are two kinds of testing possible. One is validation and other one verification. Verification is doing right things. Validation is doing things right. Here the testing process is to the validate the source code. This process is only for manual testing.

1.1. What is Testing?

Testing is the process of operating a system or component under specified conditions, observing or recording the results and making an evaluation of some aspects of the system or component. (IEEE/ANSI, Std-10.12-1990)

Testing is an important stage in the software development process. Testing consumes a significant human effort. Of the total software development effort, testing takes 35 to 50%. Testing is a very important element in the QA strategy of any software development organization.

1.2. The Testing Process

The Testing Process is generally described by the V model shown below.

Testing is normally done with a bottom-up approach. The first step in testing is Unit Testing in which the individual functions are tested. During integration, these functions are integrated into a system. During Integration testing proceeds incrementally. At the end of integration, the system as a whole is tested. This stage is called System Testing. The testing activities are thus done in three distinct stages. Each of these stages are important.

1.3. Testing Techniques

The Process of Testing checks the implementation using the desired specification as a reference with an intent to find defects in the implementation. The Implementation must fully satisfy the Requirements. While translating the requirements into implementation, the following are the possibilities:

1. Correct Implementation :Some requirements are implemented correctly

2. Incorrect Implementation :Some requirements are implemented wrongly

3. Omissions :Some requirements are not implemented and omitted by mistake

4. Surprises :Some implementation do not correspond to any requirement

Different testing techniques will have to be used to handle these different possibilities. Omissions, Surprises and Incorrect Implementation must be detected and the implementation must be corrected.

The following matrix shows which technique is useful for what type of problem.

Ability to Catch Incorrect

Testing Approach Omissions Surprises Implement.

Black box/Functional Testing yes yes

White box/Implementation based Testing yes

Heuristic Testing yes yes yes

The testing strategy must take this fact into account and use complementary testing techniques to unearth all the problems. Thus effective Testing is a combination of Black box, White box and Heuristic Testing Techniques.

1.1. Top down and Bottom Up Integration

Integration of software may be done in a top down or bottom up fashion. In Bottom up testing all lowest level units are tested first. Then the higher level units are tested. The units already tested are used during testing higher level modules. Stubs are not needed for bottom up testing. In top down testing, the top most layer of Units are tested first followed by the next lower layer of units. While testing the second layer of units, the already tested top layer units are used. There is no need for Drivers in the top down testing.

1.2. Unit Testing

An Unit is a basic functional block in software. It is a function, procedure or a subroutine in a programming language like C. It forms the basic Unit by which blocks, subsystems or systems are built. Unit Testing is the first stage of testing. It immediately follows the coding stage. As discussed before, Black box, White box and Heuristic Testing must be used to expose all the problems in the Unit under test. Every Unit occupies an unique position in a hierarchy of units. Before integrating the units together, it is very desirable that each unit is tested individually. The Unit Testing stage ensures that an Unit behaves correctly before starting to test the interactions between different units by integration testing.

Some engineers follow a testing strategy which combines unit and integration testing. Whichever strategy is followed, one objective is to look for problems within the individual units and the second objective is to look for problems in the manner in which these units interact. Unit Testing meets the first objective while integration testing meets the second objective.

1.3. Black box Testing

Black box or responsibility based testing views the unit or system under test as a Black box and involves test cases which exercise the responsibilities to be met by the unit or system under test. In the case of units, the Input output specifications of an individual unit is taken and the unit is subjected to a set of cases that attempt to find omissions and defects in the implementation vis a vis the requirements.

The most important Black box testing techniques include Equivalence Partitioning and Boundary Value Analysis.

1.4. White box Testing

White box testing assumes a knowledge of the software under test. It is also called structural testing sometimes. The structure of the software under test is used to generate test cases to problems of surprises and incorrect implementation. The white box testing techniques typically involve exercising the different sections of the code systematically. The structure of the software and ways in which they may be exercised vary so much that a large number of test cases are necessary to test “all possible combinations”. This is not necessary and instead an increasingly comprehensive coverage criteria are defined which can be used while identifying White box test cases. These criteria are called the coverage criteria. The well accepted coverage criteria include:

· Statement Coverage

· Branch Coverage

· Decision Coverage

· Loop Coverage

· LCSAJ Coverage

2. Unit testing - in General

Identify Black Box Test Cases

Identify White Box Test cases

1.1. General Procedure for Unit Testing

1. Identify Black Box Test Cases

1.1. Identify all the input and output data entry fields for the selected unit. There may be some indirect data fields also. (ex. Additem_to_a_linked_list, or a functions return value, setting a variable indirectly through the called function.)

1.2. For each of the identified data fields, identify the possible values that these data fields can hold (both in the input and output space).

Note : Here, you have to use the following testing techniques in the given order, to identify black box test cases.

Equivalence Class

Boundary values

Heuristic

1.3. Take a particular field as focus and list the possible values for that variable. Repeate the setps 2 to 5 for all the identified data fields.

2. Generate Black Box Test Cases

2.1. Generate a test case by assigning a value from the list of possible values for the selected variable and assigning any one of the possible values for all other data entry fields in input and output. This is called a test case definition.

3. Document the Generated Test Cases

3.1. The test case definitions should be entered into the Test Case Entry Template (given in subsequent sections).

4. Identify Called functions

4.1. Identify called functions and decide whether they have to be called as actual functions or they need stubbing for each test case.

4.2. If stubbing is necessary, for those function calls, select the necessary variable and decide the necessary values for those data entry fields also.

5. Write drivers and stubs

5.1. Write driver code using the above test case definitions.

5.2. Write stubs with the above test case definitions obtained from step 2.7, and modify the unit accordingly to support the stub calls instead of actual calls, is necessary. These details also should be added to the test case definition.

6. Execute Test cases

6.1. Execute the test case(s).

6.2. Analyze the result for expected result after executing all the test cases.

7. Establish coverage

7.1. To establish coverage, while generating drivers and stubs itself, instrument the test source so that it prints the branch and node it has traversed. The same thing can also be done by debugging the test source.

7.2. While executing the test cases, note down the path taken by seeing the printed mater on screen or by stepping through the FUT.

Note : Instead of the above two steps, you can use FasTest to just establish the coverage by entering the identified testcases and executing them(incase of ‘c’).

8. Identify additional test case for coverage

8.1.

If desired level of coverage is not achieved, identify additional paths and branches which are left not covered by the currently available set of test case definitions.

8.2.

Identify additional test case to cover the remaining statements/ branches /paths.

8.3.

This can be done by slightly modifying the test cases which has covered the adjacent path.

9.

Generate White box Test cases:

9.1.

Generate additional test cases by modifying existing test cases or generate new test cases to cover entirely new path. The test case should be generated in the given order

Coverage techniques

Statement coverage

Branch coverage

Path coverage.

Deccanet’s standard is Branch coverage unless otherwise specified for a particular project.

9.2.

After this goto setp PU_G5

1.1. General Rules for Unit Testing(RU-G)

RU_G1 The unit under test should be compliable

RU_G2 The unit should have passed the normal inspection review process

RU_G3 Identify and remove duplicate test cases(keep the no. of test case less)

RU_G4 Test cases should have a reasonable chance of catching defects.

RU_G5 Possible values of each data entry fields should lie with in the acceptable range of the underlying data type.

RU_G6 Consider all the direct and indirect data entry fields as a focus variable.

RU_G7 Consider both input and output variables / events as focus and generate test cases around all of them.

RU_G8 Don't omit any test case generated for each focused variable’s possible values.

RU_G9 Don’t forget to handle any of the called functions.

RU_G10 Initialize all the pointers with appropriate value.

RU_G11 Don’t generate too complex or too trivial test cases.

RU_G12 Ensure that the standards are met from coverage point of view.

1.2. General checklist for Unit testing (CU-G)

CU_G1 Whether the unit under test is compliable or not.

CU_G2 Confirm that the unit under test had already been subjected to normal inspection / review process.

CU_G3 Check whether the test case is too complex or too trivial?

CU_G4 Check whether a redundant test case exists?

CU_G5 Check whether the test cases have a reasonable change of catching defects?

CU_G6 Verify that the possible values of each data entry fields lie with in the acceptable range of the underlying data type.

CU_G7 Check whether each of the direct and indirect data entry fields are considered as a focus variable.

CU_G8 Check whether all the possible test cases are there for each of the focused variables.

CU_G9 Check whether all the called functions are handled properly by either stubbing it or calling it as an actual function.

CU_G10 Check whether all the pointers are initialized properly?

CU_G11 Verify whether the standards are met from coverage point of view.

1. Equivalence Testing

1.1. Definition

If you expect a no. of test cases to be testing the same behavior or the Function Under Test(FUT), then you must treat them as equivalent. A group of tests forms an equivalence class if you believe that :

· they all test the same thing

· if one test catches a bug, the others probably will too

· if one test doesn’t catch a bug, the others probably won’t either.

[ref. : 126 of Cem Kaner]

1.2. Procedure for Equivalence Testing (PU-EQ)

Follow the general procedure for unit testing. In addition to that, follow the procedure given below, to identify equivalence classes:

1. Analyze the possible values that can be entered for each data fields(both in input and output space) and group those values into possible equivalence classes.

2. To group the values into equivalence classes, decided whether the possible values of the identified data field fall under any one of the following category

· Range of value

· List of values

· Specific value

3. As per the rules for the corresponding category define the test cases.

4. For all the data entry field (in the input and output space) repeat the steps 2 and 3 until all the variables are subjected to this procedure.

1.1. Rules for Equivalence Class Identification (RU-EQ)

RU_EQ1 For any variable whose possible values lies in a range, then generate 3 equivalence classes.

Ex: If your possible values lies in a range (say 10 to 90) for a particular variable, then the equivalence classes are

· Less than 10 Invalid equivalence class

· Greater than 90 Invalid equivalence class

· 10 to 90 Valid equivalence class

RU_EQ2 For any variable, if the possible range of values can be further subdivided into ‘n’ sub ranges, then define n+2 equivalence classes.

If the range can be further sub divided into sub ranges(say 10-30, 30-60 60-90), then consider the following equivalence classes also

· 10 - 30 Valid Equivalence class

· 31-60 Valid equivalence class

· 61-90 Valid equivalence class

· Less than 10 Invalid equivalence

· Greater than 99 Invalid equivalence class

RU_EQ3 For any variable, if the possible value is a list of ‘n’ items, then ‘n+1’ equivalence class should be defined, if the behavior of the FUT is expected to be different for each.

Ex: If the possible values lie in a list of n item, then n+1 equivalence class should be generated with each list item as an individual valid equivalence class and an item not present in the list should form the n+1th equivalence class, which is an invalid equivalence class.

Ex: If your possible values lie in a list of items, (say a, m, k) then the equivalence classes are

· Each of the list items is a valid equivalence class (a, m, k - 3 valid ECs )

· Items which are not there in the list forms one Invalid equivalence class

RU_EQ4 If the possible value is a specific values, then two equivalence class should be generated one with the specific value and other class with any values other than the specific

Ex: If your possible value is a specific value (say 0 always), then the equivalence classes are

· specific value (i.e.) 0 Valid equivalence class

· all values other than 0 Invalid equivalence class

RU_EQ5 If the input specifies the number(N) of valid values, define one valid Equivalence class and two invalid equivalence classes.

EX : if the valid name of project requires at least one letter and not more that 8 characters then

any name with 1 - 8 letters Valid equivalence class

any other combination of characters Invalid equivalence class

RU_EQ6 Don’t forget to test invalid equivalence classes for any variable.

RU_EQ7 Don’t omit sub equivalence classes within valid equivalence classes

RU_EQ8 Identify time-determined equivalence classes(ex. press the shift key just before, during, and just after the system finishes scanning for keyword)

RU_EQ9 Don’t forget to identify indirect(hidden) equivalence classes

Ex. if the function deals with insertion of an item to linked list, then adding an item to the list when the list contains less than maximum no. of elements becomes an equivalence class. Adding an item to the list, when the list contains maximum no. of elements.

RU_EQ10 Don’t forget to focus on output variables. Consider all the output variables also for generating equivalence classes. Apply all the above rules on these variables also.

RU_EQ11 Some times, a group of variable may gives raise to a possible value for a particular situation. In this case also, apply all the above rule on these group of variables.

2. Boundary Value Testing

2.1. Definition

Boundary value testing is a variant and refinement of equivalence partitioning, with a major difference - Rather than selecting any element in an equivalence class as being representative, elements are selected such that each edge of the equivalence class is the subject of a test. Boundaries are always a good place to look for defects.

2.2. Procedure for Boundary Test case Identification

In addition to the general procedures, the following should also be considered for boundary test case generation

1. Analyze the possible values that can be entered for each fields(both in input and output space) and group those values into possible boundaries with precise lower and upper limits.

2. To group the values into boundary test cases, decide whether the possible values of the identified data filed falls under any one of the following category :

· Range of Values

· List of continuous values

3. As per the rules for the corresponding category, define the test cases

4. For all the data entry fields, repeat steps 2 to 3 until all the variables( in the inputs and output space) are subjected to this procedure.

2.3. Rules for Boundary Test case Identification :

RU_BV1 For any variable whose possible values lie in a range, then generate 3 equivalence classes.

Ex :If your possible values lies in a range, say 10 to 90 then the boundary classes are

· 9,91 Invalid boundary values

· 10, 11, 89, 90 Valid boundary values

RU_BV2 For any variable, if the possible range of values can be further subdivided into ‘n’ sub ranges, then ‘n+2’ equivalence classes should be defined.

Ex : If the range can be further sub divided into sub ranges(say 10-30, 30-60 60-90), then consider the following boundary test cases also

· 9, 91 Invalid boundary test cases

· 10, 11, 29, 30 Valid boundary test cases

· 31, 32, 59,60 Valid boundary test cases

· 61,62, 89,90 Valid boundary test cases

RU_BV3 For any variable if the possible value lies in a continuous list of n items, then define the boundary test cases by keeping the following consideration in mind.

Ex : If your possible values are a list of continuous items ( say m,n,o,p), then the boundary classes are

· Each of the list items is a valid boundary class (m,n,o,p - valid boundary cases)

· One item below the lower boundary and one item above the upper boundary list forms one Invalid equivalence class (l, q - Invalid boundary cases)

RU_BV4 Consider the no. of inputs to a program as an input data field and try defining test cases around the limit of the no. of inputs

Ex: Some times no. of inputs to a program may gives raise to a test case. In this case, consider the range from 1 to n as a valid range, give 0, 1,2,n-1, n and n+1 inputs for the program and test it.

RU_BV5 Think of boundaries on hidden data fields, which are not explicit as a defined variable in the project, and generate test cases around the boundary of possible values of that hidden variable.

Ex: If the functionality of the function is to add an item to linked_list, then add and item when the list is empty, when there is an item, when there is n-1 item where n is the maximum capacity of the list). This will form 3 valid boundary cases. Add an item when the list is full. This is an invalid boundary case.

RU_BV6 Look for boundaries around the loop variable limits( if any loop exists in the FUT).

Ex: If any loop is there in the FUT, the executed the loop for 0 times, 1 item, n-1 time and n times, where n is the maximum no. of times the loop has to be executed.

RU_BV7 Don’t forget to test invalid boundary cases for any variable.

RU_BV8 Don’t omit any valid boundary cases also.

RU_BV9 Don’t forget to generate test cases for sub ranges within a range.

RU_BV10 Look boundaries around each valid equivalence classes

RU_BV11 Identify time-determined boundary classes.

Ex: Assume, the first digit has to be dialed within 5 sec after getting dial tone, then press the first digit exactly at 5^th sec just before and after 5 sec.

3. Heuristic Testing

3.1. Definition

Heuristic Testing is an approach, based on intuition and experience, to identify tests that are considered likely to expose errors. The basic idea is to make a list of possible errors or error-prone situations and then develop tests based on the list. With the past experience we need to give some inputs to the present program, so as to check whether the most common defects still exists or not.

3.2. Procedure for Heuristic Test case identification:

Identify test case based on your judgement and previous experience

3.3. Rules for Heuristic Test cases identification (RU-HR)

In addition to the general rules, the following rules also should be followed to generate heuristic test cases

RU_HR1 Don’t omit any variable which are directly or indirectly affecting the functionality of the Function Under Test.

RU_HR2 Test the unit with some random values for all the variables, without focusing on any particular variable.

RU_HR3 Any interesting feature, combinations, sequence which you feel may hold a surprise should be exercised.

RU_HR4 Check for rarely executed conditions and sequences.

RU_HR5 Identity test cases by recollecting situations from past experiences

RU_HR6 Try to create possible error conditions.

RU_HR7 Identify test case which seem to be “interesting” to you.

4. Coverage Testing

4.1. Definition

These are the testing techniques that takes advantages of the structure of the program under test are grouped under white box or structural testing. Structural testing complements black box testing

4.2. General Procedures for all coverage test cases

Execute the setps explined in the section PU_G (PU_G14 to PU_G18)

4.3. General rules for While box testing (RW-G)

RW_G1.Confirm that the identified test cases are covering expected branches / statements.

RW_G2. Don’t generate duplicate , trivial test cases which does not give an incremental coverage

RW_G3. Ensure that the desired level coverage is achieved by meeting the organization standards

RW_G4. Give more attention to the rarely accessed branches and statements

RW_G5. Unreachable code, if any, should be properly commented, so that the coverage calculation are not affected

4.4. General check List for white box testing (CW-G)

CW_G1 Check for duplicate , trivial test cases

CW_G2 Check whether all the branches are covered

CW_G3 Check whether identified TCs are covering the expected branches /statements or not

CW_G4 Verify whether all the rarely accessed branches and statements are given more attention

CW_G5 Check whether if any unreachable code left uncommented

Note : Here there is no need for specific procedure /rule and checklist for each coverage techniques and for all techniques the same procedures and rules are applicable.

5. Glossary:

FUT : Function Under Test

LCSAJ : Linear Code Segment And Jump

EC : Equivalence Classes

BV : Boundary Values

TC : Test Case

Organizational Standards : 100 % Statement Coverage

100 % Branch Coverage