Question

Develop a test strategy for testing the entire application of a college course registration system. Specifically define roles, responsibilities, timing, and test strategy for each level of testing.

Answer 1

Test Requirements

The following have been identified as targets for testing. This gives an idea of what will be tested.

Data and Database Integrity Testing

Verify access to Course Catalog Database.

Verify simultaneous record read accesses.

Verify lockout during Course Catalog updates.

Verify correct retrieval of update of database data.

System Testing / Functional testing

Verify Login Use Case

Verify Course Registration Use Case

Verify Maintain Student Information Use Case

Verify Maintain Professor Information Use Case

Verify Submit Grades Use Case

Verify View Report Card Use Case

Verify Select Courses to Teach Use Case

Business Cycle Testing

Verify operation following download of a new course catalog.

Verify operation across multiple semesters and multiple years.

Verify correct operation when semester spans year rollover.

User Interface Testing

Verify ease of navigation through a sample set of screens.

Verify sample screens conform to GUI standards.

Performance Testing

Verify response time to access external Finance system.

Verify response time to access external Course Catalog subsystem.

Verify response time for remote login.

Verify response time for remote submittal of course registration.

Load Testing

Verify system response when loaded with more number of logged on students (say around 200).

Verify system response when more number of students (say around 50) simultaneously accesses to the Course Catalog.

Stress Testing

Verify system response during prime time use

Verify system response during maximum student logins.

Volume Testing

Verify system response when Course Catalog Database at 90% capacity.

Security and Access Control Testing

Verify Logon from a local PC.

Verify Logon from a remote PC.

Verify Logon security through user name and password mechanisms.

Failover / Recovery Testing

Verify that the system shall be available 24X7.

Configuration Testing

Verify that the system runs successfully on all the supported OS

Installation Testing

Verify installation of server portion.

Verify installation of client portion.

Test Strategy

The following test strategy is generic in nature and is meant to apply for the general requirement considerations of College Course Registration System

Data and Database Integrity Testing

The databases and the database processes should be tested as separate systems. These systems should be tested without the applications (as the interface to the data). Additional research into the DBMS needs to be performed to identify the tools / techniques that may exist to support the testing identified below.

Test Objective:	Ensure Database access methods and processes function properly and without data corruption.
Technique:	Invoke each database access method and process, seeding each with valid and invalid data (or requests for data). Inspect the database to ensure the data has been populated as intended, all database events occurred properly, or review the returned data to ensure that the correct data was retrieved (for the correct reasons)
Completion Criteria:	All database access methods and processes function as designed and without any data corruption.
Special Considerations:	Testing may require a DBMS development environment or drivers to enter or modify data directly in the databases. Processes should be invoked manually. Small or minimally sized databases (limited number of records) should be used to increase the visibility of any non-acceptable events.

System Testing

The goals of these tests are to verify proper data acceptance, processing, and retrieval, and the appropriate implementation of the business rules. This type of testing is based upon black box techniques, that is, verifying the applicationby interacting with the application via the GUI and analyzing the output. Identified below is an outline of the testing recommended for each application:

Test Objective:	Ensure proper application navigation, data entry, processing, and retrieval.
Technique:	Execute each use case, use case flow, or function, using valid and invalid data, to verify the following: The expected results occur when valid data is used. The appropriate error / warning messages are displayed when invalid data is used. Each business rule is properly applied.
Completion Criteria:	All planned tests have been executed. All identified defects have been addressed.
Special Considerations:	Access to the Server and the existing Course Catalog System and Billing System is required.

Business Cycle Testing

Business Cycle Testing should emulate the activities performed on the system over time. A period should be identified, such as one year, and transactions and activities that would occur during a year's period should be executed. This includes all daily, weekly, monthly cycles and events that are date sensitive, such as ticklers.

Test Objective	Ensure proper application and background processes function according to required business models and schedules.
Technique:	Testing will simulate several business cycles by performing the following: The tests used for application function testing will be modified / enhanced to increase the number of times each function is executed to simulate several different users over a specified period. All time or date sensitive functions will be executed using valid and invalid dates or time periods. All functions that occur on a periodic schedule will be executed / launched at the appropriate time. Testing will include using valid and invalid data, to verify the following: The expected results occur when valid data is used. The appropriate error / warning messages are displayed when invalid data is used. Each business rule is properly applied.
Completion Criteria:	All planned tests have been executed. All identified defects have been addressed.
Special Considerations:	System dates and events may require special support activities Business model is required to identify appropriate test requirements and procedures.

User Interface Testing

User Interface testing verifies a user's interaction with the software. The goal of UI Testing is to ensure that the User Interface provides the user with the appropriate access and navigation through the functions of the applications. In addition, UI Testing ensures that the objects within the UI function as expected and conform to corporate or industry standards.

Test Objective:	Verify the following: Navigation through the application properly reflects business functions and requirements, including window to window, field to field, and use of access methods (tab keys, mouse movements, accelerator keys) Window objects and characteristics, such as menus, size, position, state, and focus conform to standards.
Technique:	Create / modify tests for each window to verify proper navigation and object states for each application window and objects.
Completion Criteria:	Each window successfully verified to remain consistent with benchmark version or within acceptable standard
Special Considerations:	Not all properties for custom and third party objects can be accessed.

Performance Testing

Performance testing measures response times, transaction rates, and other time sensitive requirements. The goal of Performance testing is to verify and validate the performance requirements have been achieved. Performance testing is usually executed several times, each using a different "background load" on the system. The initial test should be performed with a "nominal" load, similar to the normal load experienced (or anticipated) on the target system. A second performance test is run using a peak load.

Additionally, Performance tests can be used to profile and tune a system's performance as a function of conditions such as workload or hardware configurations.

Test Objective:	Validate System Response time for designated transactions or business functions under the following two conditions: - normal anticipated volume - anticipated worse case volume
Technique:	Use Test Scripts developed for Business Model Testing (System Testing). Modify data files (to increase the number of transactions) or modify scripts to increase the number of iterations each transaction occurs. Scripts should be run on one machine (best case to benchmark single user, single transaction) and be repeated with multiple clients (virtual or actual, see special considerations below).
Completion Criteria:	Single Transaction / single user: Successful completion of the test scripts without any failures and within the expected / required time allocation (per transaction) Multiple transactions / multiple users: Successful completion of the test scripts without any failures and within acceptable time allocation.
Special Considerations:	Comprehensive performance testing includes having a "background" load on the server. There are several methods that can be used to perform this, including: "Drive transactions" directly to the server, usually in the form of SQL calls. Create "virtual" user load to simulate many (usually several hundred) clients. Remote Terminal Emulation tools are used to accomplish this load. This technique can also be used to load the network with "traffic." Use multiple physical clients, each running test scripts to place a load on the system. Performance testing should be performed on a dedicated machine or at a dedicated time. This permits full control and accurate measurement. The databases used for Performance testing should be either actual size, or scaled equally.

Load Testing

Load testing measures subjects the system-under-test to varying workloads to evaluate the system's ability to continue to function properly under these different workloads. The goal of load testing is to determine and ensure that the system functions properly beyond the expected maximum workload. Additionally, load testing evaluates the performance characteristics (response times, transaction rates, and other time sensitive issues).

Test Objective:	Verify System Response time for designated transactions or business cases under varying workload conditions.
Technique:	Use tests developed for Business Cycle Testing. Modify data files (to increase the number of transactions) or the tests to increase the number of times each transaction occurs.
Completion Criteria:	Multiple transactions / multiple users: Successful completion of the tests without any failures and within acceptable time allocation.
Special Considerations:	Load testing should be performed on a dedicated machine or at a dedicated time. This permits full control and accurate measurement. The databases used for load testing should be either actual size, or scaled equally.

Stress Testing

Stress testing is intended to find errors due to low resources or competition for resources. Low memory or disk space may reveal defects in the software that aren't apparent under normal conditions. Other defects might results from competition for shared resource like database locks or network bandwidth. Stress testing identifies the peak load the system can handle.

Test Objective:	Verify that the system and software function properly and without error under the following stress conditions: little or no memory available on the server (RAM and DASD) maximum (actual or physically capable) number of clients connected (or simulated) multiple users performing the same transactions against the same data / accounts worst case transaction volume / mix (see performance testing above). NOTES: Stress testing's goal might also be stated as identify and document the conditions under which the system FAILS to continue functioning properly.
Technique:	Use tests developed for Performance Testing. To test limited resources, tests should be run on single machine, RAM and DASD on server should be reduced (or limited). For remaining stress tests, multiple clients should be used, either running the same tests or complementary tests to produce the worst case transaction volume / mix.
Completion Criteria:	All planned tests are executed and specified system limits are reached / exceeded without the software or software failing (or conditions under which system failure occurs is outside of the specified conditions).
Special Considerations:	Stressing the network may require network tools to load the network with messages / packets. The DASD used for the system should temporarily be reduced to restrict the available space for the database to grow. Synchronization of the simultaneous clients accessing of the same records / data accounts.

Volume Testing

Volume Testing subjects the software to large amounts of data to determine if limits are reached that cause the software to fail. Volume testing also identifies the continuous maximum load or volume the system can handle for a given period. For example, if the software is processing a set of database records to generate a report, a Volume Test would use a large test database and check that the software behaved normally and produced the correct report.

Test Objective:	Verify that the application / system successfully functions under the following high volume scenarios: maximum (actual or physically capable) number of clients connected (or simulated) all performing the same, worst case (performance) business function for an extended period. maximum database size has been reached (actual or scaled) and multiple queries / report transactions are executed simultaneously.
Technique:	Use tests developed for Performance Testing. Multiple clients should be used, either running the same tests or complementary tests to produce the worst case transaction volume / mix (see stress test above) for an extended period. Maximum database size is created (actual, scaled, or filled with representative data) and multiple clients used to run queries / report transactions simultaneously for extended periods.
Completion Criteria:	All planned tests have been executed and specified system limits are reached / exceeded without the software or software failing.
Special Considerations:	What period of time would be considered an acceptable time for high volume conditions (as noted above)?

Security and Access Control Testing

Security and Access Control Testing focus on two key areas of security:

- Application security, including access to the Data or Business Functions, and
- System Security, including logging into / remote access to the system.

Application security ensures that, based upon the desired security, users are restricted to specific functions or are limited in the data that is available to them. For example, everyone may be permitted to enter data and create new accounts, but only managers can delete them. If there is security at the data level, testing ensures that user "type" one can see all customer information, including financial data, however, user two only sees the demographic data for the same client.

System security ensures that only those users granted access to the system are capable of accessing the applications and only through the appropriate gateways.

Test Objective:	Function / Data Security: Verify that user can access only those functions / data for which their user type is provided permissions. System Security: Verify that only those users with access to the system and application(s) are permitted to access them.
Technique:	Function / Data Security: Identify and list each user type and the functions / data each type has permissions for. Create tests for each user type and verify permission by creating transactions specific to each user type. Modify user type and re-run tests for same users. In each case verify those additional functions / data are correctly available or denied. System Access (see special considerations below)
Completion Criteria:	For each known user type the appropriate function / data are available and all transactions function as expected and run in prior Application Function tests
Special Considerations:	Access to the system must be reviewed / discussed with the appropriate network or systems administrator. This testing may not be required as it maybe a function of network or systems administration.

Failover / Recovery Testing

Failover / Recovery testing ensures that an application or entire system can successfully failover and recover from a variety of hardware, software, or network malfunctions with undue loss of data or data integrity.

Failover testing ensures that, for those systems that must be kept running, when a failover condition occurs, the alternate or backup systems properly "take over" for the failed system without loss of data or transactions.

Recovery testing is an antagonistic test process in which the application or system is exposed to extreme conditions (or simulated conditions) such as device I/O failures or invalid database pointers / keys. Recovery processes are invoked and the application / system is monitored and / or inspected to verify proper application / system / and data recovery has been achieved.

Test Objective:	Verify that recovery processes (manual or automated) properly restore the database, applications, and system to a desired, known, state. The following types of conditions are to be included in the testing: Power interruption to the client Power interruption to the server Communication interruption via network server(s) Interruption, communication, or power loss to DASD and or DASD controller(s) Incomplete cycles (data filter processes interrupted, data synchronization processes interrupted). Invalid database pointer / keys Invalid / corrupted data element in database
Technique:	Tests created for Application Function and Business Cycle testing should be used to create a series of transactions. Once the desired starting test point is reached, the following actions should be performed (or simulated) individually: Power interruption to the client: power the PC down Power interruption to the server: simulate or initiate power down procedures for the server Interruption via network servers: simulate or initiate communication loss with the network (physically disconnect communication wires or power down network server(s) / routers). Interruption, communication, or power loss to DASD and or DASD controller(s): simulate or physically eliminate communication with one or more DASD controllers or devices. Once the above conditions / simulated conditions are achieved, additional transactions should executed and upon reaching this second test point state, recovery procedures should be invoked. Testing for incomplete cycles utilizes the same technique as described above except that the database processes themselves should be aborted or prematurely terminated. Testing for the following conditions requires that a known database state be achieved. Several database fields, pointers and keys should be corrupted manually and directly within the database (via database tools). Additional transactions should be executed using the tests from Application Function and Business Cycle Testing and full cycles executed.
Completion Criteria:	In all cases above, the application, database, and system should, upon completion of recovery procedures, return to a known, desirable state. This state includes data corruption limited to the known corrupted fields, pointers / keys, and reports indicating the processes or transactions that were not completed due to interruptions.
Special Considerations:	Recovery testing is highly intrusive. Procedures to disconnect cabling (simulating power or communication loss) may not be desirable or feasible. Alternative methods, such as diagnostic software tools may be required. Resources from the Systems (or Computer Operations), Database, and Networking groups are required. These tests should be run after hours or on an isolated machine(s).

Configuration Testing

Configuration testing verifies operation of the software on different software and hardware configurations. In most production environments, the particular hardware specifications for the client workstations, network connections and database servers vary. Client workstations may have different software loaded (e.g. applications, drivers, etc.) and at any one time many different combinations may be active and using different resources.

Test Objective:	Validate and verify that the client Applications function properly on the prescribed client workstations.
Technique:	Use Integration and System Test scripts Open / close various PC applications, either as part of the test or prior to the start of the test. Execute selected transactions to simulate user activities into and out of various PC applications. Repeat the above process, minimizing the available conventional memory on the client.
Completion Criteria:	For each combination transactions are successfully completed without failure.
Special Considerations:	What PC Applications are available, accessible on the clients? What applications are typically used? What data are the applications running (i.e. large spreadsheet opened in Excel, say 100 page document in Word etc). The entire systems, network servers, databases, etc. should also be documented as part of this test.

Installation Testing

Installation testing has two purposes. The first is to insure that the software can be installed on all possible configurations, such as a new installation, an upgrade, and a complete installation or custom installation, and under normal and abnormal conditions. Abnormal conditions include insufficient disk space, lack of privilege to create directories, etc. The second purpose is to verify that, once installed, the software operates correctly. This usually means running a number tests that were developed for Function testing.

Test Objective:	Verify and validate that the client software properly installs onto each client under the following conditions: New Installation, a new machine, never installed. Update machine previously installed with same version Update machine previously installed with older version
Technique:	Manually or develop automated scripts to validate the condition of the target machine (new - never installed, same version or older version already installed). Launch / perform installation. Using a predetermined sub-set of Integration or System test scripts, run the transactions.
Completion Criteria:	Transactions execute successfully without failure.
Special Considerations:	What transactions should be selected to comprise a confidence test that the application has been successfully installed and no major software components are missing?