Summary of testing techniques

You can use a variety of testing techniques in your test plan. Table 6-1 summarizes the advantages and disadvantages of various testing techniques and tools:

Table 6-1: Testing techniques’ advantages and disadvantages

Technique

Description

Advantages

Disadvantages

Ad hoc testing

Manually walk through important application processes, screens, and reports

  • Easy to implement

  • Tests front-end applications and back-end servers

  • For complex applications, code coverage is too small

  • Difficult to distinguish front-end and back-end bottlenecks if response time is a determining factor

  • Impossible to obtain production multiuser load, which misses concurrency and capacity issues altogether

Manual performance scripts and cases

Specify input and compare with known outputs

  • Easy to implement

  • Common basis for regression test suites

  • Back-end focus may help locate the cause of a problem

  • Only tests backend server

  • Impossible to obtain production multiuser load, which misses concurrency and capacity issues altogether

  • No ad hoc query testing

  • Depends on strong analysis of process or transaction profiles

Keystroke capture

Record and replay keystrokes and mouse clicks into an application

  • Tests front-end applications and back-end servers

  • Tool may include powerful language and looping capabilities to manipulate inputs for multiuser concurrency and capacity testing

  • Heavy processing requirements, may require additional hardware

  • May increase development time for creating multiuser test simulations, and add time for debugging test harness

  • Depends on strong analysis of process or transaction profiles

Concurrency and capacity testing

Use third-party load-testing tools

  • Tests both front and back ends

  • Language and looping constructs make these tools

  • Heavy processing requirements for test tool—may even require additional hardware, or else results may be skewed

  • Learning and development curve to write multiuser test simulations

  • Risk of bugs in test harness may skew results

  • Dependent on strong analysis of process/transaction profiles

Transaction generation

Thin client to simulate user execution of transactions

  • Strong multiuser load testing

  • Focus on back-end server issues

  • May increase development time for creating multiuser test simulations, though learning and development curve generally less than keystroke capture tools

  • Adds time for debugging test harness to prevent skewed results

  • Depends on strong analysis of process or transaction profiles

Production load capture

Use tools to capture real transactions in a production environment, including performance and semantic characteristics, and resubmit in a test environment for analysis

  • Tests real production loads, including ad hoc queries

  • Especially useful when little or no analysis of transaction profiles is available

  • Introduces new software into a production environment

  • Production and test system configurations must be identical for valid performance analysis