Summary of Testing Techniques

Ad hoc testing

Manually walk through important application processes, screens, and reports

• Easy to implement

• Tests front-end applications and back-end servers

• For complex applications, code coverage is too small

• Difficult to distinguish front-end and back-end bottlenecks if response time is a determining factor

• Impossible to obtain production multiuser load, which misses concurrency and capacity issues altogether

Manual performance scripts and cases

Specify input and compare with known outputs

• Easy to implement

• Common basis for regression test suites

• Back-end focus may help locate the cause of a problem

• Only tests back- end server

• Impossible to obtain production multiuser load, which misses concurrency and capacity issues altogether

• No ad hoc query testing

• Depends on strong analysis of process or transaction profiles

Keystroke capture

Record and replay keystrokes and mouse clicks into an application

• Tests front-end applications and back-end servers

• Tool may include powerful language and looping capabilities to manipulate inputs for multiuser concurrency and capacity testing

• Heavy processing requirements - may require additional hardware

• May increase development time for creating multiuser test simulations, and add time for debugging test harness

• Depends on strong analysis of process or transaction profiles

Concurrency and capacity testing

Use third-party load-testing tools

• Tests both front and back ends

• Language and looping constructs make these incredibly powerful tools

• Heavy processing requirements for test tool—may even require additional hardware or else results could be skewed

• Learning and development curve to write multi-user test simulations

• Risk of bugs in test harness could skew results

• Dependent on strong analysis of process/transaction profiles

Transaction generation

Thin client to simulate user execution of transactions

• Strong multiuser load testing

• Focus on back-end server issues

• May increase development time for creating multiuser test simulations, though learning and development curve generally less than keystroke capture tools

• Adds time for debugging test harness to prevent skewed results

• Depends on strong analysis of process or transaction profiles

Production load capture

Using tools to capture real transactions in a production environment, including performance and semantic characteristics, and resubmit in a test environment for analysis

• Tests real production loads, including ad hoc queries

• Especially useful when little or no analysis of transaction profiles is available

• Introduces new software into a production environment

• Production and test system configurations must be identical for valid performance analysis