Testing framework

A testing framework gives you a place to organize and run your tests without worrying about the low-level plumbing. We talk about two kinds here: tests harnesses and test tools. A test harness is an API that you can use as the basis for writing your own tests. A test tool is a program that you use to create and run tests. Both have their place, depending on what you’re testing.
What’s Available (Test Harnesses):
SUnit- SmallTalk Unit is the original XUnit testing harness. This is the one that all the current unit testing frameworks copied.
JUnit- One of the more popular testing frameworks, written in Java for Java.

JUnitPerf- A nice collection of JUnit extensions that make performance and scalability measurement easier. This is a great example of why you want to use an open tool. Lots of people extend open tools, and the extensions are the sort of thing you’d have to write for yourself if you used a toolkit you wrote.

NUnit- A unit testing harness for .NET, originally ported from JUnit. It supports all the .NET languages.

MbUnit- Built on top of NUnit, MbUnit brings several higher-level testing techniques to the table. Integrated combinatorial testing, reporting, row testing, data-driven testing, and other concepts are all part of the package. Note that you must download TestDriven.NET to get MBUnit.

HTMLUnit- Used inside another test harness (such as JUnit), HTMLUnit simulates a web browser to test web applications.

HTTPUnit.- HTTPUnit is a lot like HTMLUnit, but it uses HTTP requests and responses to do its testing.

JWebUnit- JWebUnit sits on top of HTTPUnit to give you a high-level API for navigating a web app.

JSUnit - JSUnit is an incredible framework for testing JavaScript. But it's also got a distributed test runner to execute test scripts on different machines.

DBUnit- This is not a traditional testing framework but can be used to save a database to XML, restore it later, or compare the data to the XML file to verify it's integrity. This is a great tool to use for creating a clean database for a test run.

What’s Available (Additional Testing Tools):
Cobertura.- Cobertura (Spanish for “coverage”) is a code coverage tool. When you run a set of tests, it tells you how much tests exercise the tested code. IBM's DeveloperWork's has a good tutorial by Elliotte Harold.
Clover- A commercial Java code coverage tool, Clover has integrated plug-ins for most of the popular IDEs out there.

Watir- A testing tool to drive automated tests inside Internet Explorer. By driving IE, it solves the problem of imitating a specific browser’s interpretation of a web page. It’s based on Ruby and is becoming more popular.

Selenium also drives browser testing. It's more complicated than Watir but it can drive tests against browsers on different platforms.

Fit- Fit takes a unique, user-friendly, table-driven approach to acceptance tests. It’s worth reading about even if you choose not to use it.

Fitnesse- An extension to Fit. Fitnesse is both a stand-alone wiki and an acceptance testing framework.

WinRunner & QTP- Winrunner is an enterprise-class tool for functional and regression testing (and has a price tag to match).

LoadRunner- From the same company as WinRunner, LoadRunner handles performance and stress testing.

Empirix E-Tester- Empirix is a web recorder/playback tool that embeds MS Internet Explorer.

Systir- Another Ruby-based tool. This one is designed to drive tests in other languages.


Key Concepts:
API- The programmatic interface that your test code uses to access the test harness.
Methodology for creating tests- The framework that the test harness exposes.

User Interface- How you create, save, run, and maintain tests.

Test engine- The program that actually runs the tests you create.

Results display- How you find out whether the tests passed or failed.


How to Choose:
Type of testingDoes the tool or harness let you run the tests you need (e.g., functional, performance)?-
Support for the stuff you’re testing- Does the tool let you test your application code? Your web site?

Supported programming languages- Can you test the languages you use natively, or do you have to learn a new technology?

Flexibility- Can you create and run the type of tests that you need for your program?

Open formats- Can you integrate your test tool with other tools?