Deciphering Testing Jargon

Lately, we’ve been hearing and reading more and more about test-driven development. This domain, however, comes with a series of expressions and specific jargon that can be confusing to newcomers. This article will walk you through the most common definitions, test types and test parts. Use cases will be provided, and, where possible, some code in PHP will also be presented.

First Day at a New Job

A few years ago, a new programmer was hired to a development team. As any other newcomer, he was quite confused on his first day. As he listened to the discussions around him in the office, a lot of testing specific terms were used. These were expressions unknown to our fictional new programmer.

Fortunately for him, this being his first day at work, the two colleagues were later assigned to explain all this jargon to him. They began with a list of terms related to the inner workings of a test case.

Automated Test

Software testing software is practically an automated test. Test automation has been around since before the PC; the first automated testing frameworks appeared in the times of mainframes and consoles. Today, automated testing is the obvious way to go. Why? Because testing is a tedious and repetitive task – something not well suited for human beings. Automated testing is considerably faster, and more precise than manual testing. And no, it doesn’t eliminate the human tester or a QA team from the scheme. It simply makes them do a more human suited job, and allow them to do it well.

Setup / Exercise / Verify / Teardown

Any test should be breakable into four parts:

• Setup – whatever needs to be prepared before the code can be run
• Exercise – run the code we want to test
• Verify – compare the result of the run with some expected condition
• Teardown – cleanup all the extra stuff we used for testing so that the system is in the same state as it was before we started the current test (the state from before the Setup step).

We design each test to have four distinct phases that are executed in sequence: fixture setup, exercise SUT, result verification, and fixture teardown. – xUnit Test Patterns: Refactoring Test Code, by Gerard Meszaros

A Test Fixture

A Fixture represents all the information that the test needs in order to be exercised. A fixture can be as simple as creating a plain object, like $testedObject = new RealObject();, or something as complicated as populating databases and starting user interfaces.

Everything a system under test (SUT) needs to have in place so that we can exercise the SUT for the purpose of verifying its behavior. – xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros

SUT: The System Under Test

You probably observed this recurring term. Programmers will usually refer to it as SUT. It represents all the things required to be tested. Depending on the type of the test (see below for test types) the SUT can be many things from a method or a class to the whole system.

Whatever thing we are testing. The SUT is always defined from the perspective of the test. – xUnit Test Patterns: Refactoring Test Code, by Gerard Meszaros

Discovering More …

Starting his second day on the job, our programmer wrote his first test. It was more difficult than he anticipated. To write it, he needed a testing framework, and he had to create a test case and then run all the test methods. There were also a handful of strange dependencies that he needed to figure out. It seemed that learning about DOC was on schedule.

Testing Framework

A testing framework is an application that is specifically designed for testing code in a specific language. The concept of a test framework was pioneered by Kent Beck in the early ’90s. His work later led to a framework for SmallTalk, called SmalltalkUnit, and afterwards renamed to SUnit.

Smalltalk has suffered because it lacked a testing culture. This column describes a simple testing strategy and a framework to support it. The testing strategy and framework are not intended to be complete solutions, but rather a starting point from which industrial strength tools and procedures can be constructed. – Simple Smalltalk Testing: With Patterns, by Kent Beck

It was the first xUnit framework, and it defined the basic concept of testing and the terms presented above. Today, nearly every programming language offers its version of this framework: PHPUnit for PHP, JUnit for Java, ShUnit for UNIX Shell Scripts and so on. You would be surprised to find out how many things can be tested today, and how many tests can be automated.

Test Case

Originally, a “test case” was defined as the smallest unit of testing by Kent Beck.

When you talk to a tester, the smallest unit of testing they talk about is a test case. TestCase is a User’s Object, representing a single test case. – Simple Smalltalk Testing: With Patterns by Kent Beck

These days, we are using test method to define this smallest part and a test case mostly refers to a set of related test methods. For example, a typical situation is when we are unit testing our code and a test case refers to the totality of the test methods testing a particular class or whatever is the smallest unit in our programming language. A test case, many times, is simply referred to as: “a test.”

Test Method

A test method is the smallest part of a test architecture. A test method is the unit that consists of the above defined parts: setup / exercise / verify / teardown. It is the essential part of any test; the one that does the work.

A test method is a definitive procedure that produces a test result. – Form and Style Manual, ASTM, 2012

DOC – Dependent-On Component

This was easily one of the most confusing new terms for our new programmer. It represents all the other classes and system components that our SUT needs in order to properly run. But, also, the DOC has to provide specific methods that allows us to observe and test it. The concepts of mocking and test doubles are strongly related to the DOC.

An individual class or a large-grained component on which the system under test (SUT) depends. The dependency is usually one of delegation via method calls. – xUnit Test Patterns: Refactoring Test Code, by Gerard Meszaros

Are There Any Other Than Unit Tests?

Soon after writing his first few tests, the new guy realized that he is testing different logical parts of the application. Sometimes, it is best to test a small part in isolation; other times, it is required to test a group of objects together and the way they talk to one another; and other times, you need to test the whole system. Things looked more complicated than previously presumed; so our programmer went on and read a book, and another, and another, and, finally, he understood.

The Testing Pyramid

The testing pyramid was first defined in the book, Succeeding with Agile Software Development Using Scrum, by Mike Cohn, and then soon adopted by the software community.

The pyramid represents the three main testing layers: UI, Service, and Unit.

The UI layer represents the topmost testing level: when the system is exercised through the UI and the whole application is tested as one. This layer should represent the smallest amount in our multitude of tests.

The Service layer contains several different test types. It is mostly concerned with the internal communication of modules and by the correct working of the external API (application programming interface) of an application. There should be several such tests in our suites, but they should not be a base for our testing. These tests are usually training several parts of the application, and, thus, they are fairly slow. They should be run as frequently as possible, but not on every save of the code. Probably at every build of the system or when a commit happens to the versioning system.

The Unit layer refers to tests exercising the smallest possible units of our code in complete isolation. These tests should represent the vast majority of the tests. They should be very fast (1-4 milliseconds / test) and should be run as frequently as possible. Test driven development (TDD) is a good example of how to maximize the use of unit tests.

Detailing the Testing Pyramid

Based on the above example, the community devised several more detailed versions of the testing pyramid. The one I consider to be the best can be seen in the image below.

The three main layers can be clearly distinguished, but the center layer is more detailed. As time passed by, the software community discovered and defined several new testing methods. Some of them were included on the pyramid.

The Unit Test

A unit test represents the testing of the smallest unit one’s programming language allows. In object oriented programming, these are classes/objects. In other languages, they can be small modules or even functions/procedures.

A test in these definitions refer to the same thing as a test case represents.

A test that verifies the behavior of some small part of the overall system. What turns a test into a unit test is that the system under test (SUT) is a very small subset of the overall system and may be unrecognizable to someone who is not involved in building the software. – xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros

Unit tests represent the vast majority of the tests that a programmer writes. Yes, it’s true: unit tests are most of the time written by programmers. Unit test help the programmers to develop the application, prevent common bugs, typos and regressions. They are tests made by programmers for programmers.

This is why unit tests are more technical and more cryptic in nature. They are here to help programmers write better code; when something fails on a unit test level, it is usually a problem for a programmer to fix.

Component Tests

As the name suggests, component tests are written for a little bit larger chunks of the application. A component test usually exercises a whole module or a group of logically interdependent units.

The component is a consequence of one or more design decisions, although its behavior may also be traced back to some aspect of the requirements. – xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros

Surely, a component test exercises more code than a unit test. It also may test how some units work together and talk to each other.

A component can also be traced back to a requirement or a part of a requirement. This means that a component test is not just for programmers. Team leaders, scrum masters, architects and other technically involved people are surely interested by the modules, by their organization and sometime even by their inner-workings. These people are not necessary familiar with a specific programming language. The test has to concentrate more on the behavior and define the expectations in a more understandable way.

For example, a unit test may have an error message stating that:

TestFileAccessCanWriteToAFile: Failed asserting that file '/tmp/testfile' is present on the system.

Such a message would not be helpful for an architect or a manager or a team leader. A component test may fail with a more descriptive error:

Account Administration Test: Failed when we tried to specify 0 (zero) as the total money a user has in his account.

Such a test exercises a higher level functionality. Based on the error message above, there may be several layers of communication and classes / objects involved in the operation of specifying an amount as the total in someones account.

The Integration Test

This type of test takes several modules, and checks how they integrate with one another. It verifies if the internal module APIs are compatible and working as expected.

The term, however, allows a wide range of possible uses. Some software communities strongly relate integration tests with testing how our application works inside the medium it has to run. In other words, how it integrates into the higher system.

Others define integration test at different levels: anything defining the communication between two elements can be seen as an integration. These elements can be units, like classes, modules or even higher functional parts of the software.

There is no unanimously accepted definition for the term, integration test.

API Tests

The GUI of an application is talking to the software by the software’s API. Testing at this level exercises a lot of code, and can a relatively significant amount of time to run.

API is the means by which other software can invoke some piece of functionality. – xUnit Test Patterns: Refactoring Test Code by Gerard Meszaros

In object oriented programming, such APIs are defined by the public methods of the classes. However, if we take a look at a high level architectural design schema, the meaning of the API can be restricted to the public methods of the classes providing functionality through the borders of the business logic. These boundary classes represent the API and we should test that, when we call and use them, the system behaves as expected.

Usually, these tests are run periodically and take a long time to finish.

Testing the GUI

There should be only a few rare cases when you wish test to the presentation of an application. There is really no logic in the GUI layer, just presentation.

Testing if a button is green or red, or if it has 30px in width is useless, and is too much of an overhead. So don’t jump into testing your views. If something goes terribly wrong with the GUI, it will be observed on the exploratory manual testing phase.

Testing views should do only two things: test conditional presentation, and test that the expected API is called.

Jumping into testing your views can be tempting. Don’t! Test only what you think can fail. Test only for values or function calls. Never check for GUI elements or their properties. Use REGEX whenever possible to match strings and check keywords that are not probable to change.

For instance, the following pseudo code is a bad example for testing the presence of a string on the screen.

function testItCanTellTheNameOfTheUser() {
	// some rendering code logic here
	$renderedName = $this->renderName();
	$this->assertEquals('User has the name ' . $renderedName['first'] . ' ' . $renderedName['last'] . '.');
}

Not only is this test difficult to read, but it tests for an exact phrase – something like “User has the name John Doe.”, including punctuation! Why is this bad? Because someone may easily change the form of this sentence without changing its meaning.

What if our client requires Lastname, Firstname form to be presented? That would make our test fail. We have to ask ourselves: should the test fail? Did we change software logic? I say no, it should not fail. The name would still be present on the screen; the order of the two parts would simply be different. This is a more appropriate test.

function testItCanTellTheNameOfTheUser() {
	// some rendering code logic here
	$renderedName = $this->renderName();
	$this->assertRegExp($renderedName['first'], $renderedName);
	$this->assertRegExp($renderedName['last'], $renderedName);
}

This now ensures that the name is present, but it doesn’t care about the lexical construct. Someone could change the initial phrase into something, like Don, John is the name of the current user. The meaning will remain the same and the test will still pass correctly!

What’s Missing from the Pyramid?

After a month or so of working on the job, our fictional new programmer realizes that, even if the pyramid is quite cool, it’s not complete. Sometimes, there are a couple or so different test that should be execute – and they are pretty hard to place on the pyramid.

Acceptance Tests

These are one of the most controversial tests. Depending on what kind of books you are reading, acceptance tests might be referred to as Functional Tests

or

or

or