The core idea of Test-Driven Development (TDD) is writing tests before writing any functional code, and then writing only the least possible amount of code required to make the tests pass. It may sound strange to develop in this fashion, but it’s actually quite useful, as the test base doubles as a partial specification of the main code.
Given such a simple premise, however, there is an amazing amount of terminology and techniques. In this article, I gather the most important terms and buzzwords that you might hear, and define them.
Test-first programming allows for high level functional tests.
The highest level of testing validates that the software meets the customer’s requirements. Acceptance testing is commonly run in environments as close to production as possible. See functional testing and system testing.
Assertions are statements that perform an actual check on the software’s output. In general, a single function called
assert is enough to express any check. In practice, test libraries often have many assert functions to meet specific needs (such as
assertEqual and more) to offer better analysis and friendlier output.
Behavior-Driven Development (BDD)
A subset of TDD driven by the need of clearer communication and proper documentation. BDD is perhaps the biggest recent development in TDD.
Its core idea is to replace confusing and developer-centric terminology (tests, suites, assertions etc) with ubiquitous language that all participating stakeholders (including non-technical staff, and, possibly, clients) can understand.
See user story.
A general principle in testing where the person writing tests does not know or avoids the internals of the software, choosing instead to test the public interface of the software strictly by its interface or specification. See white-box testing.
A strategy for writing tests to catch off-by-one and other similar types of errors. To perform boundary-value testing, test the inputs around certain possibly problematic boundaries. In case of integers, this might be
MAX_INT and other similar values.
Assertions are statements that perform an actual check on the software’s output.
A dummy is a type of test double that is never used by the actual software, but is only used in testing to fill required parameters.
Fakes are test doubles that implement the required functionality in a way that is useful in testing, but which also effectively disqualifies it from being used in production environment. For example, a key-value database that stores all values in memory and loses them after every execution potentially allows tests to run faster, but its tendency to destroy data would not allow it to be used in production.
A particular environment that must be set up before a test can be run. It generally consists of setting up all test doubles and other dependencies for the software under test: such as inserting predefined data into a fake database, setting up certain directory structure in the fake file system, setting up properties on the dependencies of software under test.
A high level testing activity verifying that all business requirements of the product are met. Functional testing commonly involves using user stories to focus on a higher level of requirements to cover as many usage scenarios as possible. See acceptance testing and system testing. For example:
# In this example we check that the about page of the website is working as expected open 'example.com' clickOn 'about us' assertThereIs 'We are a small Example company'
A colloquialism for a passing collection of tests, or sometimes a particular passing test. See red.
A mid-level testing activity that verifies a certain set of modules work correctly together. Integration tests are like unit tests without using test doubles for a certain subset of dependencies, essentially testing the interactions between the software its dependencies. Example:
# In this example we check that the newly registered user, # who was referred by another user, gets an on-site "friendship" created. # Here we check the interaction between the form controller, # database, and a User active record db = new Fake Db u1 = db.createUser(name='john') RegistrationForm(db, name='kate', referred='john').save() assert(u1.hasFriend('kate'))
A type of test double created for a particular test or test case. It expects to be called a specific number of times and gives a predefined answer. At the end of the test, a mock raises an error if it was not called as many times as expected. A mock with strict expectations is part of the assertion framework. Example:
# In this example we use a mock database to check that the form # uses the database to store the new user. # If the database has not been called at the end of the test, # the mock itself will raise an assertion error. db = new Mock Db db.expect('save').once().with(name='john') RegistrationForm(db, name='john').save()
A way to extend and change the behavior of existing objects and classes in a programming language. Monkey-patching can be used as an alternative to dependency injection and test doubles by directly modifying existing functions that are called by the software under test (and changing them back after the test).
# In this example we replace the standard library function # to prevent the test from using a real filesystem filesystem.listdir = f (name) -> ['.', '..', 'foo', 'bar']; assertEqual(MyFileSearch('foo').count(), 1)
A colloquialism for a failing collection of tests or sometimes a particular failing test. See green.
The process of improving implementation details of code without changing its functionality.
Refactoring without tests is a very brittle process, as the developer doing the refactoring can never be sure that his improvements are not breaking some parts of functionality.
If the code was written using test-driven development, the developer can be sure that his refactoring was successful as soon as all tests pass, as all the required functionality of the code is still correct.
A software defect which appears in a particular feature after some event (usually a change in the code).
See functional testing.
# In this example we prepare a fake database with some fake values # that we will need across multiple tests db = new Fake Db db.createUser(name='john') db.createUser(name='kate') db.createFriendship('john', 'kate')
# In this example, like in an example on mock objects, # we will check that the form uses the database to store the new user. # This time we will check state, instead of behavior db = new Fake Db RegistrationForm(db, name='john').save() assertInList('john', db.listUsers())
Fakes are test doubles that are never used by the actual software.
A type of test double that can reply to the software being tested with predefined answers. Unlike mocks, however, stubs do not usually check if they have been called properly, but rather only make sure that the software can call its dependencies.
A high level testing activity when the entirety of the software is tested top to bottom. This includes functional testing, as well as checking other characteristics (such as performance and stability).
An abbreviation for software under test. Used to distinguish the software under test from its dependencies.
The smallest possible check for correctness. For example, a single test for a web form could be a check that, when given an invalid email address, the form warns the user and suggests a fix. See test case.
A collection of tests grouped by an attribute. For example, a test case for a web form could be a collection of tests checking the behavior of the form for different valid and invalid inputs.
function t1: assertNoError( RegistrationForm(name='john', password='horse battery staple correct').save() ) function t2: assertError(MissingPassword, RegistrationForm(name='john').save()) function t3: assertError(StupidPassword, RegistrationForm(name='john', password='password').save())
User stories are usually defined in human languages to focus on user experience instead.
Any kind of metric that attempts to estimate the likelihood of important behavior of the SUT still not covered by tests. Most popular techniques include different kinds of code coverage: techniques that make sure that all possible code statements (or functions, or logical branches in the code) have been executed during testing.
A process of TDD development. Given that TDD development starts with writing a few tests, it is clear that the test suite starts red. As soon as the developer implements all newly tested functionality, tests turn green. Now the developer can safely refactor his implementation without the risk of introducing new bugs, as he has a test suite to rely on. Once refactoring is complete, the developer can start the cycle again by writing more tests for more new functionality. Thus, the red-green-refactor test cycle.
Test doubles are objects the test code creates and passes to the SUT to replace real dependencies. For example, unit tests should be very fast and only test a particular piece of software.
For these reasons, its dependencies, such as a database or file system interaction libraries, are usually replaced by objects that act in memory instead of talking to a real database or file system.
A collection of test cases that test a large portion of software. Alternatively, all test cases for a particular software.
White-box testing allows a deeper analysis of possible problems in the code.
Test-first programming is a slightly broader term for test-driven development. While TDD promotes tight coupling between writing tests (usually unit tests) and writing the code, test-first programming allows for high level functional tests instead. However, the distinction in general usage is rarely noted, and two the terms are usually used interchangeably.
The lowest level testing technique consisting of test cases for the smallest possible units of code. A single unit test usually checks only a particular small behavior, and a unit test case usually covers all functionality of a particular single function or class.
A single description of a particular group of people willing to perform a particular task using the SUT to achieve a particular goal. User stories are usually defined in human languages, using simple, user-centric terms to avoid considering implementation details and to focus on user experience instead. For example:
As a user, I want to be able to find my friends on this website by my address book, instead of looking for them one by one, because that will save me a lot of time.
White-box testing is a testing technique when a person performing the testing knows about, or can read, the internals of the SUT. Unlike the more common black-box testing, white-box testing allows a deeper analysis of possible problems in the code.
For example, a particular boundary value might not look like one based solely on the specification of the software, but it might be obvious from the implementation of it.
In addition, any test coverage techniques are usually, by definition, a part of white-box testing.