The challenges of testing TCP/IP
by Joaquim Soares

The pressure on companies to reduce time-to-market may have as a side effect an increase in the number of bugs in released software. Not only because programmers have to actually program in a shorter time frame, but also because the time dedicated to testing is in the front line to suffer from time-budget cuts.

Testing a piece of communication software is difficult because that piece must be able to interact with a range of stimuli that is virtually unpredictable. The robustness principle says, “be strict with what you send and liberal with what you accept”. But it is unrealistic to expect that even a programmer with this principle in mind is able to cover all possible situations without going through the typical “code-test-code-test” spiral approach.

Why is it necessary to test TCP/IP implementations? After all, they are here. They are (apparently) working properly. For three main reasons:

First, just because an implementation seems to be working properly doesn’t mean it actually is. It might contain a defect that is hindering its performance, or will result in a fault when least expected. A proper testing tool can be used to profile an implementation, assess its possible weaknesses and, based on that, allow the choice of the best task for that implementation and/or trigger measures for the correction of those weaknesses.

Second, everything with which the pieces of communication software interact is changing. Testing is not a one-time activity. An implementation may be correct for what existed five years ago, but react badly to some particular behaviour of, say, a recent firewall. Testing tools must evolve with what they test, much like virus detectors do.

Third, TCP/IP is still an area for new developments. Not only in applications (above TCP), but also at TCP level and below. Examples of these are proxies, firewalls, traffic and rate shapers. Moreover, many of these pieces of software must, by design, operate differently from normal TCP/IP stacks, and require also specialised testing.

Up to the IP level, it is feasible to test by fabricating datagrams (and for this there are many suitable tools) delivering them to the IUT (Implementation Under Test) and verify that they are handled correctly. IP is stateless, and, apart from fragments, datagrams are independent from one another.

TCP is another story altogether. It is a stateful protocol, which means that to test a TCP implementation by manufacturing segments, and keep a meaningful “conversation” with that implementation, it is necessary to speak TCP. The number of behaviours that cannot be tested with a single segment is much larger than with IP – connection establishment and release, window management, reassembly, retransmissions, reordering of segments, and so on. Many of the TCP requirements, those specified in RFCs 1122 and 1123, demand more than merely correctly handling data in TCP segments.

Applying the traditional (ITU/ETSI) protocol testing approach to TCP would mean something like having a series of TTCN test cases (a test suite):

- for the Lower Tester (the tester that sees the IUT at its network interface);

- for the Upper Tester (the tester that sees the IUT at its user interface).

This is quite feasible for the Upper Tester, but is very limited for the Lower Tester. Unless a proper TCP behaviour is emulated by the TTCN script, this approach limits the test suite to the most simple data manipulation and state transition tests. These tests are essential but not enough to assert the correctness of an implementation. However TTCN is hardly an appropriate language for implementing a TCP behaviour.

A better solution is to place a module between the network and the Lower Tester. This module is able to speak TCP, and offers the Lower Tester a simple API, suitable for a comprehensible test suite. Better yet, let’s give this module not only normal protocol operation capability, but also extended capabilities, making it more appropriate for testing. These capabilities may include sending and receiving raw data, regardless of its correctness, and ability to, by co-ordination with the Upper Tester, freeze the IUT in any given TCP state and allow the verification of behaviour in these states (a difficult task in typically short-lived states such as SYN_RCVD).

This approach is followed by the software TICK. It is a tool that not only provides an extensive set of TTCN test cases, but is extendible, thus allowing the building of customised misbehaving applications, which can be used to torment the IUT and make sure it will not fall apart when least expected. This kind of testing tool is an invaluable aid for a hands-on approach, which, as is well known, is the best way to learn protocol behaviour!