Investigation of Performance of TCP over ATM

William M. Miles, Mary E. Kaye and Brent R. Petersen

Department of Electrical and Computer Engineering
University of New Brunswick
P. O. Box 4400
15 Dineen Drive, Head Hall, Room D-41
Fredericton, New Brunswick, Canada
E3B 5A3

Technical Report
September 4, 2000.

We gratefully acknowledge the donations of ATM switching equipment and financial support provided by Canadian Marconi Corporation and the Natural Science and Engineering Research Council of Canada (NSERC).


Table of Contents

1.     Introduction.. 1

2.     Background.. 2

2.1       TCP/IP. 2

2.1.1        Introduction. 2

2.1.2        Concepts. 2

2.2       Asynchronous Transfer Mode (ATM) 2

2.3.1        Introduction. 2

2.3.2        Classical IP over ATM.. 2

2.3.3        Alternatives to Classical IP over ATM.. 2

2.3.3        TCP with ATM.. 3

3.     TCP over ATM... 3

3.1       Concept. 3

3.2       Implementation Overview.. 3

3.3       Project Goals. 3

4.0       Implementation Details. 5

4.1       Experimental Kernel. 6

4.1.1        Selection of Base Kernel 7

4.1.2        ATM Services Under Linux. 9

4.1.3        ATM Service Initialization. 31

4.2       TCP over ATM Protocol Definition. 55

4.2.1        Callback. 56

4.2.2        Connection Sequence. 58

4.2.3        Disconnection and Error Handling. 62

4.3       Implementation Details. 65

4.3.1        Modification of TCP options. 66

4.3.2        Entry Points. 68

4.3.3        Function Breakdown. 70

4.3.4        Callback address. 72

4.3.5        System input 74

5.     Conclusions and Recommendations. 78

A.    Appendix - Source Code.. 82


1.     Introduction

ATM technologies are dropping in price, and are becoming increasingly available for entire networks down to the workstation level.  ATM is an inherently connection-based protocol.  However, existing high-level network protocols typically assume a broadcast network and require complex software support to operate on an ATM network.  With the specific case of the TCP/IP suite, the overhead is further compounded by the fact that TCP is also connection-oriented.  Thus we have a connection-oriented application (TCP) operating on an emulated broadcast network (IP) in turn running over connection-oriented hardware (ATM).  This configuration is inherently wasteful, especially because IP headers are large in comparison to ATM cells.  By building ATM support directly in to the TCP layer, substantial performance improvements may be made as both the header and processing overhead of the IP layer may be bypassed.


2.     Background

2.1    TCP/IP

2.1.1  Introduction

Modern protocol suites often try to combine the features of these two paradigms to maximize the utility to the user.  The very common TCP/IP protocol suite, developed in the 1970s and 1980s, is no exception: the TCP (Transfer Control Protocol) is designed to create reliable virtual connections over a packet switching network; while the IP (Internet Protocol) provides the services to operate that packet switching network.  TCP is responsible for the application interface, allowing programs to define connections that can be trusted to remote machines.  IP is responsible only for routing packets from point A to point B; it is connectionless, and knows nothing of the data each packet contains.

2.1.2  Concepts

While TCP/IP can be adapted to run on nearly any hardware configuration, IP does make certain assumptions about the network architecture.   An IP network is broken down into subnets with related addresses and routers (known as gateways) connecting the subnets together.  Each machine on a subnet expects to be able to contact other machines on its subnet without additional routing; that is, it can simply send a packet to the hardware interface and expect direct delivery directly.  It also expects to be able to broadcast a packet to all machines on the subnet.   On broadcast media such as Ethernet or token ring networks, this is trivial; however, Asynchronous Transfer Mode (ATM), a relatively new networking hardware protocol, does not support broadcasting, and so a traditional TCP/IP implementation is not possible.

2.2    Asynchronous Transfer Mode (ATM)

2.3.1  Introduction

ATM is a protocol designed to take advantage of modern communications techniques and hardware.  ATM operates very much like TCP in that it is inherently connection-oriented, but operates over a packet switching network.  ATM switches move tiny packets called cells at extremely high speeds, routing them based on pre-established connections; this arrangement combines the rapid transit speed of circuit-switching networks with the bandwidth efficiency of packet-switching networks.  As the reliability of modern communications hardware is now extremely high, ATM does very little in the way of error checking to minimize overhead and improve throughput.  Should an error be detected, ATM switches will simply drop (ignore) the packet; thus reliability controls must be performed by a higher-level protocol.  Additionally, ATM contains inherent support for voice and video transmission, allowing many different applications to share a common ATM network with minimal degradation.

2.3.2  Classical IP over ATM

Since there is no support for packet broadcasting on ATM networks, several different schemes have been used to run the TCP/IP protocol suite over an ATM network.  The most straightforward is called Classical IP over ATM (CLIP).  CLIP requires that one machine with a known address (either standard or manually entered at all stations) act as an address server for the network.  Using a protocol called ATMARP (ATM Address Resolution Protocol), all machines on that subnet must contact this machine to find out the ATM address registered to the IP address it wishes to contact.  The client must then create an ATM connection directly to that machine, which it passes all IP traffic destined there across.   For broadcast IP packets, the client retrieves the list of all ATM addresses for that subnet, and contacts each machine individually to deliver the packet.

2.3.3  Alternatives to Classical IP over ATM

The other schemes used to facilitate IP traffic over an ATM network are equally roundabout.  LANE (Local Area Network Emulation) operates nearly identically as CLIP, except that instead of handling just IP traffic it is generalized to emulate a broadcast network device, and thus any broadcast network protocol (IP, IPX, Netbeui, etc.) may be used over it.  This simply adds yet another layer of software and emulation that must be executed each time a packet is sent.  MPOA (MultiProtocol Over ATM) is another generalized broadcast network system, but operates more directly with the protocol rather than creating an emulated network device.  Both LANE and MPOA require servers at known addresses and more processing than CLIP, but allow other broadcast protocols to be run in tandem over the same virtual circuits.

2.3.3  TCP with ATM

The irony emerges when one considers the TCP protocol.   TCP was designed to operate in conjunction with IP, providing a simple, reliable, connection-oriented interface for applications.  TCP essentially converts an incoming data stream into packets, attaches flow control and reliability information, and then employs IP to carry those packets to their destination.  When using TCP in an ATM environment, we are therefore encapsulating a connection-oriented system (TCP) on a connectionless system (IP) on a connection-oriented system (ATM).


3.     TCP over ATM

3.1    Concept

Text Box:  Figure 1.  TCP over ATMIt is possible to reduce the overhead of TCP connections in an ATM environment by mapping them directly to ATM connections.  Not only will bandwidth be saved by not sending the IP headers with each packet, but the processing overhead of creating and parsing the IP headers will also be eliminated.  This system, called TCP over ATM (T/A), will provide a shortcut bypassing the IP layer as shown in Figure 1.

Implementation Overview

To facilitate this end, a new protocol must be designed to manage the setup, status, and teardown of ATM connections that correspond to TCP connections.   Since both TCP and ATM are connection-oriented protocols, this will be relatively simple.  The new protocol must be able to identify the ATM address of each end station, determine when a TCP connection is being opened to a machine that can support T/A connections, open an ATM connection when a TCP connection is being established, handle any errors that occur while the connection is opening or established, and tear down the connection when the TCP session is terminated.

Project Goals

Over the course of this project, steps towards writing a rudimentary implementation of T/A were taken.   This implementation will be completed as part of the requirements for the author's undergraduate thesis.  At that time, performance benchmarks will be taken to analyze the practicality of continued work in this direction and the results published.   If the improvements are deemed sufficient, then a draft submission to the appropriate standards organization will be made.


4.0   Implementation Details

Experimental Kernel

4.1.1  Select ion of Base Kernel

The Linux 2.4.0-pre5 kernel was used as the base for implementing T/A for this project.   This selection was made because it is entirely open-source, ATM services are available for this kernel, and a special software driver is available that emulates ATM hardware using traditional Ethernet hardware, thus reducing the costs of development.  The source code for the base kernel may be downloaded at http://www.kernel.org/pub/linux/kernel/v2.4/linux-2.4.0-test5.tar.bz2

4.1.2  ATM Services Under Linux

ATM services under Linux are operated through a group of daemons that interact with the kernel.  These daemons are not included as part of the server sources; they must be downloaded separately from http://icawww1.epfl.ch/linux-atm/, the ATM on Linux home page.  The version 0.78 ATM services were used during this experiment.  Figure 2 details the interaction of the kernel and the daemons to operate a CLIP system.

Text Box:  
Figure 2.  ATM Services under Linux data flow.


Blue lines indicate the flow of data from the application; orange lines indicate the flow of control information.  The purple bubbles are responsible for maintaining the devices connected to.  Red lines indicate the flow of data back through the virtual ATM device.

The following table details the function of each module.

Module

Function

Application

User application creating a TCP connection.

TCP/IP Stack

Standard Linux TCP/IP stack for internetwork communications.

IP interfaces

Active interfaces that support IP.

ifconfig

InterFace CONFIG.  Administers properties of hardware interfaces supporting IP.

eth0

Ethernet hardware interface.

atm0

CLIP interface for ATM.

atmarp

Administration program that enables CLIP interfaces.

atmarpd

ATM ARP Daemon.  Responsible for providing IP to ATM name resolution.  May maintain master ARP list for subnet, or may be a client that connects to a master ARP list elsewhere.

ATM Address List

List of ATM addresses known to be reachable through ATM interface.

ilmid

Daemon supporting ILMI, a protocol to allow ATM devices to identify the ATM addresses of other nearby devices.  Not used in this project.

atmaddr

Manual ATM address list maintenance command.

atmsigd

ATM SIGnaling Daemon.  Handles the control channel for switched virtual circuit operation, communicating circuit setup and disconnection information to the ATM switch.

ATM Devices

Active kernel ATM device drivers.

atmtcp

Daemon that uses a TCP connection to emulate an ATM device.

4.1.3  ATM Service Initialization

The following setup scripts were used to open the virtual ATM connection between the two test machines, named compeng6 and vega.  An ATM hosts file has previously been set up containing fake ATM addresses for both machines.  This file substitutes the fake ATM addresses for the strings compeng6-atm and vega-atm during the setup scripts.

Host Setup (compeng6)

Command

Function

atmtcp virtual listen &

Begin virtual ATM service.  Listen for incoming connection.

atmsigd -b -m network

Initialize ATM signaling daemon as network host.

atmaddr -a compeng6-atm

Bind the ATM address for compeng6-atm to the local machine.

atmarpd -b

Activate the ATM ARP daemon.

atmarp -c atm0

Create the atm0 IP interface using CLIP.

ifconfig atm0 10.0.0.2 up

Activate the atm0 interface using IP address 10.0.0.2

Client Setup (vega)

Command

Function

atmtcp virtual connect 131.202.135.34 &

Begin virtual ATM services.  Connect to 131.202.135.34 (compeng6).

atmsigd -b

Initialize ATM signaling daemon as network client.

atmaddr -a vega-atm

Bind the ATM address for vega-atm to the local machine.

atmarpd -b

Activate the ATM ARP daemon.

atmarp -c atm0

Create the atm0 IP interface using CLIP.

ifconfig atm0 10.0.0.3 up

Activate the atm0 interface using IP address 10.0.0.3

atmarp -s 10.0.0.2 compeng6-atm arpsrv

Configure the ATM ARM daemon to map IP 10.0.0.2 to compeng6-atm, and use it as the ATMARP server.

These scripts, when run, set up a virtual ATM connection using readily available Ethernet hardware and activate CLIP across it, providing a test bed for the T/A protocol.

4.2    TCP over ATM Protocol Definition

4.2.1  Callback

TCP uses a three-way handshake to establish a connection.  First, a packet with the SYN (synchronize, or connect) flag is sent to the remote machine to request that a communications channel be opened.  If the remote machine is available to accept this incoming connection, it will reply with a packet with both the SYN and ACK (acknowledge) flags set.  The initiating machine, to complete the connection, replies with a packet with just the ACK flag set.  This handshaking system provides an ideal mechanism to add T/A functionality.  In opening a T/A connection, it was decided that an option would be added to the SYN packet, which would then allow the remote machine the option of establishing a T/A connection if it so desired.  If the remote machine is incapable of making a T/A connection, it can simply pass the SYN-ACK packet back via regular IP.  It was originally planned that this option would contain the ATM address of the local machine, allowing the remote machine to create the connection without performing an address lookup; however, a public ATM address is too large to fit in a TCP option. Thus, only a single byte is sent, and it is up to the remote machine to determine the ATM address to connect to.

4.2.2  Connection Sequence

In a pure IP environment, all three of these packets will travel via IP.  Under T/A, the original SYN packet will also contain a TCP option requesting that a T/A connection be made, if possible.  The remote host will then attempt to determine the ATM address of the initiating machine, if possible.  If a valid address is available, the remote machine will then open an ATM connection to the initiating machine.  Once the ATM connection has been established, the SYN-ACK packet and all further TCP packets will be passed along it.  Should this ATM connection fail at any time, or in the event that it cannot be established, the TCP packets will revert to using IP for transport.

Text Box:  
Figure 3.  TCP over ATM Connection Sequence
Blue text indicates T/A connection sequence; green text indicates standard IP connection.


4.2.3      Disconnection and Error Handling

Both directions of the ATM connection must remain open while the TCP connection is active, even if one side of the TCP connection is closed; it must be possible for acknowledgements to flow back.  Thus, disconnection of the T/A connection may only take place when both sides of the connection are closed or when an error occurs.  In the event of an error, this initial implementation of T/A will immediately revert back to using standard IP to communicate and disconnect the T/A connection.  Future implementations may wish to provide more sophisticated error recovery, but as the purpose of this project is simply to benchmark the concept a conservative implementation was selected.

4.3    Implementation Details

4.3.1  Modification of TCP options

The first step taken was the addition of the new TCP option.  A value of 12 was selected for this experiment; any TCP packet with this value as a single-byte option it its header will be interpreted as a request to open a T/A connection.  Support for the generation of this value based on the value of a user-controllable TCP socket option was also added, to provide the ability to choose whether or not to use T/A to user applications.  Both the header generation and header analysis functions were modified to support this new option.

4.3.2  Entry Points

To handle each T/A connection, a series of new functions and data structures were defined.  First, an ATM circuit structure was added to the TCP status structures, allowing each TCP connection to have an independent T/A connection.  Second, the entry points for the new T/A functionality were identified: when a new SYN packet was generated with the new TCP option, the system must be configured to accept a new incoming T/A connection; when a SYN packet is received with the new option and the system is capable of opening a new T/A connection, it must identify the address to open the connection to, then open it; when a packet is sent while there is an active T/A connection, it must be sent along it; when a packet is received over a T/A connection, it must be passed up to the TCP stack; and when the socket is closed, the T/A connection must also be closed.  A set of new functions was defined in a new C file to handle the majority of these cases.  The receiving function must be embedded in the ATM receive code, while the others may fit in the TCP code; consequently, it was left to last.

4.3.3  Function Breakdown

As both the connect packets and outgoing packets ultimately pass through the IP layer, the uppermost function of the IP layer was modified to call a function named tcpatm_handle to determine whether any additional T/A processing was required.  If tcpatm_handle returned true, then it will have already handled the packet and it need not be sent through IP; if false, then send the IP packet normally.   tcpatm_handle analyzes outgoing packets for which T/A is enabled.  If it is a SYN packet, it calls a function named tcpatm_listen to prepare for an incoming T/A connection; if it is a SYN-ACK packet, it calls tcpatm_open to identify the target machine and open the T/A connection to it; if it is any other packet and a T/A connection is established, then call tcpatm_send to send the packet along that connection.  In any other case, set the socket as not using T/A to avoid unnecessary further processing for future packets.   tcpatm_listen, tcpatm_open, and tcpatm_send are not yet written, however, and simply return success.

4.3.4  Callback address

One of the issues that arose during the consideration of the content of tcpatm_listen and tcpatm_open is that in order to establish an ATM connection for T/A, one must specify the destination system address.  It would be ideal to be able to send this address along with the TCP connection packets; however, ATM addresses are much too long to fit in a TCP header.   While it is possible to define a new protocol under IP to communicate this address, this would overcomplicate this experiment.  Thus, for simplicity, IP over ATM protocol independence was sacrificed; a simple ATMARP lookup was coded into a prototype tcpatm_open function, as this implementation will only be expected to operate in conjunction with CLIP.

4.3.5      System input       

Once the functions for establishing T/A connections and sending packets along them are completed, the next problem will be identifying incoming T/A packets and passing them to the TCP stack.  All of the above problems were handled by simple modifications to the well-documented TCP code.  However, installing support to recognize T/A connections will be more complicated.  One possible solution is to write a daemon that will handle all T/A connections; however, this would require a kernel-daemon interface, which adds another level of error-control that must be built in.  A better solution is to embed some support directly in to the kernel; however, the ATM code is not well-documented and subject to change.  This problem must be addressed as the test implementation is completed.


5.     Conclusions and Recommendations

TCP over ATM provides a mechanism for reducing the overhead of TCP connections in an ATM environment.  It is a logical step, given that ATM is available to machines running TCP/IP and the complexities of IP over ATM.  No previous work has been done in this direction.  A test implementation is under way to analyze the performance benefits precisely.  Should this analysis reveal any significant benefit, it is recommended that TCP over ATM be put forward to Internet standards organizations so that others may benefit from this technology.


A. Appendix - Source Code

The complete kernel source with the modifications made at the time this document was written is available at http://vega.dyndns.org/TCPATM/linux.atmtest.tar.bz2 and http://www.ece.unb.ca/petersen/pubs/reports/nserc2000/linux.atmtest.tar.bz2. A copy of just the modified source and header files is available at http://vega.dyndns.org/TCPATM/changed.tar.bz2 and http://www.ece.unb.ca/petersen/pubs/reports/nserc2000/changed.tar.bz2. Future updates will be posted at http://vega.dyndns.org/TCPATM. All modifications were added within conditional compilation blocks to allow easy updating to new kernel versions; the flag used was named TCPATM. Printed source files are not included here due to excessive length (>200 pages).


This page is located at http://www.ece.unb.ca/petersen/pubs/reports/nserc2000/
The original HTML form of our report was prepared by William M. Miles on September 4, 2000 and is available at http://vega.dyndns.org/TCPATM/.

This original HTML form of our report was placed at this URL on September 24, 2000 by Brent Petersen, email b.petersen@ieee.org; minor formatting changes were made.

This page was updated on April 26, 2002 by Brent Petersen .
© Copyright 2000, William M. Miles, Mary E. Kaye and Brent R. Petersen. UNB Professional Page Disclaimer