Implementing A Very Simple Protocol

This page explains what would be involved in implementing a new protocol using an SPP by describing the SPP implementation of the very simple virtual circuit protocol (VSVCP). VSVCP is not a practical protocol and is incomplete. But a study of its implementation will be sufficient to understand the process of supporting a new protocol using an SPP; i.e., support a new metanet. We will also relate parts of the discussion to the IPv4 metanet implementation to give a broader perspective on implementation issues.

Example

File:A-VSVCP-Network.png

A VSVCP Network

META:

• Dumbbell topology just like the one used in the GEC 4 Demo
• Paths between Hx and Hy where x = 1,2,3 and y = x+3
• Label VCIs
• Show FTs

Two processes (endpoints) in a VSVCP network communicate over a virtual circuit. Each virtual circuit is identified by a Virtual Circuit Identifier (VCI) which is globally unique. Each VSVCP packet contains an eight-byte header that contains a VCI which is used by VSVCP routers to forward packets. (We ignore how endpoints acquire VCIs, and we assume that each VSVCP router is magically loaded with an appropriate forwarding table.)

A VSVCP forwarding table maps a VCI to a meta-interface. Note that a VSVCP header is immutable; i.e., it does not change during transit from the sender to the receiver (unlike an ATM cell header). As shown in the figure (right), a packet going from H3 to H6 has a VCI of 6 throughout its journey to H6. At R1, the forwarding table maps VCI 6 to meta-interface (MI) 4. At R2, the forwarding table maps VCI 6 to meta-interface (MI) 3. Thus, there is a continuous forwarding path from H3 to H6.

File:VSVCP-Header.png

VSVCP Header

A VSVCP header (right) contains eight bytes that make up four fields:

• 2-byte Virtual Circuit Identifier (VCI)
• 2-byte data length (DL)
• 2-byte header checksum (HC)
• 2-byte flow label (FL)

The VCI field is used by routers for forwarding. The DL field indicates the number of bytes in the data portion of the packet; i.e., it excludes the eight bytes in the header. The HC is an Internet checksum (ones complement sum of 16-bit integers) over the header. The FL is an unspecified field and exists in this example to illustrate the generation of the code-option key (to be defined later). For this example, we assume that communication patterns between hosts are such that each receiver has atmost one sender (i.e., the communication graph is injective). Furthermore, for simplicity, we assume that VCI x is used to communicate with host Hx. In the example, H1 sends to H4 using VCI 4, and H4 responds to H1 using VCI 1; H2 sends to H5 using VCI 5, and H5 responds to H2 using VCI 2; and H3 sends to H6 using VCI 6, and H6 responds to H3 using VCI 3. But a process can also send to a GPE process running at router Rx using VCI 10+x; e.g., VCI 11 is used for communication with R1.

Background

Program	Description
create_fp	Configure the fastpath and handle local delivery and exception traffic
client	Configure a meta-interface and queues
fltr	Install/Update a filter

We review the highlights from The GEC 4 Demo and Overview Of The SPP Architecture that are important for the metanet developer. First, let's assume that we want to reuse as much as possible the SPP configuration software from the The GEC 4 Demo that demonstrated the IPv4 metanet. The table (right) lists the three programs used to configure the SPP for the The GEC 4 Demo .

The SPP metanet developer should remember that:

• A fastpath (FP) contains resources such as meta-interfaces (MIs), queues and filters.
• Each packet scheduling queue is bound to a MI.
• Filters are used to forward packets to scheduling queues; i.e., the set of filters is the forwarding table.

From a developer's perspective, an IPv4 metanet router forwards an incoming packet to a queue based on five fields from the metanet header (source IP address, source port, destination IP address, destination port and protocol) and meta-level information (input MI). But a VSVCP metanet router forwards based on the VCI and FL fields from the metanet header. It does this by extracting these field from the VSVCP metanet header, searching a forwarding table containing key-result (VCI-QID) pairs, and forwarding the packet to the queue indicated by the matching forwarding table entry.

The only logical difference between an IPv4 metanet router and a VSVCP metanet router is that the VSVCP router forwards by VCI and FL instead of IPv4 header fields. So, we should be able to reuse the FP, MI and queue abstractions; i.e., reuse the create_fp and client programs from The GEC 4 Demo to configure the FP, the MIs and the queues.

The developer will have to create a version of the fltr program that can be used to insert entries into the forwarding table that has a key format consisting of a VCI and a FL and a result format that is the same as the one used in an IPv4 filter. Although there are a few other minor details, the developer will need to modify the write_fltr() routine in the fltr.cc code so that it will insert a filter with the proper format.

There are only three more changes to the SPP required to support the processing of VSVCP packets: 1) it must recognize VSVCP packets; 2) it must extract the VCI and FL fields from VSVCP packet headers and form a lookup key in a format understood by the the Lookup block; and 3) it must verify that the packet header has not been damaged. If the packet header has been damaged, it must send an exception packet to the GPE process that handles exceptions. Again, there are a few other minor details, but in essence, that's all that is needed.

In SPP terminology, the SPP needs a new code option. This code option involves modifying the code in the Parse block and the Header Format block:

• The Parse block must:
  • Check for a damaged header using the checksum field.
  • Extract the VCI and FL fields from VSVCP packets.
  • Form the code-option lookup key (VCI, FL) for the SPP's Lookup block.
• The Lookup block must:
  • Search the forwarding table for the filter key that matches the incoming packet header.
  • Return the result from the matching filter.
  • Form an exception packet if the packet header was damaged.

These modifications requires that code in the SPP's IXP 2800 microengines be modified. At this time, these modifications must be done by the SPP developers, not the metanet developer. Thus, the metanet developer must tell the SPP developer the structure of the metanet header, the fields that need to be extracted, and the format of the code-option lookup key.

Even if the metanet developer, supplies the SPP developer with the above information and the SPP developer modifies the SPP accordingly, how is it possible for the SPP to handle the new protocol? The answer is that the SPP's lookup engine treats most of the key portion of the filter table as a generic, unstructured 112-bit vector. The main job of the SPP's Parse block is to format the packet key so that a search of the TCAM's filter table makes sense. Let's digress to a short explanation of lookup keys and SPP filters to clarify this answer before returning to changing the fltr.cc code.

Lookup Keys And SPP Filters

Generic TCAM Key-Result

META:

• Show fields of substrate key and result (very light blue and very light pink)
• Code-option key is a bit array (blue)

Recall that a filter table is a set of filters stored in the TCAM of the IXP network processor. A TCAM filter consists of a key K, a mask M and a result R. The SPP's Parse block creates a key K' from a packet's header. A TCAM filter matches a packet if (K' & M) = (K & M). The TCAM returns the result R of the matching filter which specifies how the packet should be processed. For example, R includes the QID of the queue where the packet should be stored until its transmission time.

The TCAM key has two parts: 1) the substrate key (light blue), and 2) the code-option key (blue). The substrate key is 32 bits (4 bytes) and has four fields:

• Type ('t')
  • 0: Normal filter
  • 1: Substrate-only filter
• Input Interface ('if')
  • The physical interface that received the packet
  • An internal index used to map to the IP address of physical interface that received the packet
• Slice ID ('sid')
  • The ID of the slice that created the filter
• RX Port ('rxport')
  • The UDP port number of the meta-interface that received the packet

Note that the 't' and 'rxport' fields come from the user's filter entry, and the 'if' and 'sid' fields are generated by the SPP software. However, the 'if' field is indirectly affected by the configuration process since it's value depends on the RX meta-interface field of a filter.

The code-option key is 112 bits (14 bytes) and has no special meaning to the TCAM (or the SPP). The format of the code-option key is determined by the program fltr that manages the TCAM filter table.

The result is 128 bits (16 bytes) and has nine fields:

• Drop Flag ('d')
• Local Delivery Flag ('l')
• Global State Index ('sndx')
• TX Destination Address ('txdaddr')
• TX Destination Port ('txdport')
• TX Source Port ('txsport')
• Queue Manager ('qm')
• Scheduler ('sch')
• Queue ID ('qid')

XXX

VSVCP TCAM Key-Result

Modifying The fltr.cc Code

Notes to the Writer

Very Simple Virtual Circuit Protocol

THE REST OF THIS IS PROBABLY USELESS AND IS DEFINITELY OUT OF DATE.

Goals

• Describe basic user procedure
• Begin to describe underlying architecture and components
• Talk about substrate concept

Sections

• xxx

Figures

• Application header format
• xxx

Links

• Overview Of The SPP Architecture
• The GEC 4 Demo

Versions

• 0
  • Protocol header chosen to work with iperf
  • Words (4 bytes) in header
    • Pkt ID (iperf UDP compatible)
    • Timestamp tv_sec
    • Timestamp tv_usec
    • VCI
    • Flow label (used in v1)
  • VCI is globally unique
  • 0.0
    • Minimalistic example
    • 1 router (R1), 2 hosts (H1, H2)
  • 0.1
    • Add monitoring daemon
  • 0.2
    • Like GEC 4, Part 1 but routes based on VCI
    • 2 routers (R1, R2), 6 hosts (H1-H6)
• 1
  • VCI can change during transit as part of pkt processing
  • Add Flow Label field

Configuring R1.0.0

• Use 2 physical interfaces from GEC4
• Use 2 hosts H1 and H2
• Replace R2 in GEC4 with H2
• FP
  • copt = 2
  • bw = 205 Mb/s
  • fltrs = xxx
  • qs = xxx
  • buffs = xxx
  • stats = fltrs
  • sram = xxx
• MIs
  • m1 (to H1)
    • bw = 100 Mb/s
    • ipaddr = 10.1.1.1
    • port = 20000
    • i.e., socket s1 = (10.1.1.1, 20000)
  • m2 (to H2)
    • bw = 100 Mb/s
    • ipaddr = 10.1.16.1
    • port = 20000
    • i.e., socket s2 = (10.1.16.1, 20000)
• Queues
  • q48 (LD): threshold = 100 pkts, bw = 1 Mb/s, mi = 0
  • q49 (EX): threshold = 100 pkts, bw = 1 Mb/s, mi = 0
  • q1: threshold = 1000 pkts, bw = m0.bw = 100 Mb/s, mi = 1
  • q2: threshold = 1000 pkts, bw = m1.bw = 100 mb/s, mi = 2
  • q0: not used
• VCIs
  • (R1, H1, H2) = (0, 1, 2) ==> Path (x,y) uses VCI 3*x+y
    • (H1, H2) uses VCI 5 = 3*1 + 2
    • Use 6 of 9 VCIs
  • Communication matrix
    • ( R1 (-, 1, 2), H1 (3, -, 5), H2 (6, 7, -) )
    • Unused VCIs: 0, 4, 8
• Filters (key = rxmi, vci)
  • f1 (R1 to H1): (rxmi, vci) = (0, 1), s = (10.1.1.1, 20000), qid = 1, sindx = 1 # substrate only
  • f2 (R1 to H2): (rxmi, vci) = (0, 2), s = (10.1.16.2, 20000), qid = 2, sindx = 2 # substrate only
  • f3 (H1 to R1): (rxmi, vci) = (1, 3), s = (10.1.16.1, 5555), qid = 0, sindx = 3
  • f4 (H1 to H2): (rxmi, vci) = (1, 5), s = (10.1.16.2, 20000), qid = 2, sindx = 4
  • f5 (H2 to R1): (rxmi, vci) = (2, 6), s = (10.1.16.1, 5555), qid = 0, sindx = 5
  • f6 (H2 to H1): (rxmi, vci) = (2, 7), s = (10.1.1.1, 20000), qid = 1, sindx = 6

User-space pkt

• Tunnel socket appears in UDP header
• VCI is first word of UDP payload
• Payload also has timestamp (for RTT and space-time diagram)

Substrate Implementation Concepts

• Physical interfaces
• Physical MIs
• Queues
• Filters
• Code option demultiplexor
• Parse, Lookup, Header Format

Slowpath Implementation

• Filters for local delivery
• User-space process to handle LD and EX traffic
  • Has VCI-MI Socket map; i.e., M : VCI --> MI Socket
  • Forwards pkt to socket M(pkt.VCI)
• Artifacts
  • Tunnel-aware sender and receiver
  • User-space process to handle monitoring data
  • spp-config-sp-vsvc-XXX.sh (slowpath) configuration scripts

Fastpath Implementation

• Requires mods to Parse and Header Format
• Uses generic Lookup
• Additional artifacts
  • spp-config-fp-vsvc-XXX.sh (fastpath) configuration scripts
  • Monitoring

Interesting Demo

• xxx