The Remote Laboratory Interface (v6.4 or earlier)
Contents |
Getting Started
You must register as an ONL user before you can use the ONL testbed. The registration form can be found by following the an account link in the side bar of the ONL web page. After submitting your registration form, you will be notified by email that your account has been activated. Note that logging in via the web page adds links to the sidebar that are only visible to registered ONL users.
Among the ONL user links in the sidebar will be a link to the page Getting Started which gives step-by-step instructions on getting and running the RLI and logging into the testbed hosts. These instructions include the following:
- Instructions for getting the RLI.jar file from the ONL Web site.
- Instructions for getting the correct version of the Java Run-time Environment (JRE)
- Instructions for setting up your host to remotely access the ONL testbed via the RLI.
Note that the RLI is a standalone Java application that requires an up-to-date version of the JRE.
Once you are setup to run the RLI, run the RLI by enterring:
java -jar RLI.jar
or double clicking its desktop icon.
RLI Basics
Before beginning our tour of the RLI, let's go over some basic conventions regarding mouse buttons. The following list is a quick summary of the main features:
- Menus are accessible by single clicking with the left mouse button.
- Objects that have not been allocated have a light blue color.
- Allocated objects have a dark blue color.
- At all times, hosts and routers have logical names (e.g., host n1p3, NPR.2). After resource allocation, hosts and routers also have actual resource names. For example, onl053.arl.wustl.edu might be the actual n1p3 host. nprcp03.arl.wustl.edu might be the Control Processor (CP) for NPR.2.
- Clicking the center of a dark blue icon (allocated object) shows the the actual object's name. In the case of a host, the logical name (e.g., n1p2) refers to the network interface to the internal network (NPR or NSP) and the actual name (e.g., onl053.arl.wustl.edu) refers to the network interface to the external network (an Ethernet). Alternatively, you can use an environment variable like $n1p2 to refer to the external interface. See below for more details.
- A single object can be moved to a different part of the window by holding the left mouse button down on the icon and dragging. Components that are close to each other can be moved as a group by dragging a box over the objects to select the objects and then holding the left mouse button down on one of the host icons and dragging.
- A property list or menu can be hidden by clicking any open area in the RLI window.
- After configuration changes have been made, the changes must be committed by selecting File => Commit. The status box at the bottom of the main display will be outlined in red to remind you that changes have been made and need to be committed before they are actually reflected in the testbed.
These features will be pointed out again as they are encountered.
Network Interfaces
A user can SSH to onl.arl.wustl.edu (called the login host) at any time but can only SSH into other ONL hosts that have been given to the user as part of the commit process. Control messages and SSH access to a committed ONL host from the login host go over the external (control) network.
Every host given to a user as part of the commit process has two interfaces: one connected to the internal (private) testbed network and one connected to the external (control) network. Experimental traffic goes over the internal network consisting of your routers and the interconnection switch. The figure (right) shows one NPR with two hosts. Each host has been annotated with green bars depicting the network (Ethernet) interfaces to the two networks and red lines depicting the control network.
Throughout our descriptions in the NPR Tutorial, interface names that begin with a dollar sign (e.g., $h1x2, $h2x3, $n1p2, $n2p3) refer to the Ethernet interface that is connected to the control network. You should replace those symbols with the hostnames appropriate to your setup or run the source Unix command shown below to make these symbols meaningful.
Interfaces to the Internal Network
Interfaces to the internal network have IP addresses of the form 192.168.N.H
In this subnet allocation scheme, N is the logical router number and H is the host portion. Interfaces have names such as n1p2, and n2p3 (Note: not $n1p2 or $n2p3); i.e., omit the dollar sign when referring to the internal network interface. For example, if you ask for only one router, the value of N will always be 1. But NPR 3 might be the actual router. If you ask for two routers, N will be 1 for one router and 2 for the other router.
External Interfaces
Interfaces to the external (control) network have IP addresses of the form 10.0.0.H where H is the host portion and have names such as onl062.arl.wustl.edu (which has the short name onl062). These external names are determined during the commit phase when actual resources are bound to resources in your configuration.
The /users/onl/.topology File
In many cases (especially scripts), it might be easier to refer to the environment variable (e.g., $n1p2) associated with the external interface whenever you are refering to the control network interface. These variables can be imported into your environment by running ONE of these two commands (your choice will depend on your ONL shell):
source ~onl/.topology # bash
source ~onl/.topology.csh # csh, tcsh
By default, users are assigned the bash shell. Enter the ps command to identify which shell you are running. Note: The control network interface names in the environment variables are NOT automatically defined.
You can verify that this procedure has worked by entering something like:
printenv n1p2
echo "$n1p2"
which should display an external name of the form onlXXX (e.g., onl062, onl053) in both cases.
Configuring Topology
The next few sections describe how to construct and configure a basic network topology and monitor traffic and queue length. An example based on some of the concepts discussed on this page is given in Examples
Adding a Router
A user begins by interactively constructing and configuring a network of routers, hosts and links using the Topology menu. Note that this phase can be done without making any resource reservations or connecting to the ONL testbed!
You can add a router using the Topology => Add NPR menu.
An NPR has five ports.
Users can attach hosts to ports 0 through 4.
Moving Icons and Spinning a Router
In most cases, you can move icons with the left mouse button as in any other GUI by grabbing the icon and dragging it into position. You can even move a group of icons by enclosing the icons in a rectangle defined by dragging the left mouse button along the diagonal of the rectangle.
One feature peculiar to the RLI is that you can spin or rotate a router icon to improve the layout for attaching to hosts and other routers.
The figure (right) shows that we have added two NPRs and positioned
them near the center of the window.
NPR 1 (NPR 2) is on the left (right).
In anticipation of creating a dumbbell topology, we want to spin
NPR 1 clockwise and NPR 2 counterclockwise before adding four hosts
and associated links.
To rotate a router icon, select and hold on to the spin handle
(pink bullet) and then rotate the NPR icon into place.
Note that a spin handle can only be moved to the five angular
positions shown as bullets along the outer
perimeter of the icon.
Watch Out!!! A spin handle is picky about where you grab it. For best results, aim towards the southeast corner of the spin handle circle. It tells you that you have grabbed the right spot by turning itself bright red.
Adding Hosts and Links
To add a host select Topology => Add PC1core => Add Host1core and a host will appear. The host icon will appear in the upper left corner of the window. Move it into position near its attachment point on the router port. Then, select Topology => Add Link; select the host icon; and drag the left mouse button to the attachment port. Alternatively, you can drag in the opposite direction; i.e., from port to host. A dashed link will appear indicating that the hardware has not yet been allocated.
Before a host is linked to a router port, it will have a configuration name such as HST3; i.e., HSTx where x starts at 0 and is incremented for each new host. But the icon will acquire a name with the form nXpY when you link it to a router port. X will be the logical NPR number, and Y will be the port number. For example, the host name n2p3 is linked to port 3 on NPR 2.
Remember: A host name nXpY refers to the internal network interface name. The nXpY interface will also have an IP address of the form 192.168.X.Z where Z = 16 x (Y+1); e.g., n1p2 has the IP address 192.168.1.48. After resource allocation, objects get bound to other names and IP addresses. For example, a host that is labeled as n1p2 will also be bound to an external (control) network interface name and IP address. That name might be onl053.arl.wustl.edu which might have an IP address of 10.0.0.53.
The links are shown as dashed lines, and the hosts and routers are shown in a light shade to indicate that the objects have not yet been bound to actual testbed resources. When these resources get assigned actual hardware, links will become solid, and hosts and routers will darken.
Saving Your Work
Saving a configuration (even an incomplete one) allows you to resume your work from the stoppiing point and aids in recovery from unexpected interruptions. It is good habit to adopt a discipline of periodically saving your configuration as it evolves. The menu items File => Save as and File => Save allow you to save your configuration to a file. They have the same semantics as in other software tools where Save as allows you to name a file and Save assumes that you are modifying a file that was read earlier. Selecting either menu item opens up a dialogue window.
The RLI assumes that your configuration files are in the .onldir directory in your home directory (~/.onldir). A configuration can be saved at any time, even before requesting that resources be allocated for the network.
Note: Since the directory name begins with the dot character, the Unix ls command will not show this directory unless you include the -a flag (all).
The figure (right) shows that our .onldir directory already has lots of configuration files. Because these files are for the older NSP routers, we will create the npr subdirectory for all of our NPR configuration files.
The five icons at the top of the dialogue window allow you to (from left to right):
- Change to the next directory up
- Change to your home directory
- Make a new directory in the current directory
- Show the current directory in brief
- Show the current directory in detail
Note that you can always get back to the .onldir directory by selecting the Home (second) icon.
The figure (right) shows that we have changed into our npr subdirectory and want to save the configuration in the ex-rli.exp file.
Remember: All of the work up to this point can be done without connecting to the ONL testbed.
Reserving Resources
Before asking ONL to give you physical resources through the File => Commit menu item, you must make a resource reservation &mdash much like making a restaurant reservation. This approach is necessary because of the limited number of routers relative to the number of users (seven NPR router pairs and four NSP routers). Reservations are made through the File => Make Reservation menu item in the main RLI window.
There are two things to consider when making a reservation:
- The starting time must be specified in terms of the testbed's local time; i.e., our time in St. Louis which is either Central Standard Time or Central Daylight Savings Time. (We are working on fixing this constraint.)
- You must ask for the maximum number of resources of each type that you plan to use during an experiment. That's because there is no way to extend the spatial requirements after you have made a reservation except to first cancel the reservation. However, you can change internal parameter values (e.g., port rate) and components (e.g. add filters) after a reservation is made.
Note however, that the reservation daemon may impose a limit on resource demands especially for course work. The most important one that is sometimes imposed is that each student is allowed only four router-hours per reservation (e.g., one router for four hours; two routers for two hours; or four routers for one hour). Students just beginning their laboratory exercises are usually asked to reserve only one or two routers for one hour since there is no non-exclusive time-sharing mechanism.
You can check for the availability of resources through the Resource Availability link in the sidebar of the ONL website. The vertical axis of the availability chart will show the number available if you restrict the search to one type of resource (e.g., NPR). Otherwise, it will show the percentage of availability. Remember to select the Go button to refresh the graph.
To make a reservation through the RLI, you must already have an existing configuration. Then, select File => Make Reservation. A dialogue box titled "Log In" will appear. Fill out the dialogue box, and select Enter. When the Log In dialogue box appears, enter your User Name and Password. A status message will appear in a Reservation popup window once the reservation has been processed.
After a reservation has been successful, you can see the reservation by clicking the My Reservations link in the sidebar of the ONL webpage.
Committing Resources
Now we are ready to ask the ONL to assign us actual resources and implement our virtual network topology. The menu item File => Commit starts the allocation and initialization of physical resources. The RLI will send a commit message to the configuration daemon which will attempt to allocate the physical resources and configure them to match your network description. A progress bar is shown at the bottom. If any error occurs, an error log window will be displayed. The error log can also be shown at any time by selecting the Show Error Log button at the bottom right hand corner of the display.
While the RLI is committing resources, the Commit Needed box will be replaced by the progress bar showing the text "Committing."
When the configuration daemon has finished, a "Commit Completed" message will be shown. The links will be solid, black lines and the other objects will be in a darker blue color signalling that the components are now bound to actual hardware resources.
This commit process usually takes from one to four minutes. Because we use preemptive rejuvenation, the commit can take less than a minute when the testbed is lightly loaded. Longer than normal commit times usually occur when there are lots of users trying to configure their experiments or if there are hardware problems.
Clicking on the center of an icon shows an objects properties and menu items. The left window (above) shows that the host n1p3 has been assigned to an actual host with the control interface name onl052.arl.wustl.edu. Furthermore, the internal interface IP address is 192.168.1.64. The only host menu item is Monitoring => User Data (described later). NPR.1 has been actually assigned actual NPR 9 (since the name of the CP is nprcp09.arl.wustl.edu). NPR.1's base IP address is 192.168.1.0; i.e., hosts attached to port Y will have an IP address of the form 192.168.1.Z where Z is 16 x (Y+1).
The NPR menu items include a configuration menu and a monitoring menu (details not shown). The configuration menu contains two items: Plugin Table and Plugin Debugging (to be described later).
It is worth repeating that each host has two interfaces. One has an internal name (e.g., n1p2) and the other an external name (e.g., onl052.arl.wustl.edu). Each interface also has an IP address. The host IP address shown in the RLI display is the address assigned to the host interface that is internal to the testbed and will carry your data traffic. These addresses are not externally visible, and like the internal names are assigned algorithmically, making it possible to repeat an experiment in different sessions without having to modify names and addresses used in demonstration scripts. Typically, the internal interface name (e.g., n1p2) is used in commands that generate experiment traffic; and the external (control) interface name (e.g., onl052.arl.wustl.edu) is used to open SSH connections to the host for the purpose of running applications.
Watching The Commit Progress
You can observe the progress of the allocation and initialization process through the Testbed Status side bar link in the ONL Web page (after logging in).
The figure (right) shows a dumbbell configuration with two routers and four hosts. The RLI shows that the host with n2p2 internal network interface name has the control network interface name onl024.arl.wustl.edu and the n2p2 interface name has been assigned the IP address 192.168.2.48.
The Testbed Status Web page shows that there are two active experiments (users that have been assigned hardware resources). The kenw user has been assigned the hosts onl024, onl027, onl028, and other resources. The Web page also shows the status of all routers and hosts. Those marked active have been assigned to users, and those marked free have not. Nodes can also be in a repair or testing state indicating respectively that an error occurred or the node is being used by developers.
When the commit is done, hardware resources have been bound to the logical resources shown in the RLI and intialized to their default settings. So far, we have accepted the default link rates (1 Gbps) and have accepted default routing within the data network. Keep in mind that at this point, the route table is empty, and therefore, all packets will be dropped by the routers. We will show how to install routes in the next page.
Running Commands on a Host
Once the commit has completed, you can access the hosts assigned to your experiment. Below is a simple example where we use the ssh to login to the host with internal interface n2p2.
1 client> ssh onl.arl.wustl.edu # ssh to ONL user host 2 onlusr> ssh onl024 # ssh to n1p2 host 3 onl024> netstat -i # show network interfaces
Here is an explanation of each line:
- You can only ssh to the hostname onl.arl.wustl.edu
from outside of the ONL testbed environment.
The name onl.arl.wustl.edu is really an external
name that is mapped to the actual ONL host providing
a desired service (ssh in this case).
Although we have shown how to login using a command line,
you can use any ssh client tool.
You will end up on the host with name onlusr, the gateway host. Note that even though you used the name onl.arl.wustl.edu, you end up on the host onlusr. You can login to ONL's gateway host at any time: You don't have to have an active experiment running. - From the gateway host, you can ssh to any of the hosts that have been committed to your experiment without a password prompt. Note that you can use the abbreviated hostname that omits the suffix ".arl.wustl.edu"; e.g. onl024 in place of onl024.arl.wustl.edu. Furthermore, you will NOT be prompted for a password.
- In this example, onl024 is the abbreviated DNS name of the control network interface to the host logically shown as n2p2. In reality, n2p2 is the name of the internal interface which is attached to port 2 on NPR 2. The netstat command shows the network interfaces of every network interface where the command is run.
Keep in mind that a firewall prevents you from initiating a new connection from within the ONL testbed to a remote host. One implication is that if you want to transfer a file on the onlusr host to a remote host, you must do it from the remote host. Also, keep in mind that ONL hosts run the Linux operating system, and the default command interpreter is the bash shell.
The figure (above) shows that there are three network interfaces on onl024:
- eth0: This is the Ethernet interface to the control (external) network.
- eth1: This is the Ethernet interface to the data (internal) network.
- lo: This is the loopback interface which is on every Unix host.
In particular, the RX-OK and TX-OK columns indicate the number of bytes received and transmitted respectively over each interface. See the man page for the netstat command for more details about the output. As expected many packets have been received and transmitted over the control network after a commit, but the data network has hardly been used.
Here is an alternative to the above example that uses the .topology file described in <a href="./RLI_Basics.html#topology">RLI Basics</a>:
1 client> ssh onl.arl.wustl.edu # ssh to ONL user host 2a onlusr> source ~onl/.topology # use ~onl/.topology.csh if using a C-shell 2b onlusr> ssh $n2p2 # $n2p2 will evaluate to onl10 3 onl024> netstat -i # show network interfaces
Line 2a defines environment variables of the form $nXpY for all control network interfaces that have been committed to the user. In our example, $n2p2 will evaluate to onl024 in Unix command lines since it is the control network interface corresponding to the host labeled n2p2 in the RLI. This allows you to write shell scripts that survive experiment shutdowns, and reduce the need for clicking on hosts in the RLI to discover their control network interface names. The command "source ~onl/.topology" can also be used on hosts other than onlusr.
The output of the ifconfig command (above) confirms that the eth1 interface has the IP address 192.168.2.48. (Note: If you get a "command not found" error message, try using /sbin/ifconfig.)
Recap
- .onldir
The RLI assumes that your configuration files are in the .onldir directory in your home directory (i.e., ~/.onldir in *nix/Cygwin and C:\Documents and Settings\USERNAME\.onldir in Windows). Configuration files can also be stored in other directories (e.g., a subdirectory of ~/.onldir or ~/onl-configs/). - Resource Reservations
You do not need a resource reservation until you want ONL to actually give you resources; i.e., enter File => Commit. You don't even need to be connected to the Internet if you are only creating a configuration file. Warning: When you do make a reservation, make sure that it includes all of the desired components because you will not be able to use your reservation if you add resources after the reservation has been made, and you will be forced to delete the old one and add a new one. - Commit
When you Select File => Commit, the ONL resource daemon will allocate and intialize resources according to the configuration you have created using the RLI. - Remote Access to ONL
From outside the ONL testbed, you can only SSH into onl.arl.wustl.edu. Then, you can SSH to any of your committed hosts. - Two Host-Network Interfaces
Each host has an interface to the control network (e.g., onl054) and another one to the data network (e.g., n1p2). Each of these interfaces has an IP address. - Host IP Addresses
A host name nXpY refers to the internal network interface name. This interface will also have an IP address of the form 192.168.X.Z where Z = 16 x (Y+1); e.g., n1p2 has the IP address 192.168.1.48. This address is shown when you click on a host icon. - Default Route Table
Initially, the route table is empty, and all packets will be dropped by the router. - .topology File
The command "source ~onl/.topology" defines environment variable references of the form $nXpY to be their corresponding control network interface names. Use .topology.csh if you are using a c-shell instead of the bash shell. This feature allows you to avoid having to click on host icons in the RLI to discover their control network interface names and makes writing shell scripts that are portable between experiments.
Configuring Routes
If we try to send traffic from n1p3 to n2p2 using the configuration we just committed in the preceding page The Remote Laboratory Interface => Configuring Topology, NPR 1 (the left router) will drop the packets because no routes have been installed; i.e., all of its route tables are empty. This page explains conceptually how an NPR route table is used to forward packets from an input port to an output port and shows you how to install route tables.
An example based on some of the concepts discussed on this page is given in NPR Tutorial => Examples => The Remote Laboratory Interface. The sections in this page are:
- The NPR Route Table
- Generating Default Routes
- The Route Table Edit Menu
- Verifying Connectivity Using Ping
The NPR Route Table
The most basic activity of a router is to transmit incoming packets out the output port designated by the best matching entry in its route (or forwarding) table. Conceptually, each port of an NPR has a route table in which each entry has three fields:
- The Prefix/mask is a subnet address specified in Classless Inter-Domain Routing (CIDR) format. A CIDR address has the form a.b.c.d/x where a.b.c.d is an IP address prefix in dotted decimal notation (e.g., 192.168.1.0) and where x (the mask) indicates the number of bits in the network part of the IP address. For example, an x of 24 means that the leftmost 24 bits of the prefix is the network part of the address.
- The Next hop indicates the output port associated with the table entry.
- The Stats is the statistics index associated with the table entry and will be described later.
For now, suppose that we are using only a basic NPR that uses only route tables for packet forwarding decisions. (Later, we will see how NPR filters can alter packet forwarding.) When a packet arrives to port P, the NPR picks from port P's route table the best matching (P, N, S) entry using the longest prefix match (LPM) algorithm and forwards the packet knowing that the packet should go out port N.
Generating Default Routes
In order for both NPRs to forward traffic between all hosts and between the two NPRs, we need to install route tables at the inputs that have attached links; i.e., ports 1.2, 1.3, 1.4, 2.1, 2.2, and 2.3 (the notation X.Y refers to NPR X, port Y).
The figure (right) shows route tables that could be used at ports 1.3, 2.1, 2.2 and 1.4 to forward packets flowing from n1p3 to n2p2 and back to n1p3. They have been arranged to show which entries would be used to forward the packets.
Recall: The nXpY network interface is assigned the IP address 192.168.X.Z where Z = 16 x (Y+1); e.g., n2p2 has the IP address 192.168.2.48.
So, packets flowing to n2p2 will have a destination IP address of 192.168.2.48 in its header.
The packets flowing from n1p3 to n2p2 (192.168.2.48) will match the prefix/mask fields 192.168.2.0/24 and 192.168.2.48/28 at ports 1.3 and 2.1 respectively. The packets flowing in the reverse direction from n2p2 to n1p3 (192.168.1.64) will match the prefix/mask fields 192.168.1.0/24 and 192.168.1.64/28 at ports 2.2 and 1.4 respectively.
The easiest way to generate the route tables we need is to generate default route tables at every port (including ones without attached hosts):
- RLI Window:
Select Topology => Generate Default Routes - Select File => Commit
You can verify that the route tables are correct. For example, look at the routing table at port 1.3 by enterring (not shown):
- Select NPR.1:Port 3 => Configuration => Route Table
The route table at port 1.3 will appear. It should look like the figure (right).
You should see a route table similar to the one shown right. The first five entries reflect the NPR's standard IP addressing where hosts on subnet 192.168.X.Z/28 are assumed to be attached to port Y and Z = 16 x (Y+1). For example, the third (P, N, S) entry is (192.168.1.48/28, 2, 46), and means that packets destined for a host in the subnet 192.168.1.48/28 (e.g., 192.168.1.48) should be forwarded to output port (next hop) 2.
The last (P, N, S) entry (192.168.2.0/24, 4, 49) forwards packets destined to hosts connected to NPR 2 and will forward those packets out port 4. The RLI generated that entry automatically because it could tell that there is only one route to the hosts attached to NPR 2.
But even when if there were multiple paths to NPR 2, the RLI would have inserted an arbitrary entry for packets destined for NPR 2.
Watch Out!!! If there are multiple paths, the output port chosen by the RLI may not be the one you want. So, check router-to-router entries in route tables.
The Route Table Edit Menu
You can examine the route table at a single port. In the figure (right), we are about to open the NPR.1:port 3 (port 1.3) route table window. The route table window has a Route Table => Edit menu that allows the user to add a route, delete a route, stop statistics and generate local default routes. Also, you can modify a route table entry by selecting an entry field and changing only that field (e.g., next hop). You must select File => Commit before the changes take effect.
The Add Route and Delete Route menu items are straightforward to use. Note also that you can delete a block of entries (shown right):
- Select the first item in the block
- Then, select the last item in a block while holding down the SHIFT key to highlight the block of entries
- Then, select Edit => Delete Route
The Generate Local Default Routes menu item allows you to generate a route table that assumes the 192.168.X.Z/28 subnetting described earlier. Here is how you would use this menu item to install a route table at port 1.3 that is identical to the one generated earlier by the Topology => Generate Default Routes menu item in the main RLI window:
- Select Edit => Generate Local Default Routes
- Select Edit => Add Route
- Change the new route entry so that the prefix/mask and next hop fields contain the values 192.168.2.0/24 and 4 respectively by selecting each field and keying in the correct values.
- RLI Window:
Select File => Commit
The route table will be populated with the five entries for ports 0 to 4 for the NPR. But we need another entry for packets going to NPR 2.
A default entry with prefix/mask 0.0.0.0/0 and next hop 0 will be added.
All of the route entries have used non-overlapping prefix/mask entries. But we can construct a route table using the fact that route lookup uses the LPM algorithm.
Verifying Connectivity Using Ping
You can verify that packets from one host can reach another one by using the ping command. For example, if you want to see if packets can be sent from n1p3 to n2p2 (and back), ping n2p2 from host n1p3 by entering:
onlusr> source ~onl/.topology # add control network IP addresses to env
onlusr> ssh $n1p3 # ssh to host n1p3 through control network
$n1p3> ping n2p2 # send ping packets to n2p2 from $n1p3
^C # enter ctrl-c to terminate the ping command
This provides a simple test that there is network connectivity between the sender and receiver.
Watch Out!! A common mistake is to enter "ping $n2p2" when you meant "ping n2p2". The first form attempts to send five ping packets to the control network interface, not the data network interface, attached to the n2p2 host.
The figure above shows that we have used ping to send five ping packets to n2p2 from n1p3 (onl029). The -c 5 limits ping to sending five packets instead of a continuous stream of packets.
When a user enters the ping command, the sending host sends one ICMP echo request packet to the destination once per second. When the receiver gets the ICMP echo request packet, it sends back an ICMP echo reply packet to the original sender. As each response is received, ping outputs information indicating the sequence number and the round-trip time (RTT). After ping terminates, it outputs a report indicating packet loss and RTT statistics (minimum, average, maximum, standard deviation). The ping command is a simple tool for general network troubleshooting. And a successful test (right) is a good indicator that you will likely be able to send packets between the two hosts participating in the ping.
The figure above shows how you can use the "netstat -i" Unix command to verify that the proper number of ICMP echo request packets were received at and ICMP echo reply packets sent from n2p3. The netstat command window shows that we ran netstat twice on the the receiver host onl074 &mdash once before pinging from n1p3 and once after ping finished. It shows that the receiver received six (6) packets and sent six (6) packets even though we sent only five (5) ping packets. The extra packet is an ARP packet which shows up if we wait long enough between network usage.
Recap
- Route Table:
When a packet arrives to a port, the NPR uses that port's route table to determine which output port (next hop) to send the packet based on the destination IP address in the packet's IP header; i.e., where to forward the incoming packet. - LPM Route Lookup Algorithm:
The NPR uses the longest prefix match (LPM) algorithm when searching for the best matching entry in a route table. - Adding Routes:
All route tables start out empty. You can generate default route entries using either Topology => Generate Default Routes in the main RLI window or Edit => Generate Local Default Routes in the Route Table window of an input port. - ping and netstat:
The ping command is often used for testing connectivity between hosts and general troubleshooting. The "netstat -i" command can be used to verify packet count activity at a host's network interfaces.
Basic Monitoring
In the preceding page The Remote Laboratory Interface, we successfully sent ping traffic from n1p3 to n2p2 and back. But how do we know that the packets actually travelled through ports 1.3, 1.4, 2.1, 2.2 and back rather than taking some other route (e.g., over the control network)? The NPR has a large number of monitoring points that can be displayed in real-time to verify traffic behavior. There are monitoring menus associated with:
- Each port that can be viewed by clicking on the port of the router icon;
- Each router as a whole that can be viewed by clicking on the center of the router icon; and
- Each host that can be viewed by clicking on the host icon.
This page discusses the per port monitoring menu. It shows how to do packet monitoring and byte monitoring at a port and describes features of monitoring panels. The router menu will be discussed in the pages on Filters, Queues and Bandwidth.
An example based on some of the concepts discussed on this page is given in NPR Tutorial => Examples => The Remote Laboratory Interface. The sections in this page are:
- Monitoring Concepts
- Starting a Chart
- Monitoring Packet Counters
- Working With Charts
- Monitoring Byte Counters
- Other Charting Features
Monitoring Concepts
The NPR has numerous byte and packet statistics counters. Some of them are updated as a packet moves through an NPR. As a general rule, the statistics counters come in byte-packet pairs; i.e., if there is a byte counter there is also a packet counter. The RLI allows you to access all of the counters. However, some counters have specific menu items while the remaining counters must be accessed through the general ReadRegisterByte and ReadRegisterPacket facilities described in a later page.
Each NPR port has a monitoring menu that can be viewed by selecting the port of the NPR icon. The figure (right) shows the menu at port 1.3 (left NPR). The five menu items allow you to monitor the following:
- ReadQLength: Read the queue-length counter of one of the 8,192 queues of an output port.
- RXPKT: Read the packets-received counter of an input port.
- RXBYTE: Read the bytes-received counter of an input port.
- TXPKT: Read the packets-transmitted counter of an output port.
- TXBYTE: Read the bytes-transmitted counter of an output port.
This page demonstrates the use of the last four types of counters. The Filters, Queues and Bandwidth page demonstrates how to read the length of a queue.
When you ask that a statistics counter be monitored, the ONL daemon periodically reads the counter at a user-specified polling rate and sends the value to your RLI. By default, the period is one second which is adequate for most uses. (Note: You can request a smaller time interval, but keep in mind that it is unlikely that the ONL daemon can monitor more often than once every 0.25 seconds.) You can either display the data (discussed here) or log the data to a file. In displays, you have a wide range of choices such as rate versus absolute value and units (e.g., bytes, bytes/sec, Mb, Kb/s).
The concepts of rate and absolute value sometimes lead to confusion. Choosing absolute value means that at each monitoring time point, you want the value of a counter as read or a value that has been converted by your choice of unit (e.g., bytes to MB). However, choosing rate means that at each monitoring time point, you want the rate of change since the previous monitoring time point. More precisely, suppose that the counter values at monitoring time points are X0, X1, X2, etc. Then, if you chose absolute value, the values displayed will be c X0, c X1, c X2, etc. where c is a conversion constant. If you chose rate, the values displayed will be c (X1-X0)/T, c (X2-X1)/T, c (X3-X2)/T, etc. where T is the polling rate (period). When T is one second, rate is equivalent to the differences (X1-X0), (X2-X1), (X3-X2), etc.
For example, absolute value is often used when monitoring queue length. If the monitored values are 1500 bytes, 9000 bytes, 3000 bytes, etc. and the units is specified as KB (Kilobytes), the displayed absolute values will be 1.5 KB, 9.0 KB, 3.0 KB, etc. On the other hand, if rate was chosen when the monitering period was 0.25 second, the first two displayed rates would be 30 KB/s = (9-1.5)/0.25 and -24 KB/s = (3-9)/0.25. But if the monitoring period was one second, the two displayed rates would be 7.5 KB/s and 6 KB/s respectively Rate is used more often when monitoring bandwidth.
Starting a Chart
To start a new monitoring chart, select Monitoring => Add Monitoring Display in the main RLI window. And then enter the name of the chart in the Graph Title dialogue box. We are creating a chart labeled "pkts per sec" in the figure (right). Now all monitoring selections will be focussed on this chart.
If there were more than one monitoring chart/window, you can change the charting focus to a particular chart by selecting Parameter => Add Parameter in the monitoring window that should be the new focus.
The figure (left) shows us electing to monitor RXPKT (the packets-received counter) at NPR 1:Port 3 by opening the menu at port 1.3 and selecting Port 3 => Monitoring => RXPKT. An Add Parameter dialogue box will appear.
Here, we have enterred 0.25 in the Enter Polling Rate box to override the one-second default polling rate. We have also checked the rate box to override the absolute value monitoring mode.
A blue RXPKT line immediately appears in the pkts per sec chart when we select Enter (not shown) in the dialogue box. Note: If resources have already been committed, monitoring does not require a File => Commit. The Y-axis is labeled Packets, and the X-axis is labeled time(secs). The Y-axis label can be changed by selecting View => Change Y-axis label and entering the new label in the dialogue box.
The X-slider at the bottom is used to move the chart back and forth in time. If moved away from the rightmost position, the chart freezes, and new data are not shown in the chart. When moved back to the far right, the X-axis resumes its time-window advance, and new values are added to the chart. If you run long enough, you will not be able to move all the way back to the beginning of an experiment because the RLI has a fixed size data cache for each plot. When it runs out of cach space, it deletes the oldest data.
Monitoring Packet Counters
To monitor points along the path between n1p3 and n2p2, we could continue to add plots for the packets-received or packets-transmitted counters at ports 1.4, 2.1, and 2.2. But we would discover that all of the line plots were labeled as RXPKT or TXPKT. We change the labeling to eliminate this ambiguity.
The RXPKT label can be selected to open a dialogue box that allows you to change the attributes of the RXPKT plot. The figure (right) shows that we have changed the label to RXPKT 1.3 indicating that it is the RXPKT field for port 1.3.
Other fields reflect options that were specified earlier. The rate radio button is marked because we had selected rate over absolute value. And the window indicates the polling rate is 0.25 seconds. The units are packets per second and not scaled (1.0 scaling factor). We have elected to accept the option of a solid line instead of a dashed line. Selecting Change applies our changes to the current line plot.
The figure (right) shows that we have added plots for the packets-received (RXPKT) at ports 1.3 and 2.1 and the packets-transmitted (TXPKT) at ports 1.4, 2.2 and 1.3. We have also changed the default plot labels. We expect the peak packet rate to be 4 packets per second since there is one ping packet sent every second and our monitoring period is 0.25 second (4 = 1/0.25); that is, during each 0.25 second monitoring period, the packet counters will either remain the same (rate = 0) or increase by 1.
Working With Charts
The "pkts per sec" chart appears crowded, and the only two visible line plots are RXPKT 1.3 and TXPKT 2.2. Any hidden plot can be made visible (brought to the front) by passing the cursor over the line plot label (not shown). We can modify the appearance of the chart to reveal more of the ping-like behavior: bandwidth spikes appearing approximately once per second. The charting features of the RLI allow the user to focus on a portion of a chart by zooming in and out along both the X- and Y-axes.
Left-clicking the X-axis at point X zooms in so that the rightmost X-value is X and also freezes the chart so that new values are not displayed. Dragging the X-axis slider to the far right resumes the display of new values.
In the figure (top right), the original chart had a time window that ran approximately from time 299.0 to 349 with hash marks at 12.5-second intervals. Left-clicking the X-axis at point X zooms in so that the rightmost X-value is X; the leftmost X-value is unchanged; and the chart freezes so that new values are not displayed. The effect is to reduce the time interval covered by the chart. The bottom right figure shows the effect of several X-scalings followed by dragging the X-axis slider to the far right to resume the display of new values. It reveals the expected spikes separated at intervals of one-second.
Similarly, left-clicking the y-axis at point Y zooms in so that the topmost y-value is Y. Other things you can do (not shown) include:
- You can zoom out by left-clicking on the arrowhead of the X-axis or the Y-axis.
- You can zoom in on a window using the menu item View => Zoom In From Selection and then dragging along the diagonal of a rectangle that encloses the region of interest.
You can also show the approximate X- and Y-coordinates within a chart by selecting View => Show Values and then left clicking the display point.
Monitoring Byte Counters
Monitoring bandwidth instead of packet count should now be straightforward: Use the RXBYTE and TXBYTE counters instead of the RXPKT and TXPKT counters. The figure (right) shows the bandwidth of ping traffic arriving to port 1.3 after we have adjusted the X- and Y-scaling to reveal the ping behavior: there are four transmissions over a four-second period; and the peak bandwidth is almost 336 bytes/sec (the chart shows 324 bytes/sec).
We expect 336 bytes/sec since each ICMP packet carries 56 bytes of data and is encapsulated in an 8-byte ICMP header and 20-byte IP header. This means that each packet is 84 bytes. Since the monitoring period is 0.25 second, the expected byte rate is 4 x 84 or 336 bytes/sec.
The line plot dialogue box for byte counters is almost identical to the packet counter ones. The figure (right) shows that the default byte rate units of bytes/sec can be changed to other units (e.g., Mb/s, Kb/s, MB/s, KB/s).
Other Charting Features
This section briefly describes other charting features. The main RLI menu item Monitoring => Set Default Polling Rate allows you to set the default polling rate to something other than one second. In most cases, the default should be sufficient, but you can use it when you need more or less precision than offered by a one-second polling period. But note that the minimum practical monitoring period is about 0.25 second. The RLI behavior when you choose a smaller period is undefined.
We have discussed most of the menu items in the View menu. The two remaining items are:
- Reset View:
This is usually used to zoom all the way out so that the maximum Y-axis value is visible. - Set Log Scale:
This sets the Y-axis to a log scale instead of a linear one.
Parameter => Remove Parameter allows you to remove a line plot. In the figure (right), selecting Parameter => Remove Parameter shows the labels of the five line plot labels. Selecting one causes the RLI to remove that line from the chart and terminate monitoring for that line. We will describe Parameter => Add Formula in the Filters, Queues and Bandwidth. But in short, it allows you to plot a line that is an arithmetic composition of several time-series. One such use is to create a stacked appearance where one line is stacked on top of another.
We have not discussed any of the three selections from the Options menu. They are:
- Save to File ...:
This selection allows you to save the configuration of only the monitoring chart to a file. This configuration file excludes other monitoring charts and the topology. It is possible to combine several monitoring files with a configuration file by opening a configuration file and then using the Monitoring => Add From File menu item in the main RLI window to add monitoring. - Change Title:
This just allows you to change the title of a monitoring chart. - Stop:
This allows you to terminate all monitoring associated with the chart and delete the chart. Alternatively, you could terminate the monitoring by terminating the monitoring window.
Recap
- Byte-Packet Counter Pairs:
As a general rule, the statistics counters come in byte-packet pairs; i.e., if there is a byte counter there is also a packet counter. - No Commit:
If resources have already been committed, monitoring does not require a File => Commit. - Rate Versus Absolute Value:
NPR statistics counters are read to provide the raw time-series data for charting. If that raw data is X0, X1, X2, etc., choosing to display the Absolute Value means that you will display the values c X0, c X1, c X2, etc. where c is some conversion constant that depends on the selected units (e.g., bits, bytes, MB). Choosing to display the Rate means that you will display the values c (X1-X0)/T, c (X2-X1)/T, etc. where T is the length of the polling interval (i.e., a difference scaled by the monitoring period and a conversion constant). - Chart Focus:
Use Parameter => Add Parameter in the monitoring window to switch focus when there is more than one monitoring window.
Session Sharing
Alert!!! The session sharing feature described here is in beta testing. Please report any problems to testbed-ops@arl.wustl.edu. We would also like to hear about any successes.
The RLI has a session sharing feature which allows a user to observe another RLI session and converse with that user through a chat window. Session sharing can be useful in two situations:
- Timely Feedback: An instructor/TA can provide timely debugging and feedback to a student/user.
- Peer Programming: Two students (or groups) at different remote locations can work together on an assignment.
This page describes the session sharing concept, and how to share an ONL session. The subsections are:
The Session Sharing Concept
The idea behind session sharing is that you (the user) want to allow another RLI user (the observer) to see your RLI session in real-time and want to discuss what is happening in the session with another user. The figure above shows the user's windows on the left and the observer's windows on the right. Once session sharing has been started between a user (kenw) and an observer (onldemo), the observer sees all changes (configuration and monitoring) made by the user. In the figure above, the observer sees that the user has a 2-NPR configuration and that the user is charting some traffic rates in the bandwidth chart. It also shows in the Chat Received box the conversation history. Each message in the chat history box is preceded by the user's name. The observer onldemo is currently responding to the user's question (see arrow). As soon as the observer types the ENTER key, the message will move into both the user's and the observer's Chat Received box.
So, what can and can not be done in session sharing? The basic ideas are outlined below and discussed in more detail in the remainder of this page.
- Starting a Shared Session:
- Visible Actions:
- Observer Actions:
- Chatting:
A user must use the RLI to give permission for an observer to join a session AFTER he/she has committed the experiment. The observer must then join the session.
The observer will see all of the user's activities except for mouse movements and individual keystrokes. For example, if the user changes a field in the Route Table, the observer will see the change as soon as the the field value has been accepted by the user's RLI (e.g., after the ENTER or TAB key has been depressed). In most cases, this will occur without selecting File => Commit.
The observer only has read-only access to the configuration. So, the observer can not change the configuration values. But the observer can change the appearance of the windows (e.g., move or resize the RLI window and charts).
After the session setup protocol has been completed, a chat window appears that will allow both the user and the observer to send text to each other.
The figure above shows the ONL components involved in making session sharing possible: user RLI, observer RLI, ONL Proxy and ONL Central Daemon (with experiment database). The ONL Central Daemon is the master controller. It:
- authenticates users;
- maintains the state of the virtual laboratory; and
- coordinates other daemons running on endhosts and router control processors (CPs).
The ONL Proxy is a message gateway whose main purpose is to forward requests to the appropriate components. For example, it forwards File => Commit requests to the ONL Central Daemon; it forwards monitoring requests to the appropriate CP; and it duplicates and forwards session sharing data to the observer.
When session sharing is in progress, the ONL Proxy ensures that the user and the observer see a consistent view of a shared RLI session. Because ONL doesn't know a user is active until the user issues File => Commit, no session sharing can occur until the user has selected File => Commit.
In brief, these are the steps involved in sharing a user's RLI session with an observer:
- The user adds an observer to his/her experiment
- Commit an experiment
- Select Extras => Add Observer to Experiment.
- The observer selects an experiment to observe
- Start the RLI
- Select Extras => Get Observable Experiments.
- Enter password into Login dialogue box.
- Select experiment to observe.
- The user accepts the observer
- Grant access to the observer.
- Session sharing starts after the user and observer finish their acceptance protocol.
A dialogue box appears in the observed users screen informing him/her that someone wants to observe.
In the example below, the user will share a simple 2-NPR experiment. The user is kenw, and the observer is onldemo. The user's screen shots are shown on the left side of the page, and the observer's screen shots are shown on the right side.
- The user adds an observer to his experiment
- Commit an experiment
- Select Extras => Add Observer to Experiment.
- The observer selects an experiment to observe
- Start the RLI
- Select Extras => Get Observable Experiments.
- Enter the password into Login dialogue box.
- Select the experiment to observe.
- Select Yes in the Observation Request window.
- Select OK in the Processing Observation Request window.
- The user accepts the observer
- The user selects Invite to accept the observer.
- The observer is notified of the user's acceptance
- A RSVP window appears in the observer's screen.
- The observer selects OK, thus enabling sessing sharing.
Remember: You can not share an experiment with an observer until after you have committed the experiment.
A chat window labeled Observation Window appears.
Select Exps Available for Observation => ....
Note in the example, that the user's login name appears.
An Observation Request window will appear.
A dialogue box will appear in the user's screen notifying them that someone wants to observe their session. A Processing Observeration Request window will appear in the observer's screen.
Note that the observer's RLI window is empty since session sharing has not yet started.
The dialogue box shows that the observer is ondemo.
Note that the observer's RLI window shows a 2-NPR configuration with two hosts and is now identical to the user's RLI window.
The Observer's View
Shown to the left in the figure above are the user's RLI window and and the Queue Table at Port 1. On the right is the observer's version of the same Queue Table. Notice that the observer's port rate is shown in a lighter color than the user's, indicating that the field is not writeable by the observer.
Once session sharing has been established, the observer will see any configuration changes enterred by the user. A field value is considered enterred when the user has keyed in the value followed by the ENTER key or the TAB key. The only exception is the Port Rate which isn't shown in the observer's version until after a Commit.
The observer can look at configuration tables somewhat independent of the user. For example, the observer could decide to display the route table at port 1 even though the user is doing something else and not displaying that table. But any changes made by the user will be displayed in the observer's screen even if it requires a new window to appear.
The Chat Window
Either participant in a shared session can send a message to the other. The figure below shows an ongoing conversation between the user kenw and the observer onldemo. The chat history box (Chat Received) shows that the conversation was initiated by the user kenw. The observer onldemo is composing a new messagea. This message will be sent as soon as the observer depresses the ENTER key.
