This page or subpages are intended to track Wide Area Network (WAN) performance within the NRAO, primarily for improving performance instrument data mirroring and user access.



Procedures and Documentation

  • Documentation for how to run tests against
    • Internal networks
    • Point to point (AOC to CV) networks
    • External networks
  • Documentation or pointers to documentation for the affects of things like buffer size, window size, latency etc


  • Understanding what's expected
  • How to interpret results


  • Proper long haul gbit tuning (targeted at both internal and external users)

AOC-CV network

We need to support a minimum of 25MB/s transfers from AOC to CV to effectively mirror EVLA data. This may need to be as high as 75MB/s to handle peak data rates. We can potentially meet that limit buy using multiple channels. There are 4 active NGAS nodes mirroring to 4 slave nodes, in theory if each node can sustain nearly 15MB/s we will be in good shape.

Tests and Results

  • AOC test system
    • The AOC test system has an internal and external interface.
    • is connected to an internal network and is behind the site router and main building switch.
    • is connected to a switch between the AOC and New Mexico Tech (NMT) routers.
    • Note Which selecting a different interface requires changing a static route on the test node, otherwise it will reply out the default interface and the pattern will be asymmetric.
      • route add -net netmask dev eth1 would force it to route all traffic to CV test fixture out the external interface.
  • CV test system
    • CV test system has a single internal interface
    • is connected to the main building switch.

Initial tests show insufficient bandwidth between the sites. More importantly it shows highly varying rates between portions of the path. There is a minor difference when testing through the internal and external AOC test fixture described above.

A current diagram showing all the devices between the test points, their connection bandwidth and initial results can be found here: WANdiag.pdf

  • Links are color coded for 1, 10 and 100Gbit
  • Results are color coded depending on they were the result of Web100 tests or iperf tests.
  • Rates describe the input rate experienced by the receiving end.
    • For example the 106Mbit/s and 755Mbit/s rates from to the UVa Web100 server means 106Mbit/s from CV to UVa, 755Mbit/s from UVa to CV.

  • Test to/from AOC & TeraGrid show: 16.6MB/s - 32.3MB/s today on upload and 5.7MB/s to 7.4 MB/S on download ( &
  • Test to/from CV & TeraGrid show: upload fairly saturated and 6.5 MB/s - 21.0MB/s today on download ( &

To be Done

  • Contact UVa and NMT/UNM James
    • Check link congestion, UVa is the more critical issue, NMT seems fine.
    • Check traffic shaping
    • Check topology
  • Move iperf to Cato Mike
  • Allow inbound port 12345 to all test IP's from any address for symmetric iperf tests. Currently can only test outbound. Gene/Derek
  • Get UNM and Virgina NLR contact info James
    • I'll handle talking to the two sites
    • (Paul Chang)
    • Request tests between UNM-NLR and UVa-NLR
  • Contact Tang at UVa and verify the Web100 server buffer settings didn't revert. That would explain the low inbound rate. I opted not to send the email, in retrospect I don't have evidence there's anything wrong with the Web100 server at UVa

Configuration changes

Congestion control options: /proc/sys/net/ipv4/tcp_congestion_control
modprobe tcp_htcp No clear difference
modprobe tcp_cubic No clear difference

-- JamesRobnett - 2011-07-19

Topic attachments
I Attachment Action Size Date Who Comment
CV-SO-Teragrid-RoutesV1.pdfpdf CV-SO-Teragrid-RoutesV1.pdf manage 37 K 2011-07-28 - 11:40 DavidHalstead CV-SO_routes
WANdiag.pdfpdf WANdiag.pdf manage 4 K 2011-07-21 - 18:11 JamesRobnett CV-AOC Link Diagram
Topic revision: r12 - 2012-08-08, JamesRobnett
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding NRAO Public Wiki? Send feedback