Network Performance Test - Preparation

Network Performance Testing (NPT) is a TR-143 based solution that enables providers to test their service delivery speeds (upload, download and latency) via Device Manager using CPE devices deployed with the properly equipped firmware. This feature is fully compliant with FCC requirements including Connected America Fund (CAF) and Alternative Connect America Cost Model (ACAM) program testing but has been expanded to encompass the general purposes of network performance testing.

This feature requires Device Manager v10.7.0.2 or later for best results. Should your Mosaic Device Manger instance and plug-ins require an upgrade, submit an Adtran support request to ensure full support for NPT is present. You'll find complete tutorial for opening a support request including video walk-through and other helpful links here.

Three components must be configured before running NPT tests:

  1. Test Controller - The Mosaic Device Manager system needs to be configured to run Network Performance Testing. Please contact Adtran Support if assistance is needed.

  2. Test Client - Compatible device firmware must be installed on the target CPE (See Below for Compatible Firmware). For the most up to date information on which firmware supports this feature, please contact the Adtran Support team for additional assistance.

  3. Test Server: Testing infrastructure must be in place. Adtran offers test servers as a subscription service or you can set up your own servers. Please contact your Adtran Sales representative to inquire about the subscription service or if you intend to build your own, see below for test server infrastructure requirements .

Compatible Firmware

See the this article at supportcommunity.adtran.com for the most up to date firmware requirements for best NPT performance and reliability.

Firmware downloads for Broadcom-based, Adtran/SmartRG CPE are available here.

Firmware downloads for SmartOS-based, Adtran/SmartRG CPE are available here.

Once upgraded to a release at or beyond the minimum firmware version listed above, the NPT feature will be available in Mosaic Device Manager for the device.

SmartOS CPE devices may require a Factory Reset post firmware upgrade. See special bulletin for SmartOS devices here.

 

Compatible Device Manager Plug-Ins

See the this article at supportcommunity.adtran.com for the most up to date Mosaic Device Manager plug-ins requirements for best NPT performance and reliability.


Device Settings

Before proceeding with performance testing, ensure that the following is true for all devices in all tiers under test.

  • Ensure the timezone on the device is set to it's local time.

 

Required Test Server Infrastructure

The NPT feature uses three different types of tests:

  1. Download: Downloads a file from a server.
  2. Upload: Uploads a file (the firmware sends the specified number of random bytes) to a server.
  3. Latency: Devices can use Internet Control Message Protocol (ICMP) (recommended) or User Datagram Protocol (UDP) Echo.

Download and upload servers are available as a Subscription Service. Please contact your Adtran sales representative for more information about FCC Test Servers. Our observations to date suggest FTP transfer offers significantly higher performance than HTTP. However, for FCC testing at the 10/1 or 25/3 levels, HTTP will be supported on all devices. The highest speeds (50/5+) supported will be with FTP only.

Download Server

  • NPT uses time-based tests. The CPE will download bytes of the specified file until the time duration or end of file is reached. The size of the file will need to be adjusted to fill the required 10 second test duration.
  • The served test file must be accessible from the CPE’s WAN network connection.
  • TR-143 specifies that no authentication can be required for HTTP sessions, however FTP can require a login. We do not recommend requiring HTTPS sessions for performance testing.
  • HTTP is recommended for CGNAT.

Do not exceed 205Mb for Download File Size, as the device under test may become unstable due to memory available.

Upload Server

  • Due to a limitation in the CPE firmware, NPT supports file-based uploads only.
  • Careful sizing of the test file* for an HTTP upload test is important. It may take a few trials in your environment to ensure the file is sized to take only slightly longer than the specified Test Duration. When the upload initiates, the destination HTTP server requires the entire size of the file however the upload time stats Device Manager captures and reports are limited to the amount of data passed during the Test Duration. Regular directory deletions should be executed so that the server file system does not get full. See below for more about test file size.
  • HTTP is recommended for CGNAT.
  • TR-143 specifies that no authentication can be required for HTTP sessions, however FTP can require a login. We do not recommend requiring HTTPS sessions for performance testing.

*IMPORTANT: More About Test File Size

The formula for calculating upload file size is: speed * test duration / 8 (to convert from Megabits to Megabytes)

For FCC/Connect America Fund (CAF) testing, the upload must run for at least 10 seconds. However, consider a file scaled for closer to a 20 second test run to ensure test integrity.

 

Example:

Using the formula above-- if the speed is 5 Mb / second and the test duration is 20 seconds, the file length would be
100 Mb. 100 / 8 = 12.5 M.

The result of the equation indicates a file size around 12.5 M should be used.

5 * 20 = 100

100 / 8 = 12.5

For example, assume you are using either the 10/1 or 25/3 (down/up Mbps). If the target is a 15 second test run, the numbers should be 1,000,000 * 15 / 8 = 1875000 bytes or 3,000,000 * 15 / 8 = 5625000 bytes for the respective tiers. As a service provider you may have multiple uplink speeds in your test population, so plan to set up multiple configurations so that each sub-cohort gets the correct parameter value for the anticipated number of seconds.

Latency Server

  • TR-143 only allows for UDP Echo for latency tests.
  • Any UDP Echo server should be supported by Adtran firmware. Several open source solutions are available on GitHub and elsewhere if needed.
  • ICMP is also a supported alternative for latency testing and most recommended by Adtran. If this is your testing end point of choice, ensure it is configured such that the field devices to be tested are able to ping your ICMP server.

 

Establish a daily routine

Testing a pool of CPE devices may be required for multiple consecutive days. Managing the NPT process requires some manual steps that must be performed each day.

Consider this example scenario:

Evan is the Device Manager admin at Vision Broadband. His company has testing requirements from 6pm to midnight for multiple consecutive days. Here is an example of how Evan might choose to map out his day to ensure he performs the administration tasks needed to initiate testing and retrieve test results.

  1. Because the testing needs to begin at 6pm, Evan sits down at 4pm to login to Mosaic Device Manager and build the bulk operation. The bulk operation is responsible for pushing the test parameters to the CPE devices. During this step, Evan will specify the following in the Bulk Operation setup:

    1. What time the bulk operation runs (when the devices will receive their test parameters). Evan has chosen 3:30PM to 5:30 PM device local time (4:30pm).

    2. The start time and end time for when the CPEs will actually begin testing. Per the testing requirements Evan has received, The test start time is 1800 hours (6:00pm) and the test end time is 23:59:59 hours. (Note that if the test ended at 2400 hours, the date associated with the test end time would be the date following the start date.).

  2. Mosaic Device Manager runs the bulk operation at 3:30pm local time, just as Evan specified in Step 1. It takes a few minutes for the bulk op to complete and push all of the test parameters to the CPE devices in the pool. (This step is automated.) Any devices that are not Solicitable inform during the period and then receive their settings.

  3. Per the test start time that Evan specified in Step 1, the CPEs begin performing their upload, download and latency tests. (This step is automated.)

    1. The test results are temporarily stored on directly on the CPE device and are therefore not accessible until later (see step 4).

    2. While the tests are running, do not schedule another bulk operation to run against the same pool of CPEs. Doing so will disrupt the test in progress.

    3. When the testing cycle is complete for today, an automated script is triggered that downloads data from the CPEs to the Mosaic Device Manager database.

  4. Now that the testing cycle for today is complete, Evan logs in to Mosaic Device Manager to run the report which retrieves the test data for this cycle and stores it in a .csv or JSON file on his local hard drive.

    1. This is a manual step to be performed by the Mosaic Device Manager Administrator.

    2. This can be done at any time after the test cycle is complete (in this example scenario, that’s 0001 hours or 12:01am) up until the afternoon of the next day when Evan needs to build the bulk op for the next day’s cycle of tests.

      If Evan skips Step 4 and sets up another bulk operation for testing the same pool of devices again, the data from the test cycle that just completed will be lost.

You'll want to customize this example scenario to create daily routine that meets with whatever testing and data collection requirements your organization may have.

 

Managing Devices In Between Tests

Making sure the devices being tested are up and solicitable or informing is imperative to successful testing. If devices are not reachable during a bulk ops, and also fail to inform during the running period, they will not complete the NPT test.

Similarly, if a device is rebooted or unplugged by a subscriber before or during a test you will not receive the appropriate results. Adtran recommends logging in and checking each day to make sure each device is informing and is solicitable, as well as ensuring that any devices that become unreachable are investigated with the subscriber, or in the case of PMM testing, swapped out from the test pool with USAC.

 

Managing Devices That Inform, But Are Not Solicitable

In some cases, you may not be able to solicit a device, but it may still be informing. There are many reasons this can happen, such as issues with carrier grade NAT, network configuration, or firewall settings. It is recommended to utilize the troubleshooting guide located on the Adtran Support Community to assist your investigation of these issues.

If the problem persists despite following all troubleshooting measures, it is recommended to run a job to set the inform interval for the affected CPEs to 1 hour and 45 minutes. Do so only for unreachable devices in your test pool to keep the MDM from becoming unstable. Next, it is recommended to set your bulk operations to run for a duration of 2 hours for the NPT-Start Test script. This gives the device two chances to inform during the period and still successfully initiate the test parameters within the CPE.

 

What next?

After preparing your CPE devices in the field, establishing required test servers and planning out a daily routine as described above, proceed to initiate the tests.
See instructions in the Network Performance Test article for details.

Don't miss the free and comprehensive training course on the topic of Network Performance Testing offered at Adtran University. Use this link to sign up. Your adtran.com login is required to access this resource. Upon successful login, you may need to return to this page to select the link again.

 


Video Tutorial - Part 1
 
 


Video Tutorial - Part 2

RELATED ARTICLES:

Network Performance Test