Network Performance Test

Network Performance Testing (NPT) is a TR-143 based solution that enables providers to test their service delivery speeds (upload, download and latency) via Device Manager using CPE devices deployed with the properly equipped firmware. This feature is fully compliant with FCC requirements including Connected America Fund (CAF) testing but has been expanded to encompass the general purposes of network performance testing.

This feature requires Device Manager v10.5.0.2 or later for best results. Should your Mosaic Device Manger instance and plug-ins require an upgrade, submit an ADTRAN support request to ensure full support for NPT is present. You'll find complete tutorial for opening a support request including video walk-through and other helpful links here. Other requirements and preparatory steps are available in the Network Performance Test - Preparation article and should be reviewed and understood before proceeding.

Don't miss the free and comprehensive training course on the topic of Network Performance Testing offered at Adtran University. Use this link to sign up. Your adtran.com login is required to access this resource. Upon successful login, you may need to return to this page to select the link again.

Several tiers of speed testing are supported:

  • 100 megabits down and 10 megabits up
  • 25 megabits down and 3 megabits up
  • 10 megabits down and 1 megabit up
  • 4 megabits down and 1 megabit up

At this time, performance testing at gigabit speeds is not supported by ADTRAN CPE devices. If you encounter a requirement for additional tier(s) not listed above, contact ADTRAN Product Support to make your needs known.

This article assumes that all of the one-time setup steps associated with Scripts, Labels and Globals for each tier being tested have been completed in your Device Manager instance.

 

Utilizing Network Performance Testing requires a thorough understanding of the Bulk Operations and Labels features of Mosaic Device Manager. This article assumes the reader has mastery of these features. Use the links above to learn more.

 

STEP 1: Label the test group

  1. Select from existing Labels in your system such as FCC 25_3 or FCC 10_1. If these named labels are not present in your Device Manager instance, learn more about adding Labels here.
    Apply the appropriate Label to all CPE devices associated with a specific test tier. For FCC testing, test sites to be included in the sampling of your subscriber population are dictated by the information you received from the USAC/HUBB portal.

 

STEP 2: Initiate the test

This operation will overwrite results stored from the previous test, if one has been conducted. Ensure you have collected all the test data (instructions below) from your previous test run before initiating another test.

Verify a single device

Use this feature to test a single CPE and to dial in your test settings, upload and download file sizes, etc. This helps to ensure smoother operation once you transition to bulk testing with multiple devices.

  1. From the main Customer Support tab, search for and select the customer record for the individual device you wish to test.
  2. From the left navigation bar, expand the Expert drop-down list and select Network Performance Test.
  3. Add a check-mark to the Marked for Testing checkbox.
  4. From the NPT Tier Selection drop-down menu, choose from one of the pre-configured tiers or select Custom Settings.
  5. In the Parameters section, complete the fields to define your desired Start Time e.g. 2021-03-28T18:00:00 and  End Time  e.g. 2021-03-2811:59:59. 
    • Use ISO Date/Time Format as seen in the example above.  YYYY-MM-DDTHH:MM:SS (Use 24 hour time format.)
    • The time is expressed as your local time zone unless a trailing "z" is added to optionally specify UTC time.)
    • Optimal duration for testing is 5 minutes.
  6. Select the blue Save button near the upper-right. Wait for completion.

The NPT configuration is now pushed to the device. The test is started per the date and time specified after the NPT Tier Selection.

Bulk test all devices in your test pool (most common)

This operation will overwrite results stored from the previous test, if one has been conducted. Ensure you have collected all the test data (instructions below) from your previous test run before initiating another test.

  1. Use the Bulk Operations feature to create a new bulk operation.
  2. Enter a name for this Bulk Operation in the Name: field.
  3. Add a check mark to the Solicit Devices checkbox.
  4. A value for Solicit Attempts must be selected when the Solicit Devices option (above) is checked. This parameter dictates how many times the system will attempt to reach a CPE device for processing of a bulk operation. The default selection is 1 solicit attempt. We recommend a value of 3 for this setting.

  5. In the Max Sessions field, enter the number of maximum concurrent sessions that this bulk operation should be allowed to consume. For purposes of Network Performance testing, we recommend a value of 5 or fewer for this setting.

  6. From the Actions drop-down list, select NPT Start Test.
  7. In the Parameters section, complete the fields to define the Start Time e.g. 2021-03-28T18:00:00 and End Time e.g. 2021-03-28T11:59:59. 
    • Use ISO Date/Time Format as seen in the example above.   YYYY-MM-DDTHH:MM:SS. (Use 24 hour time format.)
    • The time is expressed as your local time zone unless a trailing "z" is added to specify UTC time.)
    • The optimal duration for testing is 5 minutes.
  8. In Bulk Operations, due to the nature of scripts, an error will not occur if time format is entered incorrectly. We strongly recommend testing single device to verify time entries are properly formatted before proceeding with the test for the entire tier.

  1. Under the Schedule section, select Once from the Run drop-down menu. Select the day of the week you wish to push the NPT configuration to the devices and input a Run Time from 12:15 AM or later. This helps to avoid overwriting stored results from the prior test run.
  2. Enter the Beginning Date and Ending Date when the Bulk Operation should run to program the selected pool of CPE devices to be tested. The date specified here should be the same date used in step #3 however the time should be a few minutes earlier.
  3. In the Run from field, enter the time the Bulk Operation should run. The Bulk Operation must be completed well ahead of the time specified above for when the CPE will begin executing the test.
  4. The Bulk Operation needs to happen first as its job is to push test configuration parameters to the CPE. The Parameter for Start Time in step 7 tells the CPE when to actually begin the test. Thus, the time of day entered in step 7 must fall later than the time that set forDevice Manager to run the Bulk Operation in step 10 (Run From). We recommend the Bulk Operation run at least 1 hour before the CPEs are set to begin testing.

  1. In the Filter Criteria section, create a filter for the Label you applied to your pool of test devices in the Label the test group instructions above.
  2. Select the Create button in the lower-right corner of the screen to commit your bulk operation.
  3. To confirm that the NPT parameters were accepted by the CPE, check the script log of at least one device of each model (device type) to confirm that the CPE accepted the parameters. See Working with Scripts for more information.

The CPEs will automatically begin testing at the time you specified in step 7.

 

STEP 3: Collect test data from CPE

Test results are stored on the CPE device until collected by Device Manager. This happens automatically when the “Diagnostic Complete” event triggers the execution of the “NPT - Get Results” script. This automated script transfers the test results from the CPE to Device Manager by synchronizing the NPT application. It is not necessary to run another bulk operation or take additional action.

The results are typically collected and available for viewing just a few minutes after the test End Time you specified in earlier steps described above.

 

STEP 4: Download test results from Device Manager

The following is a manual process that must be performed daily. If this step is skipped today, the data retrieved from the CPEs by Device Manager in STEP 3 from this test run will be overwritten tomorrow when the tomorrow's bulk operation runs.

  1. From the horizontal navigation bar, select REPORTS.
  2. From the left navigation bar, select Network Performance Testing Report.
  3. Select the name of the Label associated with the test group for which you wish to retrieve performance test results. This would be the same label previously applied to the pool of test devices in the Label the test group instructions above. To create report output for multiple test groups, Shift-select multiple labels from this list.
  4. Use the calendar interface at the Start Date (On or After date) field to specify the start date for a test previously conducted using the Initiate the Test instructions above.
    • A Device Query Count will appear in the Report Results box to the right.
  5. (Optional) The Filtered CSV Results option will consolidate all test telemetry down to one test per hour for both upload and download testing and one test per minute for latency tests. The data is arranged in a hierarchical fashion to highlight Success -> Deferral -> Error states of outcome for submission of CAF test data to the USAC/HUBB portal.
  1. Select the Save As JSON or Save As CSV button to save the report output to your local drive for later examination or distribution. *
    The name of the output file is displayed to the right in the Report Results box (e.g. npt-results-report-xxx). The file can then be retrieved under Download Report at the left navigation bar.
  2. Open the .csv file and note the empty column for "Location ID". This column must be manually populated by you prior to submission of these test results to the USAC/HUBB portal.

*CSV output is formatted and suitable for submission to HUBB for FCC programs only after you have completed the manual modification to the .CSV file described in step 7.

 

In coordination with the recently revised upload validation criteria performed at the USAC portal, Mosaic Device Manager, Network Performance Test reports now include a line item for bandwidth and latency tests for which no data was received. The Comment column in the report will state "Device data not received" when this condition occurs.

Mosaic Device Manager v10.7.0.1 or later is required for full FCC test data submittal compliance. In previous versions of Mosaic Device Manager, if no data was retrieved from a CPE device under test, there was no data in the report output to indicate the test was attempted.

 

Step 5: Interpreting the test results

Now that you have test output data in hand, it's a fair question to ask, "Did we pass?". Here is one method to help you to read the bandwidth report and determine if the CPE devices tested are passing or failing FCC requirements.

The data in the Mosaic Device Manager report output is represented in bytes so the first step is to convert the bytes_transferred to bits per second. The formula for this is:
bytes_transferred * 8 / test duration / 1000000. Refer to the PMM Quick Tips Guide at www.usac.org for more information.

If you are viewing the CSV data from Device Manager in Excel, try using the following formula.

=G2*8/IFS((MID(F2,18,2)&"."&MID(F2,21,3))-(MID(E2,18,2)&"."&MID(E2,21,3))<0,(MID(F2,18,2)&"."&MID(F2,21,3))+60-(MID(E2,18,2)&"."&MID(E2,21,3)),(MID(F2,18,2)&"."&MID(F2,21,3))-(MID(E2,18,2)&"."&MID(E2,21,3))>0,(MID(F2,18,2)&"."&MID(F2,21,3))-(MID(E2,18,2)&"."&MID(E2,21,3)))/1000000

Follow these steps for employing this Excel formula:

  1. Copy the formula text above and paste it into cell J2 in the bandwidth report.

  2. Copy from cell J2 then paste it down the rest of the rows in column J. The cell references in each copy must increment and become relative to the new row number for each copy of the formula.

  3. The calculation results now displayed in column J should simplify assessment of which devices fell within or outside of the FCC spec for the tier tested.




Video Tutorial - Part 1
 
 


Video Tutorial - Part 2

RELATED ARTICLES:

Network Performance Test - Preparation

Bulk Operations - Creating Bulk Operations

Labels Overview