Extreme Management Center Troubleshooting

The following sections provide information on tools that can be used when troubleshooting Extreme Management Center server issues.

System-Wide Extreme Management Center Server Diagnostics

The Administration tab in Extreme Management Center provides detailed statistical and diagnostic information regarding the overall performance and operation of significant Management Center server functions. It is important to review all the reports available on the Administration tab (Diagnostics sub-tab) so that they can be leveraged when diagnosing and troubleshooting problems.

There are three levels of information available: Basic, Advanced, and Diagnostic. The Basic level provides reports indicating the overall status of the server. Increasing the level to Diagnostic adds additional reports providing more granular insight into the overall operation of the server.

For example, the Diagnostic level adds reports on Database Disk Usage, Partition Information, and Web Application Details, providing in-depth information such as how much each Management Center component contributes to the overall size of the server database, and DWR statistics including the number of calls made and the average time to complete.

The Diagnostic level also includes a menu of Diagnostic Actions which provide the ability to Save Diagnostic information (similar to the Show Support function) and also enable additional advanced diagnostic logging levels. (Note: As with any diagnostics, care should be taken when enabling increased diagnostic logging and is advised only with the guidance of an Extreme Networks Support Engineer.)

Administration Tab Diagnostics

Extreme Management Center Server Real-Time CPU and Memory Usage

Troubleshooting Extreme Management Center server performance issues related to CPU and memory utilization is easier when you can refer to a CPU utilization baseline that was charted over a period of time.

Knowing the system's normal operating behavior makes it easier to identify when the server's behavior deviates from the norm and when exactly the change occurred. It also makes it easier to identify whether the deviation corresponds to a recent change on the system or network.

For more information on how to establish a baseline, see the Track Management Center Server CPU/Memory Trends section under Monitor Management Center Server Health.

Once you have your baseline, use these four tools to monitor the Management Center server CPU and memory usage in real-time and compare that data to the baseline.

Server Statistics Window

The Server Statistics window provides a real-time view of server CPU and memory usage. Access the window by clicking the Server Information icon on the toolbar of any Management Center application, or by selecting Tools > Server Information from the menu bar. In the Server Information window, click on the Server Stats button to open the Management Center Server Statistics window, as shown below.

The window shows real-time CPU and Object Heap Memory usage. (Heap memory refers to the amount of free memory available to the Management Center server program.) Consistently high CPU usage and/or memory usage that continues to steadily increase are indicators that the system may be under duress. This would typically manifest through degraded performance or server stability issues. Both provide valuable data points when trying to understand the root cause of an issue, for example, whether performance is tied to a CPU spike, and whether that spike is specific to Management Center or the overall system.

Management Center Server Statistics Window

Windows Task Manager

Windows Task Manager is a utility on Windows-based systems that provides a real-time view of system CPU and memory utilization. It monitors the entire system's performance to help identify whether an issue is specific to Management Center or a side effect of another process running on the system.

On the Processes tab, sorting the CPU and Memory columns lets you monitor the activity of the individual active processes, giving a real-time picture of the specific processes that may be consuming a larger than normal share of resources.

The Performance tab is an excellent overview of global CPU and Memory trends, providing a view into whether elevated usage has been sustained for a period of time or whether it changes at regular intervals, perhaps as the result of performing a specific task or action.

Windows Task Manager

Linux "top" Command

The Linux equivalent of Windows Task Manager is the "top" command, which displays the most CPU-intensive processes on the system and provides an ongoing look at processor activity in real-time.

The command displays system summary information as well as the processes currently being managed by the Linux kernel. The display continuously refreshes, providing valuable insight into the processes that are consuming the most resources and the ability to quickly monitor incremental usage of specific processes that may be slowly consuming resources over time. The output contains useful information such as CPU, Memory, Process Time, and Process ID.

Refer to Linux operating system documentation for more information about the "top" command.

Linux "top" Command Display

Linux "vmstat" Command

The Linux "vmstat" (virtual memory statistics) command displays summary information about operating system memory, processes, interrupts, paging and block I/O. The first report displayed provides the averages since the last reboot.

If you want to use the command to monitor the virtual memory activity on your system, it is best to specify a sampling interval. Executing the command at predefined intervals helps to identify the rate of change in specific values. The command "vmstat -an 10 15" executes with a display that runs every 10 seconds for 15 iterations, and includes active and inactive memory usage.

Of particular interest would be whether the system is using swap memory, and if so, how frequently and what are the overall memory conditions when this occurs. Keep in mind that it is normal for Linux-based systems to have higher memory consumption due to the operation of the OS, but further investigation is warranted when swap memory is being used.

Refer to Linux operating system documentation for more information about the "vmstat" command.

Linux "vmstat" Command Display

Extreme Management Center Server Disk Utilization

Unmanaged log data or backup files can quickly consume disk space, creating a disk space problem on the Extreme Management Center server. Disk space issues can cause problems such as an upgrade that fails to execute due to inadequate disk space or, more seriously, a Management Center server crash.

For information on managing Management Center database backup files, see Tuning Database Backup Storage. For information on managing log files, see Management Center Log Files in the Reference Information section of the Console User Guide.

Here are three tools you can use to troubleshoot disk usage problems on the Management Center server.

Windows OS Properties

For Windows-based systems, access the Properties window to view disk usage data. In Windows Explorer, right-click on the drive where the Management Center server is installed and select the Properties option from the menu.

Use the data in the window to verify that there is adequate space available for normal Management Center server logging and operations, and investigate low disk availability immediately. There are two Management Center-specific directories that you can check for excessive disk space consumption:

<install directory>\backup

<install directory>\appdata\logs

You can control the automatic file size management of these directories using Management Center options. For more information, see Tuning Database Backup Storage under Management Center Performance Tuning and Tuning Data Persistence under NAC Performance Tuning in the NAC Technical Reference.

Windows Properties

Linux "vmstat -D" and "vmstat -d" Commands

The Linux "vmstat" (virtual memory statistics) command with the "-D" or "-d" option displays disk read and write activity. The "-D" option provides a summary of total disk activity for the entire system, while the "-d" option displays individual disk activity statistics for each disk on the system.

These commands are useful when monitored disk activity (see Monitor Management Center Server Disk Access) indicates an abnormal change that corresponds with performance-related issues on the Management Center server. The disk statistics provide a real-time view into disk activity that can perhaps be tied to specific actions.

Refer to Linux operating system documentation for more information about the "vmstat" command.

Linux "vmstat -D" Command Display

Linux "df" and "du" Commands

The Linux "df" command displays how much free disk space is available on the file system. Use "df" to view the overall system disk utilization and identify whether certain directories are at or near full capacity. Use the "-h" option to display the statistics in human readable format.

The "du" command can be used to identify the size of specific files or directories within a parent directory. This command is useful when trying to narrow down specific areas of the file system that are consuming large amounts of disk space.

  NOTE: Running the "du" command from a parent directory with the "-ha" option may return a large amount of data. Use the "|more" option or redirect to a file to view the total results for the command.

Refer to Linux operating system documentation for more information about the "df" and "du" commands.

Linux "df -h" Command Display

Extreme Management Center Server Diagnostics

Use the Administration tab (Diagnostics sub-tab) to obtain detailed debug information for Extreme Management Center server processes and applications. Select from a list of diagnostic options to enable targeted debugging of specific system functions. This allows for the collection of relevant data and also helps to limit the overall impact to the system when debugging is enabled.

  CAUTION: Management Center server diagnostics should only be enabled when a specific area of an issue has been identified, or under the guidance of Extreme Networks Support. Certain debug options generate verbose output and could cause adverse effects to the system if enabled for extended periods of time. Debug options should only be enabled long enough to capture data from the behavior under investigation.
 

Access Server Diagnostics in Management Center by clicking the Administration tab and then expanding the Server section on the Diagnostics sub-tab. Enable the different diagnostic groups by setting the Diagnostic Level to "Verbose" for the applications and processes of interest. Debug diagnostic information is mainly saved to the active server.log file on the Management Center server.

Server Diagnostics

Accessing the Server Log File

The server log file records server activity such as start-up log data, unexpected behavior messaging, and server debug diagnostic log data when enabled. Reviewing previous server log file output can be helpful for determining when a specific issue started to occur or for uncovering additional system issues that may be contributing to a current problem.

The file is archived daily into the same directory and is generally named in the server.log.date format.

The server log file can be accessed from the Management Center Launch page by clicking the Administration tab and then the Server Log tab. You can also view the Server Log in the Server Information window, which can be launched from the Tools menu or toolbar in every Management Center application. The Server Log can also be accessed from the <install directory>\appdata\logs directory on Windows-based systems or from the /usr/local/Extreme_Networks/NetSight/appdata/logs directory on Linux-based systems.

Examples of the server log file locations are shown below.

Server Log File Locations

Extreme Management Center Client Diagnostics

Troubleshooting unexpected behavior encountered while running applications in Extreme Management Center is not limited to server-side investigations. Some behavior may be specific to individual end-systems or may be the result of a configuration on an end-system.

Use the following steps to enable diagnostics for Management Center client-specific debugging of individual Management Center applications.

  1. From the Management Center Launch page, click on the Administration tab. Click on Client Diagnostics. You need to log in with your username and password. Select the User Configured Diagnostics radio button and click Apply.

  1. Once client diagnostics are enabled, the Launch page displays "(User Configured Diagnostics)" following each application name. Launch the desired application (for example, Console or Policy Manager).

  2. The Diagnostics Configuration window opens where you can enable diagnostic message logging for a specific application or process. Click on the table entry for the application or process and open the table editor using the Table Editor button in the upper right corner. In the Table Editor row at the bottom, set the diagnostic level. Be sure to apply your changes using the Apply button in the upper right corner. Click OK to close the window.

  3. Perform the action that generates the undesirable/unexpected behavior. The relevant debug is saved to the corresponding application log file on the client machine, located in the \Users\<user>\AppData\Roaming\NetSight\logs directory on Windows 7 or the \Documents and Settings\<user>\Application Data\NetSight\logs directory on Windows XP.

Extreme Management Center Certificates

Extreme Management Center uses server certificates to provide secure communication for application web pages and for internal communication between server components.

On the Management Center server, use the Server Utilities page to view Management Center server communication certificates. Access the page from the Management Center Launch page by clicking the Administration tab. Click on Server Utilities. You need to log in with your username and password. Click on the Certificates link to open a new window that displays the Server Certificate Chain.

For more information on updating and configuring the Management Center server certificate, see the top-level Help topic Management Center and NAC Secure Communication and the How to Update the Management Center Server Certificate Help topic in the Server Information section of the Suite-Wide Tools User Guide.

Server Utilities

Generate a Show Support Report

It is helpful to generate a Show Support report to provide to GTAC when requesting assistance with Extreme Management Center. The report collects important statistical and diagnostic data pertaining to each of the applications within Management Center and creates a single archive that your support technician can review.

This data helps to answer many of the up-front questions typically asked when opening a support case, and also provides a comprehensive collection of key data that can reduce follow-up requests for more data and expedite a resolution to the issue.

The Show Support report is generated from the Administration tab. Click the Diagnostics sub-tab and then expand the Support section. Click the Start button to start generating the report.

Administration Tab Generate Show Support

Upon completion, archived data is stored on the Management Center server in the <install directory>\appdata\ShowSupport folder in a single zipped archive that can be sent to update an open support case.

Management Center Show Support Folder

06/2017
7.1 Revision -00
Contents Subject to Change Without Notice