Login VSI Analyzing Results

From Login VSI Documentation
(Redirected from Analyzing Results)
Jump to: navigation, search

The Login VSI Analyzer will process the data collected during a VSI workload. It will calculate if the target environment has reached its saturation point and if so, at how many concurrent sessions. This point is called VSImax.

Analyzing Results Video

Update September 3, 2014: Blog - Analyzing Login VSI Results

Interpreting Test Results

When analyzing the results of your Login VSI test, there are several things to take into consideration. The first thing to realize is that there is no silver bullet that will point to the cause of any performance degradation. You must take an overview of the whole test to find out where the problem lies. If your graphs look like Example A, then your system is working within normal parameters. However, any deviation from this result indicates that there is an issue somewhere in the environment (such as Example B or C).

                                             Example A: Good chart.png Example B: Bad chart 1.png Example C: Bad-chart.png

Once you think you have discovered a suspected bottleneck, you should then check the rest of the metrics to see if they confirm the problem. For example, if you see the IO latency spiking (anything above 50ms), this indicates the storage platform that hosts the %temp% directory of the user, i.e. LoginVSI1/2/3 is the bottleneck. Anything higher than 100ms (but constant) can be the impact of a virus scanner. In this circumstance, please also check the VSImax v4 Detailed graph for the FCTS (VSIshare to %temp%) and FCTL (%temp% to C:) scores. If the FCTS score is spiking, this indicates that the network is your bottleneck, while FCTL spikes indicate that your local disk is the problem. However, if both IO latency and CPU scores are spiking, then it is more likely a CPU bottleneck. While a true IO problem also produces a large amount of stuck sessions.

We recommend checking the summary of the test first to see the results of VSImax (maximum number of users) and the Baseline (performance of the system without stress). You should then check the most important graphs to see if any patterns and/or clear performance bottlenecks are visible. The four most important graphs are:

  • VSImax v4: If the VSImax v4 graph reaches the VSImax point very early (VSImax = 1-20), this can (it’s never certain) indicate that there is an image management problem
  • IO: If the IO graph starts steadily but then fluctuates aggressively, then this can indicate that there is a problem with the storage platform
  • CPU: If the CPU graph starts with high peaks before settling down as time passes, then this can indicate that there is a power management problem
  • AppStart: This graph can tell you if a specific application is experiencing any problems

In addition to these four graphs you should also check the Raw Data table, which can help you focus your search. For example, you can sort the computername or total_response columns to see if there were any issues with individual VMs in relation to a specific host. We also recommend importing external performance data into the Analyzer from the host level (ESXTop, PerfMon or XenTop) to help you verify your suspicions behind any performance degradation.

Please be aware that many different factors can influence your test results in a negative way, such as hardware and/or software changes, anti-virus scans, backup jobs, or scheduled tasks. We also suggest that you run several tests and compare them with each other to see if the results are reliable. Statistically speaking, running ten consecutive tests gives enough data to get a reliable result. Once you have collated the results, it is best practice to remove the highest and lowest VSImax measurements (after sorting), and then average the rest to get the definitive VSImax value.

If you want in-depth help with analyzing your Login VSI test results, then please contact a member of Login VSI Support via: support@loginvsi.com

Opening the Analyzer

Description

Screenshot

Start the Login VSI Analyzer by clicking the Analyzer button in the bottom left corner of the Login VSI Management Console.

Note: You can also open the Analyzer via the VSIshare and the shortcut on the desktop.

Analyzer1.png

The location of the cache and the location of the VSIshare will need to be specified when the Analyzer runs for the first time.

2013-11-04 01 52 21-Login VSI Analyzer.png

Specify the path to the VSIshare. Change the location of the cache if desired. Click Save to continue.

2013-11-04 01 52 53-Login VSI Analyzer.png

Select the test that you want to analyze and click Open to start the analysis.

Analyzer2.png

Analyzer Preferences

This analyzer overview is only available when the following workloads are used:

  • Task Worker
  • Knowledge Worker
  • Office Worker
  • Power Worker

These preferences are configured for best practices, by default. However, some cases do require some additional settings or modifications. This section will describe what each preference tab is used for and how to adjust them.

General

Description

Screenshot

  • Show splash screen on startup

Will show the Login VSI Analyzer splash screen when starting the Analyzer

  • Show log dialog on startup

Will show the Login VSI Analyzer's default log location at startup (with ‘default’ we refer to the primary ‘VSIShare location’)

  • VSIshare location

The location of the VSIshare to use for analyzing the logs

  • License file

The location of the Login VSI License file to use for the Analyzer (default Login VSI license location is: \\<hostname>\<VSIShare>\_VSI_Configuration\LoginVSI.lic)

Local Cache Settings

  • Local cache location

The local cache location used by the Analyzer

  • Overwrite cache entry

The overwrite cache entry method used: Always prompt user (default) / Never overwrite / Always overwrite

41AnalyzerPref1.PNG

Chart Settings

Description

Screenshot

  • Chart window size

Select the chart window size: Normal (default) / Maximized / Minimized

  • Graph line width

Select the line width value shown in the charts: Selection from 1-100 (default = 1)

  • Automatically detect axis values

This will automatically detect and set the best axis values for the charts (default)

Unmark the checkbox to manually change the axis values used to generate the charts:

 Sessions Axis (X)
 -Minimum value shown
 -Maximum value shown
 Response Axis (Y)
 -Minimum value shown
 -Maximum value shown
  • Chart saving width

The width of the chart(s) when saving chart(s) (default = 1024)

  • Chart saving height

The height of the chart(s) when saving chart(s) (default = 768)

41AnalyzerPref3.PNG

Analyzer Tabs: Troubleshooting Graphs

Description

Screenshot

These troubleshooting graphs are only available with Login VSI 4.1.25 and higher:

To enable them, simply mark the checkboxes next to each troubleshooting preference, click Apply, then click the OK button. Once a test has finished, you will see that the new graphs have been added to the list of tabs in the Analyzer.

Note: All the tabs are automatically enabled except for the three troubleshooting graphs. However, if you are running large-scale data tests and/or SOAK tests it might be useful to unmark (disable) the checkboxes of certain graphs that are not required.

Analyzer troubleshoot tabs.png

Analyzer Graphs

This analyzer overview is only available when the following workloads are used:

  • Task Worker
  • Office Worker
  • Knowledge Worker
  • Power Worker

Summary

Description

Screenshot

This tab gives you an overview of the completed test, including the VSImax result.

The Test result review table summarizes the results of the test:

  • No. of sessions launched in (amount) of seconds
  • No. of failed sessions during the test before VSImax was reached (or before maximum amount of users was reached)
  • No. of sessions that did not successfully launch
  • No. of sessions that failed to become active
  • No. of sessions that were active during the test
  • No. of sessions that got stuck during the test (before VSImax threshold)

Maximum Capacity

Tells you if the maximum capacity of the test was reached, and if so, at which point, as well as what the baseline performance score is.

Configured Sessions

Tells you the recommendation concerning the configured amount of sessions.

Baseline

Lets you know the baseline performance rating:

  • 0-799 (Rating: Very Good)
  • 800-1199 (Rating: Good)
  • 1200-1599 (Rating: Fair)
  • 1600-1999 (Rating: Poor)
  • 2000-9999 (Rating: Very Poor)
SummaryO16.png

Settings

Description

Screenshot

This tab contains the settings that were used for the test.

Description

If a description was included, it can be found here.

Custom Command Line (CCL)

The configured CCL from the completed test.

Workload Settings

Displays the configured Office version and workload language.

Scenario

Displays the configured workload, number of sessions and sequential interval.

SettingsO16.png

VSImax v4

Description

Screenshot

The VSImax v4 graph is the main tab of the Login VSI Analyzer and shows the most important information.

Note: This line graph can also be displayed in scatter format.

VSImax v4.png

Reporting, Zooming and Scaling

To generate an overview of the test results, click Generate Report.

To zoom in, click the Zoom icon, then use your mouse to highlight the area of the graph you would like to take a closer look at.

To adjust the scales of both the X-axis and Y-axis, click the Scale icon to open the GraphControls pop-up.

Scale-pop-up.PNG
Scale.png

VSImax

VSImax shows the amount of sessions can be active on a system before the system is saturated. The red X shows the point where VSImax was reached. This number gives you an indication of the scalability of the environment (higher is better).

VSIbase

VSIbase is the best performance of the system during a test (the lowest response times). This number is used to determine what the performance threshold will be. VSIbase gives an indication of the base performance of the environment (lower is better).

VSImax v4 average

VSImax v4 average is calculated on the amount of active users that are logged on the system but removes the two highest and two lowest samples to provide a more accurate measurement.

VSImax v4 threshold

VSImax v4 threshold indicates at which point the environments saturation point is reached (based on VSIbase).

Stuck sessions

How many sessions got stuck during the test. This number should be zero. Stuck sessions indicate a problem during the test. As stuck sessions do not generate load, the VSImax score will be reduced by the amount of stuck sessions.

VSImax metrics.png

Average Response

Indicates the pure average response time for all the measurements taken taken when the indicated number of sessions on the X-axis were active.

Maximum Response

Indicates the maximum response time for all the measurements taken when the indicated number of sessions on the X-axis were active.

Minimum Response

Indicates the minimum response time for all the measurements taken when the indicated number of sessions on the X-axis were active.

VSI Index Average

Indicates the average value as calculated by Login VSI. The VSI Index Average differs from Average Response due to the fact that the Average Response is the pure average. The VSI Index Average applies certain statistical rules to the average to avoid any spikes from influencing the calculation.

VSImax legend.png

VSImax v4 Detailed

Description

Screenshot

This graph shows the individual measurements taken during a test in a combined graph. It shows the minimum, maximum and average response times for each individual measurement. There is also a Total measurement that combines all the metrics. The minimum, maximum and average for this combined value is shown as well.

Note: This line graph can also be displayed in scatter format.

VSImax v4 detailed.png

The metrics are as follows.

CPU

Calculates a large array of random data.

FCTL

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS

File Copy Text Share. Copy a txt (plain text) file to server.

IO

Writes the random CPU data array to disk.

NFO

Notepad File Open. Open the file with the open dialog in Notepad.

NFP

Notepad File Print. Open the print dialog in Notepad.

NSLD

Notepad Start/Load file. Start Notepad by file type association, loading a text file.

Total

The sum of all the metrics combined.

UMEM

Percentage of memory used by the sessions.

ZHC

Zip High Compression. Zip a pst (Outlook Personal Folder) file, approximately five megabytes in size, using high compression.

ZLC

Zip Low Compression. Zip a pst (Outlook Personal Folder) file, approximately five megabytes in size, using no compression.

VSI Analyzer 4.1.x VSImaxv4 Detailed Metrics.JPG

The second graph compliments the detailed version by showing the individual measurements correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Note: This line graph can also be displayed in scatter format.

Detailed time.png

VSImax v4 Detailed Weighted

Description

Screenshot

The graph shows the individual measurements taken during a test in a combined graph where a multiplier is applied. It shows the minimum, maximum and average response times for each individual measurement. In this graph you can clearly see which measurement has an impact on your environment.

Note: This line graph can also be displayed in scatter format.

VSImax v4 weighted.png

The metrics are as follows.

CPU

Calculates a large array of random data.

FCTL

File Copy Text Local. Copy a txt (plain text) file locally.

FCTS

File Copy Text Share. Copy a txt (plain text) file to server.

IO

Writes the random CPU data array to disk.

NFO

Notepad File Open. Open the file with the open dialog in Notepad.

NFP

Notepad File Print. Open the print dialog in Notepad.

NSLD

Notepad Start/Load file. Start Notepad by file type association, loading a text file.

UMEM

Percentage of memory used by the sessions.

ZHC

Zip High Compression. Zip a pst (Outlook Personal Folder) file, approximately five megabytes in size, using high compression.

ZLC

Zip Low Compression. Zip a pst (Outlook Personal Folder) file, approximately five megabytes in size, using no compression.

VSI Analyzer 4.1.x VSImaxv4 Detailed Weighted Metrics.jpg

The second graph compliments the detailed version by showing the individual measurements correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Note: This line graph can also be displayed in scatter format.

Weighted time.png

VSImax v4 Scatter

Description

Screenshot

The scatter chart is very important. If you set it to Scatter by time, you can see here where within a test a problem arises. And since it is also available by time, you can scan easily through the results. When there are gaps within this graph, this means that at that point in time there were NO measurements, which most likely points to an over-saturated environment.

The second graph compliments the by-time version by showing the individual (Min/Max/Ave) measurement response times correlated with the amount of active sessions. To switch between each version, use the dropdown box underneath the X-axis.

Note: This scatter graph can also be displayed in line format.

VSImax v4 scatter.png
Scatter time.png

UMEM, IO, CPU, ZLC, ZHC, NFP, NFO, NSLD

Description

Screenshot

These tabs show information for the specific individual measurements taken during the test. They also display the baseline value for each individual measurement, except the UMEM graph, which is the time it takes to complete this measurement when measuring the baseline. These measurements are taken while the system is under no or very little load. It is used to see how the measurement trends against a system that isn’t under load.


All the individual measurement graphs are available in two versions:


  • Detailed: Shows min/max/ave response times correlated with active sessions (plus VSImax and Baseline average)
  • By time: Shows response times correlated with the amount of time (plus VSImax)


To switch between each version, simply open the dropdown box underneath the X-axis.


Note: These line graphs can also be displayed in scatter format.

UMEM.png
UMEM by time.png
IO detailed.png
IO detailed by time.png
CPU detailed.png
CPU by time.png
ZLC detailed.png
ZHC detailed.png
NFP detailed.png
NFO detailed.png
NSLD detailed.png

AppStart

Description

Screenshot

The AppStart (application start times) graph gives you an (indication) overview of all the application start times taken during the test.

The second version compliments the first graph by showing the application start times correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Note: This line graph can also be displayed in scatter format.

AppStart.png
AppStart time.png

LogonTimer

Description

Screenshot

The LogonTimer graph gives you an indication of the time it takes for a session to log on (in milliseconds). It shows the trend of logon times during the test.

Please note that this is an indication of the logon time. Login VSI measures the time from the logon scripts running, shortly after group policy has been processed, but before the Shell has loaded (Windows Explorer) and the Windows Shell being loaded.

The second version compliments the first graph by showing the logon times correlated with the amount of time in seconds (instead of active sessions).

To switch between each version, use the dropdown box underneath the X-axis.

Note: This line graph can also be displayed in scatter format.

LT.png
LogonTimer.png

VSImax per Computer (ver4.1.25+)

Description

Screenshot

This graph shows the amount of active sessions per computer correlated with response times in milliseconds. Each computer is juxtaposed over one another. To see individual measurements, unmark the checkboxes underneath the graph as required. If there are any horizontal lines displayed, then this computer is not working properly and requires closer inspection.


VSImax per comp.png

Stuck Sessions Over Time (ver4.1.25+)

Description

Screenshot

This graph shows the amount of stuck sessions correlated with the current number of active sessions. If you see a stuck session indicator, please take further action as this session is not functioning properly.


Stuck sessions over time graph.png

Active Sessions Over Time (ver4.1.25+)

Description

Screenshot

This graph shows the amount of active sessions correlated with response times in milliseconds. Each blue dot represents an active session. If there are any large gaps in-between the blue dots, then some of these sessions did not activate properly.


Active session OT.png

VSImax v4 Data & Raw Data

Description

Screenshot

These tabs contain the raw and processed data used to create the graphs in the Analyzer. You can use this data to run your own analysis, e.g. to see if there were any issues with individual VMs in relation to a specific host.

Raw Data columns explained:

  • Session_count = ActiveSessions: The number of sessions that were active when the the VSI timer started (the analyzer uses this to merge data)
  • Log_time: The moment that the data is written to the CSV file
  • Total_Response: The total sum of the metrics from FCTS to UMEM
  • LoopNumber: Counts from segment 1 to 4, which is one loop (the user starting in segment 3 which goes to 4, then 1 and 2 is counted as two loops)
  • CurrentSegment: The segment that the user is in at the time the session starts
  • SegmentsRun: The total number of segments the user has gone through
VSImax v4 data.png
Raw data.png

Stuck Sessions / Failed to Become Active

Description

Screenshot

The Stuck sessions tab informs the user about any sessions that got stuck during the test.

The table shows the following:

  • Username
  • Computer
  • Date/time of last report


The Failed to become active tab informs the user about any sessions that did not activate during the test.

The table shows the following:

  • Username
  • Launcher
  • Date/time of last report


Note: Both of these tabs only appear in the Analyzer when the test contains stuck/inactive sessions.

VSI Analyzer 41 StuckSessions.JPG
Failed to become active.png

Importing External Data into the Analyzer

You can also import external performance data from the host level, such as ESXTop, PerfMon or XenTop, into the Login VSI Analyzer.

Important: Login VSI now provides the option to collect and generate VMware ESXTop performance logs, instead of having to create and import a CSV file manually. For more information, see ESXTop integration and configuration (only available with version 4.1.25+).

Description

Screenshot

Open the Analyzer via the Management Console, the desktop shortcut or from the VSIshare itself.

Analyzer1.png

Select a test result to open.

Analyzer2.png

Click File, then hover over Import and select External data. Alternatively, you can use the shortcut key Ctrl+X.

Import external data.png

Open the (external) CSV file, in this example we use an ESXTop CSV file.

Ex CSV.png

Select the metrics you want to import.

Select metrics.png

A new tab titled External Data 1 will appear.

All metrics are down/up scaled to be in the range of 0-100, by finding the max value and multiplying accordingly. This feature was expanded to hold both multiple sources and multiple imports of metrics with the ability to create new tabs or merge to the existing external data tabs. This also adds the remaining external data series to moving box, and adds the source column in the data grid view.

The blue step-line in the picture represents the active sessions. Everything is recalculated to time in seconds, so the X-axis is the amount of time in seconds.

External data.png

Additional Resources

Click on the links below to read related resources: