VSImax

From Login VSI Documentation
Jump to: navigation, search

The logic behind Login VSI is different to conventional benchmarks. In general, most system benchmarks are steady state benchmarks. These benchmarks execute one or multiple processes, and the measured execution time is the outcome of the test. Simply put: the faster the execution time or the bigger the throughput, the faster the system is according to the benchmark.

Login VSI is different in approach. Login VSI is not primarily designed to be a steady state benchmark (however, if needed, Login VSI can act like one). Login VSI was designed to perform benchmarks for SBC or VDI workloads through system saturation. Login VSI loads the system with simulated user workloads using well known desktop applications like Microsoft Office, Internet Explorer and Adobe PDF reader. By gradually increasing the amount of simulated users, the system will eventually be saturated. Once the system is saturated, the response time of the applications will increase significantly. This latency in application response times a clear indication whether the system is (close to being) overloaded. As a result, by nearly overloading a system it is possible to find out what its true maximum user capacity is.

After a test is performed, the response times can be analyzed to calculate the maximum active session/desktop capacity. Within Login VSI this is calculated as VSImax. When the system is coming closer to its saturation point, response times will rise. When reviewing the average response time it will be clear the response times escalate at saturation point.

This VSImax is the “Virtual Session Index (VSI)”. With Virtual Desktop Infrastructure (VDI) and Terminal Services (RDS) workloads this is valid and useful information. This index simplifies comparisons and makes it possible to understand the true impact of configuration changes on hypervisor host or guest level.


Recommended read: Calculating Maximum Virtual Desktop Capacity – VSImax Explained


Server side response time measurements

It is important to understand why specific Login VSI design choices have been made. An important design choice is to execute the workload directly on the target system within the session instead of using remote sessions. The scripts simulating the workloads are performed by an engine that executes workload scripts on every target system, and are initiated at logon within the simulated user’s desktop session context.

An alternative to the Login VSI method would be to generate user actions client side through the remoting protocol. These methods are always specific to a product and vendor dependent. More importantly, some protocols simply do not have a method to script user actions client side.

For Login VSI the choice has been made to execute the scripts completely server side. This is the only practical and platform independent solution, for a benchmark like Login VSI. The relative overhead and footprint of a benchmark engine scripted in AutoIT is small enough (1-5% range) for Login VSI’s purposes.

Calculating VSImax v4.1

The simulated desktop workload is scripted in a 48-minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of ten specific operations are measured in a regular interval, five of which are the most important. These measurements happen sixteen times within each loop. The response times of these five important operations are used to determine VSImax.

The five operations from which the response times are measured are:

Notepad File Open (NFO)

Loading and initiating VSINotepad.exe and opening the openfile dialog. This operation is handled by the OS and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user’s point of view.

Notepad Start Load (NSLD)

Loading and initiating VSINotepad.exe and opening a file. This operation is also handled by the OS and by the VSINotepad.exe itself through execution.This operation seems almost instant from an end-user’s point of view.

Zip High Compression (ZHC)

This action copy's a random file and compresses it (with 7zip) with high compression enabled. The compression will very briefly spike CPU and disk IO.

Zip Low Compression (ZLC)

This action copy's a random file and compresses it (with 7zip) with low compression enabled. The compression will very briefly disk IO and creates some load on the CPU aswell.

CPU

Calculates a large array of random data and spikes the CPU for a short period of time.

VSImax Metrics

Once the test is finished, VSImax v4.1 can be calculated. Previous VSImax models (Classic and Dynamic) needed Microsoft Office Word to function. With the new 4.1 timers this is nolonger needed, we are therefore more flexible and applicable to a larger scale of scenario's.

The following actions are part of the VSImax v4.1 calculation and are weighted as follows (US notation):

  • Notepad File Open (NFO): 0.75
  • Notepad Start Load (NSLD): 0.2
  • Zip High Compression (ZHC): 0.125
  • Zip Low Compression (ZLC): 0.2
  • CPU: 0.75

This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

With the introduction of Login VSI 4.1 we also created a new method to calculate the baseline of an environment. With the new workloads (Taskworker, Powerworker, etc.) enabling 'basephase' for a more reliable baseline has become obsolete. The calculation is explained below.

The 13 lowest VSI Index Calculation response time samples are taken from the entire test and are averaged. The result is the Baseline. In short:

  • Sort the VSI Index Calculation values from lowest to highest
  • Take the lowest 13 samples of the VSI Index Calculation
  • Average the 13 results and the result is the baseline

Calculating VSImax v4

The simulated desktop workload is scripted in a 48 minute loop when a simulated Login VSI user is logged on, performing generic Office worker activities. After the loop is finished it will restart automatically. Within each loop the response times of six specific operations are measured in a regular interval: twelve times in within each loop. The response times of these six operations are used to determine VSImax.

The six operations from which the response times are measured are:

Starting “VSI Notepad”

This operation is handled by the OS (loading and initiating VSINotepad.exe) and by the VSINotepad.exe itself through execution. This operation seems almost instant from an end-user’s point of view.

Starting the “File Open” dialogue

This operation is handled for a small part by Notepad and a large part by the operating system. The file open dialogue uses generic subsystems and interface components of the OS. The OS provides the contents of this dialogue.

Starting the “Print” dialogue

This operation is handled for a large part by the OS subsystems, as the print dialogue is provided by the OS. This dialogue loads the print-subsystem and the drivers of the selected printer. As a result, this dialogue is also dependent on disk performance.

Compress the document into a zip file with 7-zip command line (2x)

This operation is handled by the command line version of 7-zip. The compression will very briefly spike CPU and disk IO. This action is performed twice: once with no compression (IO intensive) and with high compression (CPU intensive)

Starting Microsoft Word with a document

This operation will measure the responsiveness of the Operating System and the file system. Microsoft Word is started and loaded into memory, also the new document is automatically loaded into Microsoft Word. When the disk IO is extensive or even saturated, this will impact the file open dialogue considerably.

These measured operations within Login VSI do hit considerably different subsystems such as CPU (user and kernel), Memory, Disk, the OS in general, the application itself, print, GDI, etc. These operations are specifically short by nature. When such operations become consistently long: the system is saturated because of excessive queuing on any kind of resource. As a result, the average response times will then escalate. This effect is clearly visible to end-users. If such operations consistently consume multiple seconds the user will regard the system as slow and unresponsive.


Good-chart.png
Figure 1 Sample of a VSI max responsetime graph, representing a normal test



Bad-chart.png
Figure 2 Sample of a VSI test responsetime graph where there was a clear performance issue


Once the test is finished, VSImax v4 can be calculated. Previous VSImax models (Classic and Dynamic) could not be calculated when the system was not saturated. In VSImax v4 this is not a requirement anymore. When the system is not saturated, and it could complete the full test without exceeding the average response time latency threshold, VSImax is now considered the maximum of active sessions that were launched.

The response times are very different per measurement type, for instance Zip with compression can be around 2800 ms, while the Zip action without compression can only take 75ms. These response time of these actions are weighted before they are added to the total. This ensures that each activity has an equal impact on the total response time.

In comparison to previous VSImax models, this weighting much better represent system performance. All action have very similar weight in the VSImax total, both in VSImax classic and dynamic the opening of word had far greater impact on the total than other activities. The following weighting of the response times are applied:


The following actions are part of the VSImax v4 calculation and are weighted as follows (US notation):

• Start VSINotepad with large text file: 0.5

• Start File Open Dialogue: 1.25

• Start Print dialogue: 4

• Zip PST file without compression: 6

• Zip PST file with high compression: 0.175

• Start Word with new document from document pool: 0.15


This weighting is applied on the baseline and normal Login VSI response times.

VSImax Baseline

The average VSImax baseline response time (also called VSIbase) in Login VSI 4.0 is calculated from the results recorded in the baseline phase. In total 30 VSI response time samples are takes by 5 baseline sessions. To ensure the VSIbase represents the optimal performance of the system, the highest 15 results are removed from the average calculation. To ensure no fluke low measurements are affecting the results unrealistically, the bottom 2 results are removed from the average VSIbase calculation. Over the remaining 13 VSI response time measurements the average VSImax baseline response time VSIbase is calculated. So in short:

  • Take 30 results collected during the Baseline phase
  • Remove the top 15 results
  • Remove the lowest 2 results
  • Average the remaining results

If the baseline fase is not enabled the calculation is as follows:

  • Sort the VSImax v4 data (found in analyzer) on Session Count
  • Select the first 200 results of the VSIindex_Calculation
  • Sort the results lowest to highest
  • Select the lowest 30 Results
  • Average the remaining results

VSImax average

The VSImax average response time in Login VSI 4.0 is calculated on the amount of active users that are logged on the system.

There are always 5 Login VSI response time samples averaged and 40% of the amount of “active” sessions is added. For example, if the active sessions is 60, then latest 5 + 24 (=40% of 60) = 29 response time measurement are used for the average calculation.

To remove noise (accidental spikes) from the calculation, the top 5% and bottom 5% of the VSI response time samples are removed from the average calculation, with a minimum of 1 top and 1 bottom sample. As a result, with 60 active users, the last 29 VSI response time sample are taken. From those 29 samples the top 2 samples are removed and lowest 2 results are removed (5% of 29 = 1.45, rounded to 1). At 60 users the average is then calculated over the 28 remaining results.

Reaching VSImax

VSImax v4 is reached when the VSIbase + a 2600 ms latency threshold is reached by the average VSI response time result. Depending on the tested system, VSImax response time can grow 2 - 3x the baseline average. In end-user computing, a 3x increase in response time in comparison to the baseline is typically regarded as the maximum performance degradation to be considered acceptable.

Note: In VSImax Dynamic the latency threshold was dependent on the baseline measurement at the beginning of the test. 25% of the baseline measurement was added to a latency threshold. While this was not a problem with most tests, in some cases, especially when the systems performance was very close between two different configurations, the slower system might get a higher VSImax score simply because the higher baseline results gave the slower system a higher latency threshold.

In VSImax v4 this latency threshold is fixed to 2600ms, this allows better and fairer comparisons between two different systems, especially when they have different baseline results. Ultimately, in VSImax v4, the performance of the system is not decided by the total average response time, but by the latency is has under load. For all systems, this is now 2600ms (weighted).

The *threshold* for the total response time is: average weighted baseline phase response time + 2600ms.

When the system has a weighted baseline response time average of 1500ms, the maximum average response time may not be greater than 4100ms (1500+2600). If the average baseline is 3000 the maximum average response time may not be greater than 5600ms (3000+2600).

VSImax v4 is determined before the system has exceeded it threshold. For example, when the VSI max average on system has exceeded the VSI threshold at 80 users, the VSImax will be 79.

When the threshold is not exceeded by the average VSI response time during the test, VSImax is now considered the maximum amount of users that was launched. This approach is fundamentally different in comparison to previous VSImax methods, as is it was always required to saturate the system beyond VSImax threshold.

Lastly, VSImax v4 is now always reported with the average baseline VSI response time result. For example: “The VSImax v4 was 125 with a baseline of 1526ms”. This helps considerably in the comparison of systems and gives a more complete understanding of the system. The baseline performance helps to understand the best performance the system can give to an individual user. VSImax indicates what the total user capacity is for the system. These two are not automatically connected and related:

When a server with a very fast dual core CPU, running at 3.6 GHZ, is compared to a 10 core CPU, running at 2,26 GHZ, the dual core machine will give and individual user better performance than the 10 core machine. This is indicated by the baseline VSI response time. The lower this score is, the better performance an individual user can expect.

However, the server with the slower 10 core CPU will easily have a larger capacity than the faster dual core system. This is indicated by VSImax v4, and the higher VSImax is, the larger overall user capacity can be expected.

With Login VSI 3.6 it is was possible to choose between ‘VSImax Classic’ and 'VSImax Dynamic’. With Login VSI 4.0 a new VSImax method is introduced: VSImax v4. This methodology gives much better insight in system performance and scales to extremely large systems. ‘VSImax Classic’ and 'VSImax Dynamic’ are not suitable for large systems.

VSImax Classic

VSImax Classic is based on the previous versions of VSI, and is achieved when the average VSI response time is higher than a fixed threshold of 4000ms. This method proves to be reliable when no anti-virus or application virtualization is used.

In practice however, tests have shown a substantial increase of application response time when antivirus and/or application virtualization is used. The baseline response time is typically around 1400 - 1800 ms without application virtualization or antivirus. However, when anti-virus or application virtualization is used, the baseline response time varies between 2500 – 3500 ms.

When the baseline response time is already so high the VSImax Classic threshold of 4000ms is too easily reached. ‘VSImax Classic’ will report a maximum long before system resources like CPU, RAM or disk are actually saturated.

It was therefore decided to further optimize VSImax calculation.

VSImax Dynamic

Similar to ‘VSImax Classic’, VSImax Dynamic is calculated when the response times are consistently above a certain threshold. However, this threshold is now dynamically calculated on the baseline response time of the test. VSImax Dynamic was introduced in VSI 3.0.

For this the average VSImax response times need to consistently higher than a dynamically calculated threshold. To determine this dynamic threshold, first the average baseline response time is calculated. This is done by averaging the baseline response time of the first 15 VSI users on the system.

The formula for the dynamic threshold is: Avg. Baseline Response Time x 125% + 3000. As a result, when the baseline response time is 1800, the VSImax threshold will now be 1800 x 125% + 3000 = 5250ms.

Especially when application virtualization is used, the baseline response time can wildly vary per vendor and streaming strategy.

Baseline update patch 4.0.7

The Login VSI analyzer used to plot an imaginary line at the start of a test which interferes with the baseline when the Basephase is not used. We have noticed the line can be confusing when analyzing the results and to eliminate any confusing we have decided to remove the incorrect calculation. When the Basephase is not used there will be 6% increase in millisecond on the Baseline. Because the baseline will increase this will have an effect on the VSImax. To be precise the VSImax will increase with 1%. The following screenshot shows the imaginary line from the previous analyzer.

Imaginary line.png



VSImax 4.x Metrics

VSImax v4 is determined by the measurements in the below table.

Measurement ID

Measurement Action

Measurement Action Detailed

Measures Related Resource

WSLD

Start Microsoft Word and Load a random document

Word Start/Load a local random document file from content pool

CPU, RAM and IO

NSLD

Start VSI-Notepad and Load a document

VSI-Notepad Start/Load a local random text file from content pool

CPU and IO

WFO

Press file open in VSI-Notepad

VSI-Notepad file open [ctrl+o]

CPU, RAM and IO

NFP

Press print open in VSI-Notepad

VSI-Notepad print open [ctrl+p]

CPU

ZHC*

Compress files with high compression

Compress a local random .pst file from content pool (5mb)

CPU

ZNC*

Compress files with no compression

Compress a local random .pst file from content pool (5mb)

IO

'*'compression is measured with 7zip

Other resources

Blog - Calculating Maximum Virtual Desktop Capacity – VSImax Explained