Meltdown and Spectre: Quick start guide for testing
So you know the mitigations for Meltdown and Spectre vulnerabilities are going to have a measurable impact on the performance and efficiency of your system, not to mention user experience. The only way to get ahead of the problem is to start addressing it now. Some of the numbers we are hearing from our customers and partners in the industry are quite alarming, but your action should come from the numbers that you get through your own testing, as it will be different for just about every solution out there. You can find some information on our early Windows 7 and Windows 10 testing in our blogs.
Finding the time and resources to do this testing can be difficult, so I am hopeful this information will help you. If you get stuck at any point our support team, who has a 99% satisfaction rating and is composed of some of the most knowledgeable EUC folks in the industry, can help
Getting started is easy
I know it may seem like setting up a scalable test to determine the impact of these mitigations is pretty complex, but part of what makes Login VSI so popular is that we’ve made it super easy for you to do these tests yourself. It boils down to four simple steps… setup the test systems, setup Login VSI, test your test and execute your test. Login VSI helps by automating much of this work.
- We also have a quick setup and install guide for you here:
- Getting started videos here:
- And all the documentation you need here:
Let’s take a high level look at these 4 steps below. Keep in mind that the quick install and setup guide above will have all the details you need to get started. The videos will give you the comfort of seeing someone do it before you try, which is a big confidence booster for me ????.
The test systems
Here are the ingredients for the test:
- AD and DNS: Since you are likely an enterprise shop I’ll assume you have AD/DNS/DHCP already setup, good, the first step is done
- Login VSI Management Server or VM
- Login VSI Launcher Server(s) or VM
- System Under Test (SUT): This is the system you want to test. This could be one of the servers which hosts your desktop pool, or one of your RDS hosts
- Step 1: Get the software and your free trial license here https://www.loginvsi.com/use-cases/spectre-meltdown
- Step 2: On the management server, create the VSIshare and install
- Step 3: Create the target image (for RDS or for Client OS and clone). Just use your standard image and install the Login VSI target preparation software to it.
- Step 4: Create test user accounts in AD (we already made a script for you)
- Step 5: Create launcher VMs (you want about 1 Launcher for every 25 sessions. For example, to test an RDS host for 150 sessions, you would want 6 Launcher VMs)
A quick note on setting things up:
- Typically you want to test at the cluster level, but to understand the impact of the mitigations, let’s just start with one server and keep it simple. Once you have more insight into the impact, you will likely want to rebalance your hosts and then test at the cluster level.
- Most Login VSI customers use the Knowledge Worker workload. It closely resembles much of the workforce today and can use Microsoft Office, Adobe Reader, etc… It is ready to use, out of the box.
Test your test
I know you are excited to do your first test, but we recommend you just try to kick off 1 or 2 sessions to make sure your setup is clean. Typically, it takes a couple of iterations to get everything nice and shiny—ready to test.
Execute the test
OK, hang on tight… you are going to put a load on your server to see how it behaves. This will give you a lot of insight into the next steps you will want to take based on the results and conclusions of your tests.
When setting your tests up we are looking for the best path to consistency. You want to be able to run the test at least 3 times, seeing similar results each time.
Here are some recommendations that will help you to get consistent results:
- It is recommended to reboot your SBC, pool and/or launcher hosts prior to each test to ensure the test is not influenced by previous tests (i.e., don’t let caching skew your results)
- Give your servers 30 minutes to idle after rebooting and before kicking off you next test
- Use the defaults as much as possible. The default test duration is 48 minutes
- In most cases your desktop or application streaming host will bottleneck on a resource and we will arrive at VSImax. This will tell you how many sessions you can host before performance drops off enough to impact the end-user experience
- Keep stuck sessions to less than 5% of VSImax
- Give each of your tests a meaningful name so you can easily find them among all of the other tests
Methodology is important and you should know what you want to test, why you want to test, and what the expected outcome may be. Your goal is to determine the impact of the mitigation patches.
To start, you should know the current performance of your system. Run Login VSI against your system under test to see how it behaves. You should be able to reach a VSImax. Record that number as well as your VSIbase.
- VSImax = The number of users that can work on a system comfortably before performance is impacted (scale). Higher is better.
- VSIbase = The performance of the system when there is little or no stress. If this number increases overall performance decreases. Lower is better.
Performance after mitigation patches
You will likely have a performance impact from each patch. Cumulatively they will add up to a total impact. You can test each patch serially as they are added, or you can install all patches at once to see total impact. Patches will likely be applied to the Server or Client OS (e.g., Windows Server 2012 R2 or Windows 10), the Hypervisor (VMware vSphere or XenServer) and the hardware via a microcode update (e.g., from HP, DELL, Cisco, Nutanix, etc…).
The number of tests may add up quick as you look at the different RDS Host versions, Client OS versions, Intel processor versions (Haswell versus Skylake), etc. Keep track of what you would like to test and start with the most important systems for your business. (see sample matrices below).
Analyzing your results
Login VSI collects all of the test data you need from the end-user perspective. It can also be helpful to collect server metrics. For example, use esxtop and/or Windows Perfmon to capture other relevant performance data about the server and infrastructure. Since Meltdown/Spectre patches largely impact the performance of the CPU, make sure to keep a close eye on this metric. Login VSI will capture the performance data from the end-users’ perspective, so we have you covered there.
For help with this, see the “Analyzing Results” video here https://www.loginvsi.com/documentation/index.php?title=Getting_started
Below you will find some tables that may help to track the results you are getting. These will help you compare the results to determine an actionable plan on how to manage the performance impact of these changes.
There’s hope. You can get some performance back
So maybe the performance impact you are seeing is a bit shocking. If you haven’t optimized your server or desktop images we highly recommend you do this. We have plenty of information here. If you review the test results from our initial Meltdown/Spectre testing in (here) you may notice something. When Mark tested the patches on his Windows 7 desktop image, without using OSOT tuning, he went from a VSImax of 173 to 149 (ouch!). But, when Mark then applied OSOT tuning, he was able to get his server density back to 178 after the patches. That’s better than he started with! Make sense?
Example test matrix and data recording tables