login vsi company logo login vsi company logo 250x40
header-03.jpg

normal Unreliable VSIMax value

  • vdobhal
  • vdobhal's Avatar Topic Author
24 Apr 2014 13:38 #1740 by vdobhal
vdobhal created the topic: Unreliable VSIMax value
Hi,
I am trying to find VSI max for my application. Following is what I am doing :
1. In the test workload, I am using Sikuli to do some operations on the application. Script runs for 30 mins (approx.) and gives a VSI max of 10-11 (even on repeated runs).
2. To check the correctness of the reading, I am just launching and closing the application in a second script workload. This script launches the application, waits for approximately 25 mintues (run time of sikuli script) and then closes the application.
3. There is no basephase in either of the tests.

Problem is in second workload (launching-closing application), VSI max is changing with each run. Sometimes, it gives a VSI max of 18, other times it gives a VSI max of 24 and so on.

Can someone address following queries:
1. Is VSI max reliable for workloads that use Sikuli scripts to execute operations.
2. Why is there so much fluctuation in the VSI max reported in second workload?

Thanks!

Please Log in to join the conversation.

More
24 Apr 2014 14:33 #1741 by t.willemsen
t.willemsen replied the topic: Unreliable VSIMax value
Hello,

Can you please send us your custom workload, we cant do an accurate assessment of your situation without the knowledge of your custom workload.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
25 Apr 2014 05:08 #1745 by vdobhal
vdobhal replied the topic: Unreliable VSIMax value
Pls. find attached the custom workloads and the sikuli script.

Thanks,
Vinod

Attachment not found



Attachment not found



Attachment not found

Please Log in to join the conversation.

More
25 Apr 2014 07:01 #1746 by t.willemsen
t.willemsen replied the topic: Unreliable VSIMax value
Hello Vinod,

Can you also send us your logfiles of each different workload? Please send this to Support@loginvsi.com

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

More
25 Apr 2014 11:58 #1747 by t.willemsen
t.willemsen replied the topic: Unreliable VSIMax value
Hello Vinod,

After analyzing your data it looks like the problem is that you have too few VSI_Timer() in your workload. Our own workloads run a timer roughly every 4 minutes. With your "medcustomacrobat_774" workload you only run one timer, at the start of the workload and then never return to it. I really suggest placing more timers so the graph is allot smoother.

The other workload shows very high fluctuations in baseline. Im wondering what type of environment you are testing on. It looks like the environment is already under load. Or maybe a virusscanner is acting up.

Can you also explain to me what you are trying to achieve? Maybe we can help you get there.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
25 Apr 2014 12:55 #1748 by vdobhal
vdobhal replied the topic: Unreliable VSIMax value
Hi Tom,

Thanks for all the help.

This is what we are trying to achieve. We are trying to calculate VSI max for a real-world usage of our application, Acrobat. The application has features which are not handled by any of the Login VSI’s available methods. We use Login VSI to launch the application and then call our Sikuli script (workload "medcustomacrobat_774"). The Sikuli script simulates a real-world usage for the application. We were satisfied with the result given for this workload but wanted to confirm if the values are actually correct.

To check the correctness, we hypothesized that a simple workload that opens and closes the application should give a higher VSI max value. “AcrobatMinimal” is the workload that implements this hypothesis. However, as you might have noticed, this workload gave erratic results and we couldn’t infer the meaning of different VSI max values that it gave. This made us question the integrity of original readings (ones of "medcustomacrobat_774") as well.

I understand your point on adding more timer actions to the "medcustomacrobat_774". To do that, we will create smaller sikuli scripts (each of approx. 4-5 minutes) and then have timer in between those scripts. Will that help in getting a reliable value?

For the other workload, we don’t have any virus application or any other kind of load on the system. Login VSI users are the only ones accessing the system under test. Is it possible that fluctuations might be because there is no baseline phase?

If you have any better ideas to measure the VSI max for our application (with different kinds of actions/operations), we will appreciate it. Maybe from your knowledge of other scenarios which had similar use-case.

Thanks,

Please Log in to join the conversation.

More
25 Apr 2014 13:55 #1749 by t.willemsen
t.willemsen replied the topic: Unreliable VSIMax value
Hello!

I understand what you are trying to achieve. You can also use AutoIT for scripts like you are doing, we use the AutoIT language for our workloads as well.

But yes please create more smaller scripts and more VSI_timers. Just look at our default scripts, medium, heavy etc. and use this as example. And create more segments so its longer and more controlled instead of just loop after loop.

The same goes for the short workload, just add more app opens, app closes and segments so you stretch the test a bit. This should give more stable VSImax results.

If you need any other help, please contact us via this forum or at support@loginvsi.com

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
01 May 2014 08:14 #1755 by vdobhal
vdobhal replied the topic: Unreliable VSIMax value
Hi Tom,

We modified our workloads to include VSI_Timer() every few minutes and have smaller sikuli scripts for different operations. This solved the problem of fluctuating VSI Max for each test run. However, we saw a new problem arising.

When we tested “all operation” workload with 10 users, we got a VSI Max of 8. For 15 users, value reached to 10. For 20 users, value is 14.

For “minimal operation” workload, we got VSI max of 14 for 20 users. This confused us as we assumed that if we execute more operations which are memory consuming, VSI max should be low. Thus, we assumed “all operation” workload should have lower VSI max than “minimal operation” workload.

Can you give us insights on following aspects:
1. Does VSI Max depend on number of user sessions in a test?
2. How can we quote VSI Max as part of our marketing strategy? We were working a statement like “Our application has a VSI Max of X on a particular configuration” attributing it with our setup configuration. Since we are getting different VSI Max value, we are not sure if this statement will make sense.

Thanks!

Please Log in to join the conversation.

More
01 May 2014 08:45 #1756 by t.willemsen
t.willemsen replied the topic: Unreliable VSIMax value
Hello,

Good to hear that the results are now more stable. For the problems you are having i would like to schedule a call with you to discuss this. Please email us at support@loginvsi.com if you are willing and if so, what time / day you are available. Please also note your timezone.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

What our customers are saying

Tristan Todd - VMware

“Login VSI is my favorite test orchestration and simulation tool for large-scale reference architecture testing. Not only do we use it in our testing, but we work with partners who use Login VSI exclusively, with the idea that you create the most realistic simulated workload at scale, to really test infrastructure and software capabilities. What’s more, getting the results and be able to measure yourself across platforms and across vendor solutions is just fantastic and is unequalled in the industry.”

Tristan Todd, EUC Architect at VMware


Keith Hageman - Micron

"When I joined Micron in June 2014, the first thing I did is find out if we have a copy of Login VSI and if we could run a 1000 user test case instance (which we could). We use Login VSI to run tests for customers and for case studies and I really have loved Login VSI for the last 3 or 4 years.”

Keith Hageman, Senior Solutions Architect at Micron Technology


Todd Mace - PernixData

"Login VSI has been a great partner for us. Customers use PernixData with Login VSI to help validate their user experience. Login VSI helps to see what their performance bottlenecks are and our software helps to mediate some of that. Login VSI has been a kind of defacto standard to be able to help that validation and help the customer to be able to come up with a solution to fix it."

Todd Mace, Product Manager at PernixData

Login VSI, Inc.

300 Tradecenter

Suite 3460
Woburn, MA 01801

Phone: +1 844 828 3693

Login VSI B.V.

De Entree 85
1101 BH Amsterdam
The Netherlands

Phone: +31 20 705 1200