login vsi company logo login vsi company logo 250x40
header-01.jpg

Forum Announcement

The Login VSI forum has been disabled and now only serves as an archive. If you would like to submit a new support ticket, please send an email to support@loginvsi.com.

Unreliable VSIMax value

  • vdobhal
  • vdobhal's Avatar Topic Author
24 Apr 2014 13:38 #1740

Hi,
I am trying to find VSI max for my application. Following is what I am doing :
1. In the test workload, I am using Sikuli to do some operations on the application. Script runs for 30 mins (approx.) and gives a VSI max of 10-11 (even on repeated runs).
2. To check the correctness of the reading, I am just launching and closing the application in a second script workload. This script launches the application, waits for approximately 25 mintues (run time of sikuli script) and then closes the application.
3. There is no basephase in either of the tests.

Problem is in second workload (launching-closing application), VSI max is changing with each run. Sometimes, it gives a VSI max of 18, other times it gives a VSI max of 24 and so on.

Can someone address following queries:
1. Is VSI max reliable for workloads that use Sikuli scripts to execute operations.
2. Why is there so much fluctuation in the VSI max reported in second workload?

Thanks!

Please Log in to join the conversation.

More
24 Apr 2014 14:33 #1741

Hello,

Can you please send us your custom workload, we cant do an accurate assessment of your situation without the knowledge of your custom workload.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
25 Apr 2014 05:08 #1745

Pls. find attached the custom workloads and the sikuli script.

Thanks,
Vinod

Attachment not found



Attachment not found



Attachment not found

Please Log in to join the conversation.

More
25 Apr 2014 07:01 #1746

Hello Vinod,

Can you also send us your logfiles of each different workload? Please send this to This email address is being protected from spambots. You need JavaScript enabled to view it.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

More
25 Apr 2014 11:58 #1747

Hello Vinod,

After analyzing your data it looks like the problem is that you have too few VSI_Timer() in your workload. Our own workloads run a timer roughly every 4 minutes. With your "medcustomacrobat_774" workload you only run one timer, at the start of the workload and then never return to it. I really suggest placing more timers so the graph is allot smoother.

The other workload shows very high fluctuations in baseline. Im wondering what type of environment you are testing on. It looks like the environment is already under load. Or maybe a virusscanner is acting up.

Can you also explain to me what you are trying to achieve? Maybe we can help you get there.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
25 Apr 2014 12:55 #1748

Hi Tom,

Thanks for all the help.

This is what we are trying to achieve. We are trying to calculate VSI max for a real-world usage of our application, Acrobat. The application has features which are not handled by any of the Login VSI’s available methods. We use Login VSI to launch the application and then call our Sikuli script (workload "medcustomacrobat_774"). The Sikuli script simulates a real-world usage for the application. We were satisfied with the result given for this workload but wanted to confirm if the values are actually correct.

To check the correctness, we hypothesized that a simple workload that opens and closes the application should give a higher VSI max value. “AcrobatMinimal” is the workload that implements this hypothesis. However, as you might have noticed, this workload gave erratic results and we couldn’t infer the meaning of different VSI max values that it gave. This made us question the integrity of original readings (ones of "medcustomacrobat_774") as well.

I understand your point on adding more timer actions to the "medcustomacrobat_774". To do that, we will create smaller sikuli scripts (each of approx. 4-5 minutes) and then have timer in between those scripts. Will that help in getting a reliable value?

For the other workload, we don’t have any virus application or any other kind of load on the system. Login VSI users are the only ones accessing the system under test. Is it possible that fluctuations might be because there is no baseline phase?

If you have any better ideas to measure the VSI max for our application (with different kinds of actions/operations), we will appreciate it. Maybe from your knowledge of other scenarios which had similar use-case.

Thanks,

Please Log in to join the conversation.

More
25 Apr 2014 13:55 #1749

Hello!

I understand what you are trying to achieve. You can also use AutoIT for scripts like you are doing, we use the AutoIT language for our workloads as well.

But yes please create more smaller scripts and more VSI_timers. Just look at our default scripts, medium, heavy etc. and use this as example. And create more segments so its longer and more controlled instead of just loop after loop.

The same goes for the short workload, just add more app opens, app closes and segments so you stretch the test a bit. This should give more stable VSImax results.

If you need any other help, please contact us via this forum or at This email address is being protected from spambots. You need JavaScript enabled to view it.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

  • vdobhal
  • vdobhal's Avatar Topic Author
01 May 2014 08:14 #1755

Hi Tom,

We modified our workloads to include VSI_Timer() every few minutes and have smaller sikuli scripts for different operations. This solved the problem of fluctuating VSI Max for each test run. However, we saw a new problem arising.

When we tested “all operation” workload with 10 users, we got a VSI Max of 8. For 15 users, value reached to 10. For 20 users, value is 14.

For “minimal operation” workload, we got VSI max of 14 for 20 users. This confused us as we assumed that if we execute more operations which are memory consuming, VSI max should be low. Thus, we assumed “all operation” workload should have lower VSI max than “minimal operation” workload.

Can you give us insights on following aspects:
1. Does VSI Max depend on number of user sessions in a test?
2. How can we quote VSI Max as part of our marketing strategy? We were working a statement like “Our application has a VSI Max of X on a particular configuration” attributing it with our setup configuration. Since we are getting different VSI Max value, we are not sure if this statement will make sense.

Thanks!

Please Log in to join the conversation.

More
01 May 2014 08:45 #1756

Hello,

Good to hear that the results are now more stable. For the problems you are having i would like to schedule a call with you to discuss this. Please email us at This email address is being protected from spambots. You need JavaScript enabled to view it. if you are willing and if so, what time / day you are available. Please also note your timezone.

Regards

Tom Willemsen - Support VSI

Please Log in to join the conversation.

Start Delivering the Best End User Experience Today

Request a Demo

Login VSI, Inc.

3945 Freedom Circle
Suite 670
Santa Clara, CA 95054

Phone: +1 408 899 7418

Login VSI B.V.

De Entree 85
1101 BH Amsterdam
The Netherlands

Phone: +31 20 705 1200