Do you understand your web application performance?

In this article we are going to know about an web application performance ? what does it means by performance. This is very generic ideas , spatially for managers, decision makers or a person who is very new to performance testing

Do you really know what is you application performance? I am trying to break down, you can see each politest for understanding.

Understand your Application User Load model or Business mode : This is very important , before trying to know any of performance activity, you need to know how user interact with your system. What they expect/ what your company target. This will breakdown your performance expectation. This is non technical part. User activity monitoring from browser tools might help to know your user better. Now a days all browsers have build in tools for user experience measurement.

Understand your Application load : Do you know how much you application can take load? Load of what? User load, data load. By mean of load, i referrer to knowing the application's height usages model. To know this , you need simple load testing. You need to measure, what max user supports, max data load support. While testing the load, you need to stop when you are getting error.

Understand your Application stability : Do you know how much you application is stable. What do you mean by stable? It means can you application work properly under any circumstances. From previous load test activity, we will know application average and high load condition. So, how the application is stable under this high/avg condition. To know that you need to perform stress testing for a longer period of time. This period depend on your required environment. It can be 1 hour, 2 hours, 8 hours or even 1 week. The target of this is to know stability of application in given load situation which is normally referred to stress testing.

Understand your Application tolerance: Do you know what will happen if your application gets sudden peak of user or data transaction or low network / IO bandwidth? To know the tolerance of your web application, you need to perform spike test. A spike test refers to , sudden growth of user/ data usages for a small number of time. This will test your system , how they can tolerate sudden peak.

Understand your Application capacity : Understanding maximum capacity and stability is very important. For this you need to perform load testing and stress testing together. Than mean , long term maximum support load and stress scenario together. You will know about  , the bucket(server/IO/NET) capacity. This is very important in cloud environment spatially in scale in or scale out capacity measurement. This is measured by X, means, for example, your application supports 5000 users , but it can be scale up to 2x (2*5000=10000 users).

Understand your Application boundary : This is the boundary checking. What will happen if  application is under extreme load and stress with error/exception condition. This test will define your extreme condition and boundary. Usually this is measured with capacity (X) and expected errors. Mainly this is king of precaution test for expected error behaviors. This is very critical for , financial domain and security tests.

Understand your server throughput : Do you know what is your server (app/data etc) power or capacity? This is measured by throughput. More powered server has better throughput. It is measured by Hit per second. More often it is also expressed in Transaction per second where transaction referrers to business specified transaction. The more throughput you have does not mean the fast websites, but it means server power is more. Because, faster response time has many factors and definitely throughput is one  of them. So, that is why all Load generation tools have this measurement. And, from SLA agreement, this term is clearly specified.

Understand your resource usages : Do you know , how much resources you have to run your web application and how much your application uses? And is it enough? By resource, i want to mean
a. Network bandwidth :
b. Disk IO
c. Memory
d. CPU
Do you monitor in you system server? If not, then how can you be ensure the performance that you need.
To monitor the whole system, you need to know about your resources for each component of your whole environment. Most of OS have their own monitoring system.
You may visit Sysinternals Suite if you are monitoring/debugging windows environment.
Windows also have perfmon builtin which you can monitor the whole system.

Beside that, now a days companies use APM solution which covers all of you monitoring in single application. To use an APM, you need an agent in each server that does the job for you.
[ there are some opensource APM, you need to get & configure for your system]

Understand Application Activity: As you monitor you infrastructure, there is become essential to monitor application it self. That mean, application server monitoring, DB monitoring, middle layer monitoring etc. Based on technology each items has its own different performance counters. For dotnet, java, ruby, python, php etc has its own performance counters according to its architecture and working mechanism. To pinpoint your bottlenecks and validate improvements, you must know your application activity. Each technology has its own monitoring tools.
Java : Java Mission Control , Visual VM, Command line tools
Dotnet : PerfView & Framework tools .

And again, APM tools these days can track application activity also, with server monitoring and browser activity monitoring.

These are some primary items. To know each item deeper, don't miss this full page dedicated for performance.


Performance Reporting : KPI (key Performance Indicator)

In this article we are going to know about KPI for software, especially web application. And, we will get general idea on how to represent performance through KPI. 

What is KPI ? 
By Name , KPI refers to key performance indicator. A measurement which can shows application performance value as a whole. And, it can be represented in Graph or tabular format.

Think of KPI as abstract value which tells us about performance at a glance. We will compare with , some expected value and define acceptance of the application performances.

In general, if we think of KPI of a ball point pen, the KPI should refer to performance quality of the pen. And, performance quality of a pen is how well it can write, how long it can last etc. 

So, KPI value should contain the reference value of "how well" & "How long". So, we will provide quality measurement "how well" & "How long" and then we can add them to express KPI of the pen. You can add here color intensity that might change with a time to completely describe in pen case.

Note : we can also multiply them , this decision based on how we calculate KPI. Usually, multiplication express more importance in graph representation.

Let' s have some example for web application : In general (unless you have any specific performance target) , a web application performance quality measure by
1. How fast the response time
2. How quick it can perform basic functionality(if we think for banking domain, it is business transactions)
3. How much less it takes resource(size & memory)

More often, as web applications are multi-tier application, it is very logical to measure particular application performance. For server performance , it is used to measure
1. How much is the Server Throughput (Hit/sec)
2. How much is error rate (down time %)

So, let's think about an any simple company internal application (not public facing) build on ASP.Net and has oracle DB to serve data. If we are measuring KPI, we should come up with some values that represent the performance.

For typical web product, it is considered, as less the size of the page is , the more it is performant. Same goes for as less bandwidth it takes, it is more performant.

As we are considering internal application, we may ignore bandwidth.  So, the important measuring parameters are
>Response Time
>Size of each page/request(send + received)
And, to add server power with that, we should add
>Server throughput

If you have load balancer and serial layer for data communication(we are assuming , we have) and, application gets different type of error due to Database and legacy system dependency, we should skip error rate. If we include this error%, it might be not logical for testing web servers only. So, we skip error% from key performance indication.(when we test individual servers, then it is logical. That means, webservice, DB, middle tire all separately). 

[Note : If it is customer facing website, we should not ignore error rate ]

So, we have 3 items. And, the meaning of each item is
a. Response time : More response time less performant(inversely proportionate )
b.Size : More size, less performant(inversely proportionate )
c.Throughput: More throughput , more performant. (proportionate )

So, in here KPI for each step/transaction = Response Time x Size x(1/Throughput)

In here, you can calculate throughput inversely and then multiply all. Or, add all.
= Response Time x Size x Throughput inverse
= Response Time + Size + Throughput inverse
(addition is also used to measure the value)

[Note : To keep graph more align, we might have to change the unit, Example – millisecond to min/hour, changing items in Kilo/Mega range ]

So, KPI for a step/transaction in that application is
KPI = RT*Size*TrInv

If we have multiple steps in test case (usually we have) and each of them have different priority, then we can include priority in KPI to keep things more realistic. (it will help in release to release comparison)
Then the KPI = RT*Size*TrInv*Priority.

So, let say in the test application , we have 5 transactions to test. And in current release, we have tested and get all KPI values from test tool. How can we measure , we are going good or bad (people provide large benchmarks which is not always useful). In that case , we should use KPI delta

KPI Delta = KPI expected- KPI from test results

In here expected KPI = the same KPI value calculation from expected results. (We should include SLA specification reflecting inside this expected KPI)

For example , Let say, Log in transaction. From test results we have found ,
Response time = 5s (Expected was 2s)
Size = 1.5Mb (expected was 1MB)
Through put = 20 hit per sec (Expected was 25)
Priority = 5(Measuring 1-10)

So, KPI = (5*1.5/20)*5 = 1.875
Expected KPI = (2*1/25)*5=0.4

So, Delta KPI = 0.4-1.875  = -1.475

As Delta is below Zero, so, application performance is bad. And , really it is, as we failed to achieve all expected performance marks.

So, to measure application performance here, if we use single Performance metrics KPI Delta, we can express performance status in single graph. This graph will be shown  
a.Over time: X-Axis = time, Y-axis = KPI Delta . It means  Performance status over time passes.
b.Over user : X-Axis = User Increment, Y-axis = KPI Delta.  It means  Performance status over user increment.

Note :

1.If we have difficulties with Expected KPI, we can assume any version of application as standard. So in that case the
KPI Delta = KPI Of Previous Standard version- KPI from test results

2.While making test scripts, we need to keep arrangement in such way so that we can measure our KPI directly from raw results , which makes faster on reporting.

3.Regarding response time, from test tools, we can get Max, Min, Average, 90% response time. We might be confusion , what should we choose here. By nature, 90% response time is more accurate to real customer time, my preference is this. But, if you application is very fast and scalable , have less initiation time, you can choose average also. You may choose MAX also when you need to evaluate performance for worst case scenario.

4. Scalability is also a part of performance , but for measuring, it is ignored as that is related to capacity

For web service (as it is vastly used in web), I prefer
KPI = Response time * Error Rate* size of response*Throughput Inverse

A Example : 
In this example , we can see the performance trends. 
Total 3 test performed. And test 1 considered as standard. 
Total 4 requests are there but, see the "Ajax" request. The performance trends says, it is getting low in performance from standard mark. Other requests seems higher , that means, trends is good. 

Thanks, please let me know if you have any questions....:)  

How to run jmeter in Jnekins with Performance Plugins?

In this article we are going to see how can we run Jmeter tests using Jenkins. With jenkins we can run Jmeter in several ways. But, we only show the usages of Performance Plugins.

Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.

1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter

(And many more , depend on what plugins you use)

As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.

Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.

Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )

Step 2 : Open Jenkins in browser and go to dashboard.

Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)

-Provide a Job Name , I gave "JmeterExample-Local"

Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name

Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)

Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.

This post will help to know about Jmeter command line commands. I am using following

@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl

For this , I use simple Jmeter scripts like this,

Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".

This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.

For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.

Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.

Step 6 : In Threshold section, I use this for accepted criteria like as follows.

Step 7 : Save the job and run.

Form running build if you see console , you can see where the results are saved by jenkins.

inside job folder


Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"

If you click "Performance Trend" you can see this result dashboard.

I hope dashboard charts are self explanatory. If you need to know details, comment please.