2015-10-19
Performance analysis : Top-Down and Bottom-Up Approach
In this article we are going to see basic Performance Analysis approaches. I will be referring to Top-Down & Bottom-up approach. I will not compare them as both are used for analysis, I will only try to explain what are the basic steps and when to choose what type. This article mostly match for Java & DotNet application. You may use similar approaches for other platforms too.
Top-Down Analysis :
This is the most popular method. The idea is simple, performance monitoring application from top view. That means, client monitoring -> server monitoring in OS & Resource level, then app server-> then run time environment
And, when we get unusual behavior (or not expected or targeted), then profiling or instrument application. In this moment, we have found problems , and experiment with possible solution, choose the best one.
And then, we have to tune the system. Tune code with best solution, tune environment and tune run time.
Finally, we need to test for the impact. Typically, before starting an analysis, we should have some measurement data(or benchmarks) from performance test results. We need to retest those and compare with previous results. If that is significant, then it is done, if not, we need to get back to profiling and tuning application.
Here is a flow chart to show at a glance : Google drive link. Open with draw.io.
When we use?
-> Application causing issues, we need optimize whole or a part of application.
-> Optimize application for given resources(CPU/Disk/Memory/IO/network)
-> Tune the system & application for best performance.
-> We have access to code change, now need to tune application for specific goal.(throughput, response time, longer service etc).
-> We need Root Cause Analysis for unexpected Performance Issues(OOM, Slowness, crashing in different level or sub-systems, unwanted application behavior primary suspected for performance, etc)
Bottom-Up Analysis :
Here is a flow chart to show at a glance : Google drive link. Open with draw.io.
When we use?
-> When you need to increase performance but you cant change source code.
Top-Down Analysis :
This is the most popular method. The idea is simple, performance monitoring application from top view. That means, client monitoring -> server monitoring in OS & Resource level, then app server-> then run time environment
And, when we get unusual behavior (or not expected or targeted), then profiling or instrument application. In this moment, we have found problems , and experiment with possible solution, choose the best one.
And then, we have to tune the system. Tune code with best solution, tune environment and tune run time.
Finally, we need to test for the impact. Typically, before starting an analysis, we should have some measurement data(or benchmarks) from performance test results. We need to retest those and compare with previous results. If that is significant, then it is done, if not, we need to get back to profiling and tuning application.
Here is a flow chart to show at a glance : Google drive link. Open with draw.io.
When we use?
-> Application causing issues, we need optimize whole or a part of application.
-> Optimize application for given resources(CPU/Disk/Memory/IO/network)
-> Tune the system & application for best performance.
-> We have access to code change, now need to tune application for specific goal.(throughput, response time, longer service etc).
-> We need Root Cause Analysis for unexpected Performance Issues(OOM, Slowness, crashing in different level or sub-systems, unwanted application behavior primary suspected for performance, etc)
Bottom-Up Analysis :
This is another popular approach when we need to tune resource or platform (or hardware) for specific application. Let say, you have a java application , deployed. Now, bottom up analysis will allow you to analyze and find optimization scope for deployed system, hardware and resources. This is very common approach for application capacity planning, benchmarking for changed environments(migration). The key idea is, monitor application in specific environment and then tune environment(software+ hardware resources) that makes target application running at top performance.
Here is a flow chart to show at a glance : Google drive link. Open with draw.io.
When we use?
-> When you need to increase performance but you cant change source code.
-> You need to optimize resources & environment for specific application( deployed environment)
-> You need to have benchmark and get to know resource usages as well as find possible area for tuning.
-> You need to optimize run time (JVM/CLR) for your application. You can see resource usages and tune as your app needs.
-> When you need capacity planning for your hardware which must run application in optimal way.
-> When you have optimized your application from code and there is not visible area to tune, you can use this technique to achieve some more.
Please comment if you have any question.
Thanks.. :)
Please comment if you have any question.
Thanks.. :)
2015-10-07
2015-09-08
Offline Java Memory Analysis : Introduction to tools for Heap and Thread Dumps, GC log Analysis
In this article we are going to see different tools which can be used for different Java Memory Analysis in offline. I will try to include all kinds of offline analysis tools. OS process memory monitoring and tools are ignored due to context.
What is offline analysis?
When we do analysis on recorded data , not live data, then it refers to offline analysis.
For Java memory analysis, we will perform analysis on not live data but recorded. For Java memory analysis, we need mainly three type of information of JVM to get to bottom of it.
1. Java Heap Dump : This represents JVM heap memory information
2. Java Core /Thread Dump: This shows running thread state and conditions. The core contains more detail information. For IBM , core and thread dump are same. Different JVM may also include trace files here.
3. GC logs : This shows Garbage collection history in logs.
I will not go detail on each one, lets see the tools only
For All IBM JVM : You need to have IBM Support Assistant. This is a web distribution, you need to run this with Java and your PC will host all tools to gather. Here is the download link. How to setup IBM ISA? Unzip and run start_isa.bat .
And in browser if you go to this link http://localhost:10911/isa5 You will be redirected here (all of this link & ports are configurable)
You will get 3 type of tools, JNLP web start, eclipse plugins link, web based. If you download and open the JNLP via notepad, you can actually configure JVM configuration. I used make a separate bat file to run JNLP with IBM JVM(non environment run time environment)
Make sure, your PC don’t have global log file as environment variable(if you install HP testing product you will have), Just delete that environment variable. ISA will create a variable path for log for it self and you can run this.
For all Oracle JVM :
1. Java Mission Control(JMC), Comes free with JDK. But you need to install these tools as plugins.
a. JOverflow / Heap Analysis
b. DTrace Recorder
c. Flight Recorder Plugins
d. Console Plugins
2. Visual VM : Comes free with JDK. Download useful plugins to get best out of it.
Architecturally JMC based on eclipse & Visual VM based on netbeans., so plugins installation process follows process of those IDE in installation (network config+ tools installations)
2. Visual VM : Oracle hprof format heap dump reader
3. JOverflow/Dump Analysis as plugins of Java Mission Control : Oracle hprof format heap dump reader, provides more details analysis than Visual VM.
For IBM JVM: (Java Core file and .trc file)
1. JCA (Java Core analyzer) : This is from IBM support. Download and load text format java Core files.
2. GCMV( Garbage collector Memory Visualizer) standalone or GCMV Eclipse Plugins , comes with ISA
3. Class Loader Analyzer (IBM only) used for class loader analysis from java core and Snap<>.trc file. It comes with ISA.
4. Trace & Request analyzer(IBM only) used for reading Snap<>.trc files . Download Link
5. Thread & Monitor Dump Analyzer (TMDA, IBM only)
Note : .trc is trace file, you need to enable and configure to get proper information.
For Oracle JVM
1. .tdump format: Visual VM
2. hs_err_pid.log format : We need to use Command line tools comes free with JDK.
a. jstack -> Prints Stack Trace
b. jmap -> Heap memory details
And most of the time manually reading as this is text format. We can use filtering. There are also small parser, mainly shell scripts for different filters. Here is one example.
2. PMAT (IBM Pattern Modeling Tool) : Multiple GC log together, comes with ISA
3. One of good 3rd Party Log viewers : GCViewer , Supported formats :
Sun JDK 1.4/1.5 ( -Xloggc:<file> [-XX:+PrintGCDetails] )
Sun JDK 1.2.2/1.3.1/1.4 (-verbose:gc )
IBM JDK 1.3.1/1.3.0/1.2.2 ( -verbose:gc)
IBM iSeries Classic JVM 1.4.2 (-verbose:gc)
HP-UX JDK 1.2/1.3/1.4.x ( -Xverbosegc)
BEA JRockit 1.4.2/1.5 (-verbose:memory)
Some other GC log reader tools you may find in this link.
Note :
1. For both VMs, we can use JProfiler, which is paid tool. If we use yourkit, different version of yourKit supports different version of JVMs, see archive page of yourkit for more details
2. For IBM tools, it is better to use IBM JVM. Either you can download JDK from IBM, or , you can download their development package with eclipse where JDK is present.
3. For IBM JVMs, usually an OOM will create a phd, a javaCore & a trc file. In case you are storing GC logs, verbosegc log will be there.
Please comment if you have any question. Thanks.. :)
What is offline analysis?
When we do analysis on recorded data , not live data, then it refers to offline analysis.
For Java memory analysis, we will perform analysis on not live data but recorded. For Java memory analysis, we need mainly three type of information of JVM to get to bottom of it.
1. Java Heap Dump : This represents JVM heap memory information
2. Java Core /Thread Dump: This shows running thread state and conditions. The core contains more detail information. For IBM , core and thread dump are same. Different JVM may also include trace files here.
3. GC logs : This shows Garbage collection history in logs.
I will not go detail on each one, lets see the tools only
For All IBM JVM : You need to have IBM Support Assistant. This is a web distribution, you need to run this with Java and your PC will host all tools to gather. Here is the download link. How to setup IBM ISA? Unzip and run start_isa.bat .
And in browser if you go to this link http://localhost:10911/isa5 You will be redirected here (all of this link & ports are configurable)
You will get 3 type of tools, JNLP web start, eclipse plugins link, web based. If you download and open the JNLP via notepad, you can actually configure JVM configuration. I used make a separate bat file to run JNLP with IBM JVM(non environment run time environment)
Make sure, your PC don’t have global log file as environment variable(if you install HP testing product you will have), Just delete that environment variable. ISA will create a variable path for log for it self and you can run this.
For all Oracle JVM :
1. Java Mission Control(JMC), Comes free with JDK. But you need to install these tools as plugins.
a. JOverflow / Heap Analysis
b. DTrace Recorder
c. Flight Recorder Plugins
d. Console Plugins
2. Visual VM : Comes free with JDK. Download useful plugins to get best out of it.
Architecturally JMC based on eclipse & Visual VM based on netbeans., so plugins installation process follows process of those IDE in installation (network config+ tools installations)
Java Heap Analysis :
1. Eclipse MAT(including IBM IDDE) supports IBM (phd)& Oracle JVM heap dumps.
2. Visual VM : Oracle hprof format heap dump reader
3. JOverflow/Dump Analysis as plugins of Java Mission Control : Oracle hprof format heap dump reader, provides more details analysis than Visual VM.
Thread Analysis :
For IBM JVM: (Java Core file and .trc file)
1. JCA (Java Core analyzer) : This is from IBM support. Download and load text format java Core files.
2. GCMV( Garbage collector Memory Visualizer) standalone or GCMV Eclipse Plugins , comes with ISA
3. Class Loader Analyzer (IBM only) used for class loader analysis from java core and Snap<>.trc file. It comes with ISA.
4. Trace & Request analyzer(IBM only) used for reading Snap<>.trc files . Download Link
5. Thread & Monitor Dump Analyzer (TMDA, IBM only)
Note : .trc is trace file, you need to enable and configure to get proper information.
For Oracle JVM
1. .tdump format: Visual VM
2. hs_err_pid.log format : We need to use Command line tools comes free with JDK.
a. jstack -> Prints Stack Trace
b. jmap -> Heap memory details
And most of the time manually reading as this is text format. We can use filtering. There are also small parser, mainly shell scripts for different filters. Here is one example.
GC Log Analysis :
As , there are different type of JVM implemented from Open JDK, GC log also have different format. Mostly commonly use IBM & Oracle format. And in each JVM there are different type of GC logs but mostly they follow the general format. Which is text format. Some times , old IBM verbose GC might not be compatible for latest openJVM or Oracle JVM generated GC. How to generate log, I will go details in separate blog.
For Oracle(default : Solaris JVM format) & IBM Verbose GC log reading :
1. GCMV( Garbage collector Memory Visualizer) standalone or GCMV Eclipse Plugins , comes with ISA
For Oracle(default : Solaris JVM format) & IBM Verbose GC log reading :
1. GCMV( Garbage collector Memory Visualizer) standalone or GCMV Eclipse Plugins , comes with ISA
2. PMAT (IBM Pattern Modeling Tool) : Multiple GC log together, comes with ISA
3. One of good 3rd Party Log viewers : GCViewer , Supported formats :
Sun JDK 1.4/1.5 ( -Xloggc:<file> [-XX:+PrintGCDetails] )
Sun JDK 1.2.2/1.3.1/1.4 (-verbose:gc )
IBM JDK 1.3.1/1.3.0/1.2.2 ( -verbose:gc)
IBM iSeries Classic JVM 1.4.2 (-verbose:gc)
HP-UX JDK 1.2/1.3/1.4.x ( -Xverbosegc)
BEA JRockit 1.4.2/1.5 (-verbose:memory)
Some other GC log reader tools you may find in this link.
Note :
1. For both VMs, we can use JProfiler, which is paid tool. If we use yourkit, different version of yourKit supports different version of JVMs, see archive page of yourkit for more details
2. For IBM tools, it is better to use IBM JVM. Either you can download JDK from IBM, or , you can download their development package with eclipse where JDK is present.
3. For IBM JVMs, usually an OOM will create a phd, a javaCore & a trc file. In case you are storing GC logs, verbosegc log will be there.
Please comment if you have any question. Thanks.. :)
2015-08-27
2015-08-26
2015-08-25
2015-08-19
2015-07-27
2015-07-03
How to run jmeter in Jnekins with Performance Plugins?
In this article we are going to see how can we run Jmeter tests using Jenkins. With jenkins we can run Jmeter in several ways. But, we only show the usages of Performance Plugins.
Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.
1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter
(And many more , depend on what plugins you use)
As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.
Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.
Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )
Step 2 : Open Jenkins in browser and go to dashboard.
Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)
-Provide a Job Name , I gave "JmeterExample-Local"
Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name
Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)
Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.
This post will help to know about Jmeter command line commands. I am using following
@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl
For this , I use simple Jmeter scripts like this,
Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".
This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.
For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.
Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.
Step 6 : In Threshold section, I use this for accepted criteria like as follows.
Step 7 : Save the job and run.
Form running build if you see console , you can see where the results are saved by jenkins.
inside job folder
Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"
If you click "Performance Trend" you can see this result dashboard.
I hope dashboard charts are self explanatory. If you need to know details, comment please.
Thanks..:)
Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.
1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter
(And many more , depend on what plugins you use)
As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.
Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.
Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )
Step 2 : Open Jenkins in browser and go to dashboard.
Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)
-Provide a Job Name , I gave "JmeterExample-Local"
Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name
Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)
Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.
This post will help to know about Jmeter command line commands. I am using following
@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl
For this , I use simple Jmeter scripts like this,
Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".
This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.
For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.
Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.
Step 6 : In Threshold section, I use this for accepted criteria like as follows.
Step 7 : Save the job and run.
Form running build if you see console , you can see where the results are saved by jenkins.
inside job folder
Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"
If you click "Performance Trend" you can see this result dashboard.
I hope dashboard charts are self explanatory. If you need to know details, comment please.
Thanks..:)
Subscribe to:
Posts (Atom)