In this article we are going to see how can we run Jmeter tests using Jenkins. With jenkins we can run Jmeter in several ways. But, we only show the usages of Performance Plugins.
Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.
1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter
(And many more , depend on what plugins you use)
As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.
Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.
Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )
Step 2 : Open Jenkins in browser and go to dashboard.
Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)
-Provide a Job Name , I gave "JmeterExample-Local"
Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name
Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)
Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.
This post will help to know about Jmeter command line commands. I am using following
@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl
For this , I use simple Jmeter scripts like this,
Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".
This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.
For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.
Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.
Step 6 : In Threshold section, I use this for accepted criteria like as follows.
Step 7 : Save the job and run.
Form running build if you see console , you can see where the results are saved by jenkins.
inside job folder
Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"
If you click "Performance Trend" you can see this result dashboard.
I hope dashboard charts are self explanatory. If you need to know details, comment please.
Thanks..:)
Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.
1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter
(And many more , depend on what plugins you use)
As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.
Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.
Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )
Step 2 : Open Jenkins in browser and go to dashboard.
Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)
-Provide a Job Name , I gave "JmeterExample-Local"
Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name
Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)
Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.
This post will help to know about Jmeter command line commands. I am using following
@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl
For this , I use simple Jmeter scripts like this,
Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".
This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.
For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.
Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.
Step 6 : In Threshold section, I use this for accepted criteria like as follows.
Step 7 : Save the job and run.
Form running build if you see console , you can see where the results are saved by jenkins.
inside job folder
Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"
If you click "Performance Trend" you can see this result dashboard.
I hope dashboard charts are self explanatory. If you need to know details, comment please.
Thanks..:)
Hi Sarker,
ReplyDeleteI am getting below error. Could you please help me.FailedConsole Output
Started by user anonymous
Building in workspace C:\Users\ravichandar.d\.jenkins\Ravi
[Ravi] $ cmd /c call C:\Users\RAVICH~1.D\AppData\Local\Temp\hudson2481112220767782328.bat
C:\Users\ravichandar.d\.jenkins\Ravi>jmeter -n -t C:\Jenkins\TS01_CREATEPROJECT.JMX -l results.jtl
'jmeter' is not recognized as an internal or external command,
operable program or batch file.
C:\Users\ravichandar.d\.jenkins\Ravi>exit 9009
Build step 'Execute Windows batch command' marked build as failure
Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable
Finished: FAILURE
Jmeter is not in your path, amend jmeter bin directory in your path variable
DeleteHi Sarker,
ReplyDeleteHow can we set the command line when we have multiple thread groups? Given that the thread groups have different users and ramp up.
in my example, i used only one jmeter properties to parametrize the number of users , you can use multiple properties as input of different thread groups, or same property with different formulas in different groups. It's just a parameter, that's all.
DeleteHi Sarker,
ReplyDeleteI am getting the below error when I execute the test in jenkins:
Please help..
Started by user anonymous
Building in workspace C:\apache-jmeter-2.13\bin
[bin] $ cmd /c call C:\Windows\TEMP\hudson581471014683945096.bat
Creating summariser
Created the tree successfully using test.jmx
Starting the test @ Tue Oct 06 13:50:30 EDT 2015 (1444153830674)
Waiting for possible shutdown message on port 4445
summary + 1 in 1s = 1.3/s Avg: 589 Min: 589 Max: 589 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0
summary + 15 in 27.4s = 0.5/s Avg: 413 Min: 9 Max: 1873 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1
summary = 16 in 28.1s = 0.6/s Avg: 424 Min: 9 Max: 1873 Err: 0 (0.00%)
Tidying up ... @ Tue Oct 06 13:50:59 EDT 2015 (1444153859488)
... end of run
Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable
Performance: Parsing JMeter report file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JMeter\results.jtl'.
Performance: Failed to parse file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JMeter\results.jtl': For input string: "timeStamp elapsed label responseCode responseMessage threadName dataType success bytes grpThreads allThreads Filename Latency"
java.lang.NumberFormatException: For input string: "timeStamp elapsed label responseCode responseMessage threadName dataType success bytes grpThreads allThreads Filename Latency"
at java.lang.NumberFormatException.forInputString(Unknown Source)
at java.lang.Long.parseLong(Unknown Source)
at java.lang.Long.valueOf(Unknown Source)
at hudson.plugins.performance.JMeterCsvParser.getSample(JMeterCsvParser.java:154)
at hudson.plugins.performance.JMeterCsvParser.parse(JMeterCsvParser.java:124)
at hudson.plugins.performance.JMeterParser.parseCsv(JMeterParser.java:193)
at hudson.plugins.performance.JMeterParser.parse(JMeterParser.java:51)
at hudson.plugins.performance.AbstractParser.parse(AbstractParser.java:66)
at hudson.plugins.performance.PerformancePublisher.perform(PerformancePublisher.java:539)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
at hudson.model.Build$BuildExecution.post2(Build.java:185)
at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
at hudson.model.Run.execute(Run.java:1766)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:408)
Performance: Parsing JMeter report file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JmeterSummarizer\jmeter.log'.
Performance: no JMeterCsv files matching 'results.csv' have been found. Has the report generated?. Setting Build to FAILURE
Build step 'Publish Performance test result report' changed build result to FAILURE
Finished: FAILURE
The error is showing number format could not convert. I think it is casting error. Send me JMX, i can check in weekend.
DeleteShantonu Sarkar, did you got any solution for the above "Performance: no JMeterCsv files matching 'results.csv' have been found" error? if so, can you email me the solution to 105983@merrillcorp.com
DeleteHi Shantonu,
ReplyDeleteGetting below error. Please have a look.
Started by user anonymous
Building in workspace D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx
[Jenkins_1.jmx] $ cmd /c call C:\Users\admin\AppData\Local\Temp\hudson7423588798529245178.bat
The directory name is invalid
FATAL: command execution failed
java.io.IOException: Cannot run program "cmd" (in directory "D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx"): CreateProcess error=267, The directory name is invalid
at java.lang.ProcessBuilder.start(Unknown Source)
at hudson.Proc$LocalProc.(Proc.java:244)
at hudson.Proc$LocalProc.(Proc.java:216)
at hudson.Launcher$LocalLauncher.launch(Launcher.java:816)
at hudson.Launcher$ProcStarter.start(Launcher.java:382)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
at hudson.model.Build$BuildExecution.build(Build.java:205)
at hudson.model.Build$BuildExecution.doRun(Build.java:162)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
at hudson.model.Run.execute(Run.java:1741)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:98)
at hudson.model.Executor.run(Executor.java:408)
Caused by: java.io.IOException: CreateProcess error=267, The directory name is invalid
at java.lang.ProcessImpl.create(Native Method)
at java.lang.ProcessImpl.(Unknown Source)
at java.lang.ProcessImpl.start(Unknown Source)
... 16 more
Build step 'Execute Windows batch command' marked build as failure
Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable
you can see the error -> java.io.IOException: Cannot run program "cmd" (in directory "D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx"): CreateProcess error=267, The directory name is invalid
Deleteyour jenkins cant find the directory (or cant modify)
FailedConsole Output
ReplyDeleteStarted by user admin
Running as SYSTEM
Building remotely on linux-node (dev-label) in workspace /jenkins-home/workspace/linux-job
Installing JDK jdk-8u202-oth-JPR
Downloading JDK from https://download.oracle.com/otn/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jdk-8u202-linux-x64.tar.gz
FATAL: command execution failed
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:210)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:85)
at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:113)
at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1123)
at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1980)
at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1742)
at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1105)
at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:405)
at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)
at hudson.tools.JDKInstaller.locate(JDKInstaller.java:483)
at hudson.tools.JDKInstaller.performInstallation(JDKInstaller.java:150)
at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:69)
at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:109)
at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206)
at hudson.model.JDK.forNode(JDK.java:148)
at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:341)
at hudson.model.Run.getEnvironment(Run.java:2410)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:864)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:102)
at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
at hudson.model.Build$BuildExecution.build(Build.java:206)
at hudson.model.Build$BuildExecution.doRun(Build.java:163)
at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
at hudson.model.Run.execute(Run.java:1880)
at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
at hudson.model.ResourceController.execute(ResourceController.java:97)
at hudson.model.Executor.run(Executor.java:428)
Build step 'Execute shell' marked build as failure
Finished: FAILURE
FATAL: command execution failed
Deletejava.net.SocketException: Connection reset
these is the main problem. make sure your admin user has internet access