2015-07-03

How to run jmeter in Jnekins with Performance Plugins?

In this article we are going to see how can we run Jmeter tests using Jenkins. With jenkins we can run Jmeter in several ways. But, we only show the usages of Performance Plugins.

Jenkins is a CI server. It runs processes. It has different ways to run process. Using its different plugins we can Run Jmeter in following ways.

1. With Windows Batch (we will use this approach)
2. Linux shell
3. Groovy/Gradle
4. Maven Job
5. ANT job
6. Taurus/Blazemeter

(And many more , depend on what plugins you use)

As we know we can run Jmeter from command lines(see my old post). So, we use this command to run in windows environment.

Let't have some idea about this Performance Plugin. By the name it might be confusing that the plug in will be used for performance testing, actually it is not . It only shows current logs and results and differences with previous results to make as Trends.( Helpful for heuristic analysis). So, the steps will be,
1. Run Jmeter in standard Batch file execution
2. Add Several Jmeter reports Post build actions which will automatically show current results and Performance Trends. So, lets start.

Step 1 : Install Jenkins and Install Performance Plugins (standard way of plugins installation , I am not showing this details, get latest release hpi )

Step 2 : Open Jenkins in browser and go to dashboard.

Step 3 : Click New Item & Select Free style Project(you can select others type also, but I will use only windows batch, so this option is okay)

-Provide a Job Name , I gave "JmeterExample-Local"

Step 4 : Click Advance and select "Use custom workspace"
- Provide you script location in "Directory" text box
- You can provide name for this folder in "Display Name", I keep same as Job Name


Note : I used to make a separate folder as working directory where I keep test JMX, log file, result JTL and a Batch file(backup of the command that use in Jenkins, so that if Jenkins is down, I can still use single click execution from command line)

Step 4 : Click Add build Script Combo box and select "Execute Windows Batch Command"
This will show text box to add command.




This post will help to know about Jmeter command line commands. I am using following

@echo off
set user=20
set rampup=5
set iteration=3
jmeter -n -t JmeterPlan.jmx -Jusers=%user% -Jramp=%rampup% -Jiteration=%iteration% -l results.jtl

For this , I use simple Jmeter scripts like this,



Step 5: Now, the important part. Using Performance Plugins Reports. Click Add Post-Build action combo box and select "Publish Performance test result report".



This will enable a whole section to select different report. As this plugins include Jmeter and JUnit report types, we can see several type.



For this blog, I have used only "Jmeter" and "JmeterSummarizer" reports.



Select those report and provide file names. If you generate/save CSV format results, you can also select JmeterCSV report also to see.

Step 6 : In Threshold section, I use this for accepted criteria like as follows.



Step 7 : Save the job and run.

Form running build if you see console , you can see where the results are saved by jenkins.


inside job folder

 


Inside of the job(not dashboard) you will be able to see a new report, "Performance Trend"



If you click "Performance Trend" you can see this result dashboard.


I hope dashboard charts are self explanatory. If you need to know details, comment please.

Thanks..:) 

11 comments:

  1. Hi Sarker,

    I am getting below error. Could you please help me.FailedConsole Output

    Started by user anonymous
    Building in workspace C:\Users\ravichandar.d\.jenkins\Ravi
    [Ravi] $ cmd /c call C:\Users\RAVICH~1.D\AppData\Local\Temp\hudson2481112220767782328.bat

    C:\Users\ravichandar.d\.jenkins\Ravi>jmeter -n -t C:\Jenkins\TS01_CREATEPROJECT.JMX -l results.jtl
    'jmeter' is not recognized as an internal or external command,
    operable program or batch file.

    C:\Users\ravichandar.d\.jenkins\Ravi>exit 9009
    Build step 'Execute Windows batch command' marked build as failure
    Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
    Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable



    Finished: FAILURE

    ReplyDelete
    Replies
    1. Jmeter is not in your path, amend jmeter bin directory in your path variable

      Delete
  2. Hi Sarker,

    How can we set the command line when we have multiple thread groups? Given that the thread groups have different users and ramp up.

    ReplyDelete
    Replies
    1. in my example, i used only one jmeter properties to parametrize the number of users , you can use multiple properties as input of different thread groups, or same property with different formulas in different groups. It's just a parameter, that's all.

      Delete
  3. Hi Sarker,

    I am getting the below error when I execute the test in jenkins:
    Please help..

    Started by user anonymous
    Building in workspace C:\apache-jmeter-2.13\bin
    [bin] $ cmd /c call C:\Windows\TEMP\hudson581471014683945096.bat
    Creating summariser
    Created the tree successfully using test.jmx
    Starting the test @ Tue Oct 06 13:50:30 EDT 2015 (1444153830674)
    Waiting for possible shutdown message on port 4445
    summary + 1 in 1s = 1.3/s Avg: 589 Min: 589 Max: 589 Err: 0 (0.00%) Active: 1 Started: 1 Finished: 0

    summary + 15 in 27.4s = 0.5/s Avg: 413 Min: 9 Max: 1873 Err: 0 (0.00%) Active: 0 Started: 1 Finished: 1
    summary = 16 in 28.1s = 0.6/s Avg: 424 Min: 9 Max: 1873 Err: 0 (0.00%)
    Tidying up ... @ Tue Oct 06 13:50:59 EDT 2015 (1444153859488)
    ... end of run

    Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
    Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable

    Performance: Parsing JMeter report file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JMeter\results.jtl'.
    Performance: Failed to parse file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JMeter\results.jtl': For input string: "timeStamp elapsed label responseCode responseMessage threadName dataType success bytes grpThreads allThreads Filename Latency"
    java.lang.NumberFormatException: For input string: "timeStamp elapsed label responseCode responseMessage threadName dataType success bytes grpThreads allThreads Filename Latency"
    at java.lang.NumberFormatException.forInputString(Unknown Source)
    at java.lang.Long.parseLong(Unknown Source)
    at java.lang.Long.valueOf(Unknown Source)
    at hudson.plugins.performance.JMeterCsvParser.getSample(JMeterCsvParser.java:154)
    at hudson.plugins.performance.JMeterCsvParser.parse(JMeterCsvParser.java:124)
    at hudson.plugins.performance.JMeterParser.parseCsv(JMeterParser.java:193)
    at hudson.plugins.performance.JMeterParser.parse(JMeterParser.java:51)
    at hudson.plugins.performance.AbstractParser.parse(AbstractParser.java:66)
    at hudson.plugins.performance.PerformancePublisher.perform(PerformancePublisher.java:539)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:726)
    at hudson.model.Build$BuildExecution.post2(Build.java:185)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:671)
    at hudson.model.Run.execute(Run.java:1766)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
    Performance: Parsing JMeter report file 'C:\Program Files (x86)\Jenkins\jobs\Testproject\builds\2\performance-reports\JmeterSummarizer\jmeter.log'.
    Performance: no JMeterCsv files matching 'results.csv' have been found. Has the report generated?. Setting Build to FAILURE
    Build step 'Publish Performance test result report' changed build result to FAILURE
    Finished: FAILURE

    ReplyDelete
    Replies
    1. The error is showing number format could not convert. I think it is casting error. Send me JMX, i can check in weekend.

      Delete
    2. Shantonu Sarkar, did you got any solution for the above "Performance: no JMeterCsv files matching 'results.csv' have been found" error? if so, can you email me the solution to 105983@merrillcorp.com

      Delete
  4. Hi Shantonu,

    Getting below error. Please have a look.

    Started by user anonymous
    Building in workspace D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx
    [Jenkins_1.jmx] $ cmd /c call C:\Users\admin\AppData\Local\Temp\hudson7423588798529245178.bat
    The directory name is invalid
    FATAL: command execution failed
    java.io.IOException: Cannot run program "cmd" (in directory "D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx"): CreateProcess error=267, The directory name is invalid
    at java.lang.ProcessBuilder.start(Unknown Source)
    at hudson.Proc$LocalProc.(Proc.java:244)
    at hudson.Proc$LocalProc.(Proc.java:216)
    at hudson.Launcher$LocalLauncher.launch(Launcher.java:816)
    at hudson.Launcher$ProcStarter.start(Launcher.java:382)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
    at hudson.model.Build$BuildExecution.build(Build.java:205)
    at hudson.model.Build$BuildExecution.doRun(Build.java:162)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:537)
    at hudson.model.Run.execute(Run.java:1741)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:98)
    at hudson.model.Executor.run(Executor.java:408)
    Caused by: java.io.IOException: CreateProcess error=267, The directory name is invalid
    at java.lang.ProcessImpl.create(Native Method)
    at java.lang.ProcessImpl.(Unknown Source)
    at java.lang.ProcessImpl.start(Unknown Source)
    ... 16 more
    Build step 'Execute Windows batch command' marked build as failure
    Performance: Percentage of relative difference outside -0.0 to +5.0 % sets the build as failure
    Performance: Percentage of relative difference outside -70.0 to +95.0 % sets the build as unstable

    ReplyDelete
    Replies
    1. you can see the error -> java.io.IOException: Cannot run program "cmd" (in directory "D:\Jmeter\apache-jmeter-2.13\bin\Jenkins_1.jmx"): CreateProcess error=267, The directory name is invalid

      your jenkins cant find the directory (or cant modify)

      Delete
  5. FailedConsole Output

    Started by user admin
    Running as SYSTEM
    Building remotely on linux-node (dev-label) in workspace /jenkins-home/workspace/linux-job
    Installing JDK jdk-8u202-oth-JPR
    Downloading JDK from https://download.oracle.com/otn/java/jdk/8u202-b08/1961070e4c9b4e26a04e7f5a083f551e/jdk-8u202-linux-x64.tar.gz
    FATAL: command execution failed
    java.net.SocketException: Connection reset
    at java.net.SocketInputStream.read(SocketInputStream.java:210)
    at java.net.SocketInputStream.read(SocketInputStream.java:141)
    at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
    at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
    at org.apache.commons.httpclient.HttpParser.readRawLine(HttpParser.java:85)
    at org.apache.commons.httpclient.HttpParser.readLine(HttpParser.java:113)
    at org.apache.commons.httpclient.HttpConnection.readLine(HttpConnection.java:1123)
    at org.apache.commons.httpclient.HttpMethodBase.readStatusLine(HttpMethodBase.java:1980)
    at org.apache.commons.httpclient.HttpMethodBase.readResponse(HttpMethodBase.java:1742)
    at org.apache.commons.httpclient.HttpMethodBase.execute(HttpMethodBase.java:1105)
    at org.apache.commons.httpclient.HttpMethodDirector.executeWithRetry(HttpMethodDirector.java:405)
    at org.apache.commons.httpclient.HttpMethodDirector.executeMethod(HttpMethodDirector.java:178)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:404)
    at org.apache.commons.httpclient.HttpClient.executeMethod(HttpClient.java:330)
    at hudson.tools.JDKInstaller.locate(JDKInstaller.java:483)
    at hudson.tools.JDKInstaller.performInstallation(JDKInstaller.java:150)
    at hudson.tools.InstallerTranslator.getToolHome(InstallerTranslator.java:69)
    at hudson.tools.ToolLocationNodeProperty.getToolHome(ToolLocationNodeProperty.java:109)
    at hudson.tools.ToolInstallation.translateFor(ToolInstallation.java:206)
    at hudson.model.JDK.forNode(JDK.java:148)
    at hudson.model.AbstractProject.getEnvironment(AbstractProject.java:341)
    at hudson.model.Run.getEnvironment(Run.java:2410)
    at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:864)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:102)
    at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66)
    at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:741)
    at hudson.model.Build$BuildExecution.build(Build.java:206)
    at hudson.model.Build$BuildExecution.doRun(Build.java:163)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:504)
    at hudson.model.Run.execute(Run.java:1880)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:428)
    Build step 'Execute shell' marked build as failure
    Finished: FAILURE

    ReplyDelete
    Replies
    1. FATAL: command execution failed
      java.net.SocketException: Connection reset

      these is the main problem. make sure your admin user has internet access

      Delete