Live Performance Result Monitoring with Jmeter-Grafana-influxdb

In this article we are going to see how can we build a system where we can live monitor jmeter test results.

Jmeter GUI is well known for its huge resource intensiveness.So monitoring results with GUI become very very non realistic on massive load. In here we will use Jmeter in CLI mode to test and monitor results via Gratiana. This article is intended for installation & basic configuration only. Detail Grafana graph configuration will be given in separate post.

What is Grafana ? 
Grafana is well known industry standard monitoring dashboard used in devops. We will use this for performance monitoring. Project link here.  It needs a data source. so, we will use influxdb.

Influxdb is a database which has inbuilt http based management console and provide flexible data storage . We will use inflexdb graphite plugins to connect this with grafana. Main project link here

We will use Jmeter "Backend Listener" to generate results and influxdb to store the results. Grafana will take results with simple query and show in browser. In a nutshell , this is the procedure.




Why it is useful? 
- Grafana graphs are very good for analysis.(zooming, combined graph)
- As results are stored in DB, we can analysis when ever we can
- We can monitor results live
- We can monitor results from multiple jmeter nodes , so no need Jmeter client-server execution.


In this whole example, I am using ubuntu 14 VM, you can use real PC or even docker. These are some of properties of my VM
IP (host name) : 172.16.244.137
user name : shantonu
So, when you try this instruction, you should use your VM's IP.
I will use same VM to host influxdb & grafana. You can use different host. 

Installing InfluxDB : 
Step 1 : Download & install influxdb in command-line

$ wget https://s3.amazonaws.com/influxdb/influxdb_0.10.1-1_amd64.deb
$ sudo dpkg -i influxdb_0.10.1-1_amd64.deb

(main url : https://s3.amazonaws.com/influxdb, where you can see different packages including windows )

Note : you may start influxdb as service but before starting as service you need to configure graphite plugin format which allows us to communicate with grafana.


Configure : After installation, influxdb configured in following ports.
Http (TCP) Management Port : 8083 
DB TCP port : 8086 

Let's configure graphite :

Way 1 : 
1. We wont start influxdb as service initially, 

2. Get default configuration  : In command line ,
$influxd config

you will see the configuration. Copy this and save to a file (I am saving to influxdb.conf)

3. Append  graphite plugin configuration : I am adding  this following at the end of   influxdb.conf file. 

[[graphite]]
enabled = true
bind-address = ":2003"
database = "jmeter"


4. Start influxdb with our configuration : in command line 
$ influxd -config <path>/influxdb.conf
[in my case influxd -config /usr/local/etc/influxdb.conf]

5. Now, to make life easier, I want to start influxdb with my configuration when my VM starts. So, I make a shell file(sh) and make it executable and keep it as startup. 

My influxdb.conf file link , (contains my user name, so change before use) 

So, we are done. 

Way 2 : 
 It is mall trick, just start influxdb service with your own configuration. 
$ cd /opt/influxdb
$ ./influxd -config <path>/influxdb.conf
in here , like way 1, we are using same file with just added graphite configuration. 

Validate Installation & Create DB for Jmeter : 

Now we have configured. Let's open browser and hit localhost:8083 or http://172.16.244.137:8083/



We can see management page. From query templatex , you can see lots of important info. 

 

Lets create a database for jmeter (named as jmeter) 
CREATE DATABASE "jmeter"

(Optional) , lets create an admin user. (i have created)

CREATE USER "shantonu" WITH PASSWORD '*******' WITH ALL PRIVILEGES

So, we are done.

Important Note : (If your adding repository is miss configured , you might face error while apt-get update. To solve this, you need to edit influxdb.list file under /etc/apt/sources.list.d folder. Just open this file as root user and edit the file

deb https://repos.influxdata.com/ubuntu precise stable


Installing & Configuration of Grafana : 

Download grafana : The main url (see all installers services : https://grafanarel.s3.amazonaws.com/)
As, I am using Ubuntu , I am downloading & installing  via command line : (you need your admin credentials)

$ wget https://grafanarel.s3.amazonaws.com/builds/grafana_2.6.0_amd64.deb
$ sudo apt-get install -y adduser libfontconfig
$ sudo dpkg -i grafana_2.6.0_amd64.deb
$ curl https://packagecloud.io/gpg.key | sudo apt-key add -


or if you are getting from APT. Add Package Repository URL( in /etc/apt/sources.list)
deb https://packagecloud.io/grafana/stable/debian/ wheezy main

And install in command line : 
$ sudo apt-get update
$ sudo apt-get install grafana


Now, you need to start grafana. I prefer to start as service so that each time I turn on my VM , it actually turned on automatically.

$ sudo service grafana-server start

Now, if I goto my real PC browser and hit http://172.16.244.137:3000 (or in VM localhost:3000) , I will see grafana home page.
Use default (admin, admin) as user name & password to get the dashboard.


Adding data source : 
Click Data Sources > Add New




Then select InfluxDB from type , give any name .

Put your DB URL and port which is open for getting data(DB port)



Optional :  I used but no need: Credential Information & Name(u can use any name)

So, its done. 

My learning source : http://docs.grafana.org/datasources/influxdb/

Making graph dashboard : 

I will be brief in this section due to different scope need different graph. I will make a separate post on Jmeter Plugins equivalence graph configuration in grafana. You can use grafana graph generator (https://github.com/bhattchaitanya/Grafana-Dashboard-Generator) to auto graph generation before tests.
So, after configuring DB. Click Home & add a Dashboard (click New)



(You can see, I have added Jmeter Live Test Dashboard)

In blank dashboard add a graph


Now, I will include simple query . When you see blank graph, you will see something like this ,


add jmeter.D.a.count. What does it means, simple, it shows count of D request from Jmeter table( a refers to single object/table).

Now in "General" tab , you can can configure( I use title as Max)

Now click save & back to dashboard. You can use multiple query results in single graph.

In this example I have added, single measure graph(MAX count of D), multiple measure graph(95th % + Max response time) and three overall info(started , max, ended thread count) . So finally my dashboard looks like this




BTW, each graph is editable & re-sizable, if you click a graph, you will see this menu

 
In here you can configuration or size.

Simple Jmeter test to monitor live results: 
-> I have created a jmeter test case with single request(http, get to google.com, renamed label as D)
-> I have added Backend Listener, I have configured




graphiteMetricsSender    : org.apache.jmeter.visualizers.backend.graphite.TextGraphiteMetricsSender
My Host(VM) -> graphiteHost  :  172.16.244.137
My Host(VM) -> graphitePort   : 2003
Name Prefix of DB entry -> rootMetricsPrefix  :  jmeter.
To have detail results -> summaryOnly  :  false
The Name of sample from test cases -> samplersList  :  D
Selected Percentiles -> percentiles  :  90;95;99

Note : Multiple entry rules , add semi colon (;) as separator

Now save the test plan and run it with forever loop(1 thread, 1 sec) , so that we can see update in DB as well as grafana. So, press start. (you can run test command line also)

Checking update in InfluxDB : 
1. Goto http://172.16.244.137:8083/
2. From top right corner, select database jmeter
3. From Query template , select SHOW MEASUREMENTS. This will show jemter tests are saved in DB




Checking Update in Grafana :  

As we have configured graph,
1. if we go to url http://172.16.244.137:3000/
2. Select our dashboard (Home > Jmeter Live Test)  we should be able to see this



Test is running, so end 0.

So, we have successfully setup grafana-influxdb jmeter graph to see live results.

Useful Links : 
InfluxDB Basic  : https://www.youtube.com/watch?v=sRi64imN7xg
Auto Grafana Graph generator for JMeter Test Case (i will blog separately) : https://github.com/bhattchaitanya/Grafana-Dashboard-Generator
Integrating JMX monitoring with Grafana : https://github.com/jmxtrans/jmxtrans/wiki
Docker-grafana-influxdb (Alternative of this solution, I will do a blog post) : https://github.com/kamon-io/docker-grafana-influxdb
http://davidanguita.name/articles/simple-data-visualization-stack-with-docker-influxdb-and-grafana/


Initially I read this in here with may many people's confusion. This post was intended to clear some of confusion. I hope this will help.

-----
 Thanks.. :)

12 comments:

  1. After running Jmeter scripts ,Measurements showing:- Success! (no results to display)

    ReplyDelete
    Replies
    1. Select DB from right upper corner and do it again.

      make sure your influxdb has graphite plugins configured.

      Delete
  2. Hi Shantonu!
    thanks for good and detailed guide on this.
    However I failed to found how to see in influxDB transactions response times without percentiles. Is it ever possible?

    Thanks,
    Artem

    ReplyDelete
    Replies
    1. Hi Artem,

      in jmeter response time comes in Max,Min, or any %tile.. In influx db , data stored in all of those fields. So, just select proper parameter for the graph.
      Check details here on which items are stored. http://jmeter.apache.org/usermanual/realtime-results.html
      Select among them.

      ----------- if you want to dig inside-------------
      If you check what jmeter sends , see this https://github.com/apache/jmeter/blob/trunk/src/components/org/apache/jmeter/visualizers/backend/BackendListener.java) Method name ( static void sendToListener)
      And , in sample result(https://github.com/apache/jmeter/blob/trunk/src/core/org/apache/jmeter/samplers/SampleResult.java) , it has every thing.

      So, practically it send everything as results , you need to see your query in grafana.
      And , if you want some more info , you need to make a plugins that extends backend listener and use this predefined statistics.
      https://github.com/apache/jmeter/blob/trunk/src/components/org/apache/jmeter/visualizers/backend/SamplerMetric.java
      https://github.com/apache/jmeter/blob/trunk/src/components/org/apache/jmeter/visualizers/backend/UserMetric.java

      thanks

      Delete
  3. Hey Shantanu, this is simply great, relatively new to this area. Was looking for this kind of tool. Just wanted to confirm few things

    1- Does Influxdb picks up the data from Webserver to track the traffic/load real time?
    2- Not sure if we can configure this for realtime servers with no UI

    ReplyDelete
    Replies
    1. thanks

      1. influxdb doesn't pick, we need to send. I am sending from jmeter backend listener to the influxdb host's 2003 port. This port is opened for graphite format input. (see the configuration)
      it is possible to send /receive data without any formating/plugins. I use 8086 port to actually read data from influxdb to show in grafana. you can send data using that.
      in fact, infludb supports multiple clustered DB , so you can have distributed DB for data collection which scales.

      2. Yes, this is full non UI, see my docker example http://shantonusarker.blogspot.com/2016/04/jmeter-grafana-influxdb-docker-live-monitoring.html
      and I test with jmeter which can be run in non gui or even from jenkins jobs(maven, ant, sh)

      Delete
  4. Hi Shantonu,
    Useful article.
    Keep it up.

    ReplyDelete
  5. We are trying to disply active threads with multiple jmx script that are controlled by Ant Build.properties
    But Grafana is just displaying the Max value of active threads amongst the jmx scripts as running Users.
    How do i handle this? I want to display all running VUsers of all the jmx scripts

    Suppose i have 3 scripts
    1. 4 Users
    2. 5 Users
    3. 6 users

    now i just see 6 Users as running with test.startedT

    how do i get sum of all these 3 i.e 15 Users

    ReplyDelete
    Replies
    1. V users in jmeter are the threads. So, include thread counts.

      Delete
  6. Awesome article Shantonu dada.

    Regards,
    Jobayer

    ReplyDelete