Creating a Grafana datasource using Flask and the Simple JSON datasource plugin

I was working on a project where I was creating time series data based on data stored in Postgres. The transformations needed of the data were very complex in pure SQL and I had implemented a better performing implementation in Python. The problem was that I was able to get the result I wanted in Python, but was unsure how I could visualize that data in Grafana.

I figured there must be a way to pull generic data from any source in Grafana and a quick search on Google lead me to this blog post http://www.oznetnerd.com/writing-a-grafana-backend-using-the-simple-json-datasource-flask/. I already had a way to get the result using Python so it seemed serendipitous that this post described how to use Flask and the Grafana Simple JSON datasource plugin to make it all come together. The writings I found about using the Simple JSON datasource seemed to be either too in depth or too light, so I decided I would write a basic overview of its functionality and provide an interactive tutorial to drill points home.

The code I have created for this tutorial can be found here https://github.com/Jonnymcc/grafana-simplejson-datasource-example.

There are three endpoints that need to be created for the datasource to be useful and three others for added functionality. The three that are required are…

  • /
  • /search
  • /query

And the three that are optional are…

  • /annotations
  • /tag-keys
  • /tag-values

The repo I created has instructions for getting Grafana running locally, and getting the Flask server running and configured in Grafana.

The first endpoint that needs to be provided is /. This endpoint is used when configuring a datasource in Grafana and clicking on the “Save & Test” button. All it needs to do is return a status code 200 OK.

The next endpoint that should be configured is the /search endpoint. All of the following endpoints need to support the HTTP POST method, since Grafana will make POST and not GET calls.

The search endpoint provides a list of targets that can be chosen in queries to the Simple JSON datasource. In the project I was working on the result was dynamic and was looked up in the Postgres database that held the rest of the data. The request is not important here, it just needs to return a list.

The third and last endpoint that is necessary to have a working datasource is the /query endpoint. This endpoint receives request data formatted as JSON that informs what is being queried from the Simple JSON datasource. A full example request can be seen here.

In our example here, we see how the request can be parsed and an example of the kind of data that could be returned. This is likely to be the longest method implemented depending on how much logic is involved querying your source data. In my case I only needed to know the targets provided, and the start and end of the window being queried.

At this point you could say that you are done. This is all you need to get this datasource working and using Python you could pull your source data from anywhere else. If you need more functionality however, you can also implement annotations and adhoc filters.

The /annotations endpoint needs to return a list of annotation objects. Another example of this expected output can be seen on the plugins page. When configuring annotations using the Simple JSON datasource a query can be specified. This query will be provided in the request data sent to the annotations endpoint. The query is a simple string, so however you choose to implement that logic is dependent on parsing you do in your Flask server code.

The next two endpoints work together to support adhoc filters. Adhoc filters can be added in the dashboard’s variables configuration, adhoc filters are one type of variable that can be configured. When adhoc filters are used they will appear in the /query endpoints request object.

Here is an example of the /tag-keys endpoint. This endpoint returns a list of keys that can be used in adhoc filters. The endpoint request is empty.

The second endpoint needed is the /tag-values endpoint. This endpoint will take the chosen key that was selected in the UI from the list provided by the /tag-keys endpoint and base on that input can return different values as shown here.

I want to make a few notes about using Flask. First, Flask provides a global request context that can be seen used in the code here. The request object has a method get_json() which will deserialize the request data and is a little cleaner than using json.loads. Second, when returning JSON data use the jsonify() function from Flask. jsonify() will serialize the data object and set the response mimetype to “application/json” for you. Lastly, when deciding to run your Flask server in production do not use execute Flask directly, instead use a production ready server. In the application I made I had used uWSGI and was able to get better than expected results in terms of requests per second and latency.
Read More »

HTTP to HTTPS redirect for JIRA

While setting up a JIRA Enterprise installation in AWS we wanted a solution for redirecting users from HTTP to HTTPs. We were using an ELB so that we could point a DNS record at a longer lived resource than the application instance and to also make use of the ELB’s ability to encrypt traffic with TLS, but the ELB can not redirect traffic from HTTP to HTTPS.

What alternatives came to mind? Using NGINX or Apache to proxy the request. This adds additional overhead though, NGINX or Apache have to be installed somewhere, should they be installed on the JIRA application server? We would also need to install and configure NGINX or Apache, so more configuration code, another Ansible role… This seems like a lot extra just to redirect users from HTTP to HTTPS.

The JIRA application WAR includes a Tomcat installation and the redirect can be accomplished using only Tomcat by configuring multiple HTTP connectors in conf/server.xml.


…
<Service name="Catalina">

       <Connector
           acceptCount="100"
           connectionTimeout="20000"
           disableUploadTimeout="true"
           enableLookups="false"
           maxHttpHeaderSize="8192"
           maxThreads="150"
           minSpareThreads="25"
           port="8080"
           protocol="HTTP/1.1"
           redirectPort="443"
           useBodyEncodingForURI="true"
       />

       <Connector
           acceptCount="100"
           clientAuth="false"
           disableUploadTimeout="true"
           enableLookups="false"
           maxHttpHeaderSize="8192"
           maxThreads="150"
           minSpareThreads="25"
           port="8443"
           protocol="HTTP/1.1"
           proxyName="jira.mycompany.com"
           proxyPort="443"
           scheme="https"
           secure="true"
           useBodyEncodingForURI="true"
       />
…

A security constraint also needs to be added to the file atlassian-jira/WEB-INF/web.xml. This can be added at the end of the file. As a result, if a request is made to any of these URL patterns on the unsecure connector then a redirect will be returned sending the request to the redirectPort specified above which in this case is 443.


…
<security-constraint>
   <web-resource-collection>
      <web-resource-name>all-except-attachments</web-resource-name>
      <url-pattern>*.jsp</url-pattern>
      <url-pattern>*.jspa</url-pattern>
      <url-pattern>/browse/*</url-pattern>
      <url-pattern>/issues/*</url-pattern>
   </web-resource-collection>
   <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL</transport-guarantee>
   </user-data-constraint>
</security-constraint>

</web-app>

So, when a request is made using HTTP hitting port 80 on the ELB, the ELB will then send the request to the backend Tomcat server port 8080. The Tomcat server will then redirect the request to port 443 which hits the ELB again and sends encrypted traffic to port 8443 on the backend.