I was working on a project where I was creating time series data based on data stored in Postgres. The transformations needed of the data were very complex in pure SQL and I had implemented a better performing implementation in Python. The problem was that I was able to get the result I wanted in Python, but was unsure how I could visualize that data in Grafana.
I figured there must be a way to pull generic data from any source in Grafana and a quick search on Google lead me to this blog post http://www.oznetnerd.com/writing-a-grafana-backend-using-the-simple-json-datasource-flask/. I already had a way to get the result using Python so it seemed serendipitous that this post described how to use Flask and the Grafana Simple JSON datasource plugin to make it all come together. The writings I found about using the Simple JSON datasource seemed to be either too in depth or too light, so I decided I would write a basic overview of its functionality and provide an interactive tutorial to drill points home.
The code I have created for this tutorial can be found here https://github.com/Jonnymcc/grafana-simplejson-datasource-example.
There are three endpoints that need to be created for the datasource to be useful and three others for added functionality. The three that are required are…
//search/query
And the three that are optional are…
/annotations/tag-keys/tag-values
The repo I created has instructions for getting Grafana running locally, and getting the Flask server running and configured in Grafana.
The first endpoint that needs to be provided is /. This endpoint is used when configuring a datasource in Grafana and clicking on the “Save & Test” button. All it needs to do is return a status code 200 OK.
| @app.route('/') | |
| def health_check(): | |
| return 'This datasource is healthy.' |
The next endpoint that should be configured is the /search endpoint. All of the following endpoints need to support the HTTP POST method, since Grafana will make POST and not GET calls.
| @app.route('/search', methods=['POST']) | |
| def search(): | |
| return jsonify(['my_series', 'another_series']) |
The search endpoint provides a list of targets that can be chosen in queries to the Simple JSON datasource. In the project I was working on the result was dynamic and was looked up in the Postgres database that held the rest of the data. The request is not important here, it just needs to return a list.
The third and last endpoint that is necessary to have a working datasource is the /query endpoint. This endpoint receives request data formatted as JSON that informs what is being queried from the Simple JSON datasource. A full example request can be seen here.
| @app.route('/query', methods=['POST']) | |
| def query(): | |
| req = request.get_json() | |
| data = [ | |
| { | |
| "target": req['targets'][0]['target'], | |
| "datapoints": [ | |
| [861, convert_to_time_ms(req['range']['from'])], | |
| [767, convert_to_time_ms(req['range']['to'])] | |
| ] | |
| } | |
| ] | |
| return jsonify(data) |
In our example here, we see how the request can be parsed and an example of the kind of data that could be returned. This is likely to be the longest method implemented depending on how much logic is involved querying your source data. In my case I only needed to know the targets provided, and the start and end of the window being queried.
At this point you could say that you are done. This is all you need to get this datasource working and using Python you could pull your source data from anywhere else. If you need more functionality however, you can also implement annotations and adhoc filters.
The /annotations endpoint needs to return a list of annotation objects. Another example of this expected output can be seen on the plugins page. When configuring annotations using the Simple JSON datasource a query can be specified. This query will be provided in the request data sent to the annotations endpoint. The query is a simple string, so however you choose to implement that logic is dependent on parsing you do in your Flask server code.
| @app.route('/annotations', methods=['POST']) | |
| def annotations(): | |
| req = request.get_json() | |
| data = [ | |
| { | |
| "annotation": 'This is the annotation', | |
| "time": (convert_to_time_ms(req['range']['from']) + | |
| convert_to_time_ms(req['range']['to'])) / 2, | |
| "title": 'Deployment notes', | |
| "tags": ['tag1', 'tag2'], | |
| "text": 'Hm, something went wrong...' | |
| } | |
| ] | |
| return jsonify(data) |
The next two endpoints work together to support adhoc filters. Adhoc filters can be added in the dashboard’s variables configuration, adhoc filters are one type of variable that can be configured. When adhoc filters are used they will appear in the /query endpoints request object.
Here is an example of the /tag-keys endpoint. This endpoint returns a list of keys that can be used in adhoc filters. The endpoint request is empty.
| @app.route('/tag-keys', methods=['POST']) | |
| def tag_keys(): | |
| data = [ | |
| {"type": "string", "text": "City"}, | |
| {"type": "string", "text": "Country"} | |
| ] | |
| return jsonify(data) |
The second endpoint needed is the /tag-values endpoint. This endpoint will take the chosen key that was selected in the UI from the list provided by the /tag-keys endpoint and base on that input can return different values as shown here.
| @app.route('/tag-values', methods=['POST']) | |
| def tag_values(): | |
| req = request.get_json() | |
| if req['key'] == 'City': | |
| return jsonify([ | |
| {'text': 'Tokyo'}, | |
| {'text': 'São Paulo'}, | |
| {'text': 'Jakarta'} | |
| ]) | |
| elif req['key'] == 'Country': | |
| return jsonify([ | |
| {'text': 'China'}, | |
| {'text': 'India'}, | |
| {'text': 'United States'} | |
| ]) |
I want to make a few notes about using Flask. First, Flask provides a global request context that can be seen used in the code here. The request object has a method get_json() which will deserialize the request data and is a little cleaner than using json.loads. Second, when returning JSON data use the jsonify() function from Flask. jsonify() will serialize the data object and set the response mimetype to “application/json” for you. Lastly, when deciding to run your Flask server in production do not use execute Flask directly, instead use a production ready server. In the application I made I had used uWSGI and was able to get better than expected results in terms of requests per second and latency.
Read More »
Jon's Blog 