Python-kdb communication using WebSockets

Photo by Sarah Dorweiler on Unsplash

I consider python and kdb are a union made in heaven. Kdb/q being a terse and high performant perfectly complements python’s readable but slow code. Pyq and qpython are some of the python packages we can use to solve the interaction between programs written in these two languages. While they offer a slight speed advantage, I felt the WebSockets method to be easier to write and maintain. By the end of this article, you will be having some idea about incorporating WebSockets into your next python-kdb application.

I’ll be using Python 3.8 and kdb+ 3.6 (32 bit) in this article. If you are using other versions, you may need to make a few minor changes in the code.

Prerequisites

  • Able to program in python and kdb+/q.
  • Familiarity with WebSockets and asynchronous programming (async and await in python).

Setting up

  • Create a python virtual environment using the tools you like best.
  • Install WebSockets package using the following command.

pip3 install websockets

How it works?

Two cases where this approach is fitting are when your:

  • Python app is a client that needs to query the kdb program each time it requires data.
  • Python app is a subscriber.

So, let’s implement each of the scenarios.

Server

Before completing the client and subscriber codes, let’s review this server code written in q.

  • Line 2 (.z.ws) handles incoming socket queries/ subscriptions by evaluating the string using the value function.
  • To delete connection handles of subscribers when they close, we use the .z.wc.
  • The function gendata creates the dummy data we use here. It takes symbol(s) as input(s) and generates a jsonified table. We use this function for both the client and subscriber.
  • ClientResponse sends a table for a single symbol immediately.
  • Sub function adds a row containing the connection handle, subscribed function, and its parameters to the subs table.
  • To publish data to the subscribers, the pub executes each row of the subs table and sends the data to the corresponding subscriber.
  • Lastly, .z.ts runs the pub function periodically based on the timer parameter.

Begin the server using the following command.

q server.q -p 5511 -t 10000

To learn more about this remarkable built-in WebSocket feature of kdb, you can check out this whitepaper by Chris Scott & Michael Gracey.

Client

So, we use this design when the data is required occasionally. For example, when the user asks for it exclusively. Now that being said, let’s dissect the code:

  • The package asyncio is the asynchronous programming library for python.
  • The function query accepts a string of a symbol. The first line inside the function initializes the connection using uri, which is the server’s address. Later, it sends a query string to the server and waits for the response. Once you’ve received it, you can parse it according to your needs (like converting a JSON string).
  • The asyncio eventloop (older versions) is required to execute the async functions. In case you needed to use an async function in a blocking code, you can apply the eventloop inside that particular function.

After running the program, you should find an output similar to this.

[{“time”:”19:11:14.707",”sym”:”A”,”val”:9.155428},{“time”:”19:14:14.707",”sym”:”A”,”val”:8.025247},{“time”:”19:21:14.707",”sym”:”A”,”val”:9.061393},{“time”:”19:27:14.707",”sym”:”A”,”val”:0.961324},{“time”:”19:52:14.707",”sym”:”A”,”val”:3.548481},{“time”:”20:15:14.707",”sym”:”A”,”val”:2.502691},{“time”:”20:20:14.707",”sym”:”A”,”val”:0.3990707},{“time”:”20:34:14.707",”sym”:”A”,”val”:2.683669},{“time”:”20:41:14.707",”sym”:”A”,”val”:8.874349}]

If you are new to asynchronous programming in python, you can start with this video.

Subscriber

As you have observed, the code is very similar to the client. The only differences here are:

  • We subscribe instead of requesting data each time using the query string.
  • The receive function is awaiting inside a while loop for the next update.

As noticed, with very few changes, you were able to achieve fetching real-time data without polling. Later, you can process the obtained data and link any change to an attribute of another object. As for the output of this program, JSON data gets printed on the console every 10 seconds.

Epilogue

I came across this WebSockets method when I was working on my discord bot. I studied out other alternatives such as qpython and pyq to join my bot with the kdb backend. Pyq was my first choice because it being the fastest among the three. However, I was unable to apply it alongside my pre-existing web-app due to some package issues. Although qpython worked well initially when it came to feeding data, it became increasingly difficult to write code for data retrieval as the bot is almost entirely asynchronous. Now, this where the WebSockets fit right in. If you want a lightweight, quick to implement design for your asynchronous python web client to communicate with the kdb backend, then WebSockets is the way to go!

I love building things for myself!

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store