IPC

Background

IPC (Interprocess Communication) refers to the communication between kdb+/q processes and other processes. 

The communication occurs over TCP/IP. TCP/IP (Transmission Control Protocol/Internet Protocol) is a set of protocols governing networked devices connected to a network. It is basically a set of standards and rules that two computers can use to communicate with each other. 

For more information about TCP/IP and the basics of kdb IPC - go to the IPC Basics page.

Handles

A handle is an open connection to another process (TCP connection) or a file (file handle) and is represented by an integer value.

The first handle opened will usually be '3' because handles 0-2 are the reserved system handles.

IPC message syntax

To send messages/queries to other processes, you can use two methods:

3"1+1"


3({1+x}; 1)

Sync vs. Async messaging

There are two types of messaging within IPC, synchronous and asynchronous. 

Synchronous, or sync, messages are sent on a handle and the sending process waits until a response is received. To send a sync message, use a positive handle integer.

Asynchronous, or async, messages are sent on a handle and once the message is sent, the sending process simply moves on to the next action - it does not wait for a response or any kind of indication that the message has been received and processed successfully. To send an async message, use a negative handle integer.

It is vitally important to understand the difference between the two, and when one should be used over the other. One danger of using too many synchronous queries is that two processes send sync queries to each other at roughly the same time and become locked, with both waiting on the other to respond. 

An example of something that must be entirely asynchronous is a standard gateway/load balancer framework. The point of the framework is to provide one point of entry for all processes and users and to route their queries to the relevant processes efficiently. If the framework used synchronous queries, once a query had been requested by a process and routed to the final process (e.g. an HDB), all of the processes in the chain would have to wait for the HDB to finish querying - the source process would be busy, the HDB would be busy and so would the gateway/load balancer. But in the async scenario the GW/LB should be free to handle other queries from other processes while the HDB is running the query. 

This does raise the question of how a process can send a message asynchronously but still receive a response. Two of the methods that can be used are async callbacks and deferred sync. In the example above, async callbacks would be used within the gateway processes.

Async callbacks

An async callback is essentially a way of sending an async query to a process, along with an instruction to run a particular function on the source process when the query is complete. Basically saying, "run this query, and once it is finished, send a message back to me to run this function with the result".

How can this help us? Well let's flesh out our previous example with a diagram:


In this case a gateway process receives all user queries and directs them to the appropriate service, while also collecting the results and sending them back to the user. As previously mentioned, this must be fully asynchronous otherwise once a user sent a query, the entire flow from user>gw>service would be busy until that one query was completed. In this application, multiple users must be able to send queries without blocking the entire system, so asynchronous communication must be used.

We can use an async callback when requesting data from the RDBs and HDBs so the results can be received.

Assume we have our HDB process with some data in the trade table. On that process we can define a function for receiving a query, counting the number of records in the trade table for the date provided, and invoking the callback function provided (note when invoking the callback function we are still using an async message):

q)basicQuery:{[id;dt;callback] res:count select from trade where date=dt; neg[.z.w](callback;id;res)}

On the gateway process, we can define a callback function for receiving results, which puts them into a 'res' table:

q)returnRes:{[id;r] `res upsert (id;r)}

And then on the gateway we can send an async query to the HDB, with inputs of: query we want to run, inputs for that query, callback function to use when result is ready:

q)neg[h](`basicQuery;1;2021.11.28;`returnRes)

Running this on the GW returns no result as the message was sent asynchronously. However if we check the res table:

q)res

id| res

--| ---

1 | 53

We can see that the HDB has indeed ran the query and sent the results back via the callback function, which updated our res table. Note how every step in this process was asynchronous, this is highly desirable for gateway frameworks. 

Further functions can be implemented to return these results to the user. However, if we extend this same async callback logic to the end users, the application becomes quite complex for them to run. From an end user perspective, they don't want to be keeping track of request IDs, where results are stored, etc. In fact, the average user wants to be able to send a query, have their process wait for the response, then return the response. But this is synchronous messaging behaviour and we have already established that we must use asynchronous messaging. 

Is there a way to achieve synchronous message behaviour within asynchronous messaging?

Deferred sync/response

To achieve this we can use deferred synchronous queries.

In the above framework, the gateway process manages all queries to and from the users, decides which service (RDB, HDB or both) that query should be sent to, manages the response from the service and returns the results back to the user.

The communication between the user and the gateway is asynchronous and uses a TCP connection handle. 

If the end user was to send an async message to the gateway with an async callback to return the results directly back across the same handle, the process would not handle the results message correctly as it was sent asynchronously and the process was not expecting it. 

It throws a type error in this instance:

q)neg[h]({neg[.z.w] x};6)

q)'type

The error is thrown on the target process as it is not expecting any data to be sent across the handle.

With a deferred response, immediately after sending an async message on a handle, it is possible to tell the process to expect a response on that handle, and to block the handle until that response has been received (much like a synchronous message). This is achieved using the block handle command:

q)h[]

On the source process then, send an asynchronous message with an async callback to return the results, and immediately afterwards, block the handle, and assign it to a variable:

q)neg[h]({neg[.z.w] x};6); a:h[]

q)a

6

This functionality is used so frequently that in V3.6, an internal function was created.

-30!

With the -30! function, clients can use synchronous commands. In the target process, adding -30!(::) to the callback puts the query into a 'suspended' state, blocking the handle but allowing the target process to move on to other queries. Once the result is ready to be returned, -30! can be invoked again to send the result down the blocked handle. For example:

Handle is blocked, process is now waiting until response is received:

q)//client process

q)h({neg[.z.w] show `messageReceived; -30!(::)};1+1)


Message has been received on the target process, but process is not busy:

q)//target process

q)`messageReceived

q)

The response is sent back to the client process using -30! with inputs (handle; isError; msg):

q)//target process

q)`messageReceived

q)

q)-30!(6i;0b;2)

Client process has received result, is no longer busy:

q)//client process

q)h({neg[.z.w] show x; -30!(::)};1+1)

2

q)