In a production kdb+/q system, it will obviously not be feasible for users and process to be connecting and sending queries to each other in a haphazard manner. First, processes will need to have some form of access control, such that users cannot connect and retrieve data they shouldn't have access to, or run dangerous commands (with access to 'system' commands the possibilities are endless. Second, it will be necessary to create a connection framework that handles all aspects of IPC - how one process knows which other processes exist and their details, tracking which processes are connected to which, tracking which users are connected and where, etc.
To achieve all of this, authentication checks can be added to restrict access to processes/data, and message handlers can be added to build up a system-wide connection framework.
Authentication checks prevent unauthorised users from even connecting to a process. There are two main types of authentication checks:
The 'usr-pwd' command line option (- U) which checks the user's username and password against those in a supplied file.
The .z.pw message handler which is invoked on a process when a new user connects to it.
When starting a q process, you can provide it with various command line options. One of these is -u/U.
'-u 1' will start the process but disable file access and system commands.
'-u filename' with 'filename' being replaced by a file of username and passwords, will have the same behaviour as '-u 1' but will also require the username and password to be in the file. The file can be plain text or MD5/SHA hashed.
And finally the '-U filename' option by itself will just perform the username and password checks with one additional preventative measure - \x commands will be prevented, so the user cannot remove the message handler definitions.
Sample target/server process code:
[ec2-user@ip]$ cat passwords.txt
admin:password
adminmd5:0x5f4dcc3b5aa765d61d8327deb882cf99
adminsha:0x5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
[ec2-user@ip]$ q -U passwords.txt -p 5000
q)
Sample user/client process code:
q)h:hopen `::5000
'access
[0] h:hopen `::5000
^
q)h:hopen `::5000:admin:password
q)hclose h
q)h:hopen `::5000:adminmd5:0x5f4dcc3b5aa765d61d8327deb882cf99
q)hclose h
q)h:hopen `::5000:adminsha:0x5baa61e4c9b93f3f0682250b6cf8331b7ee68fd8
Before looking at .z.pw, let's look at message handlers in general.
Message handlers are functions that are invoked on a process when an IPC action takes place, for example when a connection is opened to the process, or a query is sent to the process. These functions can be overwritten and so can be tailored to fit any solution. The message handlers exist in the 'dot z' namespace and the main ones that will be discussed are as follows:
.z.pw is invoked after the user connects, post the -U checks but prior to the .z.po handler is invoked. The timeline of handlers invoked from user process to target process is therefore:
.z.i -> -U -> .z.pw -> .z.po ->.z.pg/s
.z.pw allows for a custom authentication function to be used for user logins. This is where authentication logic can be handled by an external service such as LDAP or more more recently, using auth tokens. It's return value is a 1b or 0b indicating if the user is authorised to connect or not.
Let's try it out:
Defining .z.pw on the server process
[ec2-user@ip]$ q -p 5000
q).z.pw:{[user;pass] $[user in `admin`admin1;1b;0b]}
Connecting from the client process
[ec2-user@ip]$ q
q)h:hopen `::5000:a:a
'access
[0] h:hopen `::5000:a:a
^
q)h:hopen `::5000:admin:a
q)
.z.po is invoked when a user has passed the authentication checks and successfully connects to a process. Usually this function is used to help maintain records of which processes are connected where.
Let's try it out:
First, set up a .z.po override on the server process, which adds some connection info to a global connection table:
q)conns:([handle:`long$()] user:`$(); connTime:`timestamp$())
q)conns
handle| user connTime
------| -------------
q).z.po:{[h] `conns upsert (h;.z.u;.z.P)}
Next, connect to the process from another process:
q)h:hopen `::5000:admin:
q)
Back on the server process, check that the table has been updated:
q)conns
handle| user connTime
------| -----------------------------------
5 | admin 2024.09.25D08:27:56.925961000
These message handlers are invoked when a synchronous or asynchronous message is received from another process. Their input is the received message, the output depends on the handler, for .z.pg the output is what is returned to the other process, for .z.ps the output is discarded. The default behaviour is just 'value' i.e. execute the query and return the result.
First let's define some functions on the server process and set a .z.pg handler such that one of them is not allowed:
q)legalOperation:{[x;y] x+y}
q)illegalOperation:{[]}
q).z.pg:{$[x[0]=`illegalOperation;`$"Error - illegal operation";value x]}
Now let's try to run that on the client process:
q)h(`legalOperation;1;2)
3
q)h(`illegalOperation;1;2)
`Error - illegal operation
If we were to do the same asynchronously (but add a print message at the start):
q).z.ps:{0N!x; $[x[0]=`illegalOperation;`$"Error - illegal operation";value x]}
And run the same commands on the client process:
q)neg[h](`legalOperation;1;2)
q)neg[h](`illegalOperation;1;2)
There is nothing returned for an asynchronous query. However we can prove that the handler was still invoked by the print messages:
q)(`legalOperation;1;2)
(`illegalOperation;1;2)
As you can guess, this is invoked on connection close. Let's update the conn table on disconnect:
q)conns
handle| user connTime
------| -----------------------------------
5 | admin 2024.09.25D08:27:56.925961000
q).z.pc:{[h] delete from `conns where handle=h}
Now close the connection on the client process:
q)hclose h
And check the conns table on the other process:
q)conns
handle| user connTime
------| -------------
This is a variable that contains the 'current' handle. Depending on the context, the 'current' handle is different. If you just run '.z.w' in a q process it will return 0 as the handle to the current process is 0. However if you run '.z.w' inside a function/callback that came from another process, it will return the handle to that process.