u.q

Note: This page goes through u.q line-by-line. For a higher level view of the logic involved in the tickerplant - see Tickerplant.

What is it?

u.q is the utilities script used by the tickerplant. It contains definitions for some of the most common functions, including the publish and end of day functions.

Much like we did with tick.q - let's break it down.

Changelog and initialisation

The first three commented lines are merely a changelog - these can be deleted from your version of u.q if you want

Action 1 - load .u namespace

\d .u

Move into the .u namespace. Henceforth, any variables created will automatically be created in the '.u' namespace. This article will refer to the variables using their full '.u' name.

Define Functions

The rest of the script contains a number of function definitions. Note that each function is contained on one line. This can be difficult to read - let's format the code for easier readability as we go. 

Action 2 - define .u.init

init:{

  w::t!(count t::tables`.)#()

 }

The .u.init function initialises the variables .u.t and .u.w in the tickerplant, these variables represent the list of table names currently recognised by the tickerplant, and a dictionary mapping each table and its subscribers. It does not take any arguments.

Although just one line of code, there's a lot going on so let's break it down:

Using the keyword tables with an input of `. will return all the tables in the root namespace - assign this to variable .u.t. Perform a count on this variable (i.e. a count of the number of tables in the root namespace) and take that number of empty lists by using #(). 

 Create a dictionary with tablenames in .u.t as the key and the empty lists created earlier as the value and assign this to variable .u.w.

So after running .u.init, we have variable .u.t containing all tables in the root namespace and variable .u.w which is a dictionary of tablenames and (at the moment) empty lists.

Action 3 - define .u.del

del:{

  w[x]_:w[x;;0]?y

 };

The .u.del function is ran when a subscriber unsubscribes from a topic/table - .u.del is used to 'delete' the subscriber from the .u.w dictionary of tables and subscribers. It takes two arguments which is the table name and handle of the subscriber.

So let's break it down, first by remembering that .u.w will contain a dictionary with table names as keys, and, if there are active subscribers, the values will be lists of (handle; topics) for each subscriber.

So, the first thing we want to do is get the list of handles subscribed to the table provided in argument x, so we have to index in two levels of that dictionary value and get the item in position 0 of each list i.e. the list of handles subscribed to that table. 

Next, use the 'find' overload of '?' to find out if the handle provided in argument y exists in the list of subscribed handles. If it does, we will use that index location in the next step. If it doesn't we will be given an index location one greater than the count of the list.

As such, when we perform the next action, the dictionary value for the table (i.e. the list of handles and subscribers) will have that index removed. If it exists it is removed. If it doesn't exist-  nothing happens.

Action 4 - define .z.pc

.z.pc:{

  del[;x]each t

 };

The .z.pc function is called when a connection is closed (see IPC). It has one input which is the handle of the closed connection.

So, this is pretty simple then, run the .u.del function for the handle that has just closed its connection, and run it for each table to ensure all table subscriptions for that handle are properly closed.

Action 5 - define .u.sel

sel:{

  $[`~y;

    x;

    select from x where sym in y

  ]

 }

.u.sel is used in the .u.pub function defined below and is as simple as it gets. Given a table 'x' and a list of syms 'y':

• If 'y' is just a backtick

  • This denotes all syms, return the entire table

• Else, return the table 'x' filtered on the list of syms in 'y'

Action 6 - define .u.pub

pub:{[t;x]

  {[t;x;w]

    if[count x:sel[x]w 1;

      (neg first w)(`upd;t;x)

    ]

  }[t;x]each w t

 }

.u.pub is used by the tickerplant to publish updates to subscribers. It takes two inputs - table name and table data, 't' and 'x'. 

  {[t;x;w]

  ...

  }[t;x]each w t

Inside this function is a lambda - another function! Indexing into .u.w with 't' (table name) will give us a list of two item lists containing all handles currently subscribed to that table and the syms to which they are subscribed. 

For each of those items in the list, run this function with 't' and 'x' inputs from .u.pub and the new input of 'w' containing (handle;subscription). As we are iterating across each item, the second function will run one time for each item, with the 'w' input changing each time. That function performs the following:

    if[count x:sel[x]w 1;

      (neg first w)(`upd;t;x)

Index into 'w' at location '1', this returns the subscribed syms. Run .u.sel with input of 'x' and those subscribed syms. This will filter the table in 'x' on the syms subscribed to by this subscriber. 

If, after performing this filtering, there is data returned, then publish this data to that subscriber asynchronously (see IPC) in the format of (`upd; tableName; data). 'upd' is the standard function called on all subscribers to receive data. For an example, see r.q.

Action 7 - define .u.add

add:{

  $[(count w x)>i:w[x;;0]?.z.w;

    .[`.u.w;(x;i;1);union;y];

    w[x],:enlist(.z.w;y)

  ];

  (x;

    $[99=type v:value x;

      sel[v]y;

      @[0#v;`sym;`g#]

    ]

  )

 }

.u.add is the function called by .u.sub when registering a new subscriber. It takes two inputs of table name and the list of symbols to subscribe to on that table (or '`' for all symbols).

  $[(count w x)>i:w[x;;0]?.z.w;

    .[`.u.w;(x;i;1);union;y];

    w[x],:enlist(.z.w;y)

  ];

This section is essentially checking if the subscriber is already subscribed to this table, and taking the appropriate action of extending its existing subscription or making a new one.

So let's break it down, first by remembering that .u.w will contain a dictionary with table names as keys, and, if there are active subscribers, the values will be lists of (handle; topics) for each subscriber.

So, the first thing we want to do is get the list of handles subscribed to the table provided in argument x, so we have to index in two levels of that dictionary value and get the item in position 0 of each list i.e. the list of handles subscribed to that table. 

Next, use the 'find' overload of '?' to find out if the handle provided in .z.w exists in the list of subscribed handles. If it does, it will return an index location, if it doesn't it will return a number greater than the count of .u.w, either away assign this to variable 'i'. 

Therefore, if the count of subscribers to table 'x' in .u.w is greater than variable 'i', then it can be determined that there is an existing subscription. Otherwise, this is a new subscription.

If there is an existing subscription:

• Using apply on .u.w, at position (x;i;1) i.e. index into .u.w for table 'x', at position 'i' and then at location 1 (this represents the list of symbols subscribed to by the subscriber in position 'i' for table 'x'), join the new list of symbols with the existing list using union.

If it is a new subscription:

• Index into .u.w for table 'x', and join to the list of (handle; symList) pairs a new pair of (.z.w; y), remembering that .z.w is the handle to the subscriber in question and 'y' is the list of symbols they want to subscribe to.

  (x;

    $[99=type v:value x;

      sel[v]y;

      @[0#v;`sym;`g#]

    ]

  )

This section deals with what is going to be returned when the .u.add function is called. Two items are going to be returned: 'x' which is the table name, with the second output depending on the type of the table:

First a check on the table is performed to determine if it is a dictionary type or not. When would a table be a dictionary in this instance? When it is a keyed table. However, it is not common to find keyed tables in the tickerplant.

If the table is a dictionary i.e. a keyed table:

• Perform .u.sel to select from the keyed table the required syms

If the table is a standard table:

• Take an empty version of the table and re-apply the grouped attribute to the sym column

Action 8 - define .u.sub

sub:{

  if[x~`;

    :sub[;y]each t

  ];

  if[not x in t;

    'x

  ];

  del[x].z.w;

  add[x;y]

 }

.u.sub is the function called when a subscriber is subscribing to the tickerplant. It will take two inputs, 'x' which is table name and 'y' which is the list of syms in that table that the subscriber wants to subscribe to. .z.w will be used in the function to get the handle of the subscriber process.

  if[x~`;

    :sub[;y]each t

  ];

First, if the subscriber has used backtick instead of a table name, they want to subscribe to all tables. If that is the case, run this exact same function again for every single table in .u.t. So, for every table in .u.t, run .u.sub with 'x' as that table name and 'y' as the original sym list. 

  if[not x in t;

    'x

  ];

Quick check to ensure that input 'x' actually exists in the list of tables .u.t, if not return the table name as an error.

  del[x].z.w;

  add[x;y]

Using .u.del and .u.add - delete the existing subscription (currently in .u.w) and add the new one. 

One thing you might notice here is that as we are deleting the old subscription before adding the new one, the section in .u.add that checks if the subscription already exists is kind of useless. 

Action 9 - define .u.end

end:{

  (neg union/[w[;;0]])@\:(`.u.end;x)

 }

.u.end is the end of day function ran by the tickerplant. It takes one input of today's date. 

This function can be looked at in the following way:

(list of handles)@\:(function and input)  i.e. call the function and its input over each handle.

In this case, the 'list of handles' is determined by indexing into .u.w to get just the handles from each subscription, joining them together and performing the 'neg' operation so that the messages will be sent asynchronously. 

The 'function and input' is to call .u.end with input 'x' which is today's date.

In plain English: get the list of subscribing handles and tell each of them to run (their version of) .u.end with input 'x'.

Comments