tick.q

Note: This page goes through tick.q line-by-line. For a higher level view of the logic involved in the tickerplant - see Tickerplant.

What is it?

tick.q is the main q script used by the tickerplant. It contains (or loads) all of the functions needed to run a vanilla tickerplant. 

As with a lot of kdb+/q code, when we first look at tick.q, it seems quite daunting: 

However, once we break it down (and add some formatting), it becomes a lot clearer. Let's look at it section by section:

Changelog, version, etc.

This section contains the following:

1. Example start command for the tickerplant

2. Changelog for the tick.q script

3. Startup banner giving version and date of the tick.q script

None of these are necessary for the script to run - in fact you can delete these from your tick.q for easier readability.

Script Initialisation

In this section, the first line is commented out - it is merely providing example syntax for how to run a q session and load tick.q. 

As you can see, when you run a q session with the tick.q script, the expected (although optional) inputs are SRC and DST. SRC refers to the name of the schema file, and DST refers to the location of the tickerplant 'log' file or 'journal' file.

The first action performed by the TP is then on the next line. 

Action 1 - load schema file

system"l tick/",(src:first .z.x,enlist"sym"),".q"

The first action is to load in the schema file using the "l" system command. It expects the file to exist in the 'tick' directory. The script adds the '.q' to the filename.

If it doesn't find the file, it will use a default value of 'sym'. This is achieved with this line of code:

(src:first .z.x,enlist"sym")

What this is saying is: take the list of command line inputs (.z.x), add "sym" to the end, and take the first value as the schema file. In other words: if there are any command line inputs, assign the first to the variable 'src' and if there are none, assign 'src' to a default value of "sym".

Action 2 - set port

if[not system"p";system"p 5010"]

The next action is to determine if a port was provided to the script by checking what the current port is. If there is none, it sets a default port of 5001.

Action 3 - load utilities script

\l tick/u.q

Load the u.q utilities script.

Action 4 - load .u namespace

\d .u

Move into the .u namespace. Henceforth, any variables created will automatically be created in the '.u' namespace. This article will refer to the variables using their full '.u' name.

Define Functions

The next 4 lines of code all define functions. Note that each function is contained on one line. This can be difficult to read - let's format the code for easier readability as we go. 

Note: although .u.ld is defined before .u.tick in the code, we will cover .u.tick first for ease of explanation and understanding.

Action 5 - define the '.u.tick' function

tick:{

  init[];

  if[not min(`time`sym~2#key flip value@)each t;

    '`timesym

  ];

  @[;`sym;`g#]each t;

  d::.z.D;

  if[l::count y;

    L::`$":",y,"/",x,10#".";

    l::ld d

  ]

};

The '.u.tick' function performs initialisation actions for the tickerplant. The function is called only in the main tick.q script with two inputs x and y: schema file name and logfile location.

init[];

First, the .u.init function is ran (see u.q).

  if[not min(`time`sym~2#key flip value@)each t;

    '`timesym

  ];

Next, each table in the schema file is checked to ensure that the first two columns are exactly `time and `sym ("2#key flip value@" will return the first two columns of the table).

If they aren't the function will fail with a `timesym error.

@[;`sym;`g#]each t;

Next, the 'grouped' attribute is applied to the sym column on each table (note the tables are empty at this point). 

See the Attributes page for more info.

d::.z.D;

The variable '.u.d' is assigned to today's date (in local time).

  if[l::count y;

    L::`$":",y,"/",x,10#".";

    l::ld d

  ]

If the logfile location (input 'y') was provided when running the process, assign that to the variable .u.l, and then:

• Assign to the variable .u.L a symbol  containing the logfile location (input 'y') followed by a '/', followed by the schema file name (input 'x') followed by ten dots ('.') e.g. `:logfile/sym..........

  • This may seem odd, and it is. It is just a placeholder assignment for now - it will be overwritten later.

  • The logfile name/location is constructed as follows: (logfile location)/(schema filename).(today's date)

  • At this point we have created the first two parts, and we have simply added a placeholder for the final part

• Assign to the variable .u.l the result of the function .u.ld when ran with input of .u.d (currently assigned to today's date)

  • The result returned will be the handle to the TP logfile

Action 6 - define the '.u.ld' function

ld:{

  if[not type key L::`$(-10_string L),string x;

    .[L;();:;()]

  ];

  i::j::-11!(-2;L);

  if[0<=type i;

    -2 (string L)," is a corrupt log. Truncate to length ",(string last i)," and restart";

    exit 1

  ];

  hopen L

};

The 'ld' function is the logging function, responsible for creating and initialising the logfile. The function is initially called in .u.tick with the input of .u.d which is today's date. Subsequently it will be called at EOD with the same input of .u.d which then contains the new day's date. 

Remember before calling this function for the first time in .u.tick, .u.L was given a temporary assignment of logfile location, schema file name and ten dots, e.g. `:logfile/sym..........

  if[not type key L::`$(-10_string L),string x;

    .[L;();:;()]

  ];

Firstly, assign to the variable .u.L a symbol containing a concatenation of the previous filename minus the last 10 characters, joined with the input 'x' which is today's date. This creates a full logfile name/location of, for example, `:logfile/sym2020.01.01. 

It is now obvious why the ten dots were added earlier - it means this exact same code can be used both when initialising and at end of day.

The keyword key is used with this new value of .u.L. When used with a filepath, key will return either the filepath or an empty list if the filepath doesn't exist. Performing the type keyword on an empty list will return the list type 0h. Performing the not keyword on a value of '0h' will return true/1b. Essentially, the 'if' statement is true if the file does not exist.

If the file does not exist, create it as an empty file. This may seem like strange syntax but it is the same as saying "L set ()". However, if we wanted to use 'set' we would have to specify the exact location on disk e.g. "`:file set ()". In this instance we only have a variable called .u.L. So, we take advantage of the fact that the 'amend' operator can update files on disk; essentially the syntax is saying, update the entire domain of L to be ().

i::j::-11!(-2;L);

Perform a -11! on the logfile with an input of -2 (see section on Tickerplant logfiles). This will give us the number of valid update messages in the logfile. If the logfile is corrupt, it will return the number of valid messages and the position of the last good message. Both .u.i and .u.j are assigned to this value.

.u.i contains the number of messages in the logfile and .u.j contains the total message count. On first run, the logfile will be empty so both of these values will be 0. If the logfile exists, both of these values will contain the number of messages in the logfile. So, at startup we would indeed expect these to be the same.

  if[0<=type i;

    -2 (string L)," is a corrupt log. Truncate to length ",(string last i)," and restart";

    exit 1

  ];

This if statement is for the scenario where the logfile is corrupt. As noted, if the logfile is corrupt, a -11! with -2 input will return two values (the number of valid messages and the position of the last good message) - as such, it will be a list and its type will be a positive number. 

If this is the case, log an error to stderr and exit with exit code 1.

hopen L

Open a handle to .u.L. The return value of 'hopen' is the handle number so the return value of .u.ld is the handle to the active tickerplant logfile.

Action 7 - define the '.u.endofday' function

endofday:{

  end d;

  d+:1;

  if[l;

    hclose l;

    l::0(`.u.ld;d)

  ]

};

The endofday function is, not surprisingly, ran at the end of the day and performs basic rollover actions such as telling the downstream subscribers that the day has ended, making a new logfile for the new day, etc. The function has no inputs.

  end d;

Run the .u.end function with input of .u.d i.e. the old date. See u.q.

  d+:1;

Update the .u.d variable to have the new date.

  if[l;

    hclose l;

    l::0(`.u.ld;d)

  ]

If an open handle to the logfile exists in .u.l:

• Close that handle 

• Assign .u.l to a new handle created by running .u.ld with an input of the new day's date on the current process

  •  Handle '0' represents the current process - in this case the tickerplant

Action 8 - define the '.u.ts' function

ts:{

  if[d<x;

    if[d<x-1;

      system"t 0";

      '"more than one day?"

    ];

    endofday[]

  ]

};

The u.ts function is used to check if it is time to run end of day, and if it is, to do so. The function has one input of date.

If the .u.d variable in the tickerplant (representing today's date) is less than the date provided in the input then:

• Check if .u.d is more than 1 day less than the date provided. If so:

  • Set the system timer to 0 i.e. turn it off 

  • Fail with a break error 

• If not:

  • Run the endofday function 

Batch vs Real-time Mode

The tickerplant can run in one of two modes: batch and real-time. For more information on this and the tickerplant in general, please see the tickerplant page. Essentially, in batch mode any updates received are kept for a predefined period of time before being published on a timer, whereas in real-time mode the updates are published as they are received.

Action 9 - define '.z.ts' and '.u.upd' functions - batch mode

If the system timer has been set, the tickerplant is in batch mode. 

if[system"t";

  .z.ts:{

    pub'[t;value each t];

    @[`.;t;@[;`sym;`g#]0#];

    i::j;

    ts .z.D

  };

  upd:{[t;x]

    if[not -16=type first first x;

      if[d<"d"$a:.z.P;

        .z.ts[]

      ];

      a:"n"$a;

      x:$[0>type first x;

        a,x;

        (enlist(count first x)#a),x

      ]

    ];

    t insert x;

    if[l;

      l enlist (`upd;t;x);

      j+:1

    ];

  }

];

Define .z.ts:

pub'[t;value each t];

For every table in t, execute the .u.pub (see u.q) function with two inputs: the table name and the 'value' of the table (i.e. the contents of the table).

@[`.;t;@[;`sym;`g#]0#];

Reapply the grouped attribute to the sym column of each table. Let's break this down:

breakdown

    i::j;

Assign the variable .u.i to the value in the variable .u.j (we have just published the latest batch of data, so if there is a new subscriber, they will have to replay the entire tickerplant logfile right up to the latest message).

    ts .z.D

Run the .u.ts function with today's (local) date.

Define .u.upd. .u.upd is the function called on the tickerplant by publishing processes such as FHs or RTEs. It takes two inputs - t for table name, and x for data (expected to be lists of column data with no column headers).

    if[not -16=type first first x;

      if[d<"d"$a:.z.P;

        .z.ts[]

      ];

      a:"n"$a;

      x:$[0>type first x;

        a,x;

        (enlist(count first x)#a),x

      ]

    ];

Check if the first item in the list (i.e. the first column of data) is of type 16 i.e. timespan. If not:

• Check if the current date is greater than the date in the .u.d variable

  • If so, run .z.ts

• Assign the current time in timespan format to the variable 'a'

• Check if the first item in the list is of atomic or list type. If atomic type:

  • Add variable 'a' to the start of the data (i.e. add a timespan as the first column)

• If not:

  • Add multiple variable 'a's to the start of the list (matching the count of the list) 

    t insert x;

Insert x into t

if[l;

      l enlist (`upd;t;x);

      j+:1

    ];

If .u.l exists i.e. if the process currently has an open handle to a logfile:

• Write the upd message to the logfile

• Increment .u.j, which contains the count of total messages processed, by 1.

Action 10 - define '.z.ts' and '.u.upd' functions - real-time mode

If the system timer has not been set, the tickerplant is in real-time mode. The first action is to set the system timer to run every second.

if[not system"t";system"t 1000";

  .z.ts:{ts .z.D};

  upd:{[t;x]

    ts"d"$a:.z.P;

    if[not -16=type first first x;

      a:"n"$a;

      x:$[0>type first x;

        a,x;

        (enlist(count first x)#a),x

      ]

    ];

    f:key flip value t;

    pub[t;

      $[0>type first x;

        enlist f!x;

        flip f!x

      ]

    ];

    if[l;

      l enlist (`upd;t;x);

      i+:1

    ];

  }

];

Define .z.ts:

  .z.ts:{ts .z.D};

Run .u.ts with input of today (local time).

Define .u.upd. .u.upd is the function called on the tickerplant by publishing processes such as FHs or RTEs. It takes two inputs - t for table name, and x for data (expected to be lists of column data with no column headers).

ts"d"$a:.z.P;

Assign the variable 'a' to the current local dateTime, cast it to a date and run .u.ts with that date as the input (today's local date).

    if[not -16=type first first x;

      a:"n"$a;

      x:$[0>type first x;

        a,x;

        (enlist(count first x)#a),x

      ]

    ];

Check if the first item in the list (i.e. the first column of data) is of type 16 i.e. timespan. If not:

• Update the variable 'a' by casting its current value (local dateTime) to a timespan

• Check if the first item in the list is of atomic or list type. If atomic type:

  • Add variable 'a' to the start of the data (i.e. add a timespan as the first column)

• If not:

  • Add multiple variable 'a's to the start of the list (matching the count of the list) 

f:key flip value t;

Get the column names of the table t

    pub[t;

      $[0>type first x;

        enlist f!x;

        flip f!x

      ]

    ];

Run .u.pub (see u.q) with arguments of table name 't' and the data in 'x' converted to a table using the column headers currently in 'f'.

The conversion is performed by first making a dictionary of column headers and column data, and then to turn it into a table either enlist the dictionary or flip it depending on whether the data in 'x' contains one 'row' of data or multiple 'rows' of data (although you will rarely hear it referred to like this i.e. in terms of number of rows).

    if[l;

      l enlist (`upd;t;x);

      i+:1

    ];

If .u.l exists i.e. if the process currently has an open handle to a logfile:

• Write the upd message to the logfile

• Increment .u.i, which contains the count of messages in the logfile, by 1.

Startup

Action 11 - start tickerplant

\d .

Move out of the .u namespace back to the global namespace.

.u.tick[src;.z.x 1];

Run the .u.tick function with two inputs: src (schema filename) and the second input provided when running the script (logfile location).

Comments