Memory Management

What is memory?

All processes use memory, especially kdb/q processes. When we talk about memory in kdb/q, we are usually referring to Random Access Memory (RAM). This is where data is stored on a computer when it is needed for immediate use by the CPU. Any data stored on disk is pulled into RAM before it is accessed by the CPU. And any data in RAM is pulled into the CPU cache memory before processing the data/instructions. 

Compared to 'cheap and deep' disk storage, RAM is relatively expensive and as such is a limited resource on any computer or server.

As kdb is an "in-memory" database,  kdb/q processes have the potential to use large amounts of memory. It is vital that we manage this effectively.

How is memory allocated to kdb processes?

Each Unix process owns a specific memory region called heap, which is used to satisfy the process’s dynamic memory requests. Kdb processes manage their own thread-local heap. As the heap size is dynamic, each kdb process can increase or decrease its own heap size as its memory requirements change. 

The OS will determine the initial heap size of the process on startup - note that the heap size will never go below this amount. 

Buddy memory allocation

Kdb processes make use of buddy memory allocation. This means that memory is allocated for objects in powers of 2. So requests will be for: 1, 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048, 4096, 8192, 16384, 32768, 65536, 131072, 262144, 524288, 1048576... etc. bytes rather than the exact amount of memory needed. This can cause confusion when it comes to tracking memory usage - why has my memory usage increased by 262144 bytes when the object I added to memory was only 140000 bytes? Because the kdb process requested the next 'step up' in powers of 2.

For example (see the Memory related utilities section below to understand the utilities used in this example):

q).Q.w[]

used| 357168

q)-22!a:10#1f

94

q).Q.w[]

used| 357296

q)357296-357168

128

q)2 xlog 128

7f

Adding a list of 10 floats with a size of 94 bytes actually added 128 bytes to memory. 128 is 2^7.

If you are wondering why 10 floats of size 8 does not result in 8*10=80 bytes used, please see here. 

Reference counting

In order for objects in memory to be accessed, a reference is created which points to that object. kdb utilises reference counting, which essentially means that when a new reference is created for existing data, it points to the same location as the previous reference for that data rather than adding new data to memory. 

For example, let's say we define a variable called firstFloat:

firstFloat: 1.234 

This creates a reference from variable 'firstFloat' to the actual data in memory which consists of 8 bytes:

If we were to define a second variable, 'secondFloat' with a value of 'firstFloat' as so:

secondFloat: firstFloat

This would not create a new data object in memory but instead a reference to the existing object that variable 'firstFloat' is pointing to:

Variables 'firstFloat' and 'secondFloat' now have a reference count of 2, as the underlying object to which they are pointing has 2 references. 

Also, the memory usage has not increased as no new objects have been created in memory.

If we were to modify the variable 'secondFloat' in any way, it would not modify the original data object but rather create a copy and modify it. For example if we changed the value of the float:

secondFloat: 1.235

A new object and a new reference would be created, and memory used would double:

If a data object loses all references, the memory is returned to the process heap memory. Any local variables defined within a function will lose their references when the function has completed.

Exactly when the memory is fully returned to the OS for re-use depends on which mode of garbage collection is active.

Garbage Collection

Garbage Collection describes the process of making memory available again to the OS when the memory is no longer being used (or when it has no references). 

On startup of a q process, garbage collection mode is one of the startup options. This uses the '-g' startup option and takes an input of 0 (deferred) or immediate (1). The default behaviour is deferred if this startup option is not given.

Deferred garbage collection is like a 'manual' garbage collection mode where the user or program must use a command to run garbage collection, whereas immediate mode is like 'automatic' mode where garbage collection runs as soon as an object has lost all references.

In both cases, only objects ≥64MB in size are returned to the OS - any smaller objects remain in the heap. Why? kdb uses slab allocation with 64MB slabs (see the section 'notes on the allocator' here for more details). Note the memory in the heap will also be re-used by the process.

In deferred mode, .Q.gc[] is used to perform garbage collection. In addition to returning objects ≥64MB back to the OS, .Q.gc[] also attempts to re-organise the smaller blocks in the heap into 64MB blocks, and if successful will return any free blocks from that operation. 

Other memory related startup options 

There are currently two other memory related startup options:

The '-w' startup option sets out the maximum memory usage of the process (in bytes). The default is 0 i.e. no limit. If this limit is breached, the process will fail with a 'wsfull error.

The '-m' startup option allows a kdb process to utilise persistent memory - for example Intel Optane.

Memory related utilities

There are several functions and utilities to assist you when it comes to memory:

-8!x returns the byte representation of x. If you count this result you will get the number of bytes that x uses in memory.

You will notice that the results are slightly different than what you would expect - see Memory used by kdb objects.

The reference count for a variable.

Run garbage collection.

Run a command and record how long it takes and how much memory it uses in bytes.

Return some memory stats:

used| memory used 

heap| current heap size

peak| max heap size

wmax| memory limit set using '-w' startup option

mmap| memory used for memory mapping purposes

mphy| physical memory available

syms| number of internalised symbols

symw| memory used by those internalised symbols

Compression and serialisation

Memory mapping

Common memory issues and questions

temp: select from data where assetClass=`EQ;

temp2: update notional:px*vol from temp;

temp3: update cumVol: sums vol by sym from temp2;

In this case, three almost identical tables have been created, essentially trebling the amount of memory used.

See Tickerplant.

See here.