Framework Design

This page contains a detailed description of the design of FAST.

Contents

• Overall design
• The execution pipeline
• Data organization on heterogeneous devices
• Concurrent data streaming
• Visualization
• Tests

---

Overall design

The framework will consist of six main layers as illustrated with colors in figure X. The first bottom layer consist of the actual hardware, i.e. the CPUs and GPUs with all of its memory. The second layer consist of drivers for this hardware and are provided by the hardware manufacturers. Next is the library layer in turquoise which consist of several libraries that are needed in the framework. The libraries in this layer consist of:

• OpenCL - Enables parallel programming of heterogeneous systems
• OpenCL Utility Library - Ease the use of OpenCL
• OpenGL - Enables cross-platform visualization
• GLEW - Ease the use of OpenGL
• Qt - Cross-platform GUI for visualization
• Boost - C++ utility library

Next is the core of the framework which enables all of the described features in the previous section. The framework is split into several groups:

• Data objects - Objects for image (both static and dynamic) and tracking data which enables the synchronized processing of these data on a set of heterogeneous devices.
• Importers/Exporters - Data import and export adapters for different formats such as metaimage, raw, ITK and VTK.
• Streamers - Objects that enable streaming of data
• Algorithms - A set of commonly used algorithms.
• Visualization - A set of renderers such as surface, volume and slice renderers.
• Benchmarks - Mechanisms for measuring, assimilating and reporting the runtime of all operations in the framework.
• Tests - A set of tests for the framework which ensures that all parts of the framework are working properly.

Finally, on top is the application. The framework may be both a 1) stand-alone application which enables benchmarking and tests of a heterogeneous system and 2) an external library/framework for another medical image computing application.

The execution pipeline

FAST uses a demand-driven execution pipeline similar to what is used in ITK and VTK. This entails that each processing step is first linked together to form a pipeline and that it does not execute until some object requests data through an update call on one of the processing objects. This can currently be done in two ways:

• Explicitly by calling the update method on a object in the pipeline.
• Implicitly with a visualization/renderer which will call update on its input data repeatedly several times per second given by its framerate.

The pipeline consist of two types of objcets: Process objects and data object, which extends the abstract base classes ProcessObject and DataObject respectively.

A data object contains data and has always exactly one parent object while a process objects is an object that performs processing and may have zero, one or several parent data objects.

The data objects have an internal timestamp (positive integer) which is always updated when the data is changed. Each process object that has parent data objects has a list of timestamps for each parent data object representing which version of the data objects was used the last time the process object was executed. In addition each process object has a flag indicating whether it has been modified or not. This could be a parameter or input change.

When the update method is called on a process object it will first call update on all its parent data objects which again will call update on their parent process objects. Thus update will be called on all object backwards in the pipeline until a process object with no parent data is encountered (i.e. an importer object).

If a process object is either modified or one of its parent data objects have changed timestamps the object will re-execute by calling its execute method.

Thus each process object will implement its own execute method while the update method is the same for each process object.

Data organization on heterogeneous devices

Data organization and synchronization is one of the key components in this framework as everything will be built on top of it. First off, image data is represented by an object called Image which are used for both 2D and 3D image data. These image objects represents an image on all devices and its data is guaranteed to be coherent on any devices after being altered. I.e. if an image is changed on one device it will also be changed on the other devices before the data is used on those devices.

Data import and export

Data can be imported and exported to and from the framework in several different forms such as:

• ITK image
• VTK image
• Metaimage (.mhd and .raw)
• Image file (.jpg, .png etc.) Image data can also be created as an empty image with a specified size and data type.

Images of different data types

Medical images are represented in different formats. Some common examples are: Ultrasound (unsigned 8 bit integer), CT (signed/unsigned 16 bit integer) and MR (unsigned 16 bit integer).

The framework currently supports the following data formats for images:

• TYPE_FLOAT - 32 bit floating point number
• TYPE_UINT8 - 8 bit unsigned integer
• TYPE_INT8 - 8 bit signed integer
• TYPE_UINT16 - 16 bit unsigned integer
• TYPE_INT16 - 16 bit signed integer

An image can also have multiple channels, or components, and currently 1-4 channels are supported.

Data access

Two forms of data access are possible in the framework: 1) Read-only, 2) Read and write. The general rule is that several devices can perform read-only operations on an image coherently. However, if any device needs to write to an image, only that device can have access to the image at that time. This is to ensure data coherency across devices. Thus, if a device wants to write to an image, it has to wait for all other operations on that image to finish. And when it is writing to the image, no other devices can read or write to that image.

To facilitate this, several DataAccess objects are introduced:

• OpenCLBufferAccess - provides access to an OpenCL buffer of the image. The pixels are stored as x+ywidth (2D) and x+ywidth+zwidthheight (3D)
• OpenCLImageAccess2D - provides access to an OpenCL 2D image object. Not valid for 3D images.
• OpenCLImageAccess3D - provides access to an OpenCL 3D image object. Not valid for 2D images.
• ImageAccess - provides access to an data array (void ) on the host. The pixels are stored as x+ywidth (2D) and x+ywidth+zwidth*height (3D).

These objects are created by calling one of the following methods on the Image object:

• getOpenCLBufferAccess()
• getOpenCLImageAccess2D()
• getOpenCLImageAccess3D()
• getImageAccess()

Arguments to these methods are which device wants to access the image and what type of access is desired (Read-only or Read-and-write). From these objects an OpenCL Image (texture on a GPU) or Buffer object can retrieved which is needed to perform OpenCL computations on the image or pointer to main memory can be request for doing processing on the CPU using C++. The DataAccess objects also have methods for releasing the access, thus enabling other devices to perform write operations on the image. The access will also be released in the destructor of this object to avoid programmers creating a deadlock. When the access is released, the OpenCL Image/Buffer object pointer is invalidated to ensure that the program can no longer manipulate the data. However, this does not delete the actual data on the device. When access to an object is requested, the framework will check that any previous access objects has been released. If not an exception should be thrown. This is ensure that the user doesn't have multiple access to one image object at the same time.

Data change

Every time an image is changed on a device, the change should be reflected on the other devices as well. However, this doesn't have to be done immediately after the change is finished. The updating of data can be done the next time the data is requested on another device. This is often referred to as lazy loading. The benefit of lazy loading is that the number of data transfers can be reduced. However, the drawback is that there will be a transfer cost the next time some processing has to be performed on another device which doesn't have the updated data.

Thus, each image object has a set of flags indicating whether the data (in the form of OpenCL buffers and images and data array) is up to date for each device. When one device has changed some data, these are set to false for all devices except the device in which the change was performed. Next time the data is requested on a device, the flag is checked and if is false a data transfer will start and the flag will be set to true for that device.

Data removal

Memory usage

Concurrent data streaming

Streamers are objects that provide access to dynamic image data (i.e. a stream of images). This can be for instance real-time images from an ultrasound probe or a series of images stored on disk. The output of streamer objects is an object called DynamicImage which has a method called getNextFrame() which returns an Image object. The streamers read image data into the DynamicImage in a separate thread so that processing and data streaming can be performed concurrently.

Currently, only two streamers have been implemented: MetaImageStreamer and ImageStreamer which both read images from disk.

Streamers have three different streaming modes:

• STREAMING_MODE_NEWEST_FRAME_ONLY - This will only keep the newest frame in the DynamicImage object. So that its size is always 1.
• STREAMING_MODE_PROCESS_ALL_FRAMES - This will keep all frames in the DynamicImage object, but will remove the frame from the object after it has been returned by the getNextFrame() method.
• STREAMING_MODE_STORE_ALL_FRAMES - This will store all frames in the DynamicImage object.

Visualization

The Qt framework is chosen as the graphical user interface (GUI) because:

• Popular C++ framework, also in the medical domain.
• Cross-platform. Supports Windows, Linux and Mac.
• Supports multi-threading. (However, the Qt main/event loop is limited to be run in the main thread)
• Allows creating widgets (QGLWidget) that can be rendered to directly by OpenGL.
• Supports event handling (keyboard and mouse).
• Enables creation of simple forms and buttons.
• Object oriented (C++).

Currently, the framework only has one type of window called SimpleWindow which contains one View while a View can contain several renderers. All windows will be implemented using a QWidget, while the View object is based on the QGLWidget which is a widget that may be rendered to by OpenGL. The renderers does the actual renderering of images.

Four different types of renderers is available in the framework (the first three are for 3D images, while the last one is for 2D images):

• Surface renderer - Extracts a surface from a volume using the marching cubes algorithm and a threshold parameter. The surface is a triangle mesh which is displayed with OpenGL.
• Slice renderer - Extracts data from a volume in an arbitrary plane using trilinear interpolation.
• Volume renderer - Creates an image of a volume using ray casting.
• Image renderer - For displaying 2D images.

Tests

FAST also comes with a large set of unit and system tests, these are all located in the Tests folder and use data which can be downloaded separatly. The testing framework Catch is used which only consist of a single header file (catch.hpp).