Fuel.pillar

{
    "metadata" : {
        "title": "Serializing Complex Objects with Fuel",
        "attribution": "Martín Dias, Stéphane Ducasse, Mariano Martinez Peck and Max Leske with Johan Fabry"
    },
    "headingLevelOffset":2
}
@cha:fuel

Fuel is a fast open-source general-purpose binary object serialization
framework developed by Mariano Martinez-Peck, Martìn Dias and Max
Leske. It is robust and used in many industrial cases. A fundamental
reason for the creation of Fuel was speed: while there is a plethora
of frameworks to serialize objects based on recursive parsing of the
object graphs in textual format as XML, JSON, or STON, these
approaches are often slow. (For JSON and STON see also Chapters  *STON>../STON/STON.pillar@cha:ston* and *NeoJSON>../NeoJSON/NeoJSON.pillar@cha:JSON*.)

Part of the speed of Fuel comes from the idea that objects are loaded more often than stored. This makes it worth to spend more time while storing to yield
faster loading. Also, its storage scheme is based on the pickle format that puts similar objects into groups for efficiency and performance. As a result, Fuel
has been shown to be one of the fastest object loaders, while still being a really fast object saver.
Moreover, Fuel can serialize nearly any object in the image, it can even serialize a full execution stack and later reload it!

The main features of Fuel are as follows:
- It has an object-oriented design.
- It does not need special VM-support.
- It is modularly packaged.
- It can serialize/materialize not only plain objects but also classes, traits, methods, closures, contexts, packages, etc.
- It supports global references.
- It is very customizable: you can ignore certain instance variables, substitute objects by others, define pre and post serialization and materialization actions, etc.
- It supports class renaming and class reshaping.
- It has good test coverage and a large suite of benchmarks.

! General Information

Fuel has been developed and maintained over the years by the following people: Martin Dias, Mariano Martinez Peck, Max Leske, Pavel Krivanek, Tristan Bourgois
and Stéphane Ducasse (as PhD advisor and financer).

The idea of Fuel was developed by Mariano Martinez Peck based on the work by Eliot Miranda who worked on the "parcels" implementation for VisualWorks. Eliot's
work again was based on the original "parcels" implementation by David Leib. "Parcels" demonstrates very nicely that the binary pickle format can be a good
alternative to textual storage and that grouping of objects makes a lot of sense in object oriented systems.

Before going into details we present the ideas behind Fuel and it's main features and give basic usage examples.

!!Goals

; Concrete
: Fuel doesn't aspire to have a dialect-interchange format. This makes it possible to serialize special objects like contexts, block closures, exceptions, compiled methods and classes. Although there are ports to other dialects, most notably Squeak, Fuel development is Pharo-centric.
; Flexible
: Depending on the context, there can be multiple ways of serializing the same object. For example, a class can be considered either a global or a regular object. In the former case, references to the class will be encoded by name and the class is expected to be part of the environment upon materialization; in the latter case, the class will be encoded in detail, with its method dictionary, etc.
; Fast
: Fuel has been designed for performance. Fuel comes with a complete benchmark suite to help analyse the performance with diverse sample sets, as well as to compare it against other serializers. Fuel's pickling algorithm achieves outstanding materialization performance, as well as very good serialization performance, even when compared to other binary formats such as ImageSegment.
; Object-Oriented
: A requirement from the onset was to have a good object-oriented design and to avoid special support from the virtual machine.
; Maintainable
: Fuel has a complete test suite (over 600 unit tests), with a high degree of code coverage. Fuel also has well-commented classes and methods.
!! Installation and Demo

Fuel 1.9 is available by default in Pharo since version 2.0 of Pharo. Therefore you do not need to install it.
The ""default packages"" work out of the box in Pharo 1.1.1, 1.1.2, 1.2, 1.3, 1.4, 2.0, 3.0 and 4.0 and Squeak 4.1, 4.2, 4.3, 4.4, 4.5.
The stable version at the time of writing is 1.9.4.

Open the ==Transcript== and execute the code below in a ==Playground==. This example serializes a set, the default ==Transcript== (which is a global) and a
block. On materialization it shows that
- the set is correctly recreated,
- the global ==Transcript== is still the same instance (hasn't been modified)
- and the block can be evaluated properly.

[[[
| arrayToSerialize materializedArray |
arrayToSerialize :=
   Array
      with: (Set with: 42)
      with: Transcript
      with: [ :aString | Transcript show: aString; cr ].

"Store (serialize)"
FLSerializer serialize: arrayToSerialize toFileNamed: 'demo.fuel'.

"Load (materialize)"
materializedArray := FLMaterializer materializeFromFileNamed: 'demo.fuel'.

Transcript
   show: 'The sets are equal: ';
   show: arrayToSerialize first = materializedArray first;
   cr;
   show: 'But not the same: ';
   show: arrayToSerialize first ~~ materializedArray first;
   cr;
   show: 'The global value Transcript is the same: ';
   show: arrayToSerialize second == materializedArray second;
   cr.

materializedArray third
   value: 'The materialized block closure can be properly evaluated.'.
]]]

!! Some Links

- The home page is *http://rmod.inria.fr/web/software/Fuel*.
- The source code is at *http://smalltalkhub.com/#!/~Pharo/Fuel*.
- The CI job is at *https://ci.inria.fr/pharo-contribution/job/Fuel-Stable/*.



%=========================================================================%


! Getting Started
@GettingStarted

!!Basic Examples
Fuel offers some class-side messages to ease more common uses of
serialization  (the ==serialize:toFileNamed:== message )
and materialization (the message ==materializeFromFileNamed:==). The next example writes to and reads from a file:

[[[
FLSerializer serialize: 'stringToSerialize' toFileNamed: 'demo.fuel'.
materializedString := FLMaterializer materializeFromFileNamed: 'demo.fuel'.
]]]

Fuel also provides messages for storing into a ==ByteArray==, namely
the messages ==serializeToByteArray:== and ==materializeFromByteArray:==. This
can be interesting, for example, for serializing an object graph as a
blob of data into a database when using Voyage (see Chapter *Voyage>../Voyage/Voyage.pillar@cha:voyage*).

[[[
anArray := FLSerializer serializeToByteArray: 'stringToSerialize'.
materializedString := FLMaterializer materializeFromByteArray: anArray.
]]]

!!FileStream

In the following example we work with file streams. Note that the stream needs to be set to binary mode:

[[[
'demo.fuel' asFileReference writeStreamDo: [ :aStream |
   FLSerializer newDefault
      serialize: 'stringToSerialize'
      on: aStream binary ].

'demo.fuel' asFileReference readStreamDo: [ :aStream |
   materializedString := (FLMaterializer newDefault
      materializeFrom: aStream binary) root ].
]]]

In this example, we are no longer using the class-side messages. Now,
for both ==FLSerializer== and ==FLMaterializer==, we first create
instances by sending the
==newDefault== message and then perform the desired operations. As we will see in the next example, creating the instances allows for more flexibility on serialization
and materialization.

!!Compression

Fuel does not care to what kind of stream it writes its data. This makes it easy to use stream compressors. An example of use is as follows:

[[[
'number.fuel.zip' asFileReference writeStreamDo: [ :aFileStream |
   |gzip|
   aFileStream binary.
   gzip := GZipWriteStream on: aFileStream.
   FLSerializer newDefault serialize: 123 on: gzip.
   gzip close ].

'number.fuel.zip' asFileReference readStreamDo: [ :aFileStream |
   |gzip|
   aFileStream binary.
   gzip := GZipReadStream on: aFileStream.
   materializedString := (FLMaterializer newDefault
      materializeFrom: gzip) root.
   gzip close ].
]]]

!!Showing a Progress Bar

Sometimes it is nice to see progress updates on screen. Use the message ==showProgress== in this case. The progress bar functionality is available from the
==FuelProgressUpdate== package, so load that first:

[[[
   Gofer it
      url: 'http://smalltalkhub.com/mc/Pharo/Fuel/main';
      package: 'ConfigurationOfFuel';
      load.

   (ConfigurationOfFuel project  version: #stable)
      load: 'FuelProgressUpdate'.
]]]

The following example uses the message ==showProgress== to display a progress bar during operations.

[[[
'numbers.fuel' asFileReference writeStreamDo: [ :aStream |
   FLSerializer newDefault
      showProgress;
      serialize: (1 to: 200000) asArray
      on: aStream binary ].

'numbers.fuel' asFileReference readStreamDo: [ :aStream |
   materializedString :=
      (FLMaterializer newDefault
         showProgress;
         materializeFrom: aStream binary) root ].
]]]



%=========================================================================%
! Managing Globals
@ManagingGlobals

Sometimes we may be interested in storing just the name of a reference,
because we know it will be present when materializing the graph. For example when the current processor scheduler ==Processor== is referenced from the graph we do not want to serialize it as it does not make sense to materialize it. Hence Fuel considers some objects as globals that may not be serialized. It also allows for you to add to this set and lastly to use a different environment when materializing globals.

!!Default Globals

By default, Fuel considers the following objects as globals, i.e., it will store just their name:

- ==nil==, ==true==, ==false==, and ==Smalltalk globals==.
- Any ==Class==, ==Trait==, ==Metaclass== or ==ClassTrait==.
- Any ==CompiledMethod==, except when either it answers false to the message ==isInstalled== or true to the message ==isDoIt==. The latter happens, for example, if this is code evaluated from a Workspace.
- Some well-known global variables: ==Smalltalk==, ==SourceFiles==, ==Transcript==, ==Undeclared==, ==Display==, ==TextConstants==, ==ActiveWorld==, ==ActiveHand==, ==ActiveEvent==, ==Sensor==, ==Processor==, ==ImageImports==, ==SystemOrganization== and ==World==.


!!Duplication of Custom Globals

With this following code snippet, we show that by default a Smalltalk global value is not serialized as a global. In such a case it is duplicated on materialization.

[[[
"Define a global variable named SomeGlobal."
SomeGlobal := Set new.

"Serialize and materialize the value of SomeGlobal."
FLSerializer
   serialize: SomeGlobal
   toFileNamed: 'g.fuel'.

"The materialized object *is not* the same as the global instance."
[ (FLMaterializer materializeFromFileNamed: 'g.fuel') ~~ SomeGlobal ] assert.
]]]

We can tell Fuel to handle a new global and how to avoid global duplication on materialization. The message ==considerGlobal:== is used to specify that an
object should be stored as global, i.e. it should only be referenced by name.

[[[
| aSerializer |

"Define a global variable named SomeGlobal."
SomeGlobal := Set new.

aSerializer := FLSerializer newDefault.

"Tell the serializer to consider SomeGlobal as global."
aSerializer analyzer considerGlobal: #SomeGlobal.

aSerializer
   serialize: SomeGlobal
   toFileNamed: 'g.fuel'.

"In this case, the materialized object *is* the same as the global instance."
[ (FLMaterializer materializeFromFileNamed: 'g.fuel') == SomeGlobal ] assert.
]]]

!!Changing the Environment
@sec:ManagingGlobals

The default lookup location for globals is ==Smalltalk globals==. This can be changed by using the message ==globalEnvironment:== during serialization and
materialization.

The following example shows how to change the globals environment during materialization. It creates a global containing the empty set, tells Fuel to consider
it as a global and serializes it to disk. A new environment is then created with a different value for the global: ==42== and the global is then materialized in
this environment. We see that the materialized global has as value ==42==, i.e. the value of the environment in which it is materialized.

[[[
| aSerializer aMaterializer anEnvironment |

"Define a global variable named SomeGlobal."
SomeGlobal := Set new.

"Tell the serializer to consider SomeGlobal as global."
aSerializer := FLSerializer newDefault.
aSerializer analyzer considerGlobal: #SomeGlobal.
aSerializer
   serialize: SomeGlobal
   toFileNamed: 'g.fuel'.

"Override value for SomeGlobal."
anEnvironment := Dictionary newFrom: Smalltalk globals.
anEnvironment at: #SomeGlobal put: {42}.

"In this case, the materialized object *is the same* as the global instance."
'g.fuel' asFileReference readStreamDo: [ :aStream |
   | materializedGlobal |
   aStream binary.
   aMaterializer := FLMaterializer newDefault.

   "Set the environment"
   aMaterializer globalEnvironment: anEnvironment.

   materializedGlobal := (aMaterializer materializeFrom: aStream) root.

   [ materializedGlobal = {42} ] assert.
   [ materializedGlobal == (anEnvironment at: #SomeGlobal) ] assert ].
]]]


%=========================================================================%

! Customizing the Graph

When serializing an object you often want to select which part of the object's state should be serialized. To achieve this with Fuel you can selectively ignore
instance variables.

!!Ignoring Instance Variables

Under certain conditions it may be desirable to prevent serialization of certain instance variables for a given class. A straightforward way to do this is to
override the hook method ==fuelIgnoredInstanceVariableNames==, at class side of the given class. It returns an array of instance variable names (as symbols) and
""all"" instances of the class will be serialized without these instance variables.

For example, let's say we have the class ==User== and we do not want to serialize the instance variables =='accumulatedLogins'== and =='applications'==. So we
implement:

[[[
User class>>fuelIgnoredInstanceVariableNames
   ^ #('accumulatedLogins' 'applications')
]]]

!!Post-Materialization Action

When materialized, ignored instance variables will be ==nil==. To re-initialize and set values to those instance variables, send the
==fuelAfterMaterialization== message.

The message ==fuelAfterMaterialization== lets you execute some action once an object has been materialized. For example, let's say we would like to set back the
instance variable =='accumulatedLogins'== during materialization. We can implement:

[[[
User>>fuelAfterMaterialization
   accumulatedLogins := 0.
]]]

!!Substitution on Serialization

Sometimes it is useful to serialize something different than the original object, without altering the object itself. Fuel proposes two different ways to do
this: dynamically and statically.

!!!Dynamically

You can establish a specific substitution for a particular serialization. Let's illustrate with an example, where the graph includes a ==Stream== and you want
to serialize ==nil== instead.

[[[
objectToSerialize := { 'hello' . '' writeStream}.

'demo.fuel' asFileReference writeStreamDo: [ :aStream |
   aSerializer := FLSerializer newDefault.
   aSerializer analyzer
      when: [ :object | object isStream ]
      substituteBy: [ :object | nil ].
   aSerializer
      serialize: objectToSerialize
      on: aStream binary ].

'demo.fuel' asFileReference readStreamDo: [ :aStream |
   materializedObject := (FLMaterializer newDefault
      materializeFrom: aStream binary) root]
]]]

After executing this code, ==materializedObject== will contain ==#('hello' nil)==, i.e. without the instance of a ==Stream==.

!!! Statically
You can also do substitution for each serialization of an object by overriding its ==fuelAccept:== method. Fuel visits each object in the graph by sending this
message to determine how to trace and serialize it. The argument of the message is an instance of a ==FLMapper== subclass.

As an example, imagine we want to replace an object directly with nil. In other words, we want to make all objects of a class transient, for example all
==CachedResult== instances. For that, we should implement:

[[[
CachedResult>>fuelAccept: aGeneralMapper
   ^ aGeneralMapper
      visitSubstitution: self
      by: nil
]]]

As another example, we have a ==Proxy== class and when serializing we want to serialize its ==target== instead of the proxy. So we redefine ==fuelAccept:== as
follows:

[[[
Proxy>>fuelAccept: aGeneralMapper
   ^ aGeneralMapper
      visitSubstitution: self
      by: target
]]]

The use of ==fuelAccept:== also allows for deciding about serialization conditionally. For example, we have the class ==User== and we want to ==nil== the
instance variable ==history== when its size is greater than 100. A naive implementation is as follows:

[[[
User>>fuelAccept: aGeneralMapper
   ^ self history size > 100
      ifTrue: [
         aGeneralMapper
            visitSubstitution: self
            by: (self copy history: #()) ].
      ifFalse: [ super fuelAccept: aGeneralMapper ]
]]]

@@note We are substituting the original user by another instance of ==User==, which Fuel will visit with the same ==fuelAccept:== method. Because of this we fall into an infinite sequence of substitutions!

Using ==fuelAccept:== we can easily fall into an infinite sequence of substitutions. To avoid this problem, the message ==visitSubstitution:by:onRecursionDo:==
should be used. In it, an alternative mapping is provided for the case of mapping an object which is already a substitute of another one. The example above
should be written as follows:

[[[
User>>fuelAccept: aGeneralMapper
   aGeneralMapper
      visitSubstitution: self
      by: (self copy history: #())
      onRecursionDo: [ super fuelAccept: aGeneralMapper ]
]]]

In this case, the substituted user (i.e., the one with the empty history) will be visited via its super implementation.

!! Substitution on Materialization
In the same way that we may want to customize object serialization, we may want to customize object materialization. This can be done either by treating an
object as a globally obtained reference, or by hooking into instance creation.

!!! Global References

Suppose we have a special instance of ==User== that represents the admin user, and it is a unique instance in the image. In the case that the admin user is
referenced in our graph, we want to get that object from a global when the graph is materialized. This can be achieved by modifying the ""serialization""
process as follows:

[[[
User>>fuelAccept: aGeneralMapper
   ^ self == User admin
      ifTrue: [
         aGeneralMapper
            visitGlobalSend: self
            name: #User
            selector: #admin ]
      ifFalse: [ super fuelAccept: aGeneralMapper ]
]]]

During serialization the admin user won't be serialized but instead its global name and selector are stored. Then, at materialization time, Fuel will send the
message ==admin== to the class ==User==, and use the returned value as the admin user of the materialized graph.

!!! Hooking into Instance Creation

Fuel provides two hook methods to customise how instances are created: ==fuelNew== and ==fuelNew:==.

For (regular) fixed objects, the method ==fuelNew== is defined in ==Behavior== as:

[[[
fuelNew
   ^ self basicNew
]]]

But we can override it to our needs, for example:

[[[
fuelNew
   ^ self uniqueInstance
]]]

This similarly applies to variable sized objects through the method ==fuelNew:== which by default sends ==basicNew:==.

!!Not Serializable Objects

You may want to make sure that some objects are not part of the graph during serialization. Fuel provides the hook method named ==visitNotSerializable:== which
signals an ==FLNotSerializable== exception if such an object is found in the graph that is to be serialized.

[[[
MyNotSerializableObject>>fuelAccept: aGeneralMapper
   aGeneralMapper visitNotSerializable: self
]]]


%=========================================================================%

! Errors

We provide a hierarchy of errors which allows one to clearly identify the problem when something went wrong:

- ==FLError==
-- ==FLSerializationError==
--- ==FLNotSerializable==
--- ==FLObjectNotFound==
--- ==FLObsolete==
-- ==FLMaterializationError==
--- ==FLBadSignature==
--- ==FLBadVersion==
--- ==FLClassNotFound==
--- ==FLGlobalNotFound==
--- ==FLMethodChanged==
--- ==FLMethodNotFound==

As most classes of Fuel, they have class comments that explain their purpose:

; FLError
: I represent an error produced during Fuel operation.

; FLSerializationError
: I represent a serialization error.

; FLNotSerializable
: I represent an error which may happen while tracing in the graph an object that is forbidden of being serialized.

; FLObjectNotFound
: I represent an error which may happen during serialization, when trying to encode on the stream a reference to an object that should be encoded before, but it is not. This usually happens when the graph changes during serialization. Another possible cause is a bug in the analysis step of serialization.

; FLObsolete
: I am an error produced during serialization, signaled when trying to serialize an obsolete class as global. It is a prevention, because such class is likely to be absent during materialization.

; FLMaterializationError
: I represent a materialization error.

; FLBadSignature
: I represent an error produced during materialization when the serialized signature doesn't match the materializer's signature (accessible via ==FLMaterializer>>signature==). A signature is a byte prefix that should prefix a well-serialized stream.

; FLBadVersion
: I represent an error produced during materialization when the serialized version doesn't match the materializer's version (accessible via ==FLMaterializer>>version==). A version is encoded in 16 bits and is encoded heading the serialized stream, after the signature.

; FLClassNotFound
: I represent an error produced during materialization when a serialized class or trait name doesn't exist.

; FLGlobalNotFound
: I represent an error produced during materialization when a serialized global name doesn't exist (at ==Smalltalk globals==).

; FLMethodChanged
: I represent an error produced during materialization when a change in the bytecodes of a method serialized as global is detected. This error was born when testing the materialization of a ==BlockClosure== defined in a method that changed. The test produced a VM crash.

; FLMethodNotFound
: I represent an error produced during materialization when a serialized method in a class or trait name doesn't exist (in ==Smalltalk globals==).


%=========================================================================%

! Object Migration
We often need to load objects whose class has changed since it was saved. For example, figure *@figClassChanges* illustrates typical changes that can happen to
the class shape. Now imagine we previously serialized an instance of ==Point== and we need to materialize it after ==Point== class has changed.

+Example of changes to a class>file://figures/ClassChanges.png|width=100|label=figClassChanges+

Let's start with the simple cases. If a variable was ""inserted"", its value will be ==nil==. If it was ""removed"", it is also obvious: the serialized value will be
ignored. The ""change of Order"" of instance variables is handled by Fuel automatically.

A more interesting case is when a variable was ""renamed"". Fuel cannot automatically guess the new name of a variable, so the change will be understood by Fuel
as two independent operations: an insertion and a removal. To resolve this problem, the user can tell the Fuel materializer which variables are renamed by
using the message ==migratedClassNamed:variables:==. It takes as first argument the name of the class and as second argument a mapping from old names to new
names. This is illustrated in the following example:

[[[
FLMaterializer newDefault
   migrateClassNamed: #Point
   variables: {'x' -> 'posX'. 'y' -> 'posY'}.
]]]

The last change that can happen is a ""class rename"". Again the Fuel
materializer provides a way to handle this: the
==migrateClassNamed:toClass:== message, and an example of its use is shown below:

[[[
FLMaterializer newDefault
   migrateClassNamed: #Point
   toClass: Coordinate.
]]]

Lastly, Fuel defines the message ==migrateClassNamed:toClass:variables:== that combines both ""class and variable rename"".

Additionally, the method ==globalEnvironment:==, shown in Section *@sec:ManagingGlobals*, is useful for migration of global variables: you can prepare an ad-hoc environment dictionary with the same keys that were used during serialization, but with the new classes as values.

@@note A class could also change its ""layout"". For example, Point could change from being ""fixed"" to ""variable"". Layout changes from fixed to variable format are automatically handled by Fuel. Unfortunately, the inverse (variable to fixed) is not supported yet.


%=========================================================================%

! Fuel Format Migration

Until now, each Fuel version has used its own stream format, which is ""not"" compatible with the format of other versions. This means that when upgrading Fuel, we
will need to convert our serialized streams. This is done by using the old version of Fuel to materialize a stream, keeping a reference to this object graph, and then loading the new version of Fuel and serializing the object graph back to a file.

We include below an example of such a format migration. Let's say we have some files serialized with Fuel 1.7 in a Pharo 1.4 image and we want to migrate them
to Fuel 1.9.

[[[
| oldVersion newVersion fileNames objectsByFileName
   materializerClass serializerClass |
oldVersion := '1.7'.
newVersion := '1.9'.
fileNames := #('a.fuel' 'b.fuel' 'c.fuel' 'd.fuel' 'e.fuel').
objectsByFileName := Dictionary new.

(ConfigurationOfFuel project version: oldVersion) load.
"Need to do it like this otherwise
   the class is decided at compile time."
materializerClass := Smalltalk at: #FLMaterializer.

fileNames do: [ :fileName |
   objectsByFileName
      at: fileName
      put: (materializerClass materializeFromFileNamed: fileName) ].


(ConfigurationOfFuel project version: newVersion) load.
"Need to do it like this otherwise
   the class is decided at compile time."
serializerClass := Smalltalk at: #FLSerializer.

objectsByFileName keysAndValuesDo: [ :fileName :objects |
   serializerClass
      serialize: objects
      toFileNamed: 'migrated-', fileName ].
]]]

We assume in this example that the number of objects to migrate can be materialized all together at the same time. This assumption may be wrong. In such case, you could adapt the script to split the list of files and do the migration in parts.

@@note This script should be evaluated in the original image. We don't guarantee that Fuel 1.7 loads in Pharo 2.0, but we do know that Fuel 1.9 loads in Pharo 1.4.

%=========================================================================%

% JF: Removed this because it is too much in depth and moreover
% FLGraphViewBuilder does not work anymore.

% !! Fuel Development: Debugging Tools

% There are a couple of packages that help us to debug Fuel, and they are loaded as follows:

% [[[
% Gofer it
%    url: 'http://smalltalkhub.com/mc/Pharo/Fuel/main';
%    package: 'ConfigurationOfFuel';
%    load.

% (ConfigurationOfFuel project version: #stable)
%    load: #(FuelDebug FuelPreview).
% ]]]

% We now first talk about the internals of Fuel before we present the different tools: a graph viewer, a serialization logger and a materialization logger.

% !!!Internal Workings

% To understand the output of these tools, you should first know some basics of how Fuel works internally, both for serialization and materialization.

% ;Serialization
% :The most important thing to know is that serialization is split in two main steps: analysis and encoding. The analysis phase consists of walking the graph from the specified root object and mapping each traversed object to its corresponding group, called a ""cluster"". The encoding phase runs after analysis, and it writes the graph to the stream linarly, in these steps:

% #header
% #for each cluster, instances part
% #for each cluster, references part
% #trailer

% ;Materialization
% : Because of the extra effort put into serialization, we can materialize the graph in a single phase. We decode the graph by reading the input stream linearly, in the same order it was written. The materialization steps are obviously analogous to the ones above:

% #header
% #for each cluster, instances part
% #for each cluster, references part
% #trailer

% It is important to understand that references are ""not"" stored together with their objects. Instead, all instances are stored together and all references are
% stored together, after the references. We use this to materialize all the references in a single step, when we know that all the objects have already been
% materialized.

% !!!FLGraphViewBuilder
% This class adds drawing capabilities to the analysis in the ==FuelDebug== package.

% Right-click a node to inspect it. Some examples:

% [[[
% (FLAnalyzer newDefault
%    setDebug;
%    analysisFor: #((1) (2) (3) (4)))
%    open.
% ]]]

% [[[
% (FLAnalyzer newDefault
%    setDebug;
%    analysisFor: #((1) (2) (3) (4)))
%    openPathsTo: 3.

% (FLAnalyzer newDefault
%    setDebug;
%    analysisFor: #((1) (2) (3) (4)))
%    openPathsToEvery: [ :object |
%       object isNumber and: [ object > 2 ] ].
% ]]]

% Figure *@figFuelPreview* shows how they look like.

% +Visual preview of graph to be serialized>file://figures/FuelPreview.png|width=60|label=figFuelPreview+
% _

% !!!FLDebugSerialization
% I am a serialization which facilitates debugging, by logging the stream position before and after main steps of ==FLSerialization==, including cluster
% information. Obviously, you should be familiar with the algorithm to understand the output log.

% To use, send the message ==setDebug== to your serializer and run as usual. For example:

% [[[
% 'debug.fuel' asFileReference writeStreamDo: [ :aStream |
%    FLSerializer newDefault
%       setDebug;
%       serialize: 'hello' on: aStream binary ].
% ]]]

% Then, inspect the output log:

% [[[
% FLDebugSerialization last log.
% ]]]

% !!!FLDebugMaterialization
% I am a materialization which facilitates debugging, by logging the stream position before and after main steps of ==FLMaterialization==, including cluster
% information. Obviously, you should be familiar with the algorithm to understand the output log.

% To use, send the message ==setDebug== to your materializer and run as usual. For example:

% [[[
% 'debug.fuel' asFileReference readStreamDo: [ :aStream |
%    FLMaterializer newDefault
%       setDebug;
%       materializeFrom: aStream binary ].
% ]]]

% Then, inspect the output log:

% [[[
% FLDebugMaterialization last log.
% ]]]


%=========================================================================%

! Built-in Header Support

It can be useful to store additional information with the serialized graph or perform pre and post materialization actions. To achieve this, Fuel supports the possibility to customize the header, an instance of ==FLHeader==.

The following example shows these features: first we add a property called timestamp to the header using the message ==at:putAdditionalObject:==. We then define some pre and post actions using ==addPreMaterializationAction:== and ==addPostMaterializationAction:==, respectively. In the latter we show how we can retrieve the property value by using the ==additionalObjectAt:== message.

[[[
| serializer |
serializer := FLSerializer newDefault.

serializer header
   at: #timestamp
   putAdditionalObject: DateAndTime now rounded.

serializer header
   addPreMaterializationAction: [
      Transcript show: 'Before serializing'; cr ].

serializer header
   addPostMaterializationAction: [ :materialization |
      Transcript
         show: 'Serialized at ';
         show: (materialization additionalObjectAt: #timestamp).
      Transcript cr;
         show: 'Materialized at ';
         show: DateAndTime now rounded;
         cr ].

serializer
   serialize: 'a big amount of data'
   toFileNamed: 'demo.fuel'
]]]

Then, you can materialize the header info only, and obtain the timestamp property, as follows:

[[[
| aHeader |
aHeader := FLMaterializer materializeHeaderFromFileNamed: 'demo.fuel'.
aHeader additionalObjectAt: #timestamp.
]]]

If we materialize the whole file, as below, the print string of the results is: =='a big amount of data'==.

[[[
FLMaterializer materializeFromFileNamed: 'demo.fuel'
]]]

And something similar to the following is shown in ==Transcript==:

[[[
Before serializing
Serialized at 2015-05-24T22:39:18-03:00
Materialized at 2015-05-24T22:39:37-03:00
]]]

! Conclusion

Fuel is a fast and stable binary object serializer for Pharo and is available by default in Pharo since 2.0. It can serialize to and materialize from any stream and the graph of objects to be serialized can be customized in multiple ways. It can serialize nearly any object in the system. For example, cases are known of an error occurring in a deployed application, the full stack being serialized and later materialized on a development machine for debugging.

% Local Variables:
% compile-command: "cd .. && ./pillar export --to=\"HTML by chapter\" Fuel/Fuel.pier"
% End: