Skip to content

Auditing

Effi Ban edited this page May 26, 2022 · 39 revisions

Overview

Many applications need to audit their DB changes, so they can review the history of changes ("change log") done to each table - whether it is for financial reasons, troubleshooting and more. To handle all this, several frameworks exist depending on the type of DB and programming language you are using. If you are using persistence-layer, and you also need one of these tools - it could be cumbersome and time-consuming to combine the two.
To solve this issue - persistence-layer has its own built-in auditing support, which will be described in detail in the next sections. For now, it's important to mention one thing: PL is not opinionated, meaning it will not force you to persist your audit data to a specific storage, or even to persist it at all. Instead - it just helps you configure which data should be audited, and then lets you define your own logic for publishing the data to some other service or external system that can handle it.

Quick Start

We'll start off with a simple example of auditing.
Let's say we have set up the Entity Persistence according to the Entity Persistence page.

Step 1: Add audit annotations to entity

First we need to add two annotations to our AdEntity:

  • @Audited - at the class level, indicating that all changes to the fields of our entity will be audited
  • @Id - on the unique id (usually primary key) that we want to use for identifying our entity together with the changes.

Our AdEntity will now look like this:

@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);

    public static final EntityField<AdEntity, String>     URL = INSTANCE.field(AdCreative.TABLE.display_url);
}

Step 2: Implement the publisher

Next, we need to implement a publisher for our audit data.
As explained above, PL does not force you to save your audit data in specific tables/storage, and in fact doesn't force you to do anything with it at all!
All PL will do, is collect the DB changes that occurred during the execution of your commands, and place them in instances of AuditRecord - one for each command (The AuditRecord class will be explained in detail in the next section).
These AuditRecord-s are available after your command has finished executing, and they are exposed through the interface AuditRecordPublisher that we need to implement. PL will call this publisher immediately after the commands have been executed.
In a real application, your publisher might save the AuditRecord-s to some database, or send them to some other system for processing. For our simple example, we will implement a publisher that simply converts the AuditRecord into JSON and prints it:

import com.fasterxml.jackson.databind.ObjectMapper;

public JsonAuditRecordPublisher extends AuditRecordPublisher {
    private final ObjectMapper objectMapper = new ObjectMapper();

    @Override
    public void publish(@NonNull final Stream<? extends AuditRecord<?>> auditRecords) {
        final var auditRecordsStr = objectMapper.writeValueAsString(auditRecords);
        System.out.println(auditRecordsStr);
    }
}

Step 3: Add the audit publisher to the PLContext

The last step is to plug our publisher into the PLContext we already created, so that PL will know about it and invoke it:

public PLContext plContext() {
    DSLContext jooq = DSL.using(new DefaultConfiguration()
        .set(SQLDialect.MYSQL)
        .set(new ThreadLocalTransactionProvider(/* and your JDBC connection provider */)));

    return new PLContext.Builder(jooq)
                        .withAuditRecordPublisher(new JsonAuditRecordPublisher())
                        .build();
}

Step 4: Try it out

var cmd = new CreateAdCommand();
cmd.set(AdEntity.SYNC_STATUS, SyncStatus.APPROVED);
cmd.set(AdEntity.URL, "http://mysite.com");

var adPersistence = new AdPersistence(plContext); 
adPersistence.create(List.of(cmd));

This will produce the following output from the publisher:
(NOTE: the default field names provide by PL are slightly different, we will assume they were customized - more on this below)

{
    "entityType": "ad",
    "entityId": "<<auto generated id>>",
    "operator": "CREATE",
    "fieldRecords": [
        {
            "field": "syncStatus",
            "newValue": "APPROVED"
        },
        {
            "field": "url",
            "newValue": "http://mysite.com"
        }                
    ]
}

The AuditRecord

PL will store the changes from each command in an AuditRecord, which contains the following fields:

  • entityType : the type of entity being audited
  • entityId : the unique id that will be used to identify the entity along with its changes. This is not required, as some entities may not have a single unique id. In that case, they should include some other fields to uniquely identify the entity as part of the mandatoryFieldValues (see below).
  • mandatoryFieldValues : A collection of field/value pairs, for mandatory fields (special fields which should always appear, even if unchanged - this is explained in more detail in the next section).
  • operator : Either CREATE, UPDATE or DELETE according to the type of command being audited. If it is an InsertOnDuplicateCommand ("upsert") then PL will automatically determine during the command execution if it should create or update, so the resulting operator will be determined accordingly.
  • entityChangeDescription : An optional description of the change itself, e.g. the reason for performing the change. It can be populated in the command (explained in more detail below).
  • fieldRecords : A collection containing all the field changes performed by the command. Each change has the following fields:
    • field - the field that was changed
    • oldValue - the old value, if operator is DELETE or UPDATE (otherwise null)
    • newValue - the new value, if operator is CREATE or UPDATE (otherwise null)
  • childRecords : A collection containing changes to children of this entity, in case there are child commands executed along with the current (parent) command. This field is recursive - meaning that each child record is itself an AuditRecord with the same fields as above (See also the Examples section below).

Configuring the Audited Fields

This section will explain in detail what options you have for configuring the fields of your audited entity.

Choosing the id

As in the quick start example above you must choose a unique id of the entity, that should appear in the AuditRecord (usually the primary key). Annotate this field with @Id in your entity. Here's a reminder of how it will look:

public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    // other fields
}

NOTE: There is a limitation currently, that only one field may be annotated with @Id and it also must be numeric.
We plan to remove these limitations in the future.

Auditing all fields

As in the quick start above, annotating the entity class with @Audited will cause all field changes to be audited:

@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    // other fields
}

Including specific fields only

Suppose you only want to audit a few specific fields. In that case you should drop the entity-level annotation and place it on the desired fields only.
In the example below, only SYNC_STATUS and URL will be audited:

public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    @Audited
    public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);

    public static final EntityField<AdEntity, String>     HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);

    @Audited
    public static final EntityField<AdEntity, String>     URL = INSTANCE.field(AdCreative.TABLE.display_url);

    public static final EntityField<AdEntity, String>     BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);

    public static final EntityField<AdEntity, String>     IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);

    public static final EntityField<AdEntity, Integer>    IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}

Excluding specific fields only

Suppose you want to audit all fields except a few specific ones. In that case you should place the entity-level annotation, but also add the @NotAudited annotation on the desired fields to be excluded. In the example below, all fields except for SYNC_STATUS and URL will be audited:

@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    @NotAudited
    public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);

    public static final EntityField<AdEntity, String>     HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);

    @NotAudited
    public static final EntityField<AdEntity, String>     URL = INSTANCE.field(AdCreative.TABLE.display_url);

    public static final EntityField<AdEntity, String>     BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);

    public static final EntityField<AdEntity, String>     IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);

    public static final EntityField<AdEntity, Integer>    IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}

Mandatory fields

Mandatory fields are special fields that can be added to the AuditRecord and they will appear always in a separate section, whether they have changed or not.
These fields can either come from the entity itself or from external entities, as long as they are linked by some chain of DB keys to the original entity.
The purpose of including such fields could be:

  • To provide a unique identifier (unique key) of the entity, in case there is no single unique id that can populated in the entity_id
  • To provide additional descriptions of the entity, as well as the hierarchy in which it resides
  • To provide additional classifications/tags of the entity, that will make it easier to search and filter changes that you want to collect together.

Mandatory fields from the same entity

To add a mandatory fields from the same entity, use the annotation @Audited(AuditTrigger.ALWAYS).
Note that these fields could appear twice in the AuditRecord: Once in the mandatory section, and once more in the changes section if they have also changed.
In the example below, HEADLINE and SYNC_STATUS are mandatory fields, while URL and IMAGE_NAME are (regular) audited fields:

public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final AdEntity INSTANCE = new AdEntity();
    private AdEntity() { super("ad"); }

    @Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
    
    @Id
    @Immutable
    public static final EntityField<AdEntity, Integer>    ID = INSTANCE.field(AdCreative.TABLE.creative_id);

    @Audited(AuditTrigger.ALWAYS)
    public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);

    @Audited(AuditTrigger.ALWAYS)
    public static final EntityField<AdEntity, String>     HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);

    @Audited
    public static final EntityField<AdEntity, String>     URL = INSTANCE.field(AdCreative.TABLE.display_url);

    public static final EntityField<AdEntity, String>     BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);

    @Audited
    public static final EntityField<AdEntity, String>     IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);

    public static final EntityField<AdEntity, Integer>    IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}

Mandatory fields from external entities

To include mandatory fields from external entities, you'll need to do the following:

  1. Make sure that PL is able to fetch all the fields you need from all the entities. This is explained in detail in the "State Consumers and Relations" section.
  2. Implement the interface AuditExtensions, providing the fields you want to include as mandatory. For this example let's say the impl is named MyAuditExtensions
  3. Annotate your entity with @Audited(extensions = MyAuditExtensions.class).

In the example below, AdEntity defines the external mandatory fields AdgroupEntity.NAME and CampaignEntity.ID_IN_TARGET:

public class CampaignEntity extends AbstractEntityType<CampaignEntity> {

    public static final EntityField<CampaignEntity, String> ID_IN_TARGET = INSTANCE.field(CustomerCampaigns.TABLE.name);

    // ...
}

public class AdgroupEntity extends AbstractEntityType<AdgroupEntity> {

    public static final EntityField<AdGroupEntity, String> NAME = INSTANCE.field(CampaignAds.TABLE.name);

    // ...
}

public class AdAuditExtensions implements AuditExtensions {
    @Override
    Stream<? extends ExternalAuditedField<?, ?>> externalMandatoryFields() {
        return Stream.of(new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET).build(), 
                         new ExternalAuditedField.Builder<>(AdgroupEntity.NAME).build());
    }
}

@Audited(extensions = AdAuditExtensions.class)
public class AdEntity extends AbstractEntityType<AdEntity> {

    // ...
}

Customizing names and values

Customizing an Entity Name

By default, the name of the entity (entity type) in the AuditRecord will be the PL name (passed to the AbstractEntityType constructor).
If you want to use a different name for auditing, use the name attribute of @Audited at the entity level.
For example, to use the name "mySpecialAd" for AdEntity, it should be defined as follows:

@Audited(name = "mySpecialAd")
public class AdEntity extends AbstractEntityType<AdEntity> {
   // ...
}

Customizing a Field Name (from the same entity)

By default, the name of a field in the AuditRecord will be the name of the field member in the EntityType, obtained by reflection.
To use a different name for auditing, use the name attribute of @Audited at the field level.
For example, to use the name "mySpecialUrl" for AdEntity.URL, it should be defined as follows:

@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {

    @Audited(name = "mySpecialUrl")
    public static final EntityField<AdEntity, String>     URL = INSTANCE.field(AdCreative.TABLE.display_url);
}

Customizing an External Mandatory Field Name

By default, the name of an external mandatory field in the AuditRecord will be the name of the field member in the (external) EntityType, obtained by reflection.
To use a different name for auditing, pass the name to the corresponding ExternalAuditedField in your implementation of AuditExtensions. Following the example from the "external mandatory" section, suppose we want to override the name of the CampaignEntity.ID_IN_TARGET to "mySpecialIdInTarget".
It can be done as follows:

public class AdAuditExtensions implements AuditExtensions {
    @Override
    Stream<? extends ExternalAuditedField<?, ?>> externalMandatoryFields() {
        return Stream.of(new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET).withName("mySpecialIdInTarget").build(), 
                         new ExternalAuditedField.Builder<>(AdgroupEntity.NAME).build());
    }
}

Customizing a Field Value

By default, the value of a field in the AuditRecord will be its string representation.
If the default representation is not good enough, it can be overriden. There are two ways of doing this:

  1. Basic custom formatting:

If you wish to override the formatting with another formatter based on the value only, you can do it by defining a "string value converter" on the field.
This can be achieved in two steps:
a. Implement ValueConverter<T, String> with the formatting logic of the field.
b. Pass the converter to the field.

For example, suppose our AdEntity has a floating-point BID field, and we want to format it to 2 fractional digits.
It can be done as follows:

public class DoubleToStringValueConverter implements ValueConverter<Double, String> {

    @Override
    public String convertTo(Double value) {
        return value == null ? null : String.format("%.2f", value);
    }

    @Override
    public Double convertFrom(final String value) {
        return value == null ? null : Double.parseDouble(value);
    }

    @Override
    public Class<Double> getValueClass() {
        return Double.class;
    }
}

@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {

    public static final EntityField<AdEntity, Double> BID = INSTANCE.field(AdCreative.TABLE.bid,
                                                                           IdentityValueConverter.getInstance(Double.class),
                                                                           new DoubleToStringValueConverter(),
                                                                           Objects::equals);

    //...

}

NOTE there are 2 additional params passed to the field() method which are unrelated. It is a technical detail - they are required for this specific overload since there is no shorter one available.

  1. Advanced custom formatting

In some cases the formatting you need will not be simple. For example, you may wish to format the values according to some logic depending on the entity or field attributes, and not just the values themselves.
This can be achieved in two steps:
a. Implement AuditFieldValueFormatter containing the formatting logic.
b. Pass your implementation class as a parameter to the 'valueFormatter' attribute of @Audited.
Note that this can be done either on the field level or on the entity level. If it is at the entity level - it will apply to all fields of the entity, unless a field also has a formatter - in which case it will override the entity-level one.

As a simple example, suppose you want to format the value by placing the string representation of the field before the value.
You can define the formatter like this:

public class MyValueFormatter implements AuditFieldValueFormatter {
    
    public <T> String format(final EntityField<?, T> field, final T value) {
        return String.format("field '%s' with value '%s'", field, value);
    }
}

Now you have three ways of applying the formatter:

Scenario 1 - Applying the formatter at the entity level:

@Audited(valueFormatter = MyValueFormatter.class)
public class MyEntity extends AbstractType<MyEntity> {
      // ....
}

Scenario 2 - Applying the formatter at the field level:

@Audited
public class MyEntity extends AbstractType<MyEntity> {
      // ...

      @Audited(valueFormatter = MyValueFormatter.class)
      public static final EntityField<MyEntity, String> NAME = INSTANCE.field(MyTable.INSTANCE.name);
}

Scenario 3 - Applying the formatter at the field level, to override another one at the entity level:

@Audited(valueFormatter = SomeOtherFormatter.class)
public class MyEntity extends AbstractType<MyEntity> {
      // ...

      @Audited(valueFormatter = MyValueFormatter.class)
      public static final EntityField<MyEntity, String> NAME = INSTANCE.field(MyTable.INSTANCE.name);
}

In this scenario, MyValueFormatter will be applied to the NAME field despite the fact that SomeOtherFormatter is applied to the others.

Customizing an External Mandatory Field Value

External mandatory field values (see section above) can also be customized, in a simple way.
You just need to implement a value formatter as described in the previous section, and set it onto the ExternalAuditedField when you build it.
Suppose you want to override the value formatter of the external field CampaignEntity.ID_IN_TARGET to be MyValueFormatter:

new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET)
        .withValueFormatter(new MyValueFormatter())
        .build();

Entity Change Description

Sometimes it will not be enough to send just the changes themselves, and you would like to record some additional information (at the entity level) about the changes.
This information typically will not come from the entity itself, but is known to the system which initiated the change. For example:

  • Who initiated the change - was it manual or automatic?
  • Why was the change performed? (client request, scheduled action, optimization algorithm, etc.)

To support such information, you can optionally provide an additional description to the audit record.
This is accomplished by setting a dedicated TransientProperty AuditProperties.ENTITY_CHANGE_DESCRIPTION on the command, for example:

var cmd = new CreateAdCommand();
cmd.set(NAME, "newName");
cmd.set(AuditProperties.ENTITY_CHANGE_DESCRIPTION, "Changed due to a scheduled nightly action");

This description will be stored in the AuditRecord in the field "entity_change_description" which was described above.

Special Cases

There are some special cases where fields will not be audited, even if the relevant annotations are in place. These are:

  • When the operation is CREATE and a field is not populated, or populated with a null value. This is because it would be redundant - you can easily tell from the fact that the field is missing, that it was created empty.
  • Virtual fields. A virtual field is a special type of field which can be added to the PL entity and is dynamically created based on values of other (regular / real) fields. Since the regular fields can be audited, the auditing of the virtual field is redundant and just adds "noise" to the audit data - so it is skipped.

Examples

TBD

Clone this wiki locally