-
Notifications
You must be signed in to change notification settings - Fork 2
Auditing
Many applications need to audit their DB changes, so they can review the history of changes ("change log") done to each table - whether it is for financial reasons, troubleshooting and more. To handle all this, several frameworks exist depending on the type of DB and programming language you are using. If you are using persistence-layer, and you also need one of these tools - it could be cumbersome and time-consuming to combine the two.
To solve this issue - persistence-layer has its own built-in auditing support, which will be described in detail in the next sections. For now, it's important to mention one thing: PL is not opinionated, meaning it will not force you to persist your audit data to a specific storage, or even to persist it at all. Instead - it just helps you configure which data should be audited, and then lets you define your own logic for publishing the data to some other service or external system that can handle it.
We'll start off with a simple example of auditing.
Let's say we have set up the Entity Persistence according to the Entity Persistence page.
First we need to add two annotations to our AdEntity
:
- @Audited - at the class level, indicating that all changes to the fields of our entity will be audited
- @Id - on the unique id (usually primary key) that we want to use for identifying our entity together with the changes.
Our AdEntity
will now look like this:
@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);
public static final EntityField<AdEntity, String> URL = INSTANCE.field(AdCreative.TABLE.display_url);
}
Next, we need to implement a publisher for our audit data.
As explained above, PL does not force you to save your audit data in specific tables/storage, and in fact doesn't force you to do anything with it at all!
All PL will do, is collect the DB changes that occurred during the execution of your commands, and place them in instances of AuditRecord
- one for each command (The AuditRecord
class will be explained in detail in the next section).
These AuditRecord
-s are available after your command has finished executing, and they are exposed through the interface AuditRecordPublisher
that we need to implement. PL will call this publisher immediately after the commands have been executed.
In a real application, your publisher might save the AuditRecord
-s to some database, or send them to some other system for processing. For our simple example, we will implement a publisher that simply converts the AuditRecord
into JSON and prints it:
import com.fasterxml.jackson.databind.ObjectMapper;
public JsonAuditRecordPublisher extends AuditRecordPublisher {
private final ObjectMapper objectMapper = new ObjectMapper();
@Override
public void publish(@NonNull final Stream<? extends AuditRecord<?>> auditRecords) {
final var auditRecordsStr = objectMapper.writeValueAsString(auditRecords);
System.out.println(auditRecordsStr);
}
}
The last step is to plug our publisher into the PLContext
we already created, so that PL will know about it and invoke it:
public PLContext plContext() {
DSLContext jooq = DSL.using(new DefaultConfiguration()
.set(SQLDialect.MYSQL)
.set(new ThreadLocalTransactionProvider(/* and your JDBC connection provider */)));
return new PLContext.Builder(jooq)
.withAuditRecordPublisher(new JsonAuditRecordPublisher())
.build();
}
var cmd = new CreateAdCommand();
cmd.set(AdEntity.SYNC_STATUS, SyncStatus.APPROVED);
cmd.set(AdEntity.URL, "http://mysite.com");
var adPersistence = new AdPersistence(plContext);
adPersistence.create(List.of(cmd));
This will produce the following output from the publisher:
(NOTE: the default field names provide by PL are slightly different, we will assume they were customized - more on this below)
{
"entityType": "ad",
"entityId": "<<auto generated id>>",
"operator": "CREATE",
"fieldRecords": [
{
"field": "syncStatus",
"newValue": "APPROVED"
},
{
"field": "url",
"newValue": "http://mysite.com"
}
]
}
PL will store the changes from each command in an AuditRecord
, which contains the following fields:
- entityType : the type of entity being audited
- entityId : the unique id that will be used to identify the entity along with its changes. This is not required, as some entities may not have a single unique id. In that case, they should include some other fields to uniquely identify the entity as part of the mandatoryFieldValues (see below).
- mandatoryFieldValues : A collection of field/value pairs, for mandatory fields (special fields which should always appear, even if unchanged - this is explained in more detail in the next section).
-
operator : Either
CREATE
,UPDATE
orDELETE
according to the type of command being audited. If it is anInsertOnDuplicateCommand
("upsert") then PL will automatically determine during the command execution if it should create or update, so the resulting operator will be determined accordingly. - entityChangeDescription : An optional description of the change itself, e.g. the reason for performing the change. It can be populated in the command (explained in more detail below).
-
fieldRecords : A collection containing all the field changes performed by the command. Each change has the following fields:
- field - the field that was changed
-
oldValue - the old value, if
operator
isDELETE
orUPDATE
(otherwisenull
) -
newValue - the new value, if
operator
isCREATE
orUPDATE
(otherwisenull
)
-
childRecords : A collection containing changes to children of this entity, in case there are child commands executed along with the current (parent) command. This field is recursive - meaning that each child record is itself an
AuditRecord
with the same fields as above (See also the Examples section below).
This section will explain in detail what options you have for configuring the fields of your audited entity.
As in the quick start example above you must choose a unique id of the entity, that should appear in the AuditRecord (usually the primary key). Annotate this field with @Id
in your entity. Here's a reminder of how it will look:
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
// other fields
}
NOTE: There is a limitation currently, that only one field may be annotated with @Id
and it also must be numeric.
We plan to remove these limitations in the future.
As in the quick start above, annotating the entity class with @Audited
will cause all field changes to be audited:
@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
// other fields
}
Suppose you only want to audit a few specific fields. In that case you should drop the entity-level annotation and place it on the desired fields only.
In the example below, only SYNC_STATUS
and URL
will be audited:
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
@Audited
public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);
public static final EntityField<AdEntity, String> HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);
@Audited
public static final EntityField<AdEntity, String> URL = INSTANCE.field(AdCreative.TABLE.display_url);
public static final EntityField<AdEntity, String> BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);
public static final EntityField<AdEntity, String> IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);
public static final EntityField<AdEntity, Integer> IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}
Suppose you want to audit all fields except a few specific ones. In that case you should place the entity-level annotation, but also add the @NotAudited
annotation on the desired fields to be excluded.
In the example below, all fields except for SYNC_STATUS
and URL
will be audited:
@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
@NotAudited
public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);
public static final EntityField<AdEntity, String> HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);
@NotAudited
public static final EntityField<AdEntity, String> URL = INSTANCE.field(AdCreative.TABLE.display_url);
public static final EntityField<AdEntity, String> BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);
public static final EntityField<AdEntity, String> IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);
public static final EntityField<AdEntity, Integer> IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}
Mandatory fields are special fields that can be added to the AuditRecord
and they will appear always in a separate section, whether they have changed or not.
These fields can either come from the entity itself or from external entities, as long as they are linked by some chain of DB keys to the original entity.
The purpose of including such fields could be:
- To provide a unique identifier (unique key) of the entity, in case there is no single unique id that can populated in the
entity_id
- To provide additional descriptions of the entity, as well as the hierarchy in which it resides
- To provide additional classifications/tags of the entity, that will make it easier to search and filter changes that you want to collect together.
To add a mandatory fields from the same entity, use the annotation @Audited(AuditTrigger.ALWAYS)
.
Note that these fields could appear twice in the AuditRecord
: Once in the mandatory section, and once more in the changes section if they have also changed.
In the example below, HEADLINE
and SYNC_STATUS
are mandatory fields, while URL
and IMAGE_NAME
are (regular) audited fields:
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final AdEntity INSTANCE = new AdEntity();
private AdEntity() { super("ad"); }
@Override public DataTable getPrimaryTable() { return AdCreative.TABLE; }
@Id
@Immutable
public static final EntityField<AdEntity, Integer> ID = INSTANCE.field(AdCreative.TABLE.creative_id);
@Audited(AuditTrigger.ALWAYS)
public static final EntityField<AdEntity, SyncStatus> SYNC_STATUS = INSTANCE.field(AdCreative.TABLE.status);
@Audited(AuditTrigger.ALWAYS)
public static final EntityField<AdEntity, String> HEADLINE = INSTANCE.field(AdCreative.TABLE.headline);
@Audited
public static final EntityField<AdEntity, String> URL = INSTANCE.field(AdCreative.TABLE.display_url);
public static final EntityField<AdEntity, String> BUSINESS_NAME = INSTANCE.field(AdCreative.TABLE.business_name);
@Audited
public static final EntityField<AdEntity, String> IMAGE_NAME = INSTANCE.field(AdCreative.TABLE.image_name);
public static final EntityField<AdEntity, Integer> IMAGE_SIZE = INSTANCE.field(AdCreative.TABLE.image_size);
}
To include mandatory fields from external entities, you'll need to do the following:
- Make sure that PL is able to fetch all the fields you need from all the entities. This is explained in detail in the "State Consumers and Relations" section.
- Implement the interface
AuditExtensions
, providing the fields you want to include as mandatory. For this example let's say the impl is namedMyAuditExtensions
- Annotate your entity with
@Audited(extensions = MyAuditExtensions.class)
.
In the example below, AdEntity
defines the external mandatory fields AdgroupEntity.NAME
and CampaignEntity.ID_IN_TARGET
:
public class CampaignEntity extends AbstractEntityType<CampaignEntity> {
public static final EntityField<CampaignEntity, String> ID_IN_TARGET = INSTANCE.field(CustomerCampaigns.TABLE.name);
// ...
}
public class AdgroupEntity extends AbstractEntityType<AdgroupEntity> {
public static final EntityField<AdGroupEntity, String> NAME = INSTANCE.field(CampaignAds.TABLE.name);
// ...
}
public class AdAuditExtensions implements AuditExtensions {
@Override
Stream<? extends ExternalAuditedField<?, ?>> externalMandatoryFields() {
return Stream.of(new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET).build(),
new ExternalAuditedField.Builder<>(AdgroupEntity.NAME).build());
}
}
@Audited(extensions = AdAuditExtensions.class)
public class AdEntity extends AbstractEntityType<AdEntity> {
// ...
}
By default, the name of the entity (entity type) in the AuditRecord
will be the PL name (passed to the AbstractEntityType
constructor).
If you want to use a different name for auditing, use the name
attribute of @Audited
at the entity level.
For example, to use the name "mySpecialAd" for AdEntity
, it should be defined as follows:
@Audited(name = "mySpecialAd")
public class AdEntity extends AbstractEntityType<AdEntity> {
// ...
}
By default, the name of a field in the AuditRecord
will be the name of the field member in the EntityType
, obtained by reflection.
To use a different name for auditing, use the name
attribute of @Audited
at the field level.
For example, to use the name "mySpecialUrl" for AdEntity.URL
, it should be defined as follows:
@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {
@Audited(name = "mySpecialUrl")
public static final EntityField<AdEntity, String> URL = INSTANCE.field(AdCreative.TABLE.display_url);
}
By default, the name of an external mandatory field in the AuditRecord
will be the name of the field member in the (external) EntityType
, obtained by reflection.
To use a different name for auditing, pass the name to the corresponding ExternalAuditedField
in your implementation of AuditExtensions
.
Following the example from the "external mandatory" section, suppose we want to override the name of the CampaignEntity.ID_IN_TARGET
to "mySpecialIdInTarget".
It can be done as follows:
public class AdAuditExtensions implements AuditExtensions {
@Override
Stream<? extends ExternalAuditedField<?, ?>> externalMandatoryFields() {
return Stream.of(new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET).withName("mySpecialIdInTarget").build(),
new ExternalAuditedField.Builder<>(AdgroupEntity.NAME).build());
}
}
By default, the value of a field in the AuditRecord
will be its string representation.
If the default representation is not good enough, it can be overriden. There are two ways of doing this:
- Basic custom formatting:
If you wish to override the formatting with another formatter based on the value only, you can do it by defining a "string value converter" on the field.
This can be achieved in two steps:
a. Implement ValueConverter<T, String>
with the formatting logic of the field.
b. Pass the converter to the field.
For example, suppose our AdEntity
has a floating-point BID
field, and we want to format it to 2 fractional digits.
It can be done as follows:
public class DoubleToStringValueConverter implements ValueConverter<Double, String> {
@Override
public String convertTo(Double value) {
return value == null ? null : String.format("%.2f", value);
}
@Override
public Double convertFrom(final String value) {
return value == null ? null : Double.parseDouble(value);
}
@Override
public Class<Double> getValueClass() {
return Double.class;
}
}
@Audited
public class AdEntity extends AbstractEntityType<AdEntity> {
public static final EntityField<AdEntity, Double> BID = INSTANCE.field(AdCreative.TABLE.bid,
IdentityValueConverter.getInstance(Double.class),
new DoubleToStringValueConverter(),
Objects::equals);
//...
}
NOTE there are 2 additional params passed to the field()
method which are unrelated. It is a technical detail - they are required for this specific overload since there is no shorter one available.
- Advanced custom formatting
In some cases the formatting you need will not be simple. For example, you may wish to format the values according to some logic depending on the entity or field attributes, and not just the values themselves.
This can be achieved in two steps:
a. Implement AuditFieldValueFormatter
containing the formatting logic.
b. Pass your implementation class as a parameter to the 'valueFormatter' attribute of @Audited
.
Note that this can be done either on the field level or on the entity level. If it is at the entity level - it will apply to all fields of the entity, unless a field also has a formatter - in which case it will override the entity-level one.
As a simple example, suppose you want to format the value by placing the string representation of the field before the value.
You can define the formatter like this:
public class MyValueFormatter implements AuditFieldValueFormatter {
public <T> String format(final EntityField<?, T> field, final T value) {
return String.format("field '%s' with value '%s'", field, value);
}
}
Now you have three ways of applying the formatter:
Scenario 1 - Applying the formatter at the entity level:
@Audited(valueFormatter = MyValueFormatter.class)
public class MyEntity extends AbstractType<MyEntity> {
// ....
}
Scenario 2 - Applying the formatter at the field level:
@Audited
public class MyEntity extends AbstractType<MyEntity> {
// ...
@Audited(valueFormatter = MyValueFormatter.class)
public static final EntityField<MyEntity, String> NAME = INSTANCE.field(MyTable.INSTANCE.name);
}
Scenario 3 - Applying the formatter at the field level, to override another one at the entity level:
@Audited(valueFormatter = SomeOtherFormatter.class)
public class MyEntity extends AbstractType<MyEntity> {
// ...
@Audited(valueFormatter = MyValueFormatter.class)
public static final EntityField<MyEntity, String> NAME = INSTANCE.field(MyTable.INSTANCE.name);
}
In this scenario, MyValueFormatter
will be applied to the NAME
field despite the fact that SomeOtherFormatter
is applied to the others.
External mandatory field values (see section above) can also be customized, in a simple way.
You just need to implement a value formatter as described in the previous section, and set it onto the ExternalAuditedField
when you build it.
Suppose you want to override the value formatter of the external field CampaignEntity.ID_IN_TARGET
to be MyValueFormatter
:
new ExternalAuditedField.Builder<>(CampaignEntity.ID_IN_TARGET)
.withValueFormatter(new MyValueFormatter())
.build();
Sometimes it will not be enough to send just the changes themselves, and you would like to record some additional information (at the entity level) about the changes.
This information typically will not come from the entity itself, but is known to the system which initiated the change. For example:
- Who initiated the change - was it manual or automatic?
- Why was the change performed? (client request, scheduled action, optimization algorithm, etc.)
To support such information, you can optionally provide an additional description to the audit record.
This is accomplished by setting a dedicated TransientProperty AuditProperties.ENTITY_CHANGE_DESCRIPTION
on the command, for example:
var cmd = new CreateAdCommand();
cmd.set(NAME, "newName");
cmd.set(AuditProperties.ENTITY_CHANGE_DESCRIPTION, "Changed due to a scheduled nightly action");
This description will be stored in the AuditRecord
in the field "entity_change_description" which was described above.
There are some special cases where fields will not be audited, even if the relevant annotations are in place. These are:
- When the operation is
CREATE
and a field is not populated, or populated with anull
value. This is because it would be redundant - you can easily tell from the fact that the field is missing, that it was created empty. - Virtual fields. A virtual field is a special type of field which can be added to the PL entity and is dynamically created based on values of other (regular / real) fields. Since the regular fields can be audited, the auditing of the virtual field is redundant and just adds "noise" to the audit data - so it is skipped.
TBD
Setting up an Entity Persistence
Query language
Building the Flow
- Adding Simple Validators
- State Consumers
- Adding a Custom Validator
- Enrichers
- Output Generators
- Customizing Flows
- Primary and Secondary Tables (detailed example)
- State Consumers and Relations (what can we fetch)
- Child Commands
- Cascade Delete