Rules Engines

Skeptical? You should be. Taking a rule-based approach to logic has serious implications for performance, quality and manageability.

And, let's be honest, rules engines have gotten a bit of a bad rap for transaction logic, mainly for performance. That's because there are different types of rules engines (process, decision and transaction), and only the latter meets the transaction requirements summarized below.

Empirical Experience - Transaction Rules

In fact, Transaction Rule engines have a proven track record of success:

• Wang Labs - the PACE DBMS enforced these rules, highly valued by over 6500 customers, and was regarded as one of Wang's top products

• Versata - a J2EE startup backed by the founders of Microsoft, SAP, Ingres and Informix, Versata had a 2000 IPO of $3.4B, with over 700 customers

  • These systems tending to be large (typically 200-800 tables), with high demands for throughput and complexity

  • Studies of a dozen systems indicated over 97% of the backend logic was automated by rules, and that each rule represented about a page of code (generally consistent with our 5 vs. 200 example)

  • So, to put an appropriately fine point on it:

    • Backends are nearly half the project

    • Rules can automate over 95% of your logic

    • With a compression factor of 40:1

    • And enable extensibility and manageability with Python

• CA Live API Creator - over 100 customers, where rules were integrated with API creation and execution.

This implementation brings the same technology to the Python / open source community.

Key Differences in Rules Engines: Process, Decision, Transaction

Let's consider these rule technologies in the light of the following requirements:

Transaction Logic Requirements

Category	Consideration
Scalable	Rules should minimize SQL overhead, automatically
Extensible	Developers must be able to extend rule-based logic
Manageable	Developers must be able to use existing developer tools and procedures for code editing, debugging, code management, etc
Integrity	The system should ensure that all the rules are consistently enforced, in all cases
Architecture	Logic should automatically enforced over all apps and APIs

Process Rules

Such rules are typically graphs governing problem areas such as work flow and data flow integration. These are completely appropriate applications.

Process rules are not well suited to highly interdependent transaction logic:

• they are user ordered, so logic changes require the diagrams be redesigned
• they are not concise - a "flowchart" of thousands of lines of procedural logic is actually less readable than code.

Transaction and process rules are synergistic:

• process diagrams may need to update rows, leveraging transaction logic
• transaction logic may need to start processes ("start order process"), or resume them ("order is approved - execute next steps").

Decision Rules

Decision rules look virtually identical to transaction rules - a set of chained derivations. The difference is in the fundamental interface:

• logic engine processes rows changed in a transaction
  • logic operation begins by obtaining old values of these rows
  • these enable pruning and sql optimizations
  • old_rows also enable state transition semantics
    • e.g., all raises must exceed 10% (one of our favorite rules)
• decision logic processes an array of objects (rows), and the name of the RuleSet to run

Decision engines cannot make presumptions about old rows, so when they encounter a rule like balance is sum of order amounts, it has no choice but to read all the Orders (and each of their OrderDetails).

This is fine for a single-user "what-if" request. But for multi-user transaction processing, this can reduce performance by multiple orders of magnitude.

We also note that decision logic is explicitly called. That means that you need to audit all of the accessing code to verify the logic is enforced. Transaction logic, by contrast, ensures that all sqlalchemy access enforces the logic.

That said, these technologies are also synergistic:

• You can invoke Decision Logic using Python in transaction logic rules

Transaction Rules

This implementation is a Transaction Rules Engine: rule execution is bound into update processing.

Category	Consideration	Transaction Logic
Scalable	Rules should minimize SQL overhead, automatically	Old row access enables pruning and sql optimizations
Extensible	Developers must be able to extend rule-based logic	Many rules (events, constraints) invoke Python, providing access to all that entails
Manageable	Developers must be able to use existing developer tools and procedures for code editing, debugging, code management, etc	Rules are Python code - use standard editors (with code completion), debuggers, and source code control systems and procedures
Integrity	The system should ensure that all the rules are consistently enforced, in all cases	All ORM access enforces the rules
Architecture	Logic should automatically enforced over all apps and APIs	Logic enforcement is factored out of UI controllers, so shared over all apps and APIs