Making entities transactable

[den1k]

(This feature would mark a diversion from the datascript/datomic API.)

I've always wondered: why are entities not transactable? I find myself converting entities to maps all the time solely to transact them. This still causes problems when entities nest other entities. So here are a few simple ideas on how entities could be treated in transactions:

1. Entities could be treated as refs in transactions

(def schema
  {:user/friends #:db{:valueType   :db.type/ref
                      :cardinality :db.cardinality/many}})

(def ent (d/touch (d/entity @conn 1)))

ent ; eval
;; =>
{:db/id 1
 :user/email "foo@example.com"
 :user/friends #{{:db/id 2} {:db/id 3}}} ; <-- nested entities

Now I convert it to a map

(def ent-map (into {:db/id (:db/id ent)} ent))
ent-map ; eval
;; =>
{:db/id 1
 :user/email "foo@example.com"
 :user/friends #{{:db/id 2} {:db/id 3}}}
;; looks the same but nested entities (under :user/friends) have not been converted

I try to transact it

(d/transact! conn [(assoc ent-map :user/email "BAR@example.com")])
;; throws:
;; Execution error (ExceptionInfo) at datalevin.db/entid (db.cljc:385).
;; Expected number or lookup ref for entity id, got #:db{:id 2}

So I can either dissoc the :user/friends map-entry or convert contained entities to refs

(d/transact! conn [(-> ent-map
                       (dissoc :genstyle.project/population)
                       ;; OR (update :user/friends #(mapv :db/id %)) 
                       (assoc :user/email "BAR@example.com"))])

We could spare ourselves from this by treating entities as refs in transactions. The database already walks nested data-structures to resolve refs so why not resolve entities as refs, also?

2. Entities to return maps on update

datalevin.impl.entity/Entity implements clojure.lang.Associative which currently only throws errors:

clojure.lang.Associative
       ;; some methods elided
       (empty [e]         (throw (UnsupportedOperationException.)))
       (assoc [e k v]     (throw (UnsupportedOperationException.)))
       (cons  [e [k v]]   (throw (UnsupportedOperationException.)))

Instead assoc could return a hashmap

(deftype Entity [db eid touched cache]
  ;; elided
  clojure.lang.Associative
  (assoc [e k v]
    (let [e-map (cond-> {:db/id eid}
                  ; add other kvals if touched
                  touched (into e))]
     (assoc e-map k v))))

This would also make update possible. Together this means that the change of email to ent from above, could look like this:

(d/transact! conn [(assoc ent :user/email "BAR@example.com")])

I would've already implemented this for my own projects but unfortunately Clojure (unlike ClojureScript) doesn't allow to overwrite a Type's methods. To achieve this one would have to for Datalevin and change the definition of datalevin.impl.entity/Entity so I wanted to raise the issue here first and see what @huahaiy's thoughts are.

[den1k]

This also possibly unlocks use of libraries like specter or meander to update entities.

[huahaiy]

Sounds like a good idea. Could you elaborate a bit more on how specter or meander could be used to update entities if the change is made?

[den1k]

Could you elaborate a bit more on how specter or meander could be used to update entities if the change is made?

Here's a basic example with specter based on the entities above:

(use 'com.rpl.specter)

(def ent-as-map-recursive
  {:db/id        1
   :user/email   "foo@example.com"
   :user/friends #{{:db/id      2
                    :user/email "ava@example.com"} {:db/id 3}}})

(setval [:user/friends ALL :user/email (pred #(= % "ava@example.com"))]
        "jenny@example.com"
        ent-as-map-recursive)

; =>
{:db/id 1,
 :user/email "foo@example.com",
 :user/friends #{{:db/id 2, :user/email "jenny@example.com"} {:db/id 3, :user/email nil}}}


(setval [:user/friends ALL :user/email (pred #(= % "ava@example.com"))]
        "jenny@example.com"
        (d/entity @conn 1)))
; => 
; Execution error (UnsupportedOperationException) at datalevin.impl.entity.Entity/assoc (entity.cljc:47).
; null

[huahaiy]

I wonder how to deal with deeply nested, or even recursively nested entities?

[den1k]

Yes, I started pondering that as well. I actually think that turning the Entity into a map is a bad idea. But we could possible extend the Entity type to hold a change-set of updates to the entity similar to the cache of touched.

The key is that the, sic, key is known, e.g. for
(update (d/entity @conn 1) :user/email str/upper-case) we know that only the value for :user/email changed, so we can translate that into [:db/add 1 :user/email "FOO@EXAMPLE.COM] or {:db/id 1 :user/email "FOO@EXAMPLE.COM} for the transaction, while ignoring unchanged keys and values.

This would be similar for nested values. Non-updated entities are treated as eids and updated ones contain a minimal change-set derived from assoc/update calls.

[den1k]

@huahaiy adapted Entity to allow assoc, dissoc, add and retract. This works by adding a immutable "stage" to Entity that does nothing until the Entity is transacted. For example:

(-> (entity @conn [:user/handle "ava"])
    (assoc :user/age 42))

; => {:db/id 1, :<STAGED> #:user{:age [{:op :assoc} 42]}}

I like this interface because it is fully backwards compatible. Entity still works read-only but can optionally act as an immutable stage until transacted.
A lot of the code has been copied from datalevin.impl.entity to match the new Entity type. However, there are maybe only 50 lines of code that are actually new.

Here's a thorough API example:

(def db-path "data/lab/entity-db")

(def schema
  {:user/handle    #:db {:valueType :db.type/string
                         :unique    :db.unique/identity}
   :user/address   #:db{:valueType   :db.type/ref
                        :cardinality :db.cardinality/one}
   :address/street #:db{:valueType :db.type/string}
   :user/friends   #:db{:valueType   :db.type/ref
                        :cardinality :db.cardinality/many}})

(def conn
  (d/create-conn db-path schema))

(transact! conn [{:user/handle  "ava"
                  :user/friends [{:user/handle "fred"}
                                 {:user/handle "jane"}]}])

;; *** Simple example
(let [ava-with-age (-> (entity @conn [:user/handle "ava"])
                       (assoc :user/age 42))]
  #spy/c (entities->txs [ava-with-age])
  ;; => [[:db/add 1 :user/age 42]]
  (transact! conn [ava-with-age])
  )


;; *** Nested entities must be transacted separately
(let [{:keys [user/friends] :as ava}
      (update (entity @conn [:user/handle "ava"]) :user/age inc)
      fred   (some
               #(when (= (:user/handle %) "fred") %)
               friends)
      bestie (assoc fred :bestie? true)]
  #spy/c (entities->txs [ava bestie])
  ;; => [[:db/add 1 :user/age 43] [:db/add 2 :bestie? true]]
  (transact! conn [ava bestie]))

;; *** `add` and `retract` are directly defined on entity
;; they differ from assoc/dissoc in that they do not overwrite
;; the attr's values
(let [ava  (entity @conn [:user/handle "ava"])
      fred (some
             #(when (= (:user/handle %) "fred") %)
             (:user/friends ava))]
  #spy/c (entities->txs [(retract ava :user/friends fred)])
  ;; => [[:db/retract 1 :user/friends 2]]
  (transact! conn [(retract ava :user/friends fred)])
  )

All the code is here: https://github.com/den1k/stuffs/blob/main/src/stuffs/datalevin/entity.clj
Thoughts?

[huahaiy]

I am all for programmer convenience (that's the whole point of this project), so I think this could be a nice addition. Would you mind sending a PR with some tests as well. Thanks a lot!