Datomic

Datomic: Querying for the history of an entity (v2)

2019-06-11T00:00:00.000+02:00

This is the updated and rewritten version of an old classic.

Intro: how is this even possible?

Datomic makes it easy to implement an "audit log" of changes over time to anything in your system.

You could implement your own. But when you have Datomic, you are one abstraction level above other databases. And there are many ways to get that wrong. So you don't have to think of this when modelling your data. The power is already there. You just have to know how to query the data out of Datomic.

The history database

Normally, you transact facts into your database, and your schema determines what the "current view" of the database is like.

Let's say we have the following history of transactions:

;; The three transactions that we write to the database

;; tx 1
[[:db/add 1 :user/email "august@augustl.com"] [:db/add 1 :user/password "hash-here"]]

;; tx 2
[[:db/add 1 :user/subscribed-to-newsletter? true]]

;; tx 3
[[:db/add 1 :user/email "august@kodemaker.no"]]

We pull out a view of the database:

;; Set up
(require '[datomic.api :as d])

(def my-db (d/db my-conn))

(def my-eid 123654) ;; The assumed entity ID created above (`[:db/add 1 ...]`)

When you query my-db, you'll see a "current view" of the data, without any history. If you ask the db for the :user/email of user `1`` you'll get:

;; Just "latest data"
(d/q '[:find ?email :in $db ?eid :where [?eid :user/email ?email]] my-db my-eid)
;; => returns
#{["august@kodemaker.no"]}

However, you can also ask for the "history" db. If you ask that for the value of the :user/email of user `1`` you'll get:

;; All historical data, just like that!
(def my-history-db (d/history my-db))

(d/q '[:find ?email :in $db ?eid :where [?eid :user/email ?email]] my-history-db my-eid)
;; => returns
#{["august@augustl.com"] ["august@kodemaker.no"]}

The "history" db is a special view of the database that doesn't filter out previously added data.

Inspect the history

All Datomic facts contains information about which transaction the fact was added in. This is particularly useful for querying the history database.

This query will return a list of all changes made to the entity`, with information about the transaction in where it was added, the attribute in question, and whether that attribute was added or removed in that transaction.

;; All changes made to entity my-eid
(def res
  (d/q
    '[:find ?tx ?attr ?added
      :in $ ?e
      :where
      [?e ?attr _ ?tx ?added]]
    (d/history db)
    my-eid))
;; => returns
#{[13194139534318 63 true]
  [13194139534313 63 true]
  [13194139534317 65 true]
  [13194139534313 64 true]
  [13194139534318 63 false]}

This data structure is pretty bare metal. What follows are some examples of how to use it.

Making sense of the daa

We don't really do anything here other than restructuring the data to make it look more intuitive.

;; More prettier
(def txs
  (->> res
       (group-by first)
       (map (fn [[k v]] [k (map (fn [[tx attr added]] {:attr attr :added? added}) v)]))
       (sort-by (fn [[k v]] k))))
;; => returns
([13194139534313 ({:attr 63, :added? true} {:attr 64, :added? true})]
 [13194139534317 ({:attr 65, :added? true})]
 [13194139534318 ({:attr 63, :added? true} {:attr 63, :added? false})])

Here we se the three transactions we did above. The first transaction contains the two "added" items, the email and the password. The second transaction contains the added newsletter subscription. The third transaction contains the removed old e-mail and the added new e-mail.

You can use this query power to do lots of neat things.

It also exposes an interesting detail about Datomic internals. Attributes themselves (?attr) are also represented entities in the Datomic database, and can be queried fully. As well as the transactions themselves, of course.

Generate a list of entities before/after, whith a timestamp

Our raw data is not that interesting, so let's do something about that.

;; List of changes, with the full entity as of before and after the change was made
(->> res
     (map first)
     (set)
     (sort)
     (map
       (fn [tx-eid]
         (let [tx (d/entity my-db tx-eid)]
           {:timestamp (:db/txInstant tx)
            :ent-before (d/entity (d/as-of my-db (dec (d/tx->t tx-eid))) my-eid)
            :ent-after (d/entity (d/as-of my-db tx-eid) my-eid))})))

Here's what we get back

;; => returns
(
;; The first TX
{:timestamp #inst"2019-06-11T13:53:09.055-00:00",
  :ent-before #:db{:id 17592186045418},
  :ent-after {:db/id 17592186045418, 
              :user/email "august@augustl.com", 
              :user/password "hash-123-abc"}}

;; The second TX
 {:timestamp #inst"2019-06-11T13:54:12.693-00:00",
  :ent-before {:db/id 17592186045418, 
               :user/email "august@augustl.com", 
               :user/password "hash-123-abc"},
  :ent-after {:db/id 17592186045418,
              :user/email "august@augustl.com",
              :user/password "hash-123-abc",
              :user/subscribed-to-newsletter? true}}
;; The third TX
 {:timestamp #inst"2019-06-11T13:54:22.530-00:00",
  :ent-before {:db/id 17592186045418,
               :user/email "august@augustl.com",
               :user/password "hash-123-abc",
               :user/subscribed-to-newsletter? true},
  :ent-after {:db/id 17592186045418,
              :user/email "august@kodemaker.no",
              :user/password "hash-123-abc",
              :user/subscribed-to-newsletter? true}}
)

There are a couple of intersting to note here.

Note how we get what seems like a full entity with an id from :ent-before in the first tx. This is because the Datomic (d/entity) API does not actually check whether any facts exists for an entity before generating an EntityMap data structure.

The essential data structure here is just a sorted list of TX ids. We can safely sort on transaction ID, since the transaction is a monotone sequence.

We also use a special trick. We have a list of transactions, but how do we get the entity as of right before that transaction? We use (d/tx->t) to get the "T" value of the tx, i.e. its internal monotone sequence number that we can pass to "as-of". If we decrement that number, we'll get the transaction before tx. This is guaranteed by Datomic.

This is a slighly more practical version of the "real" way, which would be to map over pairs of transaction A+B, B+C, C+D etc.

Generate a list of entities with the timestamp, and a list of attributes that changed, and to which value

This is pretty simple when we have our existing query over the history db.

To make things slightly more interesting, let's add a fourth transaction, that doesn't update anything, but just deletes an attribute. Let's assume that the password expired or something like that, so we retract the hash from the database.

;; Add some more data - retract (delete/remove/wipe) an attribute

;; tx 4
[[:db/retract 1 :user/password "hash-123-abc"]]

;; Pull out reference to latest db
(def my-db (d/db my-conn))

Note that if we didn't re-assign my-db, we'd still get the old view without transaction 4, even though we use the history db. The history db is still an immutable reference to a point in time, it's just that it contains all of the history up until that point, not just the data directly visible in the index in the normal db view.

Now, let's generate the data we need.

;; Complete diff of changes made over time to an entity
(->> (d/q
       '[:find ?tx ?attr ?val ?added
         :in $ ?e
         :where
         [?e ?attr ?val ?tx ?added]]
       (d/history my-db)
       my-eid)
     (group-by first)
     (map (fn [[tx v]]
            [tx
             (->> v
                  (group-by (fn [[tx attr val added]] attr))
                  (map (fn [[attr changes]]
                         [(d/ident my-db attr)
                          (map (fn [[tx attr val added?]] {:val val :added? added?})
                               changes)])))]))
     (sort-by (fn [[tx v]] tx)))

When we look at the data, we can see that we have everything we need in order to generate a useful GUI.

;; => returns
(
 ;; tx 1
 [13194139534313 (
   [:user/password ({:val "hash-123-abc", :added? true})] 
   [:user/email ({:val "august@augustl.com", :added? true})])]
  
 ;; tx 2
 [13194139534317 (
   [:user/subscribed-to-newsletter? ({:val true, :added? true})])]
 
 ;; tx 3
 [13194139534318 (
   [:user/email ({:val "august@kodemaker.no", :added? true}
                 {:val "august@augustl.com", :added? false})])]
  
 ;; tx 4
 [13194139534319 (
   [:user/email ({:val "august@kodemaker.no", :added? false})])])

Look at all the things I'm not doing

If you don't use Datomic, you can always implement this yourself. And figure out all the wrong ways to do it before you end up with something useful (and performant).

But what if you get this requirement when your app is already up and running in production?

Well, a buddy of mine has done just that. They did not have an audit log requirement up front. But since this is not an up-front decision you have to make, you can be completely oblivious to this, and just add it later, in the forms of queries.

Check it out! It starts around 24:00.

Datomic: Look at all the things I'm not doing! (Cont.)

2018-04-21T00:00:00.000+02:00

In a previous blog post I listed a bunch of things I'm not doing when I'm using Datomic.

This is a continuation of that post, where I list even more things that I'm not doing, based on feedback I got on the previous post.

Look at all the query sit-ups I'm not doing

Thanks, @newresa!

In a traditional database, the query engine runs on the database server.

Well, duh. Of course. That's how databases work.

Or do they?

With Datomic, the query engine is embedded into the client.

This is not to say that Datomic does something like SELECT * FROM * and iterates a bunch of maps and has a bunch of stupid if-statements and regexps to query your data. It's a proper, real query engine, based on the Datalog language, that converts a query into efficient index access and so on and so forth.

This concept is an unbelievably good idea that I will try to blog about in detail later.

There are a few rules that you thought were rules that aren't rules anymore with this architecture.

Fundamentally, the data structures in Datomic allows multiple queries to run simultaneously, without synchronization on the database server, or between the clients that are running queries. This means that queries are trivial to scale horizontally - you just add another instance of your app, and it'll run more queries in isolation, with no effect on the performance of other queries. The only bottleneck here is fetching data from storage and into the client. You can put memcached between storage and your apps to help alleviate this.

In traditional databases, we've learned to avoid N+1 queries. To list projects and count the todo items for each project, we do NOT run one query to list the projects, and then one more query for each project to count the todos.

Why? This is not because queries are slow. Fundamentally the same amount of work has to be done by the query engine. But there will be a round-trip cost from sending the query to the database and getting the response back. This is just fundamental networking.

This is not the case with Datomic. The query engine is right there, in your application.

If you think this sounds crazy, it helps to remember that someone has to have the working set in memory. Your database server can't seek on disk when you run a query, that would just be way too slow. So why does this have to happen on the database server? Well, it hasn't. Not when you use Datomic.

So you end up designing much simpler queries with Datomic, since you don't have to express your queries as one huge statement. You can actually get the list of projects, and then count the number of todos for each project directly in the loop.

This might sound crazy. But it isn't.

Look at all the application level caching I'm not doing

On a related note, you don't need to cache the result of database queries at the application level when you use Datomic.

Kind of.

Application level caching is a common pattern to use, where you cache the result of queries. If you cache because the query is slow, Datomic can't help you. Slow queries are slow, even if the query engine is embedded in your application. However, if you're caching in order to avoid round-trips on read-heavy and cacheable queries, Datomic can help.

It should be self evident why Datomic is different here. The query engine and your working set is already in your application. So there's no need to cache what's already cached.

Thanks, val_waeselynck!

Look at all the audit logging I'm not doing

Thanks again, val_waeselynck!

Your domain has entities. In traditional databases, you typically map an entity to a row in a table or a document in a collection.

You want to be able to know how this entity changed over time. Good luck, and try to not pick one of the horrible ways of doing this. Because there are many of those.

In order to avoid having versioned entities actually being a part of your core domain, maybe you just add a separate audit log table where you dump json blobs of each version of your entity. This can end up being many different levels of slow.

With Datomic, this is just a query away.

I wrote a separate blog post about querying for the history of an entity. This is something you can just do. You don't have to plan for it or adjust your domain to support it or add audit log tables or anything.

Look at all the change detection I'm not doing

Datomic has a queue you can consume from where you get all the changes that's happening provided lazily to you in a queue. This is built right in to the Database API. You can create many of these queues, and create filters on them so you only get information about specific attributes, and so on.

There's a more general point to be made here. The reason Datomic is even able to do this, is because it is able to provide all its data structures lazily, because all the views you get of data in Datomic are immutable.

This allows you to do things like implement SSE listeners. And you can also do this in a completely generic fashion. Typically, you have to either deliberately separate the concerns between writing to the database and sending out SSE events, or you end up having SSE change code right next to the code you have for writing data. With Datomic, you can create a completely isolated section of your code, and all it has to know about is the connection object to talk to the Database. Then you set up your Datomic change listener queue and fire away.

Additionally, you don't have to figure out yourself what specifically it was that changed. The level of granularity in Datomic are facts, which refers to a single attribute of your entities in your domain. So you don't have to do things like figuring out which of the attributes changed by comparing the previous and current version of your row or document to each other.

Thanks again, again, val_waeselynck!

Look at all the database mocking/stubbing I'm not doing

The query engine in Datomic is embedded in your app.

Internally, the query engine works on two data structures: The index trees, and a log of transactions (raw transaction data) that has not yet been merged into the tree.

Datomic calls this a "live merge", and it's borrowed from Google's Bigtable. It can't just merge new data into the index trees immediately, because that merge is an expensive operation and it would dramatically lower write throughput.

Thankfully, it has been found that it's very efficient to do a "live" combination of the latest version of the index trees, and the raw transaction data that has not yet been merged into the tree. This is partly because the amount of data in the unmerged log is relatively low, since it is merged in, only periodically, not immediately after the transaction commits.

(Note: Obviously, the transaction log is persisted to disk as well, alongside the index trees.)

This means that one of the things the query engine can do, is to have a view of a database, take a list of raw transactions, and run queries as if those transactions are present in the database.

This also means that Datomic can very easily do this without actually writing to the transaction log, but instead keeping a in-memory list of "fake" transactions.

The only difference is that it's in-memory. The actual query engine performs the same operations.

This is very useful for writing tests. You can create an in-memory Datomic database, call the code that generates your transactions, use the withAPI to add these transactions to the "fake" transaction log, and run queries against it.

So you are able to test writes against your database without any mocking or stubbing at all.

Yes, you read that right.

It'll be a unit test, not an integration test, but still. Pretty cool.

Thanks again, again, again, val_waeselynck!

Look at all the up-front schema planning I'm not doing

Wait, what? Isn't this about what I'm not doing?

In my previous post, I said the following:

You create structure when you read, not when you write. Therefore, you get to come up with new ways of thinking about your data arbitrarily, after the fact. This is just a logical conclusion of having all the decisions about structure being put in the query engine, and not the write API. Contrary to traditional databases, where so much structure is defined up-front when you insert the data.

/u/dustingetz on Reddit had some very interesting insight regarding this statement.

This is a Datomic schema made by Rich Hickey himself, for Codeq, the Datomic layer on top of git:

https://github.s3.amazonaws.com/downloads/Datomic/codeq/codeq.pdf

That sure looks like a lot of up-front planning of schemas to me.

I'm not sure how to reconcile this yet. I think I'm right in asserting that you create structure when you read, not when you write, in Datomic. At least more so than in other databases. But I believe that /u/dustingetz is also right. Hopefully this will culminate into a new blog post some time in the future.

Datomic: Look at all the things I'm not doing!

2018-04-20T00:00:00.000+02:00

UPDATE: This post has a follow-up: http://augustl.com/blog/2018/datomic_look_at_all_the_things_i_am_not_doing_cont/

Original post follows.

This is the video that got me into programming:

(This embed should take you directly to 01:52, where it's all at)

It's DHH's epic first Ruby on Rails demo from 2005. His unique selling point (or one of them, at least) was "look at all the things I'm not doing"

At the time, this was ground breaking in its implementation and accessibility. "Convention over configuration" became a thing.

However, a portion of the developer community now seems to agree that hiding all the complexities of large CRUD apps has its trade offs, especially when it comes to large and long running projects. You end up in a situation where you have to be as knowledgeable about the conventions as if you wrote the code yourself.

Whoops!

Any sufficiently advanced technology is indistinguishable from magic. But is magic always bad? Do we like magic, as long as it hits the sweet spot where it is able to remain hidden from us?

Why do Clojure devs despise the magic in Ruby on Rails, but love the magic of auto-promoting persistent data structures?

Let's explore all the things you're not doing when you're using Datomic.

Look at all the data modeling I'm not doing

Datomic has made a decision for you. When inserting data, everything is completely and utterly flat, with no structure.

[:db/add -1 :person/name "August Lilleaas"]
[:db/add -1 :person/age 31]

This is it. All you have for writing data are these facts. A fact consists of:

[:db/add 
  -1                 ;; Entity (the id of the thing we're talking about)
  :person/name       ;; Attribute (defines the type and some other properties)
  "August Lilleaas"  ;; Value (self evident)
  ]
  
  ;; No table, no nesting, no depth, no ...

Any structure comes later, from queries and reads.

Datomic isn't free of data modeling. You have to choose if an attribute has a cardinality of one or many. You need to define the type of your attributes. But the API you get for writing data is extremely limited. There's no structured schema.

So, essentially: Look at all the up front decisions I'm not making about data modeling.

This has one enormous benefit. You create structure when you read, not when you write. Therefore, you get to come up with new ways of thinking about your data arbitrarily, after the fact. This is just a logical conclusion of having all the decisions about structure being put in the query engine, and not the write API. Contrary to traditional databases, where so much structure is defined up-front when you insert the data.

Look at all the time modeling I'm not doing

So you write these facts into the database, and there's almost no structure when you write.

There are 50 ways to get time modeling wrong in a traditional database, and a few ways to get it right. And even when you got it right, maybe your specific implementation of time management didn't cut it so you need to update it. And maybe you didn't add time management to your entire domain, just the crucial parts.

Given Datomic's extremely simple write API, it is able to implement time management for you, without any additional cost of data modeling or query hatching on your part.

So, essentially: Look at all the up front decisions I'm not making about time modeling.

It's difficult to talk about Datomic without at one point mentioning the time model. The writer is single threaded. You get a completely serialized time-line of your writes, without any special coordination or re-ordering happening. There's no such thing in Datomic as "read this thing, write that thing, read this other thing" and have that be inconsistent in any way shape or form. There's no such thing as "isolation level" (alternatively, Datomic is isolation level 9000), and transactions can't see what happens in other transactions. This is fundamentally solved by having only one transaction being processed at a time.

When you query Datomic, you query against what Datomic calls a "database". This is essentially a view of a point in time of the time-line of writes that have occurred so far. This view is completely immutable, and requires no synchronization with the writer process because Datomic is an append-only database with append-only semantics.

Are you thinking what I'm thinking?

Did you notice the detail there that makes Datomic amazing?

You ask Datomic for a "database", and you get an immutable reference to the value of that database at that point in time. You get to hold onto this value forever, no matter how many writes that occur in the future.

Getting it yet?

You get to hold onto a view of the database that's immutable, no matter what happens to it in the future.

This means that it's extremely cheap for Datomic to let you get a view of a previous state of the database. In fact, internally, the API is exactly the same. You get a reference to the database data structure, with a T value (a reference or id to a point in time in the transaction time-line), and all your queries will ensure that you get the data as of that T value.

So it's really cheap to just set the T value to an earlier point in time, say two weeks ago, or whatever, and get a completely immutable and consistent view of the entire database as of two weeks ago.

Woha.

Look at all the consistency management I'm not doing

In that vein, the consistency model in Datomic is also extremely simple. As mentioned above, the writes happen in a single thread, writing one transaction at a time. This is isolation level 9000. There's no extra management to be done on your part to ensure that you use the correct isolation level or that you do things in the correct order to get the expected result or any of that. Datomic is consistent, period. There's no way to screw that up by misunderstandings or bugs made by the developer that uses Datomic in their app.

The only trade-off here is that you need to resort to stored procedures, VoltDB style, when you want to do anything other than setting a value, deleting (retracting) a value, and doing a few built-in operations like compare-and-swap and identity based upsert. If you want to do things like "increment by 10" or "take N from account X and put it into account Y", you need to express this as a transaction function (Datomic's name for stored procedures), written in Clojure or Java, and ship it off to the transactor (Datomic's name for the database server).

Look at all the time management I'm not doing

This is a bit of a minor point, but I still like it. It is an example of the many things that are suddenly fundamentally solved when you have a database that manages time for you.

Exhibit A: Pagination. Pagination has the issue of being a non-atomic operation. You load the first 50 items. Then you go forward, and load the next 50 items. But in between those two fetches of items, the database might have changed. With Datomic, you can have the T value be a factor in your pagination. You can use this to get a consistent view of the database, so that page two is actually page two. You can also run queries to see if your data set has changed since the provided T value, and inform the user. "Click here to update your view of the data to the most recent changes".

Exhibit B: Return values. Let's say you have a method that takes a bunch of arguments and inserts a new thing into your database. What should it return? Maybe just the ID, so we don't force all users of the API to wait for us to reconstruct a view of the thing that was just inserted. With Datomic, however, this is a non-issue. You simply return the ID and a reference to the T-value as of after your insert (which is something Datomic provides automatically). Then it is up to the consumer to do what it wants with the T-value. And this is another example of the extreme agility you get with Datomic. Because time is managed for you, you get to make a whole lot of decisions later on, since you don't have to worry about the rug getting pulled from underneath your feet.

Look at all the event sourcing I'm not doing

This is a big one.

If you're familiar with event sourcing, you might wonder what the big deal is. "One single threaded time-line of writes" is exactly what event sourcing is?

What does Datomic have that event sourcing doesn't?

Answer: An absence of hand written views.

With event sourcing, you have raw data in your stream of events. This data is useless until you create a view of it. A view is typically a left fold reducing function that gets the previous state and the next event in the list passed to it, and returns a new and updated state.

This means that you have to decide which views you want to have. And that if there's a bug in your view code, you have to traverse the entire time-line again to build up the state.

With Datomic, you have the stream of events (facts/transactions), but there is one crucial but fundamentally important difference:

You don't have to decide what your views are up-front

So, essentially: Look at all the up front decisions I'm not making about which views to have.

Again, Datomic provides us with extreme agility. Because the data we write is completely flat, but is not arbitrary data but a well known structure (facts), and Datomic have generic indices to query these facts, and because Datomic keeps these indices up to date when we add data to our system, you get the benefits of event sourcing (time travel, a consistent time-line, all your historical data easily traversable), without having to worry about creating views for your data. The views are already there, in form of Datomic's query engine, its built-in time travel features, and efficient indices of historical data.

Look at all the file writing I'm not doing

(Implementation detail)

Internally, Datomic doesn't write to the file system directly. It writes to existing storage engines and databases. You can set up Datomic to write to any JDBC database (mysql, postgres, h2, oracle, db/2, ...), dynamodb, riak + zookeper, and a few other combinations.

This means that Datomic doesn't have to implement its own code for properly writing to disks, which is a huge pain to get right. And it means that if you already use postgres, you don't have to worry about backup routines and operations and what not, other than running the Datomic transactor itself. You can just use your existing routines for postgres operations.

Look at all the database concept reinvention I'm not doing

(Implementation detail)

Datomic doesn't actually invent a whole lot from scratch.

I've already mentioned VoltDB. VoltDB also writes in a single thread, after research revealed that multi-threaded RDBMS databases spend about 75% of their time doing coordination and only 25% of their time actually doing reads and writes. There's this instinct that single threaded processes can't possibly be efficient. But you'd be surprised how much efficiency you can get out of a thread that spends close to 100% of its time doing useful work without much time spent on coordination between multiple threads.

There's also a concept borrowed from Google's Big Table. The indices in Datomic aren't updated live for every write, that would be way too much overhead. So a query in Datomic runs against the latest index + a list of transactions that are written to a transaction log but not yet merged into the index data structures. Then, Datomic will periodically take the unmerged transaction log and merge it into the main index. Big Table does the same, after Google learned that live-merging an index of the entire internet was maybe not going to work.

Look at all the .... ok, I'm done

In my experience, Datomic is indistinguishable from magic, in a good way. It's one level of abstraction above traditional databases, but manages to hide those abstractions since you at no point worry about postgres or zookeeper internals when you make use of Datomic on top of the existing storage engines. You get a whole lot of stuff for free, without Datomic getting in your way.

And to me, the most important point is all of the things you're not doing and all of the decisions you are not making up front, but rather in an agile fashion as your application grows.

Update: even more things I'm not doing

I wrote a follow-up, with even more things I'm not doing:

http://augustl.com/blog/2018/datomic_look_at_all_the_things_i_am_not_doing_cont/

Datomic: Traditional databases only spend about 25% of their time actually writing to and reading from the database

2018-04-20T00:00:00.000+02:00

This post is about how Datomic (and Volt DB, and a few other databases) are different from traditional databases.

It turns out that OLTP databases spend 75% of their time coordinating reads and writes. That means that 75% of the time, the database isn't actually running queries and writing data, but is instead figuring out safe ways to allow multiple concurrent clients to perform reads and writes in parallel.

Volt DB figured out that with modern hardware, you can get much better performance if you instead partition your dataset across N cores, and have those cores do uncoordinated reads and writes to their respective partitions, and then when you do need consistency across partitions, the coordinator can step in.

Datomic is completely single threaded when it comes to writes. (And infinitely multi threaded, without coordination, when it comes to reads.)

It turns out that even on today's multi threaded hardware, having a single core do useful work close to 100% of the time, is actually quite efficient.

So when you hear that Datomic has a single threaded writer, maybe that's not such a bad idea after all. You get unparalleled consistency (pun intended), on top of a writer that might actually be performing better than what your gut feeling tells you.

Datomic: The most innovative DB you've never heard of

2016-01-20T00:00:00.000+01:00

Once upon a time (late 2011, to be exact), a small group of programmers got blown away by Rich Hickey's talk Simple Made Easy. This group of programmers became passionately obsessed with the programming language Hickey created: Clojure. When Hickey then introduced Datomic a year later, reconstructive jaw surgery became a meme in the Clojure community.

Why did our jaws suddenly become so useless? And why hasn't Datomic become an A-level option in the database space?

Let's investigate.

Datomic stores the actual data directly in (multiple) indexes

Datomic doesn't have the data in one place and the indexes elsewhere. In Datomic, data and indexes are the same thing, and data is stored directly in the index. This is called a covering index.

Not only that, the data is stored multiple times, once for each index, of which there are four and only four.

Datomic only stores its data in other databases

Datomic does not write directly to the file system. Instead, you can use a number of different databases as a storage backend for Datomic: Any SQL JDBC database, CouchBase, Riak, Cassandra, DynamoDB, Infinispan, and a "dev" mode that writes to a file backed H2 database.

Datomic has a completely different API for reads vs. writes

In most databases, you use the same API for reading and writing. Typically in the form of a DSL, like SQL, Cypher in Neo4j, the JavaScript based MongoDB API, and so on.

In Datomic, you use a "transactor" API for writes and datalog for reads.

Reads and writes have completely different performance characteristics, semantics, APIs, operational details - you name it.

Datomic is single threaded

To be specific, writes are single threaded.

Note that reads are however multi-threaded. As I already mentioned, reads and writes have completely separate characteristics in Datomic.

It turns out that traditional multi threaded databases only spend about 25% of their time actually writing to and reading from the database. The rest of the time is spent coordinating multiple clients reading and writing in parallell.

So a single threaded writer, like Datomic, is able to completely saturate a CPU core for writes, spending 100% of the time doing actual useful writes.

This also means that you get what Stuart Halloway calls isolation level 9000. A single thread doesn't have concurrency problems.

Datomic runs queries on the client, not on the server

This tends to seriously creep out most experienced programmers. Including myself when I first heard of Datomic.

First, keep in mind that if you want your reads to be fast, you have to keep the working dataset in RAM on the database server anyway.

Second, realize that this means you can infinitely scale your reads to as many clients as you want.

This doesn't mean that the entire dataset has to fit in memory, by the way, only the part that you actively query. And, as mentioned, for speed, you want the entire active dataset in RAM anyway. So why does this absolutely have to happen on the database server?

The Datomic server is not concerned with reads whatsoever! You can run the nastiest long running query you can think of and not affect anyone but yourself.

Remember how Datomic doesn't write to its own storage, but stores data in other databases? As an extension of that, when a client (or peer in Datomic lingo) needs to fetch some data from storage to run a query, the Datomic server is not involved in any way at all! The client instead connects directly to the underlying storage engine. Furthermore, you can put memcached between you and storage, so you don't even have to talk directly to the storage engine for reads.

Datomic has no round-trip open long running transactions

In Datomic, you can't open a transaction, do some reads, some writes, some more reads, and then commit. You have to create a complete transaction statement and send it off whole sale to the server.

This goes back to what I've already mentioned, reads and writes are completely separated.

Fear not: It is completely doable to write transactions that say "increment existing value by 10" and "replace whatever list that might exist with this new list" consistently. It's just that you have to write this in a complete statement that you send off to the Datomic server, so that Datomic can put your transaction on a queue and run it when it's ready (previous writes have finished).

Sending off a transaction like this is what makes writes so efficient. The Datomic server requires no coordination to perform writes, all it needs to do is to read the next transaction off of the queue and execute it.

Datomic treats the database as one huge immutable map

Or, well, not a map, it's actually a sorted set implemented as a persistent red-black tree traversed with a O(log n) binary search. And I'm actually not 100% sure about the red-black tree part. It's a proprietary database, so self taught programmers like myself have no chance of grokking the details completely.

To run a query, you ask the Database connection "give me the current database". What you get back, is a single data structure, a "db", that represents your entire database, consistently, at a point in time.

As mentioned, queries are executed on the client, not the server. This "entire database value" is of course lazy, backed by a LRU cache, so you only actually pull in the subset that you operate on.

Datomic has time travel built in

You can ask the database about its current value and get a immutable value back, representing the entire database.

Then, you can say "give me the database as of <timestamp>". What you get back, is another immutable value, representing your entire database, at a previous point in time.

This is one of those things that is so utterly sexy and useful and absolutely a killer feature that you'll end up loving so much you'll have problems using other databases.

At no point can you say "damn, I wonder what actually happened yesterday at 22:30 when nobody was in the office". With Datomic, all you need to do is to ask "give me the database as of 22:30 yesterday". Zomg!

I can't praise this feature enough. And honestly, it's difficult to explain how awesome it is until you've become familiar with it yourself. It's one of those things that seems so fundamental, and there's no going back.

Wait, you actually consistently delete data en masse from your customers database? Are you crazy?

Datomic has "what if" for all database operations

As I've mentioned, a write is represented as a single operation that you ship off to the Datomic server.

When you have one of those writes available, you can also ask Datomic for a database (the current one, or any previous version) and say "give me a database as if this transaction/write was applied".

This is also kind of difficult to adequately communicate so that you'll grasp how completely awesome this is

Since all your queries run against a "db", an immutable value representing your entire database, your codebase typically ends up having a lot of methods/functions that run their business logic by getting a "db" passed in.

So let's say you have a method like that for generating a report, or maybe running some analytics and doing some number crunching.

With Datomic, you can use the exact same function and run it against a "what if" database - it doesn't care if the database value is a "real" one from the Datomic server or a "what if" one generated locally!

Datomic lets you query the history of your database

I won't go into details, since this can get quite technical, but I've written a separate article about this subject which goes into detail how this works.

Datomic has actually invented very little from scratch

Covering indices are nothing new, you can do that with plain old SQL databases. VoltDB is also single threaded and has a similar queue based transaction API without round-trips. I haven't talked about Datomic's periodic merging and live merging of the transaction log, but that is borrowed from Google BigTable.

That's nice. So why the heck are so few people using Datomic?

I think there are a number of reasons.

Datomic is a proprietary database. I've used it in production without paying a dime. There is a free version of Datomic and it works just fine for startups and smaller apps without a gazillion users. But developers generally find proprietary systems off-putting, so it has some difficulty in gaining traction because of that.

Datomic is made by Cognitect, and I suspect that they want to stay small. I suspect that they have a few 100s of Datomic customers, and that's more than enough for them to make Datomic a sustainable business. It is actually possible for companies to not want to grow - I know, I'm an owner and stakeholder in such a company, Kodemaker (in fact, all of us are owners and stakeholders, that's the model).

But most importantly, I think: Datomic solves a lot of problems I didn't know I had, until I learned Datomic.

I didn't learn Datomic because its promises of time travel or separation of reads and writes and what not. I learned Datomic because I was passionate about Clojure, and was pretty open to whatever Rich Hickey created. So I toyed with it, wrote some dummy stuff, and quickly began to love to hate other databases. A lot of the problems I had at work with our RDBMS actually weren't fundamental problems! And now, at least once or twice a week, I encounter a problem that Datomic would have solved at a fundamental level.

I blogged some about Datomic a couple of years ago, and this post (i hope) marks the first of many blog posts about Datomic I'll write in 2016. I have no problems calling myself a Datomic evangelist. I would quite frankly like to see an open source alternative, or at least one other database similar to Datomic, and for now the best way of achieving that (short if making one) is to evangelize Datomic and make developers around the world aware of the problems you don't have to solve yourself.

...

Here's to an ACID future where databases are immutable values and time travel is for free!

Datomic: Direct index lookup

2013-07-17T00:00:00.000+02:00

A couple of years ago, I read about someone whose MySQL performance bottleneck was SQL parsing. Most of the queries were straightforward primary key lookups, so the author wrote code that called InnoDB (MySQL's storage engine) directly and removed the bottleneck. I thought this was really cool, and I feel like I've reached a small milestone today for doing the same thing myself, only in Datomic.

In our system, we have a querying API that lists everything if an empty query is passed. This means we need to ask datomic for potentially 1000s of entities, and also make it happen in a reasonable amount of time. In the future, we we'll probably be more clever and lazily load stuff, but I wanted to see if there was a quick fix, since lazy loading in the UI takes a lot of work to get right.

Here's the old code:

(require '[datomic.api :as d])

(map
  (fn [public-id]
    (d/entity
      db
      (ffirst
        (d/q
          '[:find ?attendant
            :in $ ?pub-id
            :where
            [?attendant :attendant/public-id ?pub-id]]
          db
          pub-id))))
  public-ids)

The attribute :attendant/public-id is a string attribute where we insert Datomic squuids. We only expose these IDs to the outside world, the Datomic entity IDs stay internal to Datomic.

In the extreme cases, 2600 such public IDs is what we mapped over, resulting in 2600 queries being performed. This took about 1 second, and was not acceptable. The solution is to not query at all, but just look up the value in the Datomic index. This ended up taking about 0.1 seconds, so a ten-fold performance increase. Not bad at all! Here's the updated code:

;; Yields exactly the same output as the above query
(map
  (fn [public-id]
    (d/entity db (:e (first (d/datoms db :avet :attendant/public-id public-id)))))
  public-ids)

Datomic has multiple indexes, and all of them are full copies of the data. So looking things up in an index is the only thing you need to do in order to access the data, and this is what the Datomic query engine does under the hood anyway, since all your data in the database is available to you as an immutable (and lazy) value.

The code to d/datoms flows as following:

Pass the datomic db whose indexes we want to look up.
Pass the index to use. We want the :avet index. (Attribute, value, entity, transaction)
- The following arguments are the components to look up in the index.
The first entry in :avet is :a. So we pass :attendant/public-id, the attribute we want to look up.
The second entry in :avet is :v. So we pass public-id, the value we want to look up.

This will give us a raw seq of facts. Since we know that the attribute :attendant/public-id is set to unique, we can just get the (first) one.

Aside: We could also just have passed one component, the attribute, and gotten back a sequence of facts for all facts for that attribute.

When we have our fact, we extract the entity ID from it. The fact is map-like, so all we need to do is (:e fact).

Then we proceed as normal, and look up the full aggregated entity just as we do with the entity IDs we get from Datomic query results.

So there you go! Again, we should probably be more clever and lazy load stuff, but we just wanted a quick fix, and a ten-fold performance increase is more than good enough for now.

Ordering of `:db.cardinality/many` attributes in Datomic

2013-05-04T00:00:00.000+02:00

In this post, I assume you already know about Datomic, how to use it, and the difference of the two settings for :db/cardinality on an attribute.

Tree structures and ordering

:db/cardinality.many values are sets. One of the big differences between Datomic and in a typical RDBMS, is that Datomic uses tree structures for pretty much everything. In a RDBMS you typically get insertion order when querying without specifying what to sort on, but Datomic can't do that since a tree structure isn't ordered in the same way.

Your business logic might require ordering, though, so to achieve this you need to add a :position attribute to your entities. So the question becomes: how to we consistently and safely manage this :position attribute in a way that ensures we don't get inconsistent values in a multi-peer world?

Transaction functions to the rescue

In any situation where multi-peer consistensy is required, a transaction function is the solution. Most of the developers I've spoken to that aren't familiar with Datomic despises the idea of storing and executing code on the database server. But since I'm in Clojure land in my system, and the transactor is too, it doesn't really feel as foreign as PLSQL might do, and since Datomic is free of time complexities, maintaining and synchronizing changes in the transaction functions is pretty much hassle free.

Here's an example of a transaction entry that adds a transaction function for ordering to the transactor.

{:db/id #db/id[:db.part/user]
  :db/ident :append-position-in-scope
  :db/doc "Atomically adds to the end of a list of sorted cardinality/many lists"
  :db/fn #db/fn {:lang "clojure"
                 :params [db scope-id scope-attr new-id pos-attr]
                 :code (let [children (scope-attr (datomic.api/entity db scope-id))
                             highest-pos (reduce max 0 (map pos-attr children))]
                         [[:db/add new-id pos-attr (inc highest-pos)]])}}

The entities at work here is a "todo list" that has a :db/cardinality.many attribute called :todolist/items.

children essentially translates to (:todolist/items todolist-entity), and then we calculate the highest :position, pos-attr, currently in the list and increments that by one.

On the peer side, we can now create a transaction for adding a new todo item and setting its position to be at the end of the list.

[{:todoitem/text "This needs to be done"
 :db/id new-todo-item-tempid}
 [:append-position-in-scope todolist-eid :todolist/items new-todo-item-tempid :position]]

Since the Datomic transactor ensures consistency, the transaction function is guaranteed to expand to a :position value of the currently highest position in the list plus one. There's absolutely no risk of race conditions here.

Filling gaps

The other function I have for working with sorted sets is a function that I use when I delete an individual item, to fill in the gaps.

{:db/id #db/id[:db.part/user]
  :db/ident :reset-position-in-scope
  :db/doc "Goes through existing positions and sequentializes them"
  :db/fn #db/fn {:lang "clojure"
                 :params [db scope-id scope-attr retracted-eid sorted-attr]
                 :code (let [children (scope-attr (datomic.api/entity db scope-id))
                             sorted-children (sort-by sorted-attr children)
                             without-retracted (filter #(not= (:db/id %) retracted-eid)
                                                       sorted-children)]
                         (map-indexed
                          (fn [idx entity]
                           [:db/add scope-id sorted-attr idx])))}}

When a positioned entity is deleted, it would normally leave a gap in the position attribute. This function goes through all of what's left and sets the positions to 0 and up, with zero gaps. Note that this creates facts for _all_ the remaining entities. It could probably be improved to only state facts for the items from retracted-eid and below.

Since this is always used when an entity is retracted, we need to pass in the entity that is being retracted as well, so we can remove it from the set. The transaction gets the database as of before the transaction commits, so even if we have a retract entity command in the transaction, this function won't know. So we manually remove that from the list.

Finding by ID (primary key) in Datomic

2013-03-10T00:00:00.000+01:00

Datomic has no essential type for entities, so how do you find the user with id X with Datomic?

Finding by internal Datomic entity ID

All entities in datomic gets the attribute :db/id, which is Datomic's own primary key attribute. You might be surprised that (d/entity db 123) for an entity id that doesn't exists doesn't return nil.

(require ['datomic.api :as 'd])

(def db (d/db my-datomic-connection))
(def non-existing-id 123123)
(d/entity db non-existing-id)
;; Returns an actual entity
;; Even if there's no entity with id 123123

To actually find an entity by ID, you need to run a query.

;; Assuming an entity with id 123 exists

(d/q '[:find ?eid :in $ ?eid :where [?eid]] db 123)
;; HashSet [[123]]

(d/q '[:find ?eid :in $ ?eid :where [?eid]] db non-existing-id)
;; HashSet [[]]

Also, just for fun, we can create a reproducible example that doesn't depend on having an actual Datomic database around, since Datomic is able to query any data strucutre that consists of a list of lists.

;; This is what an actual Datomic DB actually looks like.
(def my-mock-db [
  [123 :name "Test"]
  [123 :email "test@test.com"]
  [456 :name "Foo"]
  [456 :email "Hello"]])

;; A "normal" query, for good measure.
(d/q '[:find ?person
       :in $ ?name
       :where [?person :name ?name]]
     my-mock-db "Test")
;; HashSet [[123]]

(d/q '[:find ?eid :in $ ?eid :where [?eid]] my-mock-db 123)
;; HashSet [[123]]

(d/q '[:find ?eid :in $ ?eid :where [?eid]] my-mock-db 999)
;; HashSet [[]]

However, how do we know that we get an entity of the type we want? Datomic entities doesn't actually have a type, they're just entities with attributes, so we're at a loss.

Using manual "public ID" attributes

I like to have a public ID field per entity type in my system. Again, Datomic entities has no type, so this is all on us, Datomic doesn't care.

I use a string type that's set to unique. Setting it to unique is obviously important. You could also use the built-in UUID type if you want to, but I find it more convenient to just work with strings.

;; Inserting data
(d/transact
  my-datomic-conn
  [{:person/name "Test"
    :person/email "test@test.com"
    :person/public-id (str (java.util.UUID/randomUUID))
    :db/id (d/tempid :db.part/user}])

;; ...

(d/q
 '[:find ?person
   :in $ ?pub-id
   :where
   [?person :person/public-id ?pub-id]]
 db
 person-id)

Benefits:

You get an entity type system for free. If you use a Datomic ID, how do you query for people entities only for GET /people/123?
Datomic entity IDs stay internal, meaning you don't have to worry about possible entity ID changes when you import/export the database.
You don't have to convert from a string type to a numeric type when you get the entity ID from a URL. Not a very big deal, but since it's a very common use case I'd say it counts.

Querying Datomic for the history of an entity

2013-01-20T00:00:00.000+01:00

This post has been rewritten and updated.

https://augustl.com/blog/2019/datomic_querying_for_history_of_entity_v2/

The main reason for choosing Datomic for our project was our requirement to have a page that lists all the changes that ever occurred to a specific entity. This turned out to be a bit more tricky than I anticipated, but after I finally wrapped my head around the basics, the solution ended up being quite elegant.

The first step is to get a hold of all the transactions related to the entity in question. This is done with a query on (datomic.api/history db), a special version of the database containing the full history of assertions and retractions.

(:require ['datomic.api :as 'd])

(d/q
 '[:find ?e ?a ?v ?tx ?added
   :in $ ?e
   :where
   [?e ?a ?v ?tx ?added]]
  (datomic.api/history db)
  123123)

This is the generic query to find them all. This query returns the datoms in the history for the specified entity only.

My actual query was a bit different though. I had the entity ID, and needed to find all datoms that touched a referred attribute for that entity. So simply finding all transactions for my entity would not suffice. Here's a short summary of my schema:

;; Attendants

 {:db/id #db/id[:db.part/db]
  :db/ident :attendant/public-id
  :db/valueType :db.type/string
  :db/cardinality :db.cardinality/one
  :db.install/_attribute :db.part/db}

 {:db/id #db/id[:db.part/db]
  :db/ident :attendant/attributes
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/many
  :db.install/_attribute :db.part/db}

 ;; Attendant attributes

 {:db/id #db/id[:db.part/db]
  :db/ident :attendant-attr/value
  :db/valueType :db.type/string
  :db/cardinality :db.cardinality/one
  :db.install/_attribute :db.part/db}

 {:db/id #db/id[:db.part/db]
  :db/ident :attendant-attr/schema-attr
  :db/valueType :db.type/ref
  :db/cardinality :db.cardinality/one
  :db.install/_attribute :db.part/db}

The entity I'm tracking is the attendant, but the attributes of an attendant isn't stored directly on the attendant entity. The attributes are a reference, in order to join the attribute values to the schema-attr for the event the attendant belongs to. The user of the system is able to specify, per event, the attributes that are available to an attendant, so this join maintains data integrity.

Here's the actual query we run:

(d/q
 '[:find ?tx ?attr ?added
   :in $ ?attendant
   :where
   [?attendant :attendant/attributes ?attr]
   [?attr _ _ ?tx ?added]]
  (d/history db)
  123123)

This gets all the transactions that touches the attribute :attendant/attributes for the specified attendant.

And this is where the fun begins.

The next step is to take the tuples of entity IDs we get from the query, and turn it into a full changeset of entities we can use to display the list of changes to the user. Essentially, we want to return a list of maps, where each map contains the timestamp of the change, and the actual list of changes. We also need to handle all the cases: an attribute being created for the first time, and attribute changing from something to something else, and an attribute being removed entirely. As a word of caution, this means we have to be careful when we create the actual transactions for our attendant, and only create transactions for the attributes that actually change, and make sure we retract facts when fields are blanked out. What we're about to do wouldn't be possible if the blanking out of a field simply blanked out the attribute in the database. We need to have a clean transaction history.

We extract the entities for the changeset with a map over the result of the query above.

(map
 (fn [[tx attr added]]
   (let [tx-db (d/as-of db tx)]
     {:tx (d/entity tx-db tx)
      :attr (if added
              (d/entity tx-db attr)
              (d/entity (d/as-of db (dec (d/tx-t tx))) attr))
      :added added}))
 result-of-transaction-query)

It's important to make sure you use the database as of the transaction inside the map. If you just use db as of now (specifically, the same db as you used when querying for all the transactions just before), you'll get the entities as of now, which you don't want. So in the map, we make sure to always use (db/as-of db tx) when we retrieve entities, so we get the entity as of that transaction. This "small" detail caused me lots of pain and trouble when implementing this feature, so make sure you get it right.

I haven't explained what added actually means. In the history db we queried for transactions, ?added is a boolean that tells us whether the datom was an assertion or a retraction.

If the datom was an assertion to the database, the entity is just the entity as of the transaction.

If the datom was a retraction, the entity as of the transaction will (perhaps surprisingly) not have any attributes other than the entity id. When we show the details of the change to the user, we want to show what the old values for the entity were, so we need to know what the entity looked like prior to deletion. Thankfully, Datomic doesn't delete any old data, even when we retract entities from the database. So to get the values, we simply get the entity as of the transaction before the transaction where the entity was retracted. d/tx-t is used to turn our transaction entity id into a timestamp, and dec decrements it by one. So our call to d/as-of gets the instant just before the transaction, leaving us with the database as it was the moment before our transaction occurred.

Note: I'm not entirely sure how safe this is; it's probably better to somehow get the transaction entity ID from before. For all I know, there was a large amount of writes going on at the time of our transaction, so the dec of the transaction instant might be inaccurate.

Phew! We now have a list of {:tx tx-entity :attr attr-entity :added true/false}. I'll leave it as an exercise to the reader to morph this list into something you can show to your users. Essentially, you need to group by :tx, sort by the transactions (:db/txInstant tx) (all transactions automatically gets a timestamp associated with them by Datomic), and traverse the data to figure out which combination of added you have in order to determine whether the transaction added, updated or removed the value.

We're really happy with Datomic so far. We haven't hit production yet, so I'm sure this query will be tweaked even further (we already know we want to add more stuff, such as tracking the user that made the change). But the fact that this feature is just a query away is very convenient indeed.

Show full article

Go to updated article