Data points required for distributed Orient DB in Production

[maruthikusoji]

Hello OrientDB community,

We are looking to migrate from current standalone Orient DB v3.0.37 to 5 node distributed DB v3.2.x. While testing (v3.2.5+ migration from v3.0.37) this, we are encountering quite a lot of issues and we are working with SAP ODB team to resolve them. While we continue exploring to migrate to distributed orient DB, can anyone give us following information w.r.t distributed DB to evaluate the stability of the distributed DB performance in your Org/Project?

Which version of the distributed ODB is in production currently?
Was the data exported from v3.0.x and imported in v3.1.x or v3.2.x?
Please list last 5 versions that was used in production, if any. Trying to understand upgrade path here.
How many total servers, how many masters and replica amongst them.
What is the size of DB on file system.
What are total number of vertices and edges.
What is the Transactions (read / write ) Per Second on DB.

Any input is much appreciated, looking forward to it.

The releases made after v3.2.5 have a most fixes to the problem reported by our team.

thanks,

[timw]

We've been running a distributed cluster in production for a few months.

Some general observations from our experience:

• We have a fairly modest deployment at present, < 200MB data at the moment, 4 nodes across 2 AZs in AWS (don't ask why, we know), quite modest transaction volumes, so I can't really comment on scalability, but we do have a bunch of durability observations.
• We've been running 3.1.x, since until 3.2.8 the 3.2.x branch was missing a bunch of fixes that stopped the storage going into read-only mode when an error occurred (like a parse exception in a Lucene query).
• 3.1.x escalates write quorum to all online masters (i.e. all online for non sharded configurations) if there are any unique indexes in the update, which means you have 0 node redundancy in the case of a node becoming unresponsive. 100% of our update transactions had unique indexes, so this was not good news for us.
• We've encountered a bunch of issues specific to distributed deployments that caused major outages. Significant ones include: a TCP connection leak in distributed comms (that ultimately caused one side of distributed connections to stop working), and a thread safety issue that broke Lucene indexes (which then broken the storage, which then broke the entire cluster). The latest 3.1.x and 3.2.x have our fixes for these issues
• Not distributed, but the automatic disk cache calculations aren't particularly safe in production (they can overcommit memory, leading the Linux OOM killer doing nasty things), so definitely set your disk cache explicitly.
• We've found that OrientDB is on the delicate side when it's shutdown uncleanly (we had various issues with our setup that have caused unclean shutdown - OOM killer, systemd terminating process, but even CTRL-C in Docker windows can break things). Lucene indexes in particular get wedged quite badly in various ways we haven't yet fully debugged, and we've seen broken OSystem databases, NPEs in storage etc. Primary fault here is the unclean shutdowns, but more resiliency/recoverability would have saved some all-nighters.
• We've also observed the distributed deployment can get unstable if you restart all of the nodes at once (we had some over-eager automation scripts that did this, as well as system updates), we haven't fully debugged these scenarios, but some of the required shutting down all nodes to resolve (cluster state was tainted somehow, and rolling node restarts didn't fix it).
• We're not running at large scale, but we have noticed during code inspection that a lot of the thread pools in OrientDB don't scale as intended (there's an in-progress PR to fix these).
• Backup without the Enterprise Agent installed is a PITA - there's no practical way to do a backup without shutting down one of the distributed nodes so you can backup safe-pointed data or use the console backup commands (which in 3.1.x will cause in-flight transactions to fail during the shutdown due to the ALL_ONLINE quorum). We're integrating the (now open source) enterprise agent into our deployments to get around this.
• We're in the process of upgrading to 3.2.8 now that it's got all the fixes from 3.1.x that we needed.

In summary, there's a lot we like about OrientDB (the data model and query model are really good), but you need to treat it with care on the operational side. We're fairly confident on the operational resilience side now (i.e. staying running once it's in a good state), but durability across failure still needs a bunch of testing and work (I'm contemplating doing some crash fuzzing to see if we can reproduce and hunt down some of the failures). The overall code and documentation quality is inconsistent - great in some places and pretty rough in other areas - and it's obvious the project has run into difficulties in recent years balancing quality and feature development. In particular it doesn't look like the 3.1+ versions have been used in anger in distributed deployments (the TCP connection leak alone essentially time-bombed every distributed deployment to about a week unless you increased token timeouts) so it felt like we were running into a lot of issues.

3.2.8 looks like a credible candidate for production based on our experience, but prior to that release (i.e. in the past few days), but all the 3.1/3.2 releases prior to this had some significant issues for our use cases.

Hope this experience helps you - if you're able to share the issues you've encountered it might be useful to the community (and from a selfish point of view, we'd like to know about and avoid/fix them before they hit us as well)... :)

[maruthikusoji]

@timw thanks a lot for the detailed response. I will provide details on issues we faced and that got fixed etc. in few days.