Alternatives to SQL Databases

Posted Apr 16, 2009 4:16 UTC (Thu) by jordanb (guest, #45668)
Parent article: Alternatives to SQL Databases

One thing to consider when looking at storing persistent data is how much effort the data store developers have put into maintaining data integrity. This was driven home to me when I was administering a computer running OpenLDAP and Fedora. The computer was in a situation where it would occasionally be hard-booted, and that combined with some (apparent) version incompatibilities with the Fedora repos would cause the sleepycat backend for the LDAP server to corrupt itself.

Our eventual (horrible, rube-goldbergian) workaround was to store the actual information in MySQL/innodb and have a script nuke the LDAP database and re-inject the data whenever it fell over.

I decided after that experience that if the data has even a chance of being important, then the most important property of any datastore is that the information should always be there and never be corrupt, regardless of if the computer is hard-booted or if the disk drive lies or if Ted Tso decides that his ideology is more important than your data.

A bit later I heard an interview with Richard Hipp in which he discussed how much effort they put into making sure data in sqlite is "in the oxide" -- going so far as simulating hard boots during writes in their testing procedures. I've since made sqlite my default data store whenever the data might be important and it's not in an RDBMS and would hesitate to go to something else without some assurance that they take a similar amount of interest in data integrity.

That said, memcached is awesome for caching, and some of these things do sound interesting for storing unimportant data like search indicies.

to post comments

Alternatives to SQL Databases

Posted Apr 16, 2009 17:00 UTC (Thu) by intgr (subscriber, #39733) [Link] (2 responses)

I think the problem you were having with Sleepycat BDB wans't "corruption" per se. BDB requires an explicit recovery command after an unclean shutdown and many applications using it do not automatically run this.

This annoying feature has probably driven off many of Sleepycat's potential customers.

Alternatives to SQL Databases

Posted Apr 16, 2009 19:43 UTC (Thu) by jordanb (guest, #45668) [Link] (1 responses)

Yeah in this case we tried making the LDAP init script do that, but it seemed to make the problem even worse. We began to suspect that the problem was that there was a version incompatibility between the bdb tools we had and the library being used, which was likely Fedora's fault, but we weren't able to verify that.

But anyway, 'need to explicitly run recovery before attempting to use the database again' is a design failure, imho. The database should be able to recognize that it is not completely consistent and recover itself on startup, and there should always be enough data to reach a consistent state that's not too far from the state when the system crashed.

Alternatives to SQL Databases

Posted Apr 23, 2009 14:44 UTC (Thu) by joib (subscriber, #8541) [Link]

At least on RHEL, you need to use slapd_db_recover rather than dbxx_recover, and similar for the other db_* commands, this will use the correct version of the BDB libraries that openldap was built against.

But yeah, needing manual recovery after a crash is incredibly annoying. Though we have replicated openldap servers that reduce the likelihood of service disruptions due to this.

Alternatives to SQL Databases

Posted May 1, 2009 18:00 UTC (Fri) by gdamjan (subscriber, #33634) [Link]

Well, at least CouchDB is very robust.

CouchDB will only append to the storage file on disk, and will add a token at the end of the append. So in the wors case you only loose the last "transaction".

CouchDB does not need any special recovery tools, and instatly restarts