A New Database!

Databases are a core building block for most developers. The chosen database system will usually also describe the future of the entire project. This is because you will be required to stick to this decision, since changing the database later will end up in a rewrite, not to talk about data migration.

So why is that? Well it is because you have to choose between parameters like these:

  • Cloud (server) or local (client)?
  • Relational (SQL) or document based (noSQL)?
  • Schema or schema-less?
  • Isolated (only local) or replicating (sync)?
  • Cross-platform or OS specific (e.g. CoreData)?

You see, there is a lot to consider.

CoreData

For developers for Apple devices the choice usually is simple. Inside of the app you will use CoreData. It is well supported in Xcode and samples and documentation are in a good state. You might come into areas where life becomes harder e.g. if you do a some multi-threaded stuff or when the dataset becomes large and queries become slow. But in general it is a good choice.

But what to do, if it comes to sync? No problem you say, I'll sync with a remote server or use CloudKit. Well, good luck, this is not trivial.

CouchDB

So CoreData is an example for a local database. Lets pick CouchDB as an example for a remote database. It has a lot of features you may desire. It can sync data and has a way to do conflict resolution quite easily. You are free to add properties to the documents it holds and it supports assets very well. Many other databases offer similar features, including CloudKit from Apple.

But what to do if privacy is a requirement? And why do I still need to write that much code in my app to sync and support offline?

All New - The Idea

Ok, so what would be the perfect criteria for an app developer like for iOS and macOS?

  • Serverless i.e.
    • no additional costs
    • no 24/7 service
    • no worries about scaling
  • Sync i.e.
    • the user can connect devices and have data in sync
    • conflict resolution is simple or even better comes out of the box
  • Platform agnostic i.e.
    • I start my project writing in Swift or ObjC but once I want to add other platforms, the database supports that
  • Privacy compatible i.e.
    • the user "owns" his data, he decides where to put it and has full control over it
    • between the devices everything is end-to-end encrypted
  • Reliable i.e.
    • it should store data in a redundant way, like on multiple locations; such each copy would also work as a backup
    • data should not be easy to manipulate; inconsistencies should be detected

How can a database do this?

A Solution

Actually I started working on such a database solution and what I did is the following:

  • Locally have a key-value-storage that functions as the database layer, in the case of macOS and iOS this is SQLite, for other platforms it can be another appropriate solution
  • The database holds records which are schema-less. You can put any object into it that serializes to a plain data format, for now these are strings, numbers, decimal numbers, dates, asset references and record references (yes, you can build relations)
  • Each record has a unique identifier as its key and a revision info. It also points to its parent revision. Conflict resolution is done similar to the CouchDB way.
  • Even though the database is schemaless, on macOS and iOS you can inherit from the record object and directly access properties you defined. This way the work with record objects feels like working with CoreData. Even KVO is working well.
  • The changes, called transactions, are also stored in a blockchain on the local file system. The blockchain can be encrypted using state of the art implementation.
  • Large binary objects like pictures etc. should be stored as assets
  • Both transactions and assets make a container
  • The database can have access to multiple containers at the same time, they can be placed on Cloud Drive, Dropbox, NAS or any other file like system or specialized service.
  • Via these containers sync is implemented

Does it work?

See for yourself! Currently I use it in my demo todo application Zutun. I will use it in the next major release of Receipts as well.

I'm curios to hear about your opinion on it. Please comment below or drop a line on Twitter @holtwick.