Applications store data

Sublime Text loads all files, buffers, layout, and other data back from the last time you quit. This allows you to quickly jump into action. I'd like Todool's save format to be just as good.

Here's how my first file format looked like:

• data: stored general info about the editor
• tree: stored task related info like indentation, time info, string content, special content in ^1.123123123.1^ which wasnt in the example image

Let's keep it short

• Pros
  • quick & easy to start with
  • manually editable to add new features
  • loose structure
  • raw text viewable
• Cons
  • users could modify the data, potentially deleting data
  • hard to change: manually edit, scrap your save file or go the old read -> new write -> new read route
  • hard to read: odin does help since it has a basic scanner
  • structure has to be implied by i.e. indentation, scopes or numbers in my case
  • hard to write: conversions to strings all over the place, again odin helps with strconv
  • big file size
  • slow read / write in comparison to binary

No backwards compatibility killed a lot of my save files, usually because of my negligence when adding new data to the format.

Exploration

Heres a list of some file formats. JSON like formats offer a sense of structure through scopes. A format like that makes sense for my theme and option files.

These are just plain structs that could be dumped to JSON. I looked into alternatives to JSON and found out about msgpack, which is pretty much JSON in binary with a simple spec.

I implemented msgpack and all the goodies you'd want odin-msgpack. There you can read / write entire structs and not worry too much.

Important

Option / Theme Data that does not exist in a old / new version will simply be ignored ~ Default values will stay. A format like this does not work for a lot of structured data like a tree though. JSON like formats would store lots of similar data like variable names to associate back.

A custom file format makes sense here, since you can tailor that towards all you need. Or you could go for a format with schemas. These allow to predefine structures that will reappear throughout your file and be written / read back the same expected way.

Interestingly Databases work via these schemas

Database?

Some applications actually use databases as an application file format. sqlite is a C library that you can easily use, so I did in odin-sqlite.

Here's how you work with databases

• open DB
• optionally add table / add data
• modify data throughout the program
• close DB

The DB does all the hard work for you. You have to conform to the way the DB works, which then allows an easy way to store data.

Working with a DB is a painful process. The constant back and forth messaging via SQL kills usability to me. Transitioning a functioning codebase to a DB is also terrible, since you need to restructure everything. I actually tried this out in the Wheel Reinvention Jam where I transitioned a week old codebase to using a DB. That process took almost a week... Clearly not suited for a jam or a long existing project.

Reflect & Goals

What I wanted from a new file format:

• small size
• direct read / writes (memcopy) data sections
• direct tree read / write which holds structure i.e. MetaDesk
• some form of backwards compatibility
• extendable for additions to the format

Throughout the jam I looked into how milton is setup. It actually has an interesting way to deal with savefiles, that I think I've seen before but I can't recall where exactly.

It uses versioned flat structures which don't change too often. I assume it copies them directly into / out of memory, which is pretty clever.

Here is my data layout for attemping something like milton's approach

This binary based file format achieves all the goals I stated above. For a new version you have to declare new structs or reuse previous Data* structs. Write always uses the most recent Data* structs.

End

Thank you for reading this, turned out a bit longer than expected. Feel free to share your thoughts to me