Applications store data
Sublime Text loads all files, buffers, layout, and other data back from the last time you quit. This allows you to quickly jump into action. I'd like
Todool's save format to be just as good.
Here's how my first file format looked like:
data:stored general info about the editor
tree:stored task related info like indentation, time info, string content, special content in
^1.123123123.1^which wasnt in the example image
Let's keep it short
- quick & easy to start with
- manually editable to add new features
- loose structure
- raw text viewable
- users could modify the data, potentially deleting data
- hard to change: manually edit, scrap your save file or go the
old read -> new write -> new readroute
- hard to read: odin does help since it has a basic scanner
- structure has to be implied by i.e. indentation, scopes or numbers in my case
- hard to write: conversions to strings all over the place, again odin helps with strconv
- big file size
- slow read / write in comparison to binary
No backwards compatibility killed a lot of my save files, usually because of my negligence when adding new data to the format.
Heres a list of some file formats.
JSON like formats offer a sense of structure through scopes. A format like that makes sense for my theme and option files.
These are just plain structs that could be dumped to
JSON. I looked into alternatives to
JSON and found out about msgpack, which is pretty much
JSON in binary with a simple spec.
I implemented msgpack and all the goodies you'd want odin-msgpack. There you can read / write entire structs and not worry too much.
Option / Theme Data that does not exist in a old / new version will simply be ignored ~ Default values will stay.
A format like this does not work for a lot of structured data like a tree though.
JSON like formats would store lots of similar data like variable names to associate back.
A custom file format makes sense here, since you can tailor that towards all you need. Or you could go for a format with schemas. These allow to predefine structures that will reappear throughout your file and be written / read back the same expected way.
Interestingly Databases work via these schemas
Here's how you work with databases
- open DB
- optionally add table / add data
- modify data throughout the program
- close DB
The DB does all the hard work for you. You have to conform to the way the DB works, which then allows an easy way to store data.
Working with a DB is a painful process. The constant back and forth messaging via
SQL kills usability to me. Transitioning a functioning codebase to a DB is also terrible, since you need to restructure everything. I actually tried this out in the
Wheel Reinvention Jam where I transitioned a week old codebase to using a DB. That process took almost a week... Clearly not suited for a jam or a long existing project.
Reflect & Goals
What I wanted from a new file format:
- small size
- direct read / writes (memcopy) data sections
- direct tree read / write which holds structure i.e. MetaDesk
- some form of backwards compatibility
- extendable for additions to the format
It uses versioned flat structures which don't change too often. I assume it copies them directly into / out of memory, which is pretty clever.
Here is my data layout for attemping something like
This binary based file format achieves all the goals I stated above. For a new version you have to declare new structs or reuse previous
Data* structs. Write always uses the most recent
Thank you for reading this, turned out a bit longer than expected. Feel free to share your thoughts to me