Skytrias | Michael Kutowski — 11 months, 1 week ago

Applications store data

Sublime Text loads all files, buffers, layout, and other data back from the last time you quit. This allows you to quickly jump into action. I'd like Todool's save format to be just as good.

Here's how my first file format looked like:

data: stored general info about the editor
tree: stored task related info like indentation, time info, string content, special content in ^1.123123123.1^ which wasnt in the example image

Let's keep it short

Pros
- quick & easy to start with
- manually editable to add new features
- loose structure
- raw text viewable
Cons
- users could modify the data, potentially deleting data
- hard to change: manually edit, scrap your save file or go the old read -> new write -> new read route
- hard to read: odin does help since it has a basic scanner
- structure has to be implied by i.e. indentation, scopes or numbers in my case
- hard to write: conversions to strings all over the place, again odin helps with strconv
- big file size
- slow read / write in comparison to binary

No backwards compatibility killed a lot of my save files, usually because of my negligence when adding new data to the format.

Exploration

Heres a list of some file formats. JSON like formats offer a sense of structure through scopes. A format like that makes sense for my theme and option files.

option file

These are just plain structs that could be dumped to JSON. I looked into alternatives to JSON and found out about msgpack, which is pretty much JSON in binary with a simple spec.

I implemented msgpack and all the goodies you'd want odin-msgpack. There you can read / write entire structs and not worry too much.

Important

Option / Theme Data that does not exist in a old / new version will simply be ignored ~ Default values will stay. A format like this does not work for a lot of structured data like a tree though. JSON like formats would store lots of similar data like variable names to associate back.

A custom file format makes sense here, since you can tailor that towards all you need. Or you could go for a format with schemas. These allow to predefine structures that will reappear throughout your file and be written / read back the same expected way.

Interestingly Databases work via these schemas

Database?

Some applications actually use databases as an application file format. sqlite is a C library that you can easily use, so I did in odin-sqlite.

Here's how you work with databases

open DB
optionally add table / add data
modify data throughout the program
close DB

The DB does all the hard work for you. You have to conform to the way the DB works, which then allows an easy way to store data.

Working with a DB is a painful process. The constant back and forth messaging via SQL kills usability to me. Transitioning a functioning codebase to a DB is also terrible, since you need to restructure everything. I actually tried this out in the Wheel Reinvention Jam where I transitioned a week old codebase to using a DB. That process took almost a week... Clearly not suited for a jam or a long existing project.

Reflect & Goals

What I wanted from a new file format:

small size
direct read / writes (memcopy) data sections
direct tree read / write which holds structure i.e. MetaDesk
some form of backwards compatibility
extendable for additions to the format

Throughout the jam I looked into how milton is setup. It actually has an interesting way to deal with savefiles, that I think I've seen before but I can't recall where exactly.

It uses versioned flat structures which don't change too often. I assume it copies them directly into / out of memory, which is pretty clever.

Here is my data layout for attemping something like milton's approach

This binary based file format achieves all the goals I stated above. For a new version you have to declare new structs or reuse previous Data* structs. Write always uses the most recent Data* structs.

End

Thank you for reading this, turned out a bit longer than expected. Feel free to share your thoughts to me

Update + Discord

Skytrias | Michael Kutowski — 1 year, 2 months ago

Hey everyone, i have 2 weeks of vacation so i plan to work more on Todool these weeks.

I opened a Discord if anyone is interested on following Development more closely there.

The design decision of sticking to Trees + different Sub Node Types holds up really nicely. Through that the editor starts to feel really solid. So thinking about the design of the project you're making is something I'd really suggest, otherwhise you'll end up like me abondoning ideas completly and having to start over. That's why the project has gone on for so long.

I implemented multiple Views (only vertical) as well, in which you can open trees independently and focus on them.
The search mechanism was recently in a Tree too, which i thought was clever but didnt quite work out. So instead I pulled that out into its own mechanism + keymapping.
Same thing with the (Date) Time Tree, which presented uneditable data in a tree. The idea was that you could open that in a split, but thats quite useless. Now I've been thinking of making something like a vim bottom bar in case you want to see other kinds of data quickly. Maybe holding shift will expand that bottom bar / hide it, since shift isnt used in the editor.

Also recorded some showcase + working on Task Pointers https://www.youtube.com/watch?v=LZAVyqv7sEI

PS: if you only use the forum, you can stay up to date with videos / gifs that i send by clicking on my profile

Focusing on Trees

Skytrias | Michael Kutowski — 1 year, 5 months ago

Todool's old structure relied on Task and Text Nodes alone.

TaskNode :: struct {
    prev, next: ^TaskNode,
    parent: ^TaskNode,
    
    task_head, task_tail, task_jump: ^TaskNode,
    text_head, text_tail: ^TextNode,
    
    ...
}

TextNode :: struct {
    parent: ^TaskNode,
    prev, next: ^TextNode,
    
    length: u8,
    bytes: [BOX_CAP]byte,
}

This was obviously bad. It seriously limited the more advanced things i wanted to do. I wanted TextNode's to be more generic. Since i got really frustrated i began thinking about another issue i didnt like about Todool, the UI.

I had built a Retained Mode UI (RMGUI) around the tree data and a completly seperate Immediate Mode UI (IMGUI) for the sidebar, menu, notifications, etc. This was terrible since i had 2 different systems all working in a different way and not playing well together.

So i started experimenting with RMGUI again, since i really liked how powerful they can be. Take Luigi from nakst for example, simple yet powerful. I tried pulling in different RMGUI experiments into Todool but again, they were seperate systems which i hated.

I would have rather liked forming the UI around Todool itself.

So i experimented trying to pull everything together, while also making the SubNodes in Tasks more generic. Here's where I'm at right now.

Twitter for videos

On my twitter you can see the proof of concept and now I'm diving more deeply into the concept since i really like it.

The new data layout is the following (using Odin).

TaskNode :: struct {
	parent, prev, next: ^TaskNode,
	head, tail: ^TaskNode,
	
	sub_head, sub_tail: ^SubNode,

	...

	custom_calls: map[ShortcutCall]proc(),
	custom_draw: proc(node: ^TaskNode),
}

SubNode :: struct {
	parent: ^TaskNode,
	prev, next: ^SubNode,
	data: SubData,
	...
}

SubData :: union {
	SubText,
	SubButton,
	SubToggle,
	SubU8,
	SubInt,
	SubF32,
	SubHex,
	
	SubColor,
	SubPadding,
	...
}

SubText :: struct {
	text: string,
}

SubButton :: struct {
	call: proc(),
}

SubToggle :: struct {
	state: ^bool,
}

SubU8 :: struct {
	ptr: ^u8,
	clamped: bool,
}

SubInt :: struct {
	ptr: ^int,
	start, end: int, // range
	clamped: bool,
}

SubF32 :: struct {
	ptr: ^f32,
	start, end: f32, // range
	clamped: bool,
}

SubColor :: struct {
	color: ^Color,
}

SubHex :: struct {
	ptr: ^Color,
	clamped: bool,
}

SubPadding :: struct {
	width: int,
	height: int,
}

You can see this allows for way more than my old code ever could've and I'm not even done adding different SubNode types yet.

In this case SubText is the general text node that can be edited. SubValue types are values that get converted to text when edited and reconverted back to their pointer locations. Also can do special types like padding or colors.

Here's some usage code for creating the Options tree seen in some of the videos on twitter.

create_option_nodes :: proc(x, y: int) -> ^TaskNode {
	options := task_scoped();
	options.rect.l = x;
	options.rect.t = y;
	options.disabled = true;
	sub_init(SubText { "options" }).disabled = true;;
	
	{
		task_scoped().disabled = true;
		sub_init(SubText { "visual" }).disabled = true;;
		
		{
			task_scoped().disabled = true;
			sub_init(SubText { "tab" }).disabled = true;
			sub_init(SubInt { ptr = &ui.tab, start = 10, end = 100 });
		}
		
		{
			task_scoped().disabled = true;
			sub_init(SubText { "line_height" }).disabled = true;
			sub_init(SubInt { ptr = &ui.line_height, start = 20, end = 50 });
		}
		
		{
			task_scoped().disabled = true;
			sub_init(SubText { "font" }).disabled = true;
			font := &core.fonts[.Editor];
			sub_init(SubF32 { ptr = &font.temp_size, start = font.min_size, end = font.max_size });
			
		}
	}
	
	return options;
}

The disabled params are still temporary - they just disable things from being changed, which i could probably make on a parent task basis.

I know this was long, what do you guys think though? Jumping into Theme editing all with the same movement for the actual program itself feels really powerful and for everything else.

first post

Skytrias | Michael Kutowski — 1 year, 7 months ago

Welcome to Todool

I've been working on this project since August 2019 in my free time. I started prototyping with Raylib, afterwards I wanted to go more lower level and started implementing rxi's cached software renderer. Took a bit of time but at the end I'm happy with it.

Remember I work on this project alone, so progress is slow. You can follow my progress on Twitter, I will use the blogs here to show more of the technical / code aspects.

Suggestions are welcome on the forum or general discussion.

This week i worked on implementing networking for collaboration in the editor. I want it to be kind of live, i.e. Google Docs. It's coming along slowly, still goes out of sync a few times. My idea is to sync all users to the host file, simulate key / mouse input across the network.