Officially I hate my notes

Recently I came across an interesting way of working with personal notes called Zettelkasten. I resisted it for a while, but finally jumped the bandwagon and converted my whole knowledge database to zettels, which is a German word for small post-it notes, usually kept in a box.

I am plaintext junkie. New fancy formats come and go and basic plaintext files have stayed with me for my whole life. When I started the process my whole knowledge database was kept in roughly 20-30 unordered pages, with strong “unmaintained wiki” vibes. A month later I have ~150 notes in 2 “notebooks”, 3 scripts which help me handling renaming of my notes and synchronizing them, Hugo theme, ctags language definition, freaking custom git-submodules replacement, a few custom Vim functions and mappings on top of that and strong WTF vibes. Everything resides comfortably (yeah, right) in 3 or 4 different git repositories, which is probably the biggest downside of my setup.

And I hate it. And then I love it. And then I hate it again. But ultimately I love it.

The hatred comes probably from my anxiety which I get when I start looking at it. It is complex and hacky and most probably would call it unnecessary or “you don’t have anything to do with your free time, don’t you”. But at the end of the day it just works for me. It even works on my phone terminal, which is a great achievement of modern-day technology, but not very useful as I consume my notes differently on laptop and phone. *Nively put, but as young Cerro said to King Vridank on their wedding night: “does it have any practical uses?”1

Now it’s so easy to take the notes that I actually started taking them. And thanks to Hugo, when I’m not in a plaintext mood and want to just watch these not-so-beautiful images I embed from time to time to some of my notes, I can just generate my custom not-very-unpretty static webpage in a matter of milliseconds, copy them to /var/www, where I have Python’s simple HTTP server2 constantly running on port 9999, type that 9999 in a web browser and bam! watch my glorious notes.

Structure

I store notes in Git, which is the only system I wholeheartedly trust to help me recover from my stupidity. I believe that my SSD will die sooner than any of my Git repos lose integrity and thanks to its decentralized nature I usually have copies on 2, 3 or 4 different, geographically distant devices. I already deleted some of my notes by accident but thanks to Git they’re versioned, so it was trivial to recover them.

In the past, when I keptn notes synchronized with syncthing I sometimes run to synchronization conflicts. These weren’t that bad. Bad was when empty copy of single file overrided copies with content and OF COURSE I haven’t thought about enabling Syncthing’s file versioning.

My notes are stored in several notebooks and each one of those is a separate Git repo which contains notes only, optionally attachments. All notebooks are bundled inside a parent Git repo with Hugo theme and configuration files. It looks like this:

./              <parent repo>
  content/
    _index.md   <main page of generated site>
    0/          <personal git repo>
      files/    <attachments>
    other/      <work git repo>
      files/    <attachments>

Sounds like a job for Git Submodules, right? Yeah, almost. There are 3 reasons why I don’t want submodules:

  1. They complicate the workflow of notes syncing because I have to remember to set-up branch-tracking submodules, update them and then update them (commit + push) in parent repo.
  2. I want to have conditional submodules. Why? Because I want to keep my work-related notes only on one laptop and specifically keep them as far as I can from my phone to avoid any IP theft accusations.
  3. My work forces me to connect to internet via a proxy, which only opens ports 80 and 443, so connecting to git via ssh is no-go at work but it is my preferred method on any other machine.

So I created csync script which reads submodule definitions from a file named modules. It is a very simple text file which contains 3 columns: hostname, local path and repo URL. csync iterates this file and when hostname matches with machine’s hostname, it synchronizes URL under the requested path. For example:

work    content/0       https://personal.git
any     content/0       ssh://personal.git

And csync:

#!/bin/bash

synchronized=()

while IFS=' ' read -r host dir url
do
    # skip if host doesn't match; "any" is special host which always matches
    if [[ "$(hostname)" != "$host" ]] && [[ "$host" != "any" ]]; then
        continue
    fi

    # skip if directory was already synchronized
    if [[ " ${synchronized[@]} " =~ " $dir " ]]; then
        continue
    fi

    if [[ ! -e "$dir" ]]; then
        git clone "$url" "$dir" && synchronized+="$dir"
    else
        pushd "$dir" > /dev/null

        # Better use git-sync instead; read note below!
        git pull --rebase --autostash && synchronized+="$dir"
        if [ -n "`git status --porcelain`" ]; then
            git add -A && git commit -m "sync from `hostname`" && git push
        fi

        popd > /dev/null
    fi
done < "modules"

2020-02-10: Instead of manual git operations as presented above, which are far from being perfect (I once ended with a conflict report instead of file), I switched to the superior git-sync script.

Cross References

2020-02-12: I changed my approach for Cross References. You can read about it in my next post.

One of central ideas of Zettelkasten is creating and maintaining a graph of connections between notes. One way to do it is by tagging each note. I use literal tags for this purpose: ctags, which I already described in my other post. Because 95% of them live in notes’ frontmatters, they are also understood by Hugo, which correctly renders them the to HTML.

Notes cross referencing is equally important. Markdown allows creating hyperlinks in form of [descr](file.md). It’s super handy when I read it in Vim, because I can place cursor over file.md part, press gf and Vim opens that file for me. That’s all blows and whistles, but URLs written this way don’t go along with Hugo because they’re interpreted as relative paths. That’s why I use shortcodes for cross references provided by Hugo:

[descr]({{</* ref "file.md" */>}})

I combine them with UltiSnips, because shortcode syntax is a little too much writing for me:

snippet ref "Hugo reference"
[$1]({{</* ref "$2" */>}})$0
endsnippet

Additionally I created a markdown-local mapping which helps me with gf part:

" after/ftplugin/markdown.vim
nnoremap <buffer> <cr> f"gf

Cross-linking cannot work without permalinks, period. Niklas Luhmann, who initially developed the system, assigned subsequent letters and numbers, indicating positions of notes in a box (like 21/3d7, 21/3a1). Today people usually recommend using some kind of timestamp for note IDs, which I tried and hated. After a quick brainstorm I invented my own format: some-title-{8-digits-of-UUID}. For example: android-51fa716a.md, ipfs-82a24c1.md etc. Pretty unconventional, I say, but I had good reasons to do it this way:

  1. File names must start with human readable string (note topic). That’s because sometimes I want to browse them from the phone (with limited search capabilities) or with a file manager (usually some kind of netrw clone to quickly go to the next note on the topic).
  2. Files must be unique. I don’t want conflicts when I create 2 notes on a single topic on 2 different devices.
  3. File names must not be too long or unreadable. Timestamps look horrible, but UUIDs have a nice property of being a mix between letters and numbers which is easier for eye in my opinion.

That’s why I use first 8 hexadecimal digits of generated random UUID. It allows storing up to 168 notes of the same topic which is unique enough for me3. Even if I run into a collision (I doubt it), it will be spotted by me so I can just generate next short UUID.

Of course I don’t create those UUIDs by hand. I wrote a vim function which does that for me. For example I can write :Note foo and it’ll create a file named 0/aaa-{short-uuid} (0 is my default notebook, but the function detects if I specified any other notebook). When available, it uses uuidgen and Python uuid.uuid4() as a fallback.

To ease referencing newly created notes in other notes I also created a command which yanks basename of current buffer: YankNoteName. The command name itself doesn’t look very friendly, but actually is the only user command on my system which starts with Y, so I only have to type :Y<Tab><CR> to call it.

I admit that I’m no vimscript pro and that I hate every moment spent writing vimscript, so the code might be below any standards. But hey, it works so let’s call it a day!

" 99notes.vim
func! s:shortuuid()
    let l:uuid = ""
    if executable('uuidgen')
        let l:uuid=system('uuidgen')[:-2]
    else
python << ENDPY
import vim
from uuid import uuid4
vim.command("let l:uuid = '%s'"% str(uuid4()))
ENDPY
    endif

    let l:spl = split(l:uuid, '-')
    return tolower(l:spl[0])
endfunction

func! custom#99notes#edit(name)
    let l:uuid = s:shortuuid()

    " assign note to a default notebook
    let l:default_notebook = "0/"
    let l:name = a:name =~ '.\+\/.\+' ? a:name : l:default_notebook . a:name
    let l:fname = expand("~/docs/content/") . l:name . "-" . l:uuid . ".md"
    exec "e " . l:fname
    exec "keepalt r " . expand("~/docs/archetypes/default.md")
    exec "normal ggdd"
endfunc

func! custom#99notes#copynotename()
    let @" = expand("%:t:r")
endfunc

command! -nargs=1 Note call custom#99notes#edit(<q-args>)
command! YankNoteName call custom#99notes#copynotename(<f-args>)

But even with all of that sometimes I might have to rename a note. For example, when I create a note from Markor. That’s when rename script comes into play. It not only renames files, but also searches and replaces all their back-references:

#!/bin/bash

from=$1
to=$2
frombase=${from##*/}  # remove largest prefix
tobase=${to##*/}

dir="$(dirname "$from")"

for cf in content/*; do
    if [[ ! -d "$cf" ]]; then
        continue
    fi

    files=$(git -C "$cf" ls-files "*.md")

    pushd "$cf" > /dev/null
    sed -i "s/$frombase/$tobase/g" $files
    popd  > /dev/null
done

git -C "$dir" mv "$frombase" "$tobase"

But manual searching of files that don’t conform to my filename spec is still too much work to do, so I have bulkrename script which fixes my laziness. Well, not exactly fixes laziness itself, but rather its consequences. It takes a note’s title and creates a correct file name from it, with spaces replaced by dashes and all.

#!/bin/bash

skip=("content/_index.md" "content/0/quicknotes.md")

for dir in content/*; do
    if [[ ! -d "$dir" ]]; then
        continue
    fi

    files=$(git -C "$dir" ls-files "*.md")
    for f in $files; do
        oldf="$dir/$f"

        # skip explicitly listed files
        if [[ " ${skip[@]} " =~ " $oldf " ]]; then  # check if array contains a value
            echo "Skipping $oldf (excluded)"
            continue
        fi

        # skip files which already match a filename pattern (file-shortuuid.md)
        # where shortuuid are first 8 hexadecimal numbers od full UUID
        if [[ "$f" =~ ^.+-[a-f0-9]{8}.md$ ]]; then
            echo "Skipping $oldf (name correct)"
            continue
        fi

        shortuuid=$(uuidgen | sed 's/-.*//')
        title="$(awk -F '[ ().?!,/-]+' '/^title:/ {for(i=2;i<NF;i++) printf "%s-", tolower($i); print tolower($NF)}' "$oldf")"
        newf="$dir/$title-$shortuuid.md"
        echo "Renaming $oldf -> $newf"
        ./rename "$oldf" "$newf"
    done
done

One thing it lacks is a progress bar. HAHA. HAHA. HA. And now I’m thinking about implementing one (which shouldn’t be that hard). Thank you very much, brain, I love you too, that’s exactly what my notes setup lacks: a progress bar in a script which will be used maybe once a year or less. Oh boy, have you seen these nice progress bars that recent pytest versions have? I think they even have CONCURRENT progress bars in pytest-xdist! Maybe I can get one too? Because I certainly can run this script concurrently!

  1. Andrzej Sapkowski - The Lady of the Lake ↩

  2. python3 -m http.server 9999 ↩

  3. Trivia: it’d take over 136 years to create that many notes if a note was created every second. ↩