Synchronizing code development workspace with Syncthing

I switch between my desktop PC and laptop quite often, especially when I work and travel. This needs to be a frictionless process as otherwise I could lose work progress or simply get discouraged from using the best tool for the job. Achieving convenience in this matter had cost me a bit of effort, but the result is well worth the price.

The easiest solution would be to use the PC as a server and connect to it whenever I work from another device. This doesn’t suit me for the following reasons:

• I don’t always have the best internet connection and need to be able to access the data offline.
• I write for various platforms and need to be able to run stuff on a locally connected device.
• Latency matters. It’s exhausting to code over an SSH session for me. Running stuff in the browser solves this issue partially, but web technology adds to the delay on its own.

The proper solution is to replicate the state between the computers and use them as independent devices. For that purpose, I decided to use Syncthing. It’s an open-source, performant, and easy-to-use tool suited for continuous file synchronization.

# Synchronizing configuration

I expect the programming setup to be the same on every computer I use. With the help of Nix and Home Manager, this is the easiest thing to achieve. All the declarative configuration is stored in a git repository. It’s not the fanciest¹ dotfiles project but is simple enough and gets the job done. Whenever I need to make some changes, I edit the config and push it. When I want to update the system on some device I just pull and rebuild it.

# Synchronizing regular data (documents, pictures, etc.)

This is where Syncthing shines the most. The configuration is plain and simple. The overall setup consists of another computer that runs 24/7 and contains its own copy of all the files. This yields two benefits: I don’t need to have other computers turned on simultaneously to sync them, and I can have an easy backup system² that runs on a single machine.

# Synchronizing workspace

All the projects that I work on have a root in a single directory (~/Workspace). Files there have high churn relative to the rest of the filesystem. Aside from making small, manual changes, there are also various byproducts of compilations, dependency fetching, or other processes related to building a program.

The most popular way of syncing project data is to use a version control system (duh). I use git for almost all the work, but it’s not what I want here. It does give me the benefit of ignoring anything that I don’t care about, is highly reliable and somewhat of an industry standard. But I don’t want to be forced to commit and push whenever I plan to stop working on a particular device. And I sometimes work on stuff that doesn’t need any version control.

Using Syncthing like in the case of regular data is also not a solution. Tracking all the changes to the build files would kill my laptop battery and cause heavy network traffic - which is not nice when using mobile internet. Although such a solution is discouraged, this tool can be adapted to handle this use case by leveraging its ignoring mechanism.

# stignore and gitignore

Syncthing ignore file (.stignore) syntax is loosely modeled on gitignore. There are a few quirks and differences. Here are some examples:

• .stignore must be placed in the root of the synced directory. Any other ignore file won’t be processed. All the rules must be relative to the directory root.
• If a negation (!) happens after related parent directory is ignored then it will have no effect.
• .stignore file isn’t synced (it’s ignored on its own). a/**/b will not match a/b.
• a**b is a valid regex in Syncthing and will match a/b, ab, a/c/b.

I wrote a simple Python script that merges all the unignored .gitignore files and makes them relative to the workspace root directory. The resulting file is stored as stignore, which is, in turn, statically linked as .stignore and thus synced between devices. Here is its outline³:

 1from gitignore_parser import parse_gitignore
 2
 3matchers = []
 4
 5def matches(path: str) -> bool:
 6    return any(matcher(path) for matcher in matchers)
 7
 8for gitignore_path in glob("**/.gitignore", recursive=True, include_hidden=True):
 9    if matches(gitignore_path):
10        continue
11    matchers.append(parse_gitignore(gitignore_path, base_dir=os.getcwd()))
12
13    with open(gitignore_path) as gitignore:
14        gitignore_dir_path = os.path.dirname(gitignore_path)
15        for line in gitignore.readlines():
16            line = line.strip()
17
18            if not line or line.startswith("#"):
19                continue
20
21            if line.startswith("!"):
22                print("!", end="")
23                line = line[1:]
24
25            if line.startswith("/"):
26                print(f"{gitignore_dir_path}{line}")
27                continue
28
29            if "/" in line and line[-1] != "/":
30                print(f"{gitignore_dir_path}/{line}")
31                continue
32
33            print(f"{gitignore_dir_path}/{line}")
34            print(f"{gitignore_dir_path}/**/{line}")

# Watching for changes

The script above works well for processing all the .gitignore files. Another problem is to run it only when necessary.

The most straightforward solution would be to run it periodically, but it has a significant flaw. Whatever the time interval is, Syncthing may begin syncing files before .stignore regeneration. This will cause conflicts when the directory is removed - other devices won’t know what to do with it as ignored files are already there. And, of course, it adds a lot of unnecessary I/O traffic.

A better solution is to watch for file changes and run the script whenever some .gitignore is modified. The most popular tool for that purpose that I’ve read about is entr, but it doesn’t trigger when a new file is added. Another solution is to use systemd’s path unit, but it doesn’t monitor the directory recursively. Finally, I found out about inotifywait (part of inotify-tools) which is able to monitor for specific file events at some directory, recursively. All that was left was to enable a small systemd service running the following script:

1inotifywait --format "%f" -e 'modify,moved_to,create,delete' -m -r ~/Workspace |
2  while read line; do
3    if [[ "$line" == ".gitignore" ]]; then
4      echo ".gitignore update"
5      # 1. Run the Python script
6      # 2. Check if there is any content difference against current stignore.
7      #    If there is then replace it.
8    fi
9  done

This causes a lot of file watches to be started. Fortunately, this is offloaded to the server machine, so I never worry about it on a computer I work with. A better solution could be to utilize Syncthing API and monitor for changes there⁴. But for the time being, I’m happy with what I have.