Some thoughts on Rust

Some backstory

I have needed a tool to manage my local Ubuntu and Debian mirrors. Aptly is probably the best tool available to do the job right now. However, none of the available tools are actually great. Most are in varying degrees of non-maintenance or throw out hordes of errors because they look for every file variation even though the Ubuntu and Debian mirror creation tools only release 1 or 2 variations.

Why the new tool?

Since I run several physical and virtual machines on my local network, I need a local software mirror. I am also packaging a few updated or tweaked Debian packages, so I need a storage location for those packages. As a result, I am mirroring the stable and testing versions of Debian, and all the versions of Ubuntu back to the current LTS, Focal Fossa. Add to those a few extra repositories I mirror, PPA’s basically, and my maintenance scripts are getting ridiculous. I am getting tired of directly modifying that massive pile of shell scripting every time a new distro release happens. To help with that, I have released a new tool that will create the non-PPA chunk of the shell scripts from scratch.

Why Rust?

After trying to build the same piece of software in Go, PHP, and actual proper shell, this time, I decided to try Rust. The result was pretty interesting. Importing dynamic data structures from any configuration file sucks. While I spent about 12 hours figuring out that the TOML would take longer than I originally wanted to implement. The hard-coded configuration took about 16 hours to build and test. You can get a copy from https://github.com/rrbrussell/aptly_manager.

I would describe using Rust so far as enjoyable frustration. I use a code, run, evaluate, repeat development cycle. While cargo build is not as fast as go build, it is quick enough that I can work at a fairly productive rate. When I understand the language in front of me. The completion of the first sub-command to the final v0.1.0 took about one, maybe two hours. I rapidly completed the last part compared to the fourteen or fifteen hours I spent bumbling around parsing command lines and figuring out Rust’s borrow checker.

Again Why Rust?

I am more familiar with Go, PHP, Java, and C# now than with Rust. However, even accounting for the learning curve, I prefer Rust. First off, it integrates better into Linux than C# or .NET. This would not be a big issue, except I like using one language for most tasks, and it appears to be easier to get Rust and GTK working on Windows than C# and GTK working on Linux. There are a couple of larger projects that I have on the back burner. My options for those projects boil down to three options

  • using PHP for the back end, with a .NET UI,
  • use .NET for both the front and back ends,
  • use Rust for both the front and back ends.

Go unfortunately does not work for any of the options because it does not support dynamically loaded plugins or have a useable GUI library available. Rust may not fully support dynamically loading plugins, but it is probably as safe and idiot-proof as Go and supports generics. Rust supports C’s foreign function interface natively, so the worst case is I may write the plugin management code in C and everything else in Rust.

ZFS Backup Tool Part 2

Recognizing a snapshot made by zfs-auto-snapshot.

First, what does a list of these snapshots look like?

robert@mars:~/src/go/zfs_backup$ zfs list -Hrt snapshot dpool
dpool@zfs-auto-snap_monthly-2020-05-12-1245     96K     -       148K    -
dpool@zfs-auto-snap_monthly-2020-06-11-1248     8K      -       23.3G   -
dpool@zfs-auto-snap_monthly-2020-07-11-1245     0B      -       23.3G   -
dpool@zfs-auto-snap_weekly-2020-07-26-1242      0B      -       30.5G   -
dpool@zfs-auto-snap_daily-2020-07-27-1238       4.74G   -       31.3G   -
dpool@zfs-auto-snap_daily-2020-08-02-1235       0B      -       143G    -
dpool@zfs-auto-snap_weekly-2020-08-02-1240      0B      -       143G    -
dpool@zfs-auto-snap_hourly-2020-08-03-1117      0B      -       143G    -
dpool@zfs-auto-snap_daily-2020-08-03-1236       0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2030    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2045    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2100    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2115    0B      -       143G    -
dpool@zfs-auto-snap_hourly-2020-08-04-2117      0B      -       143G    -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1717 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1817 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1917 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-2017 0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2030       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2045       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2100       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2115       0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-2117 0B      -       96K     -
dpool/home/robert@zfs-auto-snap_frequent-2020-08-04-2115        0B      -       69.1G   -
dpool/home/robert@zfs-auto-snap_hourly-2020-08-04-2117  0B      -       69.1G   -
dpool/plex@snap1        116G    -       442G    -
dpool/plex@zfs-auto-snap_monthly-2020-05-12-1245        8K      -       344G    -

I trimmed the previous list down a bit. So what is a regular expression that will match this? The first question is which regular expression library am I using? I am writing this tool in Go. Thus I will use the regexp Go package. Go’s regexp package is based on Google’s RE2 library. The syntax for it is here.

I will start with the snapshot names. The part after the @. Those start with zfs-auto-snap so "zfs-auto-snap" will match it.

The next section is which timer made the snapshot. This section can also be called the increment. The valid timers are yearly, monthly, weekly, daily, hourly, and frequent for a default install of zfs-auto-snapshot. The regex "yearly|monthly|weekly|daily|hourly|frequent" will match these timers. However, I would like to get which timer created the snapshot without further parsing. That is the perfect job for a capturing sub match. After adding the capturing sub match, the regex looks like "(?P<increment>yearly|monthly|weekly|daily|hourly|frequent)".

The final section is the timestamp of the snapshot. Like with the timer section, it is useful not to have to parse this data a second time. With the sub matches "(?P<year>[[:digit:]]{4})-(?P<month>[[:digit:]]{2})-(?P<day>[[:digit:]]{2})-(?P<hour>[[:digit:]]{2})(?P<minute>[[:digit:]]{2})" will work.

With the snapshot names completed, I need to capture the zfs tree structure before the @ symbol. I haven’t found a reliable regular expression that will capture that tree but "(?:[[:word:]-.]+)+(?:/?[[:word:]-.]+)*" will recognize a subset of all valid zfs trees. Avoid using anything it won’t recognize, or you may end up with inaccessible files.

Including some test code the tool’s source code looks like this so far.

package main

import (
	"bufio"
	"fmt"
	"os"
	"regexp"
)

const zfsRegexStart string = "zfs-auto-snap"
const zfsRegexIncrement string = "(?P<increment>yearly|monthly|weekly|daily|hourly|frequent)"
const zfsRegexDateStamp string = "(?P<year>[[:digit:]]{4})-(?P<month>[[:digit:]]{2})-(?P<day>[[:digit:]]{2})-(?P<hour>[[:digit:]]{2})(?P<minute>[[:digit:]]{2})"

var zfsRegex = regexp.MustCompile(zfsRegexStart + "_" + zfsRegexIncrement + "-" + zfsRegexDateStamp)

func testSnapshot(possible string, increment string) (bool, bool) {
	var matches = zfsRegex.FindStringSubmatch(possible)
	if matches == nil {
		return false, false
	}
	var isASnapshot = true
	if matches[1] == increment {
		return isASnapshot, true
	}
	return isASnapshot, false
}

func isAYearlySnapshot(possible string) bool {
	_, isYearly := testSnapshot(possible, "yearly")
	return isYearly
}

func isAMonthlySnapshot(possible string) bool {
	_, isMonthly := testSnapshot(possible, "monthly")
	return isMonthly
}

func isAWeeklySnapshot(possible string) bool {
	_, isWeekly := testSnapshot(possible, "weekly")
	return isWeekly
}

func isADailySnapshot(possible string) bool {
	_, isDaily := testSnapshot(possible, "daily")
	return isDaily
}

func isAnHourlySnapshot(possible string) bool {
	_, isHourly := testSnapshot(possible, "hourly")
	return isHourly
}

func isAFrequentSnapshot(possible string) bool {
	_, isFrequent := testSnapshot(possible, "frequent")
	return isFrequent
}

const poolNameRegex string = "(?:[[:word:]-.]+)+(?:/?[[:word:]-.]+)*"

var snapshotLineRegex = regexp.MustCompile("^" + poolNameRegex + "@" + zfsRegex.String() + ".*$")

func main() {
	//fmt.Println(snapshotLineRegex.MatchString("dpool/www@zfs-auto-snap_frequent-2020-08-04-1830\t0B\t-\t201M\t-"))
	input := bufio.NewScanner(os.Stdin)
	for input.Scan() {
		if snapshotLineRegex.MatchString(input.Text()) {
			fmt.Println(snapshotLineRegex.FindStringSubmatch(input.Text()))
			fmt.Println(snapshotLineRegex.SubexpNames())
		} else {
			fmt.Printf("%s\t%s\n", input.Text(), "Is not a snapshot.")
		}
	}
	if err := input.Err(); err != nil {
		fmt.Fprintln(os.Stderr, "reading Standard Input:", err)
	}
}

I will continue this tomorrow. See you then!