Posted on : By : Robert Posted in : Medium|Thoughts

Some thoughts on Rust

Some backstory

I have needed a tool to manage my local Ubuntu and Debian mirrors. Aptly is probably the best tool available to do the job right now. However, none of the available tools are actually great. Most are in varying degrees of non-maintenance or throw out hordes of errors because they look for every file variation even though the Ubuntu and Debian mirror creation tools only release 1 or 2 variations.

Why the new tool?

Since I run several physical and virtual machines on my local network, I need a local software mirror. I am also packaging a few updated or tweaked Debian packages, so I need a storage location for those packages. As a result, I am mirroring the stable and testing versions of Debian, and all the versions of Ubuntu back to the current LTS, Focal Fossa. Add to those a few extra repositories I mirror, PPA’s basically, and my maintenance scripts are getting ridiculous. I am getting tired of directly modifying that massive pile of shell scripting every time a new distro release happens. To help with that, I have released a new tool that will create the non-PPA chunk of the shell scripts from scratch.

Why Rust?

After trying to build the same piece of software in Go, PHP, and actual proper shell, this time, I decided to try Rust. The result was pretty interesting. Importing dynamic data structures from any configuration file sucks. While I spent about 12 hours figuring out that the TOML would take longer than I originally wanted to implement. The hard-coded configuration took about 16 hours to build and test. You can get a copy from https://github.com/rrbrussell/aptly_manager.

I would describe using Rust so far as enjoyable frustration. I use a code, run, evaluate, repeat development cycle. While cargo build is not as fast as go build, it is quick enough that I can work at a fairly productive rate. When I understand the language in front of me. The completion of the first sub-command to the final v0.1.0 took about one, maybe two hours. I rapidly completed the last part compared to the fourteen or fifteen hours I spent bumbling around parsing command lines and figuring out Rust’s borrow checker.

Again Why Rust?

I am more familiar with Go, PHP, Java, and C# now than with Rust. However, even accounting for the learning curve, I prefer Rust. First off, it integrates better into Linux than C# or .NET. This would not be a big issue, except I like using one language for most tasks, and it appears to be easier to get Rust and GTK working on Windows than C# and GTK working on Linux. There are a couple of larger projects that I have on the back burner. My options for those projects boil down to three options

  • using PHP for the back end, with a .NET UI,
  • use .NET for both the front and back ends,
  • use Rust for both the front and back ends.

Go unfortunately does not work for any of the options because it does not support dynamically loaded plugins or have a useable GUI library available. Rust may not fully support dynamically loading plugins, but it is probably as safe and idiot-proof as Go and supports generics. Rust supports C’s foreign function interface natively, so the worst case is I may write the plugin management code in C and everything else in Rust.

Posted on : By : Robert Posted in : Medium

Don’t leave web server performance on the table.

If you followed the WordPress Install Guide you are leaving performance on the table for your website. The WordPress Install Guide defaults to Apache’s mod PHP which is definitely leaving performance on the table.

Here is how to fix that in Ubuntu 20.04:

  1. install php-fpm: sudo apt install php-fpm
  2. disable mod_php and mpm_prefork: sudo a2dismod php mpm_prefork
  3. enable mpm_event proxy_fcgi and setenvif: sudo a2enmod mpm_event proxy_fcgi setenvif
  4. enable the php-fpm configuration file: sudo a2enconf php7.4-fpm
  5. mark apache2 as a manually installed package: sudo apt-mark manual apache2
  6. remove mod_php: sudo apt purge libapache2-mod-php*

That takes care of the general Apache configuration. The next configuration adjustments need to be done at the vhost level. Substitute the correct information for YOUREMAIL, YOURDOMAIN, and WPDIR, and you are good to go.

<VirtualHost *:443>
Protocols h2 http/1.1
ServerName YOURDOMAIN
ServerAdmin YOUREMAIL@YOURDOMAIN.COM

DocumentRoot "WPDIR"

<Directory "WPDIR">
    <FilesMatch "\.php$">
 #Proxy php files to php-fpm
        SetHandler  "proxy:unix:/run/php/php-fpm.sock|fcgi://localhost/"
    </FilesMatch>
    Options All
    AllowOverride All
    DirectoryIndex index.php
    Require all granted
</Directory>

#Protect the uploads directory.
<Directory "WPDIR/wp-content/uploads/">
    <FilesMatch "\.php$">
        SetHandler  none
        Require all denied
    </FilesMatch>
</Directory>

#snip extraneous stuff
</VirtualHost>

Let me take a minute to talk about performance. I have a small Atom 230 based ITX computer that I use for some network administration tasks and as a CPU limited system for benchmarking.

robert@pallas:~$ lscpu
Architecture:                    x86_64
CPU op-mode(s):                  32-bit, 64-bit
Byte Order:                      Little Endian
Address sizes:                   32 bits physical, 48 bits virtual
CPU(s):                          2
On-line CPU(s) list:             0,1
Thread(s) per core:              2
Core(s) per socket:              1
Socket(s):                       1
NUMA node(s):                    1
Vendor ID:                       GenuineIntel
CPU family:                      6
Model:                           28
Model name:                      Intel(R) Atom(TM) CPU  230   @ 1.60GHz
Stepping:                        2
CPU MHz:                         1595.763
BogoMIPS:                        3191.77
L1d cache:                       24 KiB
L1i cache:                       32 KiB
L2 cache:                        512 KiB
NUMA node0 CPU(s):               0,1
Vulnerability Itlb multihit:     Not affected
Vulnerability L1tf:              Not affected
Vulnerability Mds:               Not affected
Vulnerability Meltdown:          Not affected
Vulnerability Spec store bypass: Not affected
Vulnerability Spectre v1:        Not affected
Vulnerability Spectre v2:        Not affected
Vulnerability Srbds:             Not affected
Vulnerability Tsx async abort:   Not affected
Flags:                           fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat clflush dts acpi mmx fxsr sse sse2 ss h
                                 t tm pbe syscall nx lm constant_tsc arch_perfmon pebs bts nopl cpuid aperfmperf pni dtes64 monitor ds_cpl
                                  tm2 ssse3 cx16 xtpr pdcm movbe lahf_lm dtherm
robert@pallas:~$

Pallas isn’t fast but it does make a good test bench for what performance difference you might see on a very limited server. I will leave the full benchmark tests below. To summarize the performance looks pretty similar from a users perspective. However, if I point out that the mpm_prefork test had a load average of 88.72 while the mpm_event test had a load average of 11.04 you can see that while mpm_event may not be faster than mp_prefork from a users perspective it does help with server load.

You can expect a return to this topic once I find a way to stress test HTTP 2 connections.

Prefork
robert@Venus:~$ sudo ab -c 100 -n 100000 -r http://192.168.1.12/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.12 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        Apache/2.4.41
Server Hostname:        192.168.1.12
Server Port:            80

Document Path:          /
Document Length:        10918 bytes

Concurrency Level:      100
Time taken for tests:   110.875 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      1119200000 bytes
HTML transferred:       1091800000 bytes
Requests per second:    901.91 [#/sec] (mean)
Time per request:       110.875 [ms] (mean)
Time per request:       1.109 [ms] (mean, across all concurrent requests)
Transfer rate:          9857.63 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    4  14.6      3    1039
Processing:     2  107  77.7    108     611
Waiting:        1   29  27.3     21     403
Total:          3  111  78.9    112    1260

Percentage of the requests served within a certain time (ms)
  50%    112
  66%    151
  75%    170
  80%    182
  90%    212
  95%    239
  98%    273
  99%    298
 100%   1260 (longest request)
robert@Venus:~$
Event
robert@Venus:~$ sudo ab -c 100 -n 100000 -r http://192.168.1.12/
This is ApacheBench, Version 2.3 <$Revision: 1843412 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking 192.168.1.12 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Completed 100000 requests
Finished 100000 requests


Server Software:        Apache/2.4.41
Server Hostname:        192.168.1.12
Server Port:            80

Document Path:          /
Document Length:        10918 bytes

Concurrency Level:      100
Time taken for tests:   118.068 seconds
Complete requests:      100000
Failed requests:        0
Total transferred:      1119200000 bytes
HTML transferred:       1091800000 bytes
Requests per second:    846.97 [#/sec] (mean)
Time per request:       118.068 [ms] (mean)
Time per request:       1.181 [ms] (mean, across all concurrent requests)
Transfer rate:          9257.12 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0    3  40.6      1    3143
Processing:    11  115  25.5    110     454
Waiting:        2  108  21.4    107     384
Total:         19  118  47.7    111    3235

Percentage of the requests served within a certain time (ms)
  50%    111
  66%    115
  75%    121
  80%    128
  90%    147
  95%    165
  98%    192
  99%    216
 100%   3235 (longest request)
robert@Venus:~$

—Robert

Posted on : By : Robert Posted in : Medium

ZFS Backup Tool Part 3

Today’s project is parsing a snapshot into a custom datatype that gives us more accessible options to manipulate snapshots. First, the regular expression strings need to be moved into separate files so I can reference them across other files.

The essential parts of a snapshot are:

  • the pool name
  • the filesystem tree
  • the Interval
  • the TimeStamp

To parse a snapshot out of a string.

  • Confirm that the string matches our regular expression for snapshots and only contains the regular expression for a snapshot.
  • If it does not return an error otherwise continue
  • Split the input string into the path and the snapshot name
  • Parse the snapshot name into the interval and timestamp fields
  • Split the path into the pool name and any filesystem tree portions.

Below is a function that implements the listed requirements.

/*
ParseSnapshot parses a string into a Snapshot.

It returns nil on error.
*/
func ParseSnapshot(input string) *Snapshot {
	var snapshotOnly, err = regexp.Compile("^" + PoolNameRegex + "@" + ZfsSnapshotNameRegex + "$")
	if err != nil {
		return nil
	}
	if !snapshotOnly.MatchString(input) {
		return nil
	}
	var snapshotPieces []string = snapshotOnly.FindStringSubmatch(input)
	var theSnapshot = Snapshot{}
	theSnapshot.Interval = intervalStringToUInt(snapshotPieces[1])
	var year, month, day, hour, minute int
	year, err = strconv.Atoi(snapshotPieces[2])
	if err != nil {
		return nil
	}
	month, err = strconv.Atoi(snapshotPieces[3])
	if err != nil {
		return nil
	}
	day, err = strconv.Atoi(snapshotPieces[4])
	if err != nil {
		return nil
	}
	hour, err = strconv.Atoi(snapshotPieces[5])
	if err != nil {
		return nil
	}
	minute, err = strconv.Atoi(snapshotPieces[6])
	if err != nil {
		return nil
	}
	theSnapshot.TimeStamp = time.Date(year, time.Month(month), day, hour, minute, 0, 0, time.UTC)
	var splitInput []string = strings.Split(input, "@")
	if len(splitInput) != 2 {
		return nil
	}
	var paths []string = strings.Split(splitInput[0], "/")
	theSnapshot.pool = paths[0]
	if len(paths) > 1 {
		copy(theSnapshot.fsTree, paths[1:])
	}
	return &theSnapshot
}

func intervalStringToUInt(input string) uint64 {
	switch input {
	case "yearly":
		return 0
	case "monthly":
		return 1
	case "weekly":
		return 2
	case "daily":
		return 3
	case "hourly":
		return 4
	}
	return 5
}

Now that I can create a Snapshot structure I need some utility methods for them.

  • Read the pool snapshot and file system tree as a single string.
  • Compare two snapshots by date and interval
  • Get the snapshot name
  • Get the full snapshot string. <path>@<snapshot name>

The following code will implement those utility methods.

/*
Path returns a string containing the path of the snapshot
*/
func (s Snapshot) Path() string {
	var temp strings.Builder
	temp.WriteString(s.pool)
	if len(s.fsTree) > 0 {
		for _, v := range s.fsTree {
			temp.WriteString("/" + v)
		}
	}
	return temp.String()
}

/*
Name returns a string containing the full name of snapshot
*/
func (s Snapshot) Name() string {
	var temp strings.Builder
	temp.WriteString("zfs-auto-snap_")
	temp.WriteString(intervalUIntToString(s.Interval) + "-")
	fmt.Fprintf(&temp, "%d-%d-%d-%d%d", s.TimeStamp.Year(), s.TimeStamp.Month(), s.TimeStamp.Day(), s.TimeStamp.Hour(), s.TimeStamp.Minute())
	return temp.String()
}

func intervalUIntToString(x uint64) string {
	switch x {
	case 0:
		return "yearly"
	case 1:
		return "monthly"
	case 2:
		return "weekly"
	case 3:
		return "daily"
	case 4:
		return "hourly"
	}
	return "frequent"
}

/*
String returns a string equal to s.Path() + "@" + s.Name() for Snapshot s
*/
func (s Snapshot) String() string {
	return s.Path() + "@" + s.Name()
}

/*
CompareSnapshotDates returns -2 if x occured before y and would include y in its interval
returns -1 if x occured before y
returns 0 if x and y are the same snapshot
returns +1 if y occured after x
err is non nill if the snapshots do not have the same path
*/
func CompareSnapshotDates(x Snapshot, y Snapshot) (int, error) {
	if x.Path() != y.Path() {
		return 0, errors.New("Can only compare snapshots with the same path")
	}
	if x.Interval == y.Interval {
		if x.TimeStamp.Equal(y.TimeStamp) {
			return 0, nil
		}
		if x.TimeStamp.Before(y.TimeStamp) {
			return -1, nil
		}
		return 1, nil
	}
	if x.Interval < y.Interval { // y is from a more frequent backup interval than x
		var interval time.Time
		switch x.Interval {
		case 0:
			interval = x.TimeStamp.AddDate(-1, 0, 0)
		case 1:
			interval = x.TimeStamp.AddDate(0, -1, 0)
		case 2:
			interval = x.TimeStamp.AddDate(0, 0, -7)
		case 3:
			interval = x.TimeStamp.AddDate(0, 0, -1)
		case 4:
			interval = x.TimeStamp.Add(time.Hour * -1)
		case 5:
			interval = x.TimeStamp.Add(time.Minute * -15)
		}
		if x.TimeStamp.Before(y.TimeStamp) {
			return 1, nil
		}
		if interval.Before(y.TimeStamp) {
			return -2, nil
		}
		return -1, nil
	}
	// y is from a less frequent backup interval than x
	if x.TimeStamp.Before(y.TimeStamp) {
		return -1, nil
	}
	if x.TimeStamp.After(y.TimeStamp) {
		return 1, nil
	}
	return 0, nil
}

You can get the entire source code for the tool below.

zfs_backup-part3Download
Posted on : By : Robert Posted in : Medium

ZFS Backup Tool Part 2

Recognizing a snapshot made by zfs-auto-snapshot.

First, what does a list of these snapshots look like?

robert@mars:~/src/go/zfs_backup$ zfs list -Hrt snapshot dpool
dpool@zfs-auto-snap_monthly-2020-05-12-1245     96K     -       148K    -
dpool@zfs-auto-snap_monthly-2020-06-11-1248     8K      -       23.3G   -
dpool@zfs-auto-snap_monthly-2020-07-11-1245     0B      -       23.3G   -
dpool@zfs-auto-snap_weekly-2020-07-26-1242      0B      -       30.5G   -
dpool@zfs-auto-snap_daily-2020-07-27-1238       4.74G   -       31.3G   -
dpool@zfs-auto-snap_daily-2020-08-02-1235       0B      -       143G    -
dpool@zfs-auto-snap_weekly-2020-08-02-1240      0B      -       143G    -
dpool@zfs-auto-snap_hourly-2020-08-03-1117      0B      -       143G    -
dpool@zfs-auto-snap_daily-2020-08-03-1236       0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2030    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2045    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2100    0B      -       143G    -
dpool@zfs-auto-snap_frequent-2020-08-04-2115    0B      -       143G    -
dpool@zfs-auto-snap_hourly-2020-08-04-2117      0B      -       143G    -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1717 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1817 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-1917 0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-2017 0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2030       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2045       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2100       0B      -       96K     -
dpool/home@zfs-auto-snap_frequent-2020-08-04-2115       0B      -       96K     -
dpool/home@zfs-auto-snap_hourly-2020-08-04-2117 0B      -       96K     -
dpool/home/robert@zfs-auto-snap_frequent-2020-08-04-2115        0B      -       69.1G   -
dpool/home/robert@zfs-auto-snap_hourly-2020-08-04-2117  0B      -       69.1G   -
dpool/plex@snap1        116G    -       442G    -
dpool/plex@zfs-auto-snap_monthly-2020-05-12-1245        8K      -       344G    -

I trimmed the previous list down a bit. So what is a regular expression that will match this? The first question is which regular expression library am I using? I am writing this tool in Go. Thus I will use the regexp Go package. Go’s regexp package is based on Google’s RE2 library. The syntax for it is here.

I will start with the snapshot names. The part after the @. Those start with zfs-auto-snap so "zfs-auto-snap" will match it.

The next section is which timer made the snapshot. This section can also be called the increment. The valid timers are yearly, monthly, weekly, daily, hourly, and frequent for a default install of zfs-auto-snapshot. The regex "yearly|monthly|weekly|daily|hourly|frequent" will match these timers. However, I would like to get which timer created the snapshot without further parsing. That is the perfect job for a capturing sub match. After adding the capturing sub match, the regex looks like "(?P<increment>yearly|monthly|weekly|daily|hourly|frequent)".

The final section is the timestamp of the snapshot. Like with the timer section, it is useful not to have to parse this data a second time. With the sub matches "(?P<year>[[:digit:]]{4})-(?P<month>[[:digit:]]{2})-(?P<day>[[:digit:]]{2})-(?P<hour>[[:digit:]]{2})(?P<minute>[[:digit:]]{2})" will work.

With the snapshot names completed, I need to capture the zfs tree structure before the @ symbol. I haven’t found a reliable regular expression that will capture that tree but "(?:[[:word:]-.]+)+(?:/?[[:word:]-.]+)*" will recognize a subset of all valid zfs trees. Avoid using anything it won’t recognize, or you may end up with inaccessible files.

Including some test code the tool’s source code looks like this so far.

package main

import (
	"bufio"
	"fmt"
	"os"
	"regexp"
)

const zfsRegexStart string = "zfs-auto-snap"
const zfsRegexIncrement string = "(?P<increment>yearly|monthly|weekly|daily|hourly|frequent)"
const zfsRegexDateStamp string = "(?P<year>[[:digit:]]{4})-(?P<month>[[:digit:]]{2})-(?P<day>[[:digit:]]{2})-(?P<hour>[[:digit:]]{2})(?P<minute>[[:digit:]]{2})"

var zfsRegex = regexp.MustCompile(zfsRegexStart + "_" + zfsRegexIncrement + "-" + zfsRegexDateStamp)

func testSnapshot(possible string, increment string) (bool, bool) {
	var matches = zfsRegex.FindStringSubmatch(possible)
	if matches == nil {
		return false, false
	}
	var isASnapshot = true
	if matches[1] == increment {
		return isASnapshot, true
	}
	return isASnapshot, false
}

func isAYearlySnapshot(possible string) bool {
	_, isYearly := testSnapshot(possible, "yearly")
	return isYearly
}

func isAMonthlySnapshot(possible string) bool {
	_, isMonthly := testSnapshot(possible, "monthly")
	return isMonthly
}

func isAWeeklySnapshot(possible string) bool {
	_, isWeekly := testSnapshot(possible, "weekly")
	return isWeekly
}

func isADailySnapshot(possible string) bool {
	_, isDaily := testSnapshot(possible, "daily")
	return isDaily
}

func isAnHourlySnapshot(possible string) bool {
	_, isHourly := testSnapshot(possible, "hourly")
	return isHourly
}

func isAFrequentSnapshot(possible string) bool {
	_, isFrequent := testSnapshot(possible, "frequent")
	return isFrequent
}

const poolNameRegex string = "(?:[[:word:]-.]+)+(?:/?[[:word:]-.]+)*"

var snapshotLineRegex = regexp.MustCompile("^" + poolNameRegex + "@" + zfsRegex.String() + ".*$")

func main() {
	//fmt.Println(snapshotLineRegex.MatchString("dpool/www@zfs-auto-snap_frequent-2020-08-04-1830\t0B\t-\t201M\t-"))
	input := bufio.NewScanner(os.Stdin)
	for input.Scan() {
		if snapshotLineRegex.MatchString(input.Text()) {
			fmt.Println(snapshotLineRegex.FindStringSubmatch(input.Text()))
			fmt.Println(snapshotLineRegex.SubexpNames())
		} else {
			fmt.Printf("%s\t%s\n", input.Text(), "Is not a snapshot.")
		}
	}
	if err := input.Err(); err != nil {
		fmt.Fprintln(os.Stderr, "reading Standard Input:", err)
	}
}

I will continue this tomorrow. See you then!