Written by: Robert R. Russell on Thursday, August 6, 2020.
Today’s project is parsing a snapshot into a custom datatype that gives us
more accessible options to manipulate snapshots. First, the regular expression
strings need to be moved into separate files so I can reference them across
other files.
The essential parts of a snapshot are:
the pool name
the filesystem tree
the Interval
the TimeStamp
To parse a snapshot out of a string.
Confirm that the string matches our regular expression for snapshots and only
contains the regular expression for a snapshot.
If it does not return an error otherwise continue
Split the input string into the path and the snapshot name
Parse the snapshot name into the interval and timestamp fields
Split the path into the pool name and any filesystem tree portions.
Below is a function that implements the listed requirements.
/*
ParseSnapshot parses a string into a Snapshot.
It returns nil on error.
*/funcParseSnapshot(input string) *Snapshot {
var snapshotOnly, err = regexp.Compile("^" + PoolNameRegex + "@" + ZfsSnapshotNameRegex + "$")
if err != nil {
returnnil }
if !snapshotOnly.MatchString(input) {
returnnil }
var snapshotPieces []string = snapshotOnly.FindStringSubmatch(input)
var theSnapshot = Snapshot{}
theSnapshot.Interval = intervalStringToUInt(snapshotPieces[1])
var year, month, day, hour, minute int year, err = strconv.Atoi(snapshotPieces[2])
if err != nil {
returnnil }
month, err = strconv.Atoi(snapshotPieces[3])
if err != nil {
returnnil }
day, err = strconv.Atoi(snapshotPieces[4])
if err != nil {
returnnil }
hour, err = strconv.Atoi(snapshotPieces[5])
if err != nil {
returnnil }
minute, err = strconv.Atoi(snapshotPieces[6])
if err != nil {
returnnil }
theSnapshot.TimeStamp = time.Date(year, time.Month(month), day, hour, minute, 0, 0, time.UTC)
var splitInput []string = strings.Split(input, "@")
iflen(splitInput) != 2 {
returnnil }
var paths []string = strings.Split(splitInput[0], "/")
theSnapshot.pool = paths[0]
iflen(paths) > 1 {
copy(theSnapshot.fsTree, paths[1:])
}
return &theSnapshot
}
funcintervalStringToUInt(input string) uint64 {
switch input {
case"yearly":
return 0
case"monthly":
return 1
case"weekly":
return 2
case"daily":
return 3
case"hourly":
return 4
}
return 5
}
Now that I can create a Snapshot structure I need some utility methods for them.
Read the pool snapshot and file system tree as a single string.
Compare two snapshots by date and interval
Get the snapshot name
Get the full snapshot string. <path>@<snapshot name>
The following code will implement those utility methods.
/*
Path returns a string containing the path of the snapshot
*/func (s Snapshot) Path() string {
var temp strings.Builder
temp.WriteString(s.pool)
iflen(s.fsTree) > 0 {
for _, v := range s.fsTree {
temp.WriteString("/" + v)
}
}
return temp.String()
}
/*
Name returns a string containing the full name of snapshot
*/func (s Snapshot) Name() string {
var temp strings.Builder
temp.WriteString("zfs-auto-snap_")
temp.WriteString(intervalUIntToString(s.Interval) + "-")
fmt.Fprintf(&temp, "%d-%d-%d-%d%d", s.TimeStamp.Year(), s.TimeStamp.Month(), s.TimeStamp.Day(), s.TimeStamp.Hour(), s.TimeStamp.Minute())
return temp.String()
}
funcintervalUIntToString(x uint64) string {
switch x {
case 0:
return"yearly"case 1:
return"monthly"case 2:
return"weekly"case 3:
return"daily"case 4:
return"hourly" }
return"frequent"}
/*
String returns a string equal to s.Path() + "@" + s.Name() for Snapshot s
*/func (s Snapshot) String() string {
return s.Path() + "@" + s.Name()
}
/*
CompareSnapshotDates returns -2 if x occured before y and would include y in its interval
returns -1 if x occured before y
returns 0 if x and y are the same snapshot
returns +1 if y occured after x
err is non nill if the snapshots do not have the same path
*/funcCompareSnapshotDates(x Snapshot, y Snapshot) (int, error) {
if x.Path() != y.Path() {
return 0, errors.New("Can only compare snapshots with the same path")
}
if x.Interval == y.Interval {
if x.TimeStamp.Equal(y.TimeStamp) {
return 0, nil }
if x.TimeStamp.Before(y.TimeStamp) {
return -1, nil }
return 1, nil }
if x.Interval < y.Interval { // y is from a more frequent backup interval than x
var interval time.Time
switch x.Interval {
case 0:
interval = x.TimeStamp.AddDate(-1, 0, 0)
case 1:
interval = x.TimeStamp.AddDate(0, -1, 0)
case 2:
interval = x.TimeStamp.AddDate(0, 0, -7)
case 3:
interval = x.TimeStamp.AddDate(0, 0, -1)
case 4:
interval = x.TimeStamp.Add(time.Hour * -1)
case 5:
interval = x.TimeStamp.Add(time.Minute * -15)
}
if x.TimeStamp.Before(y.TimeStamp) {
return 1, nil }
if interval.Before(y.TimeStamp) {
return -2, nil }
return -1, nil }
// y is from a less frequent backup interval than x
if x.TimeStamp.Before(y.TimeStamp) {
return -1, nil }
if x.TimeStamp.After(y.TimeStamp) {
return 1, nil }
return 0, nil}
You can get the entire source code for the tool below.
I will start with the snapshot names. The part after the @. Those start with
zfs-auto-snap so “zfs-auto-snap” will match it.
The next section is which timer made the snapshot. This section can also be
called the increment. The valid timers are yearly, monthly, weekly, daily,
hourly, and frequent for a default install of zfs-auto-snapshot. The regex
“yearly|monthly|weekly|daily|hourly|frequent” will match these timers. However,
I would like to get which timer created the snapshot without further parsing.
That is the perfect job for a capturing sub match. After adding the capturing
sub match, the regex looks like
“(?P<increment>yearly|monthly|weekly|daily|hourly|frequent)”.
The final section is the timestamp of the snapshot. Like with the timer section,
it is useful not to have to parse this data a second time. With the sub matches
“(?P<year>[[:digit:]]{4})-(?P<month>[[:digit:]]{2})-(?P<day>[[:digit:]]{2})-(?P<hour>[[:digit:]]{2})(?P<minute>[[:digit:]]{2})”
will work.
With the snapshot names completed, I need to capture the zfs tree structure
before the @ symbol. I haven’t found a reliable regular expression that will
capture that tree but “(?:[[:word:]-.]+)+(?:/?[[:word:]-.]+)*” will recognize
a subset of all valid zfs trees. Avoid using anything it won’t recognize, or you
may end up with inaccessible files.
Including some test code the tool’s source code looks like this so far.
Written by: Robert R. Russell on Tuesday, August 4, 2020.
I haven’t seen a lot of tools that are designed to backup ZFS snapshots to
removable media. So, I am writing my own. I am going to document the process
here.
The basic loop for a backup tool is
Read a list of snapshots on the source.
Read a list of snapshots on the destination.
Find the list of snapshots on the source that are not on the destination.
These are the non backed up snapshots.
Find the list of snapshots on the destination that are not on the source.
These are the aged out snapshots.
Copy all non backed up snapshots to the destination, preferably one at a time
to make recovery from IO failure easier.
Remove the aged out snapshots.
I am designing this tool to only backup snapshots taken by zfs-auto-snapshot.
These are named
<pool|filesystem>@zfs-auto-snap_<time interval>-<year>-<month>-<day>-<hour><minute>.
The command zfs -Hrt snapshot <source poolname> will generate a list of
all snapshots in a pool in a machine parseable format.
Issuing the command zfs send -ci <old snapshot> <pool|filesystem>@<new snapshot>
will send an incremental snapshot from old to new to the commands standard
output. I can estimate the amount of data to be transferred by replacing
-ci with -cpvni in the zfs send command.
Issuing the command zfs receive -u <backup location> will store a snapshot
from its standard input to the backup location.
Snapshots are removed by zfs destroy -d <pool|filesystem>@<snapshot name>.
The snapshot name is the portion of the snapshot pattern mentioned above after
the @ symbol.
Written by: Robert R. Russell on Friday, July 31, 2020.
I am planning on a video for today’s post. It is processing right now, and that
will take some time. Hopefully, it will be prepared and uploaded before 17:00
tonight.
Written by: Robert R. Russell on Thursday, July 30, 2020.
There will not be a new post today. I have been running into several problems
trying to get a video done for the Friday post. The Saturday post maybe a rant
about getting OBS to record in a better intermediate format than x264.
Written by: Robert R. Russell on Wednesday, July 29, 2020.
I am using Gulp.js or just Gulp for automating the compilation of CSS
stylesheets for the upcoming custom WordPress theme for this blog. In the
process of getting that automation setup, I have concluded that NPM’s extremely
lax requirements for adding a package to their servers have resulted in an
explosion of abandonware.
One of Gulp’s useful advantages over a more traditional solution like make
is its choice to pass virtual files around between stages of the processing
chain. Parts of the Gulp chain can modify the contents of a file and pass on
those modifications without writing them to disk and creating dozens or more
temporary files that require exclusion from git and other tools. The most
constructive use of this ability I have found so far is a tool that can replace
strings in files based on variables I setup.
NPM manages Gulp’s dependencies and plugins. Since creating a new public NPM
package is pretty easy, sharing a plugin you wrote doesn’t take any time.
That all sounds great until you end up trying to use a plugin and find that
no one has updated it for one or two major versions of Gulp. Worse yet is the
situation where some dependency is several versions behind, and either is a
security vulnerability itself or requires another dependency that is.
I don’t have time to maintain a public NPM package. I may fix one or two
outdated plugins I am probably not going to share those fixes on NPM.
Written by: Robert R. Russell on Tuesday, July 28, 2020.
I have been designing websites as a side gig for about a year now. Most of that
design work has been CSS modifications to existing themes. Since January, I have
needed to do more extensive design changes. That work culminated in a
scratch-made WordPress theme designed for people using WordPress as a CMS, not a
blog platform.
During that time, I have begun preferring Gutenburg’s design philosophy over
Elementor’s. Gutenburg does a better job of separating content, and
content-specific layout from the general theme layout than Elementor does.
Gutenburg also seems less opinionated about its block styling than Elementor.
The downside to flexibility is a lack of capability to micromanage the layout. I
don’t see the appeal of complicated website designs that demand pixel alignment
from individual paragraphs or worse letters. I tend towards a utility first
approach. That utility first approach doesn’t mean that I do not appreciate any
artistry. I cannot entirely agree with form over function.
I like the mobile-first approach to design. However, I do get frustrated at the
limitations of mobile devices because they can complicate the implementation of
proper form and function. A navigatable mobile interface for tabular data is one
example.