I Nick Urbanik — Free Software Written by Me

Free Software Written by Nick Urbanik

I have written a number of programs over the years to help with system administration and anything else I need to do. Here are just a few of them.

Emacs 29.0

Emacs 28.2 was going into 100% CPU with some Python files. It is triggered when opening a docstring in a function definition in some particular files. The bug is fixed in Emacs 29, but that hasn't been released, so is not packaged by Fedora or other providers. I built a package that works for me. Here is the spec file I made to build emacs under mock on Fedora 37. I reported the bug upstream as bug#62794, and to Fedora as bug 2187041.

Ethernet device renamer

There are problems when you want to keep your network devices named eth0, eth1, eth2, eth3, ... and you are using RHEL 7, and the network cable is not plugged into the device that the kernel things is eth0.

udev says: 

Mar 14 16:58:19 servername systemd-udevd: Error changing net interface name 'eth1' to 'eth0': File exists
Mar 14 16:58:19 servername systemd-udevd: could not rename interface '3' from 'eth1' to 'eth0': File exists
Mar 14 16:58:19 servername systemd-udevd: Error changing net interface name 'eth0' to 'eth1': File exists
Mar 14 16:58:19 servername systemd-udevd: could not rename interface '2' from 'eth0' to 'eth1': File exists
This little program works because it uses a temporary name

The idea is to install this little thing as a singleshot service that runs early in the boot process, and renames the Ethernet interfaces, matching their actual MAC addresses with the corresponding HWADDR and DEVICE fields in their /etc/sysconfig/network-scripts/ifcfg-* files.

Note: it does not deal with selinux.

DNS malware blocker

Are you getting lots of DNS queries that look something like this: 

abatopuvohyhyjoj.www.xixiwg.com
dcpdljmyesi.www.xixiwg.com
guynjvtwwzylqlq.www.xixiwg.com
mvvckisovcs.www.xixiwg.com
ntl.www.xixiwg.com
ysqkyciyjjpdhcc.www.xixiwg.com
and your DNS caches are running out of resources, with many thousands of outstanding queries?

  1. Turn on query-errors on your BIND DNS caching server.
    2.     	    channel "query-errors" {
                file "/var/log/queries/errors" versions 5 size 1g;
    	        severity debug 2;
    	        print-category yes;
    	        print-time yes;
    	    };
  2. Install Debug/OnTheFly.pm into your Perl path
  3. Install rcs, which is used to track the history of changes to see which domains were blocked when;
  4. Install the Perl module Linux::Inotify2
  5. Install a zone file that is empty to return NXDOMAIN to all queries 
    $ sudo cat /var/named/chroot/var/named/blank-zone
@ 10800 in soa . . ( 1 3600 1200 604800 10800 )
@ 10800 IN NS .
  6. Touch the file /var/named/chroot/etc/named/bad-zones.conf. We are running BIND chrooted into /var/named/chroot.
  7. Touch the file /var/cache/bind/bad-zones, which keeps a record of when each naughty zone was last seen. Create the directory first.
  8. Add to your named.conf
    include "/etc/named/bad-zones.conf";
  9. Run block-fast-flux from cron every minute
  10. The current settings are for a very busy DNS cache. See output of block-fast-flux --help
  11. The code writes to /var/named/chroot/etc/named/bad-zones.conf for each bad zone it finds; it removes them if they have not been seen with SERVFAILs in the last two hours. It writes entries into that file like this: 
    zone "www.qixingfz.cc" {
    type master;
    file "/var/named/blank-zone";
    allow-query { 127.0.0.1; };
};
  12. If you put the code into a directory baddies, then it will write a verbose log to /var/log/baddies/block-fast-flux.log, which you need to write a logrotate file for: 
    $ cat /etc/logrotate.d/baddies
# /etc/logrotate.d/baddies

/var/log/baddies/*.log {
    weekly
    rotate 10
    nocreate
    missingok
    notifempty
    compress
}

How it works

  1. The program reads the last 20MB of the query error logs. That's about two minutes or less of data. If your DNS server is less busy than that, You probably need to read a shorter length of the logs. Change that with the --maxsize=BYTES option.
  2. It decides a zone is bad if:
    • There are more than 200 error queries for subdomains of that parent zone.
    • If the parent zone has two dots or more, then 50% of the queries for subdomains under that zone are unique (only asked for once), otherwise it has 75% are unique;
    • The parent zone is not in a whitelist;
  3. If the zone is found to be bad, then /var/cache/bind/bad-zones (the cache file) is updated, with current statistics of that zone, including the current time. The file is a tab-separated text file.
  4. The code then decides to fill /var/named/chroot/etc/named/bad-zones.conf with records for each zone that has a time less than two hours old, and tells named to re-read its configuration.
  5. The code decides from /var/cache/bind/bad-zones which zones have recently been unblocked, and tails the query error logs for sixty seconds, looking for SERVFAILs for each of those zones.
  6. If any reoffenders are found, they have their timestamp in the cache file updated, and they are re-blocked in the bad-zones.conf file.
Linux::Inotify2 is used in the log tailing; much more responsive than sleeping.

Salted hash checker

this simple salted hash checker is useful when working with MD5, SHA256, SHA512 salted hashes, as used in /etc/shadow files.

It can also help with checking whether a plain text matches a particular salted hash.

DHCP next file patch

We have had problems with ISC DHCP in failover mode when the size of the dhcpd.leases file was in the order of 250 MB or so. Every hour, the ISC dhcp server stops to dump all its leases to a new dhcpd.leases file, during which time, it enters "Communications-Interrupted" mode. About once a week, the two machines would fail to re-establish normal communication. This caused much consternation and phones rang under pillows at 2 AM.

A colleague of mine had "a very cunning plan" to solve this problem. I coded up a patch, which is licenced under the GPL of course. I also wrote a program in Perl to aggregate the dhcpd.leases files.

The basic idea of this is that the DHCP server never stops to rewrite its leases files; the external process, dhcp-aggregator.pl does that instead.

There is a debug module used by dhcp-aggregator.pl to implement on-the-fly debugging.

Please note that you may wish to change the value of $START_DIR, the location of of the dhcpd.leases files, from /var/lib/dhcp to /var/lib/dhcpd (or something else) in dhcp-aggregator.pl.

Updates to Next File Patch

2007-06-12

This version fixes two bugs:

The counting = 1; fix
The patch previous to this had a serious omission that prevented dhcpd from starting a new dhcpd.leases file every hour. That has been fixed with a one-line addition to the code.
Locking during startup
This allows the external program performing the aggregation to know when dhcpd is in its startup phase, so that it knows when not to rename the old lease files. dhcpd flock(8)s a lock file during its startup, and the external program cooperatively does not rename the old files until this lock is released.

Updates to dhcp-aggregator

2007-06-15

The current code now uses a module OIE::Debug::OnTheFly to implement logging and on-the fly debugging.

It also fixes a bug that would allow another dhcp-aggregator process to start while the previous dhcp-aggregator process was waiting for a lock.

2008-01-18

Now dhcp-aggregator reads the leases from the /etc/dhcpd.conf configuration file to determine whether a lease remains within one of the ranges of IP addresses. If it does not, the lease is removed during the aggregation process.

Used by several hundred thousand customers

This code is currently serving several hundred thousand happy cable customers, so it may serve your purposes.

Hope for patch to be incorporated in ISC DHCP server

It is my hope that the ISC will incorporate this patch into the official ISC DHCP server, and I am happy to cooperate to improve it, and would like to hear any suggestions for its improvement.

Chroot a Program

Sometimes you want to chroot a program. It's a nuisance to put all the required libraries into the chroot. Here's a little simple minded program that will take care of most of the boring details.

With any of these programs, if they nearly but not quite do what you want, write to me, and I'll (probably) make it do that.

CA and Certificate Creation

The convenience of using your own long-lasting root CA certificate and distributing that to your customers compared with giving them a copy of a self-signed certificate is that you only need to give them the root certificate every decade.

Another benefit is that you do not have an outage while the local self-signed certificate does not match that held by the customer. The upgrade needs to be simultaneous to avoid that outage.

To that end, I wrote make-new-ca.pl and make-server-cert.pl to help automate this process. I know that the CA.pl script with openssl is designed to help, but I wanted something that makes it even more straight forward.

Here is the output of the CA program (my root CA file). (PEM)

LDAP Directory Management

LDAP diff

When working with slurpd, I wanted to have a simple way to produce diffs bwtween two LDIF entries that would be suitable for feeding to ldapmodify, so I wrote ldap-diff that produces such LDIF output. The first version of this program loads everything into memory, but handles DNs properly even if they are wrapped over more than one line.

Net::LDAP programming

At the Department of Information and Communications Technology at the Hong Kong Institute of Vocational Education (Tsing Yi) I implemented a single sign on solution for our full-time and part-time students and staff. Here are some of the programs I wrote to manage this.

I discussed this at a Sydney Linux User's Group talk in June 2005. The PDF slides are available, as is the LaTeX source. Immediately after this talk, I was offered a teaching job at TAFE.

DHCP and DNS Management System

Please see a detailed description of this system. It combines Perl programs, Samba, DHCP, BIND and some shell scripts to create a simple way to keep track of computers, and maintain network information and configuration for those computers.

Copy podcasts to music player

I listen to a lot of Radio National podcasts and wanted to make maintaining the podcasts on my music player easier. I wrote get-podcasts to this end.

Maintain Apache Password File from Mailman Addresses and Passwords

It would be nice to use the usernames and passwords from a Mailman mailing list to control access to a web site. I wrote this program mm_htaccess to do that. I based this on a collection of shell scripts, a Makefile builder, and other components in the Mailman FAQ and thought I might do it simpler. It can also send this file by ftp to a remote server if required.

URL munging

Inspired by the example eg/hrefsub with HTML::Parser I have written href-level, which munges all the links, including those in comments (for recalcitrant IE) and those in <script>@import url(...)</script>, as well as with Apache SSI includes. It's very much oriented to what I want, but then, so is everything else!

GnuCash XML File Munging

Tired of trying to get GnuCash to generate reports that are useful to me, I decided to write an GnuCash XML munger. What I wanted was really simple, and so what parse-gnucash.pl actually does is very simple too; it simply gives the amount for each account over any period of time.

Randomly catting a file

Needing to simulate how a dhcp server writes its lease files, I wrote random-cat.pl which divides its input files into a specified number of random pieces, then writes them to standard output with random delays.

Generating Type Map Files for Apache Content Negotiation

When writing strict xhtml web pages such as this one, it is nice to serve it as application/xhtml+xml to web browsers that Accept it. This is called content negotiation. Apache provides two methods to do this:

  1. using Multiviews, where Apache looks for all files that start with the name provided in the URL,
  2. Using Type Maps.

Since type maps are slightly more efficient, and give more control over the content negotiation, I decided to write a program gen-type-map.pl to implement type maps automatically for each named file.

Note that if you view this page with Firefox, and select Tools => Page Info, you will see that the Content type is application/xhtml+xml, while Internet Explorer will get text/html.

Online man page viewer

I needed a simple way to present documentation for system configuration and system commands, so I wrote a simple cgi program man to allow reading man pages over a web browser. Look up the man page for the Apache::AuthNetLDAP module to which I contributed.

Editing Large numbers of Web Pages

The CAMWEST web site previously used a table-based layout. I wrote this program to edit all the 116 web pages to simplify them to use CSS positioning: fix-camwest-pages.pl.

Rotating Mail Folders

When subscribing to many mailing lists, they can easily grow to the point where mutt slows down a bit when opening the folder. I wrote a program rotate-mail-folders.pl to run from cron to rotate them when they reached a suitable size.

Generate Quiz questions in Moodle

Moodle is an open source learning system used for education and training systems around the world. I wrote this program to convert questions in one common format to one that Moodle can import simply.

Noughts and Crosses

I wrote this game noughts-and-crosses.pl to play with my son.

Make Phonogram Cards

After buying a copy of The Writing Road to Reading, I decided to write a program to generate the phonogram cards: parse-phonograms.pl

Make Arithmetic Exercise Sheets

When our child was in kindergarten, I wrote make-sums.pl to generate sheets of arithmetic exercises for him.

Calculate volume, weight of sand, cement and gravel to mix volume of concrete

I've mixed a bit of concrete lately, and prefer to buy the right amount of sand, cement and gravel to mix it, since it is a nuisance having too much left over. So I wrote a little program to calculate the correct volumes and weights of sand, cement and gravel for a required volume of concrete: calc-concrete.pl

Examine and Change EXIF Data in Photographs

When rotating JPEG images lossleesly with gthumb, the mtime date stamp is changed. I wrote three little programs:

Generate Web Pages Given A Directory Of Photos

This started because I was very busy after my son was born, and we took zillions of photos, and I just wanted any family member to be able to just plug the cable into the camera and for the photos to be automatically downloaded, sorted into directories, and a web page made for each day, showing thumbnails of all the photos.

Limit Time A User Spends Running Particular Software

We decided to limit the time my son spends playing computer games each day. This set of programs implements control on games that he can play. After time is up, then games in the group whose time has expired will be killed in less than thirty seconds.

His ambition is to be able to write a more powerful program to beat this program. I'm sure he will succeed one day.

Count Number Of Elements In Training Packages

When doing the Certificate 4 in Workplace Training and Assessement for TAFE teaching, there is an assignment that involves assessing someone on their competency in some particular skill that is listed among the thousands on the website http://www.ntis.gov.au/. So I decided to automate selecting the optimal training packages by reading them all and looking for the minimum number of elements. Hence count-elements-in-training-packages.pl.

Make a Convenient Phone Book Sheet From Abook Database

I like abook, since it can be integrated with my mutt email client, and so decided to write software to produce a nice sheet oc paper that I can glue into my diary containing contact information that matters to me.

Convert Between Time Zones, include Daylight Saving

It is often important to check the time in another timezone: when is it too late to phone my friend in France? What time will the Fedora Core 6 release be made? It is often a problem to think about daylight saving. It would be nice for the computer to use its own time zone database to work all that out for me. I wrote a little program to solve this little problem: time-convert.pl

Get My IP Address

A simple shell script to show the IP address of the interface connected to the default gateway by parsing the output of the ip program: get-my-ip.sh.

Licenses of the Software

All the software written by me, linked to on this machine from this page, is is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version.