I'm not a pro administrator. I do have several computers running in the background, though. The number has exploded (to "five") since the Raspberry Pi became a thing. I've had machines running in the background for years.

Recently I rigged two new ones and found that I have pretty much settled on a list of things that need to be in place from the start, and maybe that's helpful for someone out there.

What do I want in a server?

Most of this is independent of the purpose that the machine actually "serves". Some items though are less important in some contexts.

Backup, for example, is a staple. On some machines it's critical to the point that it needs to be frequent, redundant, and encrypted. On other machines it's unnecessary, for example because there's no local data and the machine can be set up from scratch in thirty seconds.

The order of things here is a nonsensical mix of importance and chronology.

Documentation

At the beginning, fiddling with the new machine will be exciting and fun, and I'll check up on it at least once a day. After a while, though, I'll be convinced that it "works", and I'll almost forget about it. Fast forward a year later: Something doesn't work. What exactly did I do in the first place?

Before I start: Journal

When you're setting up the first Linux machine in your life, or the first public-facing webserver, start a journal. Start a new entry every day, tag it with the date. Write down what you did. Be specific and exact. Wherever possible, copy-paste the actual command lines. Keep track in chronological order. When it makes sense, aggregate: What are the packages I installed on which day ... what are all the packages that are installed now? Maintain the journal right on the new machine, but keep a copy of it somewhere else.

Reflect on every step:

Should anything go wrong, there's a step-by-step instruction on how to recreate what I had.

vi ~/journal.txt

If you want to know what vi means, open a terminal and type man vi. man is for manual. This works for almost every command line tool. If you've never done this before, do that for every program that you see me using in this article.

Also, ~ (tilde) is short for your home directory, typically in /home/<username>, and also called $HOME. So the file journal.txt will sit in my home directory.

Oh, and of course never put passwords or other sensitive information in a plain text file.

git in /etc

There is a pragmatic adage that goes "version everything". Do that.

As you might have noticed, in Linux every configuration goes in a plain-text file - most of it in /etc. This is different from Windows, where all the settings are spread out in registry, configuration files in weird places, and sometimes proprietary databases that you can't even read without a special tool.

I love the Linux way because it is dead simple to point at the place where a configuration needs to be made. Even better: I can put the whole system configuration under version control.

apt install git
cd /etc
git init
git add .
git commit -m 'starting system configuration'

Updates

Up-to-date system programs are your first line of defense against intrusion. Make a plan on when you run updates. Consider how exposed and how sensitive the machine is.

How to not forget to do updates?

# in /etc/cron.d/check-for-updates
43 21 * * * root /usr/bin/apt update && /usr/bin/apt list --upgradable

A cronjob is a little instruction for your system to run regularly. In this case, every night at 9:43 p.m. If any of the commands produce any output, that will be emailed to the root user account. Make sure you have it set up so that the root email gets forwarded to your single inbox that you read at least once every day.

How to make sure you don't miss the most important updates? Subscribe to the mailing list where critical updates are announced for your system, e.g. debian-security-announce .

Minimalism

What is the reason you are maintaining this machine? Remove all non-essential services. Every service that is running can be a backdoor, needs to be kept up-to-date, and possibly occupies storage, memory, or even CPU power.

That's why I like to start with a minimal system and add what I need, rather than starting with a desktop and forgetting to remove what I don't need.

For Debian, the installer has a step called "tasksel" that looks like this:

Tasksel presents an abstract way of selecting sets of packages for
 installation.

Audit your server from time to time. Look at running services, orphaned accounts, and whatever else you can think of, and for each of them check whether you still need it.

Look for open ports:

$ netstat -l | sed -n '1,2 p ; /tcp.*LISTEN/ p'
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State
tcp        0      0 0.0.0.0:ssh             0.0.0.0:*               LISTEN
tcp        0      0 0.0.0.0:ipp             0.0.0.0:*               LISTEN
tcp        0      0 localhost:postgresql    0.0.0.0:*               LISTEN
tcp        0      0 localhost:smtp          0.0.0.0:*               LISTEN
tcp        0      0 localhost:3551          0.0.0.0:*               LISTEN
...

"Local address" here shows all the services that are open for business. Do I really need postgresql to listen on a tcp socket?

What's 3551? Normally you want to run netstat with an option to prevent name lookups. Name lookups can be timeconsuming. Use netstat -n for "no names". You'll then find 5432 instead of postgresql. For most ports, there's an easy way to find out what they do:

$ grep 5432 /etc/services
postgresql      5432/tcp        postgres        # PostgreSQL Database
postgresql      5432/udp        postgres

That's what netstat does if you don't put -n.

Now what's 3551? Turns out it's not in /etc/services. Search the web for port 3551 and you'll find that I have an uninterruptible power supply attached to this server.

Also look for stale users.

less /etc/shadow
ls /home/*/.ssh/authorized_keys

See man shadow for what this means. Do you still know every user who has a password or an authorized ssh key?

Access

As is widely acknowledged, there's a tradeoff between security and convenience. I want my computer to be as safe as possible - that's best achieved by air gap. But I want to access my server at any time, from any point in the world. I mean, otherwise it wouldn't be much use, right?

Obviously, choose good passwords. I'm not going to dwell on that - there's plenty of info, and controversy, on what constitutes a good password.

Change default passwords

For example, in the standard Raspberry Pi operating system, there's a user with a standard name, and with a standard password. That's important in order to get started, but you should change the password as soon as you get a chance.

Secure shell

For unix, use ssh. For Windows, use PowerShell through SSL. However you do it, create a channel that's sufficiently encrypted and guarded by keys.

For ssh, switch off password authentication and root login. But first, we need to make sure we get in ourselves.

On my laptop, I generate a pair of a public key and a private key:

$ ssh-keygen
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in id_rsa.
Your public key has been saved in id_rsa.pub.
The key fingerprint is:
SHA256:gqq78leroQlgJWchpkX1sSXkEIrRM25bgFhUWzKjIJY jan@g2
The key's randomart image is:
+---[RSA 2048]----+
|=X*+Oo= .        |
|BE*o.X =         |
|oooB. +          |
|  B ..           |
|.o o. . S        |
|o ..  ..         |
|. . .. .         |
|.o o...          |
|=++...           |
+----[SHA256]-----+

You can leave the passphrase empty. If you decide to access the server with a script, that's easier with an empty passphrase. Since I have the key on a laptop, and I take that laptop on the train, and I don't take the laptop with me when I'm using the lavatory on the train, and every time I hope, hope, hope that it's still there when I get back ... I'm using a passphrase. Well, I've also encrypted my file system. Call me paranoid.

This has created two files in a subdirectory of my home directory:

/home/jan/.ssh/id_rsa       # private key
/home/jan/.ssh/id_rsa.pub   # public key

Copy the contents of the file id_rsa.pub (it's a single line) from the laptop into the file ~/.ssh/authorized_keys on the server. Keep one ssh connection open while you log in from another terminal. Once you're sure that works, we can finally switch off password authentication:

# in /etc/ssh/sshd_config
PasswordAuthentication no

Make sure you have a way of becoming root on the server. Have a somewhat complex root password. If you like that thing, put your user in the sudoers files. For example, this is the standard setting on a Raspberry Pi:

#  in /etc/sudoers.d/pi - not really a good idea
pi ALL=(ALL) NOPASSWD: ALL

This is not necessarily a good idea. You did change the default password for user pi, didn't you?

Now to get root privileges, first log in as the normal user. Then, depending on the path you chose, go:

$ sudo su -                             $ su -
# enter user's password                 # enter root's password

However, once you made sure you can still become root, we can make sure nobody logs in as root directly:

# in /etc/ssh/sshd_config
PermitRootLogin no

Here's a simple measure out of the "security by obscurity" toolbox, but it goes a long way because apparently ssh worms are too lazy: Switch the exposed port to something that's non-standard.

# in /etc/ssh/sshd_config
Port 333

Watch your logfiles with the standard ssh port and with a different ssh port. For me that's in /var/log/auth.log. A few minutes should show you the difference.

There's a whole bunch of cool things you can do with ssh in addition to just accessing your server. To name just two:

See man ssh to learn how this works.

Also, check out tmux. With this "Terminal multiplexer" you can open several shells in one terminal and switch back and forth between them. More importantly, you can detach from that session and reattach later, while the shells - and programs running inside of them - continue running. It's a bit like disconnecting from a remote desktop session, and reconnecting later to find all your open windows where they were before. Only here they're not windows, they're command line programs.

Tracking

I want graphs for some vital parameters, e.g. core temperature and disk usage. Looking at those graphs from time to time, I'll learn what they are supposed to look like when everything's OK, and I might just notice when something's out of order. Munin does that.

Munin shows temperature over time in a graph.

For a web server, I might want to keep an eye on:

Behold one of the many graphs that awstats generates:

  Unique visitors: 591Number of visits: 1170Pages: 7588Hits: 9694Bandwidth: 78.33 MB Unique visitors: 692Number of visits: 1327Pages: 10573Hits: 13006Bandwidth: 64.10 MB Unique visitors: 836Number of visits: 1653Pages: 11237Hits: 14842Bandwidth: 95.37 MB Unique visitors: 142Number of visits: 178Pages: 2174Hits: 5514Bandwidth: 61.15 MB Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0 Unique visitors: 0Number of visits: 0Pages: 0Hits: 0Bandwidth: 0  
 Jan
2018
Feb
2018
Mar
2018
Apr
2018
May
2018
Jun
2018
Jul
2018
Aug
2018
Sep
2018
Oct
2018
Nov
2018
Dec
2018
 

Note that I've set up the apache configuration so that awstats is only accessible from localhost (127.0.0.1), and I'm using ssh port forwarding to look at those pages from my laptop.

# in /etc/apache2/sites-enabled/eucli.de.conf
<Directory /usr/lib/cgi-bin>
        Require ip 127.0.0.1
</Directory>

# in ~/.ssh/config
Host ...
        LocalForward 8080 127.0.0.1:80

Monitoring

It's one thing to look at pretty pictures while you're in the mood and while they're pretty, but if all goes pear-shaped you probably want to know as soon as possible. You don't? Yeah, maybe you can sleep better if you don't know that your disk is running full. Or that there's an unusual amount of traffic going on.

Munin can send emails when some limits are exceeded.

Nagios can check even more, and has a choice of notification backends.

Backup

Your data is at risk. We're protecting it against accidental messup by using version control. Against evil access by encryption. In order to protect it against hardware failure, we need copies on a different medium; even better in a different physical place.

There are some very good off-the-shelf solutions. Look into tarsnap for one.

Basic approaches can use tar or rsync over ssh.

Frills

The uninterruptible power supply (UPS) keeps the server running for a while in the case of a power failure. In case the blackout goes on for too long, it'll notify my system in time to shut down gracefully.

An overly paranoid tripwire system is described in this other entry here.

Further reading

Linux Security HOWTO

Clifford Stoll: The Cuckoo's Egg

jan 2018-04-09