I'm not a pro administrator. I do have several computers running in the background, though. The number has exploded (to "five") since the Raspberry Pi became a thing. I've had machines running in the background for years.
Recently I rigged two new ones and found that I have pretty much settled on a list of things that need to be in place from the start, and maybe that's helpful for someone out there.
What do I want in a server?
Most of this is independent of the purpose that the machine actually "serves". Some items though are less important in some contexts.
Backup, for example, is a staple. On some machines it's critical to the point that it needs to be frequent, redundant, and encrypted. On other machines it's unnecessary, for example because there's no local data and the machine can be set up from scratch in thirty seconds.
The order of things here is a nonsensical mix of importance and chronology.
At the beginning, fiddling with the new machine will be exciting and fun, and I'll check up on it at least once a day. After a while, though, I'll be convinced that it "works", and I'll almost forget about it. Fast forward a year later: Something doesn't work. What exactly did I do in the first place?
Before I start: Journal
When you're setting up the first Linux machine in your life, or the first public-facing webserver, start a journal. Start a new entry every day, tag it with the date. Write down what you did. Be specific and exact. Wherever possible, copy-paste the actual command lines. Keep track in chronological order. When it makes sense, aggregate: What are the packages I installed on which day ... what are all the packages that are installed now? Maintain the journal right on the new machine, but keep a copy of it somewhere else.
Reflect on every step:
- Why was it hard?
- What did I learn?
- How did I find the answers?
Should anything go wrong, there's a step-by-step instruction on how to recreate what I had.
If you want to know what
vi means, open a terminal and type
man is for
manual. This works for almost every command line
tool. If you've never done this before, do that for every program
that you see me using in this article.
~ (tilde) is short for your home directory, typically in
/home/<username>, and also called
$HOME. So the file
journal.txt will sit in my home directory.
Oh, and of course never put passwords or other sensitive information in a plain text file.
git in /etc
There is a pragmatic adage that goes "version everything". Do that.
As you might have noticed, in Linux every configuration goes in a
plain-text file - most of it in
/etc. This is different from
Windows, where all the settings are spread out in registry,
configuration files in weird places, and sometimes proprietary
databases that you can't even read without a special tool.
I love the Linux way because it is dead simple to point at the place where a configuration needs to be made. Even better: I can put the whole system configuration under version control.
apt install git cd /etc git init git add . git commit -m 'starting system configuration'
Up-to-date system programs are your first line of defense against intrusion. Make a plan on when you run updates. Consider how exposed and how sensitive the machine is.
How to not forget to do updates?
# in /etc/cron.d/check-for-updates 43 21 * * * root /usr/bin/apt update && /usr/bin/apt list --upgradable
A cronjob is a little instruction for your system to run regularly. In this case, every night at 9:43 p.m. If any of the commands produce any output, that will be emailed to the root user account. Make sure you have it set up so that the root email gets forwarded to your single inbox that you read at least once every day.
How to make sure you don't miss the most important updates? Subscribe to the mailing list where critical updates are announced for your system, e.g. debian-security-announce .
What is the reason you are maintaining this machine? Remove all non-essential services. Every service that is running can be a backdoor, needs to be kept up-to-date, and possibly occupies storage, memory, or even CPU power.
That's why I like to start with a minimal system and add what I need, rather than starting with a desktop and forgetting to remove what I don't need.
For Debian, the installer has a step called "tasksel" that looks like this:
Audit your server from time to time. Look at running services, orphaned accounts, and whatever else you can think of, and for each of them check whether you still need it.
Look for open ports:
$ netstat -l | sed -n '1,2 p ; /tcp.*LISTEN/ p' Active Internet connections (only servers) Proto Recv-Q Send-Q Local Address Foreign Address State tcp 0 0 0.0.0.0:ssh 0.0.0.0:* LISTEN tcp 0 0 0.0.0.0:ipp 0.0.0.0:* LISTEN tcp 0 0 localhost:postgresql 0.0.0.0:* LISTEN tcp 0 0 localhost:smtp 0.0.0.0:* LISTEN tcp 0 0 localhost:3551 0.0.0.0:* LISTEN ...
"Local address" here shows all the services that are open for business. Do I really need postgresql to listen on a tcp socket?
What's 3551? Normally you want to run netstat with an option to
prevent name lookups. Name lookups can be timeconsuming. Use
netstat -n for "no names". You'll then find
5432 instead of
postgresql. For most ports, there's an easy way to find out what
$ grep 5432 /etc/services postgresql 5432/tcp postgres # PostgreSQL Database postgresql 5432/udp postgres
netstat does if you don't put
Now what's 3551? Turns out it's not in
/etc/services. Search the
port 3551 and you'll find that I have an uninterruptible
power supply attached to this server.
Also look for stale users.
less /etc/shadow ls /home/*/.ssh/authorized_keys
man shadow for what this means. Do you still know every user who
has a password or an authorized ssh key?
As is widely acknowledged, there's a tradeoff between security and convenience. I want my computer to be as safe as possible - that's best achieved by air gap. But I want to access my server at any time, from any point in the world. I mean, otherwise it wouldn't be much use, right?
Obviously, choose good passwords. I'm not going to dwell on that - there's plenty of info, and controversy, on what constitutes a good password.
Change default passwords
For example, in the standard Raspberry Pi operating system, there's a user with a standard name, and with a standard password. That's important in order to get started, but you should change the password as soon as you get a chance.
For unix, use
ssh. For Windows, use PowerShell through SSL.
However you do it, create a channel that's sufficiently encrypted and
guarded by keys.
For ssh, switch off password authentication and root login. But first, we need to make sure we get in ourselves.
On my laptop, I generate a pair of a public key and a private key:
$ ssh-keygen Generating public/private rsa key pair. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in id_rsa. Your public key has been saved in id_rsa.pub. The key fingerprint is: SHA256:gqq78leroQlgJWchpkX1sSXkEIrRM25bgFhUWzKjIJY jan@g2 The key's randomart image is: +---[RSA 2048]----+ |=X*+Oo= . | |BE*o.X = | |oooB. + | | B .. | |.o o. . S | |o .. .. | |. . .. . | |.o o... | |=++... | +----[SHA256]-----+
You can leave the passphrase empty. If you decide to access the server with a script, that's easier with an empty passphrase. Since I have the key on a laptop, and I take that laptop on the train, and I don't take the laptop with me when I'm using the lavatory on the train, and every time I hope, hope, hope that it's still there when I get back ... I'm using a passphrase. Well, I've also encrypted my file system. Call me paranoid.
This has created two files in a subdirectory of my home directory:
/home/jan/.ssh/id_rsa # private key /home/jan/.ssh/id_rsa.pub # public key
Copy the contents of the file
id_rsa.pub (it's a single line) from
the laptop into the file
~/.ssh/authorized_keys on the server. Keep
one ssh connection open while you log in from another terminal. Once
you're sure that works, we can finally switch off password
# in /etc/ssh/sshd_config PasswordAuthentication no
Make sure you have a way of becoming
root on the server. Have a
somewhat complex root password. If you like that thing, put your user
in the sudoers files. For example, this is the standard setting on a
# in /etc/sudoers.d/pi - not really a good idea pi ALL=(ALL) NOPASSWD: ALL
This is not necessarily a good idea. You did change the default
password for user
pi, didn't you?
Now to get root privileges, first log in as the normal user. Then, depending on the path you chose, go:
$ sudo su - $ su - # enter user's password # enter root's password
However, once you made sure you can still become root, we can make sure nobody logs in as root directly:
# in /etc/ssh/sshd_config PermitRootLogin no
Here's a simple measure out of the "security by obscurity" toolbox, but it goes a long way because apparently ssh worms are too lazy: Switch the exposed port to something that's non-standard.
# in /etc/ssh/sshd_config Port 333
Watch your logfiles with the standard ssh port and with a different
ssh port. For me that's in
/var/log/auth.log. A few minutes should
show you the difference.
There's a whole bunch of cool things you can do with ssh in addition to just accessing your server. To name just two:
Port forwarding allows you to access services through the secure tunnel of ssh. The simplest variant is
ssh -X ....
authorized_keys supports a bunch of weird tricks to restrict from where you can log in, or run a single specific program when you access the account with a specific key.
man ssh to learn how this works.
Also, check out tmux. With this "Terminal multiplexer" you can open several shells in one terminal and switch back and forth between them. More importantly, you can detach from that session and reattach later, while the shells - and programs running inside of them - continue running. It's a bit like disconnecting from a remote desktop session, and reconnecting later to find all your open windows where they were before. Only here they're not windows, they're command line programs.
I want graphs for some vital parameters, e.g. core temperature and disk usage. Looking at those graphs from time to time, I'll learn what they are supposed to look like when everything's OK, and I might just notice when something's out of order. Munin does that.
For a web server, I might want to keep an eye on:
- Which pages are popular?
- Which pages do people try to access and fail?
- How does traffic spread out over time?
Behold one of the many graphs that awstats generates:
Note that I've set up the apache configuration so that awstats is only accessible from localhost (127.0.0.1), and I'm using ssh port forwarding to look at those pages from my laptop.
# in /etc/apache2/sites-enabled/eucli.de.conf <Directory /usr/lib/cgi-bin> Require ip 127.0.0.1 </Directory>
# in ~/.ssh/config Host ... LocalForward 8080 127.0.0.1:80
It's one thing to look at pretty pictures while you're in the mood and while they're pretty, but if all goes pear-shaped you probably want to know as soon as possible. You don't? Yeah, maybe you can sleep better if you don't know that your disk is running full. Or that there's an unusual amount of traffic going on.
Munin can send emails when some limits are exceeded.
Nagios can check even more, and has a choice of notification backends.
Your data is at risk. We're protecting it against accidental messup by using version control. Against evil access by encryption. In order to protect it against hardware failure, we need copies on a different medium; even better in a different physical place.
There are some very good off-the-shelf solutions. Look into tarsnap for one.
Basic approaches can use
rsync over ssh.
The uninterruptible power supply (UPS) keeps the server running for a while in the case of a power failure. In case the blackout goes on for too long, it'll notify my system in time to shut down gracefully.
An overly paranoid tripwire system is described in this other entry here.
Clifford Stoll: The Cuckoo's Egg