install.jarvis.ubuntu.18.04

I.) Base install and configs:

1.) Install Ubuntu server as normal from an Ubuntu install CD.

Partition as follows:

Disks 1 and 2:

- BOS compat boot partition (grub_bios) - 100MB
/boot - 1GB (physical RAID)
- rest (physical RAID)
  - Make this RAID LVM, partitioned as follows:
    / - 50GB
    /tmp - 50GB
    /var - 50GB
    swap - 4GB
    /home - rest

Disks 3 and 4:
- whole disk (physical RAID)
  - Make this RAID LVM, partitioned as follows:
    /mnt/time_machine - 2TB
    /mnt/shared - 1TB

When it asks for what to install, select "standard system utilities"
and "OpenSSH server" and leave everything else blank.

2.) Networking is already set up with a reserved DHCP lease on the
router. It is accessible as jarvis.

3.) Bootstrap install of useful things:

sudo apt-get install synaptic

4.) After machine is up, run synaptic, go to settings->repositories
make sure the following are enabled:

- main
- universe
- restricted
- multiverse
- source code

Select other software tab and enable/add:
- partner
- partner (Source Code)
- independent
- independent (Source code)

Select updates tab and enable:
- bionic-security
- bionic-updates
- bionic-backports

(or just grab sources.list from some reasonable machine)

Select Statistics and enable:
- submit statistical information

5.) Do:
apt-get update
apt-get dist-upgrade

6.) Install more useful things

sudo apt-get install tree unison atop nmap emacs lm-sensors ntp ssmtp logcheck gdisk git-core gitk iftop mailutils ppa-purge xsltproc smartmontools

7.) Add accounts

8.) Make ssh work:
## For an old machine, use the old keys - you did save /etc, didn't you?

## For a new machine, use the new keys generated by the distro.

- make sure to add to the firewall

  sudo ufw allow ssh

- set:
- PermitRootLogin no

- once you've set up public key auth, turn off password access. edit
    /etc/ssh/sshd_config
and set
    PasswordAuthentication no

sudo service ssh restart

9.) Set up logcheck
- edit /etc/cron.d/logcheck and set it to @daily and not every 2 hours

10.) Set up Samba

sudo apt-get install cifs-utils samba

- and either set up a config file or copy one from
  ~/system_stuff/samba (I have several machine specific ones in there)

    sudo ufw allow from 192.168.9.0/24 to any port bootps
    sudo ufw allow from 192.168.9.0/24 to any port netbios-ns
    sudo ufw allow from 192.168.9.0/24 to any port netbios-dgm
    sudo ufw allow from 192.168.9.0/24 to any port netbios-ssn
    sudo ufw allow from 192.168.9.0/24 to any port microsoft-ds

- and set Samba to start on boot:

    sudo systemctl enable smbd
    sudo systemctl enable nmbd

- and restart them all now:

    sudo service smbd restart
    sudo service nmbd restart

- and, for this server, we do not need the AD DC server:

    sudo systemctl disable samba-ad-dc
    sudo service samba-ad-dc stop

- Make sure to add accounts with:

    smbpasswd -a

for each user
 
11.) Enable firewall (after allowing some other things through)

    sudo ufw allow from 192.168.9.0/24 to any port mdns

- enable:

    sudo ufw enable

Note that we install the Xerox 8580 as the Phaser 8500DN, using the
PPD from OpenPrinting, which I have stored in
~/system_stuff/printer_drivers/xerox.

- Under Server->Settings select:
    Show printers shared by other systems
- For servers, under "Server Settings", select:
    Publish shared printers conected to this system
- make sure to enable it in the ufw
    sudo ufw allow from 192.168.9.0/24 to any port ipp

- Fix up avahi's publishing of addresses:

Some services suck at IPv6, and, for some reason, the IPv4 multicast
propagation is very laggy. This leads to unreliable lookups of
hostnames via mDNS (you can check with `avahi-resolve-host-name -4
machine.local` and it will likely timeout). Anyway, this can just make
for a bad user experience, so have the server publish its IPv4 info in
the multicast packets.

    edit /etc/avahi/avahi-daemon.conf and set "publish-a-on-ipv6=yes"

12.) Removed. Printers are smart enough these days that they just show
     up and don't need CUPS anymore.

13.) Add dashpodder to crontab

crontab -e

Then add a line like this:

@daily  /home/matt/workspace/code/dashpodder/dashpodder.sh -v -c /home/matt/workspace/code/dashpodder/mp.conf

14.) Set up email backups (amahi only)

sudo apt-get install offlineimap libexpect-perl

crontab -e

then add:

@hourly              /home/matt/bin/offlineimap_helper

so it will sync email hourly

Of course, this is only half of it - it snapshots email, but doesn't
really do an archive. For that, we do an rsnapshot

sudo apt-get install rsnapshot

and then set up crontab:

0 */1  * * *  /usr/bin/rsnapshot -c /home/matt/.rsnapshot.conf hourly
30 3   * * *  /usr/bin/rsnapshot -c /home/matt/.rsnapshot.conf daily
0  3   * * 1  /usr/bin/rsnapshot -c /home/matt/.rsnapshot.conf weekly
30 2   1 * *  /usr/bin/rsnapshot -c /home/matt/.rsnapshot.conf monthly

15.) Set up sensors:

- For amahi / ASROCK E350
- add the following to /etc/modules:
w83627ehf

16.) Set up ssmtp
    cd /etc/ssmtp
    mv ssmtp.conf ssmtp.conf.old
    cp ~/system_stuff/ssmtp/ssmtp.conf .
    chgrp mail ssmtp.conf

17.) Add fstab line for external backup drive (because there's no
     automounter)

/dev/sde1          /mnt/external_backup  ext4   defaults,noauto  0       0

- Make sure to make the mountpoint:

sudo mkdir /mnt/external_backup

18.) Add UPS monitoring

- From:

http://blog.shadypixel.com/monitoring-a-ups-with-nut-on-debian-or-ubuntu-linux/

- The first bit, with GNOME, works for desktops, not server. Anyway,
  install things:

sudo apt-get install nut

- Edit /etc/nut/ups.conf

[ups]
    driver = usbhid-ups
    port = auto

- There's only one UPS hooked to this guy, so we don't need to worry
  about disambiguation.

- Also, if you just installed nut, but the UPS is already plugged in,
  you'll need to unplug and replug it to fire the notplug events.

- Start it:

sudo upsdrvctl start

- Add the following to /etc/nut/upsd.conf

ACL all 0.0.0.0/0
ACL localhost 127.0.0.1/32
ACCEPT localhost
REJECT all

- This will reject all nonlocal traffic

- Add the following to /etc/nut/upsd.users

[local_mon]
    password = PASSWORD_HERE
    allowfrom = localhost
    upsmon master

- Obviously, make PASSWORD_HERE some random password

- Add the following to /etc/nut/upsmon.conf, at the bottom of the
  MONITOR section:

MONITOR ups@localhost 1 local_mon PASSWORD_HERE master

- Edit /etc/nut/nut.conf and set

MODE=standalone

- Enable and start it:

sudo systemctl enable nut-server
sudo systemctl restart nut-server
sudo systemctl enable nut-client
sudo systemctl restart nut-client

- You can print statistics via:

upsc ups

- To do control things (changing batteries, etc.) you need to:

sudo apt-get install apcupsd

- then edit /etc/apcupsd/apcupsd.conf and find the UPSTYPE
  line. Commend out that and the DEVICE line, and set it as follows:

UPSTYPE usb

19.) Set up linode backups

 - make target dir
mkdir ~/attic/backup/linode

- on the remote server, you need to add the public key to
  authorized_keys, with the:

command=rsync --server --sender -vlHogDtprRze.iLsf . /etc /home /var/lib/mysql /var/lib/syma

- in front of it.

- add to cron
@daily               /home/matt/bin/linode_backup

- add to my rsnapshot config:
backup  /home/matt/attic/backup/linode/         localhost/

23.) Add monitoring (sortof):

- make sure landscape is installed (to get landscape-sysinfo):

sudo apt-get install landscape-common

- Then add the following to my crontab:

@daily               /usr/bin/ntpq -p; echo; df -lh; echo; cat /proc/mdstat; landscape-sysinfo

24.) Common shared bind mounts:

     sudo mkdir /home/matt/shared
     sudo mkdir /home/liz/shared

- Add remount lines:

/mnt/shared        /home/matt/shared/      none   bind             0       0
/mnt/shared        /home/liz/shared/       none   bind             0       0

- Then remount:

   sudo mount -a

- And we want guests to *actually* be able to write to it, so we need
  to fix those perms:

sudo chmod o+w /mnt/shared

- But we don't want random users deleting things, so set the sticky
  bit for a modicum of security:

sudo chmod +t /mnt/shared

- And we want the users group to be able to manipulate things:

sudo chmod g+s /mnt/shared

25.) Install experimental minidlna (now readymedia) branch with transcoding.

- Install dependencies (both for building and running)

sudo apt-get install autoconf autopoint gcc libavutil-dev libavcodec-dev libavformat-dev libjpeg-dev libsqlite3-dev libexif-dev libid3tag0-dev libogg-dev libvorbis-dev libflac-dev pkgconf libmagickwand-dev ffmpeg

- Get source:

git clone git clone https://github.com/necropotame/readymedia.git

- Build it:

./autogen.sh
./configure --prefix /usr/local/apps/readymedia
make
make install

- Fix perms
sudo chown -R root:root /usr/local/apps/readymedia
sudo chmod -R a+rX /usr/local/apps/readymedia

- Symlink it into place

cd /usr/local/sbin
sudo ln -s /usr/local/apps/readymedia/sbin/* .
cd /usr/local/share
sudo ln -s /usr/local/apps/readymedia/share/minidlna .

- Set up the config file

- Copy the minidla.conf out of the source code

And then you need to configure the transcoding on the file types you
want to transcode in the config file.

IMPORTANT: This transcoding only happens when the file is accessed. If
you server queries the file metadata or name and filters out things it
doesn't think it can play based on that, you won't see those files, so
the transcoding won't even get a chance to happen. This means it works
on stuff like my Onkyo receiver, but not on my Roku.

- edit /etc/minidlna.conf

- comment out all the media_dir lines

- add this one:

   media_dir=/mnt/shared/dlna

- And set the friendly name and other such things.

- And we need to set the amount of inodes that can be watched:

sudo sysctl fs.inotify.max_user_watches=100000

- And it needs to persist:
 - add /etc/sysctl.d/90-inotify.conf and set the contents to:

# Increase inotify max watchs per user for local minidlna
fs.inotify.max_user_watches = 100000

- Create an init script for it (or copy and modify one)

- If you modified one, it likely wants defaults in
  /etc/default/minidlna, so add that too.

- Make sure to make a privsep group with a locked password

sudo useradd minidlna
sudo usermod -L minidlna

- Make the cache dir and set it to be owned by minidlna

sudo mkdir /var/cache/minidlna
sudo chown minidlna:minidlna /var/cache/minidlna

- start and set it to start on boot
sudo systemctl enable minidlna 
sudo systemctl start minidlna 

- Allow through firewall

sudo ufw allow from 192.168.9.0/24 to any port 8200

- If you ever need to rebuld the DB:

sudo service minidlna stop
sudo -u minidlna minidlnad -R
   (and this can take awhile, so you kind of need to wait for it to finish)
sudo service minidlna start

26.) Time Machine

From:

http://dae.me/blog/1660/concisest-guide-to-setting-up-time-machine-server-on-ubuntu-server-12.04

- Install the daemons:

sudo apt-get install netatalk avahi-daemon

- fix perms so Liz can write to it:

sudo chown -R liz /mnt/time_machine/

- Allow through firewall

sudo ufw allow from 192.168.9.0/24 to any port 548

- Edit /etc/netatalk/AppleVolumes.default

- Add the following line which sets default settings for all shares:

:DEFAULT: options:upriv,usedots

- Add this line for Liz:

/mnt/time_machine "Liz's Time Machine" options:tm allow:liz

- Restart it:

sudo service netatalk restart

28.) Fix things not starting

After the 18.04 upgrade, ntp seems to like to die on startup. Add the
following to /etc/rc.local to kick it in the head:

sudo service ntp restart

29.) Fix DNS

Ref: https://github.com/systemd/systemd/issues/7009

It was set to:

    resolv.conf -> ../run/resolvconf/resolv.conf

by default, but this uses a local proxy which isn't otherwise set up
correctly (doesn't get local DNS server at home). Switching it to use
the systemd provided nameserver works better:

cd /etc &&  sudo rm resolv.conf && \
   sudo ln -s ../run/systemd/resolve/resolv.conf .

I really do wish Ubuntu would stop breaking things that have worked
for over a decade.

APPENDIX:

1.) Backup scripts

To run a backup, insert the external drive in to the cradle and run:

    sudo ~/bin/server_backup

2.) To set up a new drive for the above:

a.) Partition

    sudo gdisk /dev/sde

(gdisk because MBR tops out at 2TB)

create one big partition for the whole drive. No need to split it.

b.) make the filesystem

    sudo mkfs.ext4 -m 0 -L external_backup /dev/sde1

-m 0 => no reserved blocks; it's a backup drive.

-L external_backup = volume label. This is to make it consistent with the
   		     mount point, but the