2023 October 14
Began the process of migrating the SCANALYST site to Amazon Linux 2023. Created a new instance with: AMI: Amazon Linux 2023 AMI 2023.2.20231011.0 x86_64 HVM kernel-6.1 ami-065ab11fbd3d0323d Clicked "Launch instance from AMI": Instance type: t3.medium Key pair name: Scanalyst Instance details: Network: (default) Subnet: eu-central-1b Auto-assign IPv6 IP: Enable Firewall: Existing security group, Scanalyst, sg-049b61db659446aab All other: (default) Storage: Root /dev/xvda snap-0b108c803dcd979d5 96 Gb No delete on termination Server /dev/sdb No snapshot 2 Gb No delete on termination Tags: Name: Scanalyst L2023 Selected Launch. New instance was created as i-092217f3135c36549 with: IPv4 public address: 184.108.40.206 IPv6 address: 2a05:d014:d43:3101:d577:c011:85c2:e9db /dev/xvda vol-0cf623d818601d186 Scanalyst root L2023 /dev/sdb vol-03cabd4e67ff7ffc8 Scanalyst server L2023 Made an /etc/hosts entry on Hayek: 220.127.116.11 sc2 to reduce the amount of typing in system configuration. Logged in with: $ ssh -i Scanalyst.pem ec2-user@sc2 X11 forwarding request failed on channel 0 , #_ ~\_ ####_ Amazon Linux 2023 ~~ \_#####\ ~~ \###| ~~ \#/ ___ https://aws.amazon.com/linux/amazon-linux-2023 ~~ V~' '-> ~~~ / ~~._. _/ _/ _/ _/m/' Ran: sudo su dnf update which reported nothing to update. uname -a reports: Linux ip-172-31-19-123.eu-central-1.compute.internal 6.1.55-75.123.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 26 20:06:16 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux Created a mount point and mounted /dev/sdb. sudo su mkdir /server mkfs -t ext4 /dev/sdb fsck -f /dev/sdb mount /dev/sdb /server Added: /dev/sdb /server ext4 defaults 1 2 to /etc/fstab. Note that the file system on the root device is now xfs, not ext4 as it was on the previous Linux 2 AMI. Someday we might want to migrate /server to xfs, but this is not that day. Set /etc/hostname to "scanalyst". Rebooted to make sure it was re-mounted. It was. The system came up with the hostname changed and /server mounted. Added accounts to /etc/passwd: kelvin:x:500:500:John Walker:/server/home/kelvin:/bin/bash Added corresponding entries to /etc/shadow: kelvin:!!:18902:0:99999:7::: then /etc/group: kelvin:x:500: and /etc/gshadow: kelvin:!:: Transferred the current contents of the /server file system from the production Scanalyst site to /server on sc2. This transfers my /server/home/kelvin directory under which everything I need to log in and build Discord should be installed. Indeed, I can now log in to sc2 via ssh without a password. Added kelvin to /etc/sudoers.d/90-cloud-init-users to permit sudo without a password. # User rules for kelvin kelvin ALL=(ALL) NOPASSWD:ALL Tested: it works. Installed our magic /bin/super utility. I simply copied the binary from the production Fourmilab AWS server. Transferred the /root/.ssh/authorized_keys file from production Scanalyst to sc2, saving the original as authorized_keys_ORIGINAL. Edited /etc/ssh/sshd_config and set: PermitRootLogin yes Restarted: systemctl restart sshd Now I can log in as root from local machines without a password. Verified that regular user logins continue to work. This will allow a mirror backup from Juno. Rebooted to confirm that all of the configuration and permission changes so far persist. I can now log in with my regular account and use super when I get there. From now on, we shouldn't need to use ec2-user, but it's there if necessary. Configured AWS command line: aws configure AWS Access Key ID [None]: REDACTED AWS Secret Access Key [None]: REDACTED Default region name [None]: eu-central-1 Tested with: aws s3 ls and it seems to be working. Installed: super dnf install git and tested: it works. Installed: dnf install docker Installed: dnf install nmap-ncat Set up Docker to start at boot time. systemctl start docker systemctl status docker # Looks OK systemctl enable docker systemctl is-enabled docker # enabled Now reboot again to make sure it comes up after a boot. Installed: dnf install xauth in the hope this will allow X11 forwarding on SSH logins. After logging out and back in, it re-created ~/.Xauthority and now X11 tunnelling works. Installed: dnf install gtk3-devel # This installed 147 dependencies. dnf install gcc dnf install gcc-c++ dnf install intltool This permitted re-building and installing Geany, which had been downloaded from: https://www.geany.org/download/releases/ into ~/linuxtools/geany-1.38 on the production Scanalyst and I re-built with: cd ~/linuxtools/geany-1.38 make distclean ./configure make super make install This installs in the /usr/local directory tree. Since Geany is not available as an AWS package, this locally-built version will not be automatically updated by the dnf package manager (although all of its dependencies will be). But then Geany is a stable package which changes only very slowly. I chose to rebuild from source to avoid library version dependency Hell later on as the system is updated. I tested it after installation and it works, which also confirmed that X11 forwarding is working. Installed: super dnf install "perl(JSON)" dnf install "perl(CGI)" This is required by "credit", which now works. Rebuilt the Bacula file daemon in /server/src/bacula to verify all dependencies present and library compatibility. The daemon source was copied over from production Scanalyst with the rest of /server. cd /server/src/bacula/bacula-5.2.10 ./BuildFourmilabClient This does the complete configuration, make, and installation process and worked with no problems. Started the Bacula file daemon: super /server/init/bacula start and verified it is running. Discovered that when I set up the server, I neglected to add a line in a newly created /etc/rc.d/rc.local to start our local services (which consist only of Bacula). Consequently, Bacula was not restarted automatically after a reboot. Added: /server/init/servers $1 to /etc/rc.d/rc.local (who knew it still existed in these dark days of systemd?) to start and stop our local servers at boot and shutdown. Set the file executable, which systemd requires: chmod 755 /etc/rc.d/rc.local Now it dies with: systemctl start rc-local Job for rc-local.service failed because the control process exited with error code. See "systemctl status rc-local.service" and "journalctl -xeu rc-local.service" for details. And the systemctl status..." says: scanalyst (rc.local): rc-local.service: Failed to execute /etc/rc.d/rc.local: Exec format error scanalyst (rc.local): rc-local.service: Failed at step EXEC spawning /etc/rc.d/rc.local: Exec format error After further psychoanalysis, I determined it need a "shebang" line at the start of rc.local to believe it's really executable. I added: #! /bin/bash and now systemctl start and stop work. But (and there's always a but, isn't there?) "systemctl enable" continued to fail with an eight line error message worthy of Microsoft. According to: https://www.linuxbabe.com/linux-server/how-to-enable-etcrc-local-with-systemd the way around this is to create a /etc/systemd/system/rc-local.service file containing the following incantations to demonic systemd (example updated per comments below): [Unit] Description=/etc/rc.d/rc.local Compatibility ConditionPathExists=/etc/rc.d/rc.local [Service] Type=forking ExecStart=/etc/rc.d/rc.local start TimeoutSec=0 StandardOutput=tty RemainAfterExit=yes SysVStartPriority=99 [Install] WantedBy=multi-user.target Bit happens, and who ya gonna call? With this file in place, I can now perform: systemctl enable rc-local Created symlink /etc/systemd/system/multi-user.target.wants/rc-local.service → /etc/systemd/system/rc-local.service. and "systemctl status rc-local" reports: Loaded: loaded (/etc/systemd/system/rc-local.service; enabled; preset: disabled) enabled ... disabled. What the Hell, it's systemd. All right, it's time to once again reboot and see if all of this stuff works after a reboot. Nope. Bacula was not started. systemctl status explains: ConditionPathExists=/etc/rc.local was not met so I guess we need to modify that /etc/systemd/system/rc-local.service file to set: Description=/etc/rc.d/rc.local Compatibility ConditionPathExists=/etc/rc.d/rc.local ExecStart=/etc/rc.d/rc.local start instead. (I have modified the model file above accordingly to avoid repeating this problem is it is copied blindly.) Let's reboot again! And, finally, this time it started the Bacula file daemon. The systemd status report looks normal. And, with that, I'm going to call it a night (or, more precisely, not-so-early morning). You may have noticed I haven't yet done a single thing about bringing up Discourse, its automatic installation of Nginx, or the forbidding tower of Let's Encrypt, not to mention how we synchronise this new, pristine Discourse installation with the database of the production site. That is a matter for another day.