2021 October 2
Applied for Maxmind GeoLite2 account at:
https://www.maxmind.com/en/geolite2/signup?lang=en
and received confirmation. Set up account as follows:
E-mail: REDACTED
Password: REDACTED
Account/User ID: 616763
License key:
Name: Scanalyst geolocation
Key: REDACTED
Created a new instance with:
AMI: Amazon Linux 2 AMI
Instance ID: i-082d04418d82bcfa1
Instance type: t3.medium
Instance details:
Private IP address: 172.31.7.90
Network: (default)
Subnet: eu-central-1a subnet-df741db6
All other: (default)
Storage:
Root /dev/xvda snap-03df1111d702f5161 8 Gb No delete on termination
EBS /dev/sdb snap-08ecc2d9b324d4225 128 Gb No delete on termination
Tags:
Name: Scanalyst
Security group: sg-049b61db659446aab Scanalyst, launch-wizard-2 created 2021-10-02T21:11:21.242+02:00
Selected Launch. Created a new key pair, which I called Scanalyst.pem
and saved in the Amazon_AWS/SCANALYST development directory. Went into
launching state.
Configuration user data is:
#cloud-config
package_update: true
package_upgrade: true
runcmd:
- yum install -y amazon-efs-utils
- apt-get -y install amazon-efs-utils
- yum install -y nfs-utils
- apt-get -y install nfs-common
- file_system_id_1=fs-6f65ec34
- efs_mount_point_1=/mnt/efs/fs1
- mkdir -p "${efs_mount_point_1}"
- test -f "/sbin/mount.efs" && printf "\n${file_system_id_1}:/ ${efs_mount_point_1} efs tls,_netdev\n" >> /etc/fstab || printf "\n${file_system_id_1}.efs.eu-central-1.amazonaws.com:/ ${efs_mount_point_1} nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev 0 0\n" >> /etc/fstab
- test -f "/sbin/mount.efs" && grep -ozP 'client-info]\nsource' '/etc/amazon/efs/efs-utils.conf'; if [[ $? == 1 ]]; then printf "\n[client-info]\nsource=liw\n" >> /etc/amazon/efs/efs-utils.conf; fi;
- retryCnt=15; waitTime=30; while true; do mount -a -t efs,nfs4 defaults; if [ $? = 0 ] || [ $retryCnt -lt 1 ]; then echo File system mounted successfully; break; fi; echo File system not available, retrying to mount.; ((retryCnt--)); sleep $waitTime; done;
Launched a new instance, i-082d04418d82bcfa1, which is reported as
running. Our temporary IPv4 address is 18.184.208.249.
$ ssh -i Scanalyst.pem ec2-user@18.184.208.249
X11 forwarding request failed on channel 0
__| __|_ )
_| ( / Amazon Linux 2 AMI
___|\___|___|
https://aws.amazon.com/amazon-linux-2/
11 package(s) needed for security, out of 35 available
Run "sudo yum update" to apply all updates.
Apply the updates:
sudo su
yum update
It updated the kernel among 28 packages updated and 7 installed.
Rebooted.
uname -a reports:
Linux ip-172-31-7-90.eu-central-1.compute.internal
4.14.246-187.474.amzn2.x86_64 #1 SMP
Tue Sep 7 21:48:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
Created a mount point and mounted /dev/xvdb.
sudo su
mkdir /server
mkfs -t ext4 /dev/sdb
fsck -f /dev/sdb
mount /dev/sdb /server
Added:
/dev/sdb /server ext4 defaults 1 2
to /etc/fstab. Note that the file system on the root device is
now xfs, not ext4 as it was on the previous Linux AMI. Someday
we might want to migrate /server to xfs, but this is not that
day.
Set /etc/hostname to "scanalyst".
Rebooted to make sure it was re-mounted. It was.
The system came up with the hostname changed and /server
mounted.
Added accounts to /etc/passwd:
REDACTED:x:500:500:John Walker:/server/home/kelvin:/bin/bash
Added corresponding entries to /etc/shadow:
REDACTED:!!:REDACTED:0:99999:7:::
then /etc/group:
REDACTED:500:
and /etc/gshadow:
REDACTED:!::
Transferred over relevant /server/home files from AWS with:
(on AWS)
super
cd /server/home/REDACTED
tar cfv /tmp/h.tar .aws .bash_history .bash_logout .bash_profile .bashrc .config .lesshst .nedit .ssh .vim .viminfo .vimrc bin
(on Hayek)
scp -p aws:/tmp/h.tar /tmp
scp -i Scanalyst.pem /tmp/h.tar ec2-user@18.184.208.249:/tmp
(on scanalyst)
cd /server
sudo su
mkdir home
mkdir home/REDACTED
chown REDACTED:REDACTED home/REDACTED
cd home/REDACTED
tar xfv /tmp/h.tar
Now I can:
sudo su
su - REDACTED
and get to my /server/home/REDACTED directory.
Added REDACTED to /etc/sudoers.d/90-cloud-init-users to permit
sudo without a password.
# User rules for kelvin
REDACTED ALL=(ALL) NOPASSWD:ALL
Installed our magic /bin/super utility. I simply copied the
binary from the production Fourmilab AWS server.
Transferred the /root/.ssh/authorized_keys file from AWS to
scanalyst, saving the original as authorized_keys_ORIGINAL.
Edited /etc/ssh/sshd_config and set:
PermitRootLogin yes
Restarted:
systemctl restart sshd
Now I can log in as root from local machines without a
password. Verified that regular user logins continue to work.
This will allow a mirror backup from Juno.
Rebooted to confirm that all of the configuration and
permission changes so far persist.
I can now log in with my regular account and use super when I get
there. From now on, we shouldn't need to use ec2-user, but it's there
if necessary.
Assigned a new permanent Elastic IP address:
Allocation ID: eipalloc-80070b8d
IP Address: 18.195.73.61
Association ID: eipassoc-043e70a41c071006b
Instance: i-082d04418d82bcfa1
Private IP address: 172.31.7.90
Network interface ID: eni-07554278b655ddbc1
Added records to DNS for fourmilab.ch
scanalyst.fourmilab.ch A 18.195.73.61
scanalyst.fourmilab.ch TXT "v=spf1 include:_spf.google.com ip4:193.8.230.0/24 ip4:18.195.73.61 ~all"
Verified that I can ssh to scanalyst.fourmilab.ch.
Configured AWS command line:
aws configure
AWS Access Key ID [None]: REDACTED
AWS Secret Access Key [None]: REDACTED
Default region name [None]: eu-central-1
Tested with:
aws s3 ls
and it seems to be working.
Installed:
yum install git
and tested: it works.
Cloned current Discourse Docker image with:
cd
mkdir discourse
mkdir discourse/image
super
git clone https://github.com/discourse/discourse_docker.git discourse/image
Signed in the GoogleApps Admin Console:
https://admin.google.com/fourmilab.ch/AdminHome
Clicked Security (you have to click "More" in the left panel to find it).
Went to Basic Settings/Go to settings for less secure apps. This took
me to:
https://admin.google.com/fourmilab.ch/AdminHome#ServiceSettings/notab=1&service=securitysetting&subtab=lesssecureappsaccess
where I confirmed the setting for "Less secure apps" was:
"Allow users to manage their access to less secure apps"
as I set it when setting up Ratburger.
Under API controls, "Trust internal, domain-owned apps" is checked.
The Cloud Platform Console is:
https://console.cloud.google.com/cloud-resource-manager?previousPage=%2F%3Fpli%3D1&pli=1
In that console clicked "+ Create project" to add a new project. This took me
to:
https://console.cloud.google.com/projectcreate?previousPage=%2Fcloud-resource-manager%3FpreviousPage%3D%252F%253Fpli%253D1%26pli%3D1&defaultProjectName&organizationId=1029803844244
where I entered a "Project name" of
Scanalyst Mail
which maps to a project ID of "scanalyst-mail". The Organization was
pre filled-in as "fourmilab.ch".
Clicked Create.
Now, back to the Cloud Platform Console, set the Organization to
fourmilab.ch and the new project was displayed.
There is a three dots menus which takes you to a page:
https://console.cloud.google.com/iam-admin/settings?project=scanalyst-mail
I cannot figure out how to get to the Dashboard I used before with
Ratburger, so I just edited the project name into the URL I had noted
down before:
https://console.cloud.google.com/apis/dashboard?project=scanalyst-mail&organizationId=1029803844244&duration=PT1H
and found it, which has an "Enable APIs and Services" item in the top bar.
I searched for Gmail API and found it (no change in URL).
Clicked "Enable". This took me to:
https://console.cloud.google.com/apis/api/gmail.googleapis.com/overview?project=scanalyst-mail
which had a "Create credentials" button. Pushed it. A long wait ensued
with a spinner. Finally, the "Create credentials" page appeared:
https://console.cloud.google.com/apis/credentials/wizard?project=scanalyst-mail
Selected:
Which API are you using? Gmail API
What data will you be accessing: User data
Application type: Web application Name: Discourse
and clicked "What credentials do I need?". This displays:
https://console.cloud.google.com/apis/credentials/wizard?project=scanalyst-mail
"Create credentials".
Set:
Name: SMTP with Google
Authorized JavaScript origins:
http://www.ratburger.org
Authorized redirect URIs: (From Gmail-SMTP plug-in)
http://www.ratburger.org/wp-admin/options-general.php?page=gmail-smtp-settings&action=oauth_grant
Clicked "Create client ID". It responded:
Client ID:
REDACTED-6nuvkdjugt4amffq2smrig1dsmdusevd.apps.googleusercontent.com
Downloaded credentials file:
client_secret_REDACTED-6nuvkdjugt4amffq2smrig1dsmdusevd.apps.googleusercontent.com.json
which I saved in the SCANALYST directory. In this file we have:
"client_id" : "REDACTED-6nuvkdjugt4amffq2smrig1dsmdusevd.apps.googleusercontent.com",
"client_secret" : "REDACTED"
Installed:
yum install docker
Rebooted after all these changes to make sure they persisted.
Installed:
yum install nmap-ncat
Attempted Discourse installation:
[root@scanalyst image]# ./discourse-setup
Found 3GB of memory and 1 physical CPU cores
setting db_shared_buffers = 768MB
setting UNICORN_WORKERS = 2
containers/app.yml memory parameters updated.
Hostname for your Discourse? [discourse.example.com]: scanalyst.fourmilab.ch
Checking your domain name . . .
Connection to scanalyst.fourmilab.ch succeeded.
Email address for admin account(s)? [me@example.com,you@example.com]: REDACTED@fourmilab.ch
SMTP server address? [smtp.example.com]: smtp.google.com
SMTP port? [587]: 465
SMTP user name? [user@example.com]: REDACTED@fourmilab.ch
SMTP password? [pa$$word]: REDACTED
notification email address? [noreply@scanalyst.fourmilab.ch]: noreply@fourmilab.ch
Optional email address for Let's Encrypt warnings? (ENTER to skip) [me@example.com]: REDACTED@fourmilab.ch
Optional Maxmind License key (ENTER to continue without MAXMIND GeoLite2 geolocation database) [1234567890123456]: REDACTED
Does this look right?
Hostname : scanalyst.fourmilab.ch
Email : REDACTED@fourmilab.ch
SMTP address : smtp.google.com
SMTP port : 465
SMTP username : REDACTED@fourmilab.ch
SMTP password : REDACTED
Notification email: noreply@fourmilab.ch
Let's Encrypt : REDACTED@fourmilab.ch
Maxmind license: REDACTED
ENTER to continue, 'n' to try again, Ctrl+C to exit:
letsencrypt.ssl.template.yml enabled
Configuration file at containers/app.yml updated successfully!
Updates successful. Rebuilding in 5 seconds.
Building app
Device "docker0" does not exist.
Cannot connect to the docker daemon - verify it is running and you have access
Died because Docker was not running. OK, let's:
super
systemctl start docker
systemctl status docker
# Looks OK
systemctl enable docker
systemctl is-enabled docker
# enabled
Now reboot again to make sure it comes up after a boot.
After the boot, restarted ./discourse-setup. Now it appears to be
running.
Here we go again.
[root@scanalyst image]# ./discourse-setup
The configuration file containers/app.yml already exists!
( thousands of messages )
You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
Filesystem Size Used Avail Use% Mounted on
/dev/nvme0n1p1 8.0G 4.7G 3.4G 59% /
Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)n
Well, it looks like there isn't any way to get it to use anything other
than the root volume for the installation, and the detault 8 Gb root
isn't big enough
Resized the root volume to 32 Gb. Did you know you can do this while
the system is running?
Made snapshot backup of root volume before hideously dangerous resizing
of partitions and file system.
Snapshot: snap-04347d032d7206aa9
Volume: vol-02ccb8d0ba2bd3ef9
Description: Scanalyst root backup 2021-10-03 01:11
Instructions on growing EBS volumes are in:
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/recognize-expanded-volume-linux.html?icmpid=docs_ec2_console
Partitions on root drive are:
# fdisk /dev/nvme0n1
Welcome to fdisk (util-linux 2.30.2).
Changes will remain in memory only, until you decide to write them.
Be careful before using the write command.
GPT PMBR size mismatch (16777215 != 67108863) will be corrected by w(rite).
GPT PMBR size mismatch (16777215 != 67108863) will be corrected by w(rite).
Command (m for help): p
Disk /dev/nvme0n1: 32 GiB, 34359738368 bytes, 67108864 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: 0A26FA1B-8D9E-492E-9CA8-225C968C2DC9
Device Start End Sectors Size Type
/dev/nvme0n1p1 4096 16777182 16773087 8G Linux filesystem
/dev/nvme0n1p128 2048 4095 2048 1M BIOS boot
Partition table entries are not in disk order.
Attempted to grow partition 1:
growpart /dev/nvme0n1 1
CHANGED: partition=1 start=4096 old: size=16773087 end=16777183 new: size=67104735 end=67108831
Now fdisk says:
Device Start End Sectors Size Type
/dev/nvme0n1p1 4096 67108830 67104735 32G Linux filesystem
/dev/nvme0n1p128 2048 4095 2048 1M BIOS boot
and lsblk agrees:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
nvme1n1 259:0 0 128G 0 disk /server
nvme0n1 259:1 0 32G 0 disk
|-nvme0n1p1 259:2 0 32G 0 part /
`-nvme0n1p128 259:3 0 1M 0 part
All right, now let's try growing the XFS root partition while it's
running:
# xfs_growfs -d /
meta-data=/dev/nvme0n1p1 isize=512 agcount=4, agsize=524159 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=1 spinodes=0
data = bsize=4096 blocks=2096635, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal bsize=4096 blocks=2560, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
data blocks changed from 2096635 to 8388091
and now, df -h says:
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 1.9G 0 1.9G 0% /dev
tmpfs 1.9G 0 1.9G 0% /dev/shm
tmpfs 1.9G 492K 1.9G 1% /run
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/nvme0n1p1 32G 4.7G 28G 15% /
/dev/nvme1n1 126G 72M 120G 1% /server
127.0.0.1:/ 8.0E 0 8.0E 0% /mnt/efs/fs1
tmpfs 389M 0 389M 0% /run/user/500
I'm going to reboot before trying anything else.
After the reboot, it still says 32 Gb, so I guess we survived.
Try once again.
[root@scanalyst image]# ./discourse-setup
The configuration file containers/app.yml already exists!
. . . reconfiguring . . .
nginx: [emerg] cannot load certificate "/shared/ssl/scanalyst.fourmilab.ch.cer": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)
It appears Let's Encrypt is screwing up (surprise, surprise, surprise!)
I commented out the two lines that enable it in
~/discourse/image/containers/app.yml and tried again.
This time it built and started, but the registration confirmation
E-mail was never received and the log indicated it was timing out every
30 seconds with no response. My guess was that the screwball Gmail
send procedure was not working, as Discourse doesn't support its whacko
Oauth2 humbuggery. Enough. Time to move to to Amazon Simple E-aail
Service (SES).
Created verified E-mail on Amazon SES:
kelvin@fourmilab.ch
Amazon Resource Name: arn:aws:ses:us-east-1:812306733122:identity/kelvin@fourmilab.ch
Region: US East (N. Virginia)
Created SES SMTP credentials:
ses-smtp-user.20211003-REDACTED
SMTP Username: REDACTED
SMTP Password: REDACTED
SMTP endpoint: email-smtp.us-east-1.amazonaws.com
STARTTLS Port: 25, 587 or 2587
Transport Layer Security (TLS): Required
TLS Wrapper Port: 465 or 2465
Downloaded to:
SCANALYZER/AWS_SES_credentials.csv
For information about how to graduate from the AWS/SES sandbox, see:
https://docs.aws.amazon.com/ses/latest/dg/request-production-access.html
Configured in ~/discourse/image/containers/app.yml as:
DISCOURSE_SMTP_ADDRESS: email-smtp.us-east-1.amazonaws.com
DISCOURSE_SMTP_PORT: 587
DISCOURSE_SMTP_USER_NAME: REDACTED
DISCOURSE_SMTP_PASSWORD: REDACTED
#DISCOURSE_SMTP_ENABLE_START_TLS: true # (optional, default true)
DISCOURSE_SMTP_DOMAIN: fourmilab.ch
DISCOURSE_NOTIFICATION_EMAIL: REDACTED@fourmilab.ch
Rebuilt again. This time it was able to send the E-mail via SES and
I was able to complete administrator account registration.
Register Admin Account
Email: REDACTED@fourmilab.ch
Username: Fourmilab
Password: REDACTED
So, in summary, we're running, without SSL, and with E-mail through
Amazon SES.
So I’m not the only geek who makes notes in a text file as I work on new technologies. I’m in good company, I guess. (:
Though I’m not quite as verbose as you, John.
1 Like