Development log: 2021-10-02

2021 October 2

Applied for Maxmind GeoLite2 account at:
and received confirmation.  Set up account as follows:
    E-mail:     REDACTED
    Password: REDACTED
    Account/User ID:  616763
License key:
    Name:   Scanalyst geolocation
    Key:    REDACTED

Created a new instance with:
    AMI:            Amazon Linux 2 AMI
    Instance ID:    i-082d04418d82bcfa1
    Instance type:  t3.medium
    Instance details:
        Private IP address:
        Network:    (default)
        Subnet:     eu-central-1a  subnet-df741db6
        All other: (default)
        Root    /dev/xvda   snap-03df1111d702f5161  8 Gb    No delete on termination
        EBS     /dev/sdb    snap-08ecc2d9b324d4225  128 Gb  No delete on termination
        Name:   Scanalyst
    Security group: sg-049b61db659446aab     Scanalyst, launch-wizard-2 created 2021-10-02T21:11:21.242+02:00
Selected Launch.  Created a new key pair, which I called Scanalyst.pem
and saved in the Amazon_AWS/SCANALYST development directory.  Went into
launching state.

Configuration user data is:
    package_update: true
    package_upgrade: true
    - yum install -y amazon-efs-utils
    - apt-get -y install amazon-efs-utils
    - yum install -y nfs-utils
    - apt-get -y install nfs-common
    - file_system_id_1=fs-6f65ec34
    - efs_mount_point_1=/mnt/efs/fs1
    - mkdir -p "${efs_mount_point_1}"
    - test -f "/sbin/mount.efs" && printf "\n${file_system_id_1}:/ ${efs_mount_point_1} efs tls,_netdev\n" >> /etc/fstab || printf "\n${file_system_id_1} ${efs_mount_point_1} nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,_netdev 0 0\n" >> /etc/fstab
    - test -f "/sbin/mount.efs" && grep -ozP 'client-info]\nsource' '/etc/amazon/efs/efs-utils.conf'; if [[ $? == 1 ]]; then printf "\n[client-info]\nsource=liw\n" >> /etc/amazon/efs/efs-utils.conf; fi;
    - retryCnt=15; waitTime=30; while true; do mount -a -t efs,nfs4 defaults; if [ $? = 0 ] || [ $retryCnt -lt 1 ]; then echo File system mounted successfully; break; fi; echo File system not available, retrying to mount.; ((retryCnt--)); sleep $waitTime; done;

Launched a new instance, i-082d04418d82bcfa1, which is reported as
running.  Our temporary IPv4 address is

    $ ssh -i Scanalyst.pem ec2-user@
    X11 forwarding request failed on channel 0

           __|  __|_  )
           _|  (     /   Amazon Linux 2 AMI
    11 package(s) needed for security, out of 35 available
    Run "sudo yum update" to apply all updates.

Apply the updates:
    sudo su
    yum update
It updated the kernel among 28 packages updated and 7 installed.


uname -a reports:
        4.14.246-187.474.amzn2.x86_64 #1 SMP
        Tue Sep 7 21:48:11 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

Created a mount point and mounted /dev/xvdb.
    sudo su
    mkdir /server
    mkfs -t ext4 /dev/sdb
    fsck -f /dev/sdb
    mount /dev/sdb /server

    /dev/sdb   /server     ext4    defaults        1   2
to /etc/fstab.  Note that the file system on the root device is
now xfs, not ext4 as it was on the previous Linux AMI.  Someday
we might want to migrate /server to xfs, but this is not that

Set /etc/hostname to "scanalyst".

Rebooted to make sure it was re-mounted.  It was.

The system came up with the hostname changed and /server

Added accounts to /etc/passwd:
    REDACTED:x:500:500:John Walker:/server/home/kelvin:/bin/bash

Added corresponding entries to /etc/shadow:
then /etc/group:
and /etc/gshadow:

Transferred over relevant /server/home files from AWS with:
    (on AWS)
    cd /server/home/REDACTED
    tar cfv /tmp/h.tar .aws .bash_history .bash_logout .bash_profile .bashrc .config .lesshst .nedit .ssh .vim .viminfo .vimrc bin
    (on Hayek)
    scp -p aws:/tmp/h.tar /tmp
    scp -i Scanalyst.pem  /tmp/h.tar ec2-user@
    (on scanalyst)
    cd /server
    sudo su
    mkdir home
    mkdir home/REDACTED
    cd home/REDACTED
    tar xfv /tmp/h.tar

Now I can:
    sudo su
    su - REDACTED
and get to my /server/home/REDACTED directory.

Added REDACTED to /etc/sudoers.d/90-cloud-init-users to permit
sudo without a password.
    # User rules for kelvin

Installed our magic /bin/super utility.  I simply copied the
binary from the production Fourmilab AWS server.

Transferred the /root/.ssh/authorized_keys file from AWS to
scanalyst, saving the original as authorized_keys_ORIGINAL.

Edited /etc/ssh/sshd_config and set:
    PermitRootLogin yes

    systemctl restart sshd

Now I can log in as root from local machines without a
password.  Verified that regular user logins continue to work.
This will allow a mirror backup from Juno.

Rebooted to confirm that all of the configuration and
permission changes so far persist.

I can now log in with my regular account and use super when I get
there.  From now on, we shouldn't need to use ec2-user, but it's there
if necessary.

Assigned a new permanent Elastic IP address:
    Allocation ID: eipalloc-80070b8d
    IP Address:
    Association ID: eipassoc-043e70a41c071006b
    Instance: i-082d04418d82bcfa1
    Private IP address:
    Network interface ID: eni-07554278b655ddbc1

Added records to DNS for  A  TXT "v=spf1 ip4: ip4: ~all"

Verified that I can ssh to

Configured AWS command line:
    aws configure
    AWS Access Key ID [None]: REDACTED
    AWS Secret Access Key [None]: REDACTED
    Default region name [None]: eu-central-1
Tested with:
    aws s3 ls
and it seems to be working.

    yum install git
and tested: it works.

Cloned current Discourse Docker image with:
    mkdir discourse
    mkdir discourse/image
    git clone discourse/image

Signed in the GoogleApps Admin Console:

Clicked Security (you have to click "More" in the left panel to find it).

Went to Basic Settings/Go to settings for less secure apps.   This took
me to:
where I confirmed the setting for "Less secure apps" was:
    "Allow users to manage their access to less secure apps"
as I set it when setting up Ratburger.

Under API controls, "Trust internal, domain-owned apps" is checked.

The Cloud Platform Console is:
In that console clicked "+ Create project" to add a new project.  This took me
where I entered a "Project name" of
    Scanalyst Mail
which maps to a project ID of "scanalyst-mail".  The Organization was
pre filled-in as "".
Clicked Create.

Now, back to the Cloud Platform Console, set the Organization to and the new project was displayed.
There is a three dots menus which takes you to a page:

I cannot figure out how to get to the Dashboard I used before with
Ratburger, so I just edited the project name into the URL I had noted
down before:
and found it, which has an "Enable APIs and Services" item in the top bar.
I searched for Gmail API and found it (no change in URL).
Clicked "Enable".  This took me to:
which had a "Create credentials" button.  Pushed it.  A long wait ensued
with a spinner.  Finally, the "Create credentials" page appeared:
    Which API are you using?  Gmail API
    What data will you be accessing: User data
    Application type: Web application   Name: Discourse
and clicked "What credentials do I need?".  This displays:
"Create credentials".

    Name: SMTP with Google
    Authorized JavaScript origins:
    Authorized redirect URIs: (From Gmail-SMTP plug-in)
Clicked "Create client ID".  It responded:
    Client ID:
Downloaded credentials file:
which I saved in the SCANALYST directory.  In this file we have:
    "client_id" : "",
    "client_secret" : "REDACTED"

    yum install docker

Rebooted after all these changes to make sure they persisted.

    yum install nmap-ncat

Attempted Discourse installation:
    [root@scanalyst image]# ./discourse-setup
    Found 3GB of memory and 1 physical CPU cores
    setting db_shared_buffers = 768MB
    setting UNICORN_WORKERS = 2
    containers/app.yml memory parameters updated.

    Hostname for your Discourse? []:

    Checking your domain name . . .
    Connection to succeeded.
    Email address for admin account(s)? [,]:
    SMTP server address? []:
    SMTP port? [587]: 465
    SMTP user name? []:
    SMTP password? [pa$$word]: REDACTED
    notification email address? []:
    Optional email address for Let's Encrypt warnings? (ENTER to skip) []:
    Optional Maxmind License key (ENTER to continue without MAXMIND GeoLite2 geolocation database) [1234567890123456]: REDACTED

    Does this look right?

    Hostname          :
    Email             :
    SMTP address      :
    SMTP port         : 465
    SMTP username     :
    SMTP password     : REDACTED
    Notification email:
    Let's Encrypt :
    Maxmind license: REDACTED

    ENTER to continue, 'n' to try again, Ctrl+C to exit:

    letsencrypt.ssl.template.yml enabled

    Configuration file at containers/app.yml updated successfully!

    Updates successful. Rebuilding in 5 seconds.
    Building app
    Device "docker0" does not exist.
    Cannot connect to the docker daemon - verify it is running and you have access

Died because Docker was not running.  OK, let's:
    systemctl start docker
    systemctl status docker
        #   Looks OK
    systemctl enable docker
    systemctl is-enabled docker
        #   enabled
Now reboot again to make sure it comes up after a boot.

After the boot, restarted ./discourse-setup.  Now it appears to be

Here we go again.
    [root@scanalyst image]# ./discourse-setup
    The configuration file containers/app.yml already exists!
    ( thousands of messages )
    You have less than 5GB of free space on the disk where /var/lib/docker is located. You will need more space to continue
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/nvme0n1p1  8.0G  4.7G  3.4G  59% /

    Would you like to attempt to recover space by cleaning docker images and containers in the system? (y/N)n

Well, it looks like there isn't any way to get it to use anything other
than the root volume for the installation, and the detault 8 Gb root
isn't big enough

Resized the root volume to 32 Gb.  Did you know you can do this while
the system is running?

Made snapshot backup of root volume before hideously dangerous resizing
of partitions and file system.
    Snapshot:   snap-04347d032d7206aa9
    Volume:     vol-02ccb8d0ba2bd3ef9
    Description: Scanalyst root backup 2021-10-03 01:11

Instructions on growing EBS volumes are in:
Partitions on root drive are:
    # fdisk /dev/nvme0n1

    Welcome to fdisk (util-linux 2.30.2).
    Changes will remain in memory only, until you decide to write them.
    Be careful before using the write command.

    GPT PMBR size mismatch (16777215 != 67108863) will be corrected by w(rite).
    GPT PMBR size mismatch (16777215 != 67108863) will be corrected by w(rite).

    Command (m for help): p

    Disk /dev/nvme0n1: 32 GiB, 34359738368 bytes, 67108864 sectors
    Units: sectors of 1 * 512 = 512 bytes
    Sector size (logical/physical): 512 bytes / 512 bytes
    I/O size (minimum/optimal): 512 bytes / 512 bytes
    Disklabel type: gpt
    Disk identifier: 0A26FA1B-8D9E-492E-9CA8-225C968C2DC9

    Device           Start      End  Sectors Size Type
    /dev/nvme0n1p1    4096 16777182 16773087   8G Linux filesystem
    /dev/nvme0n1p128  2048     4095     2048   1M BIOS boot

    Partition table entries are not in disk order.

Attempted to grow partition 1:
    growpart /dev/nvme0n1 1
    CHANGED: partition=1 start=4096 old: size=16773087 end=16777183 new: size=67104735 end=67108831

Now fdisk says:
        Device           Start      End  Sectors Size Type
    /dev/nvme0n1p1    4096 67108830 67104735  32G Linux filesystem
    /dev/nvme0n1p128  2048     4095     2048   1M BIOS boot
and lsblk agrees:
    nvme1n1       259:0    0  128G  0 disk /server
    nvme0n1       259:1    0   32G  0 disk
    |-nvme0n1p1   259:2    0   32G  0 part /
    `-nvme0n1p128 259:3    0    1M  0 part

All right, now let's try growing the XFS root partition while it's
    # xfs_growfs -d /
    meta-data=/dev/nvme0n1p1         isize=512    agcount=4, agsize=524159 blks
             =                       sectsz=512   attr=2, projid32bit=1
             =                       crc=1        finobt=1 spinodes=0
    data     =                       bsize=4096   blocks=2096635, imaxpct=25
             =                       sunit=0      swidth=0 blks
    naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
    log      =internal               bsize=4096   blocks=2560, version=2
             =                       sectsz=512   sunit=0 blks, lazy-count=1
    realtime =none                   extsz=4096   blocks=0, rtextents=0
    data blocks changed from 2096635 to 8388091
and now, df -h says:
    # df -h
    Filesystem      Size  Used Avail Use% Mounted on
    devtmpfs        1.9G     0  1.9G   0% /dev
    tmpfs           1.9G     0  1.9G   0% /dev/shm
    tmpfs           1.9G  492K  1.9G   1% /run
    tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
    /dev/nvme0n1p1   32G  4.7G   28G  15% /
    /dev/nvme1n1    126G   72M  120G   1% /server     8.0E     0  8.0E   0% /mnt/efs/fs1
    tmpfs           389M     0  389M   0% /run/user/500

I'm going to reboot before trying anything else.

After the reboot, it still says 32 Gb, so I guess we survived.

Try once again.

    [root@scanalyst image]# ./discourse-setup
    The configuration file containers/app.yml already exists!

    . . . reconfiguring . . .

    nginx: [emerg] cannot load certificate "/shared/ssl/": PEM_read_bio_X509_AUX() failed (SSL: error:0909006C:PEM routines:get_name:no start line:Expecting: TRUSTED CERTIFICATE)

It appears Let's Encrypt is screwing up (surprise, surprise, surprise!)
I commented out the two lines that enable it in
~/discourse/image/containers/app.yml and tried again.

This time it built and started, but the registration confirmation 
E-mail was never received and the log indicated it was timing out every 
30 seconds with no response.  My guess was that the screwball Gmail 
send procedure was not working, as Discourse doesn't support its whacko 
Oauth2 humbuggery.  Enough.  Time to move to to Amazon Simple E-aail 
Service (SES).

Created verified E-mail on Amazon SES:
    Amazon Resource Name: arn:aws:ses:us-east-1:812306733122:identity/
    Region: US East (N. Virginia)

Created SES SMTP credentials:
    SMTP Username: REDACTED
    SMTP Password: REDACTED
    SMTP endpoint:
    STARTTLS Port: 25, 587 or 2587
    Transport Layer Security (TLS): Required
    TLS Wrapper Port: 465 or 2465
Downloaded to:

For information about how to graduate from the AWS/SES sandbox, see:

Configured in ~/discourse/image/containers/app.yml as:
    #DISCOURSE_SMTP_ENABLE_START_TLS: true           # (optional, default true)

Rebuilt again.  This time it was able to send the E-mail via SES and
I was able to complete administrator account registration.

Register Admin Account
    Username:   Fourmilab
    Password:   REDACTED

So, in summary, we're running, without SSL, and with E-mail through
Amazon SES.

So I’m not the only geek who makes notes in a text file as I work on new technologies. I’m in good company, I guess. (:

Though I’m not quite as verbose as you, John.

1 Like