Amazon AWS Linux 2023 Migration System Narrative

All of Fourmilab’s public-facing servers run on Amazon Web Services (AWS), currently using their free (other than fees for resources used) Linux 2 operating system. AWS have announced an end-of-life date for Linux 2 of 2025-06-30, and the release of a new free operating system platform, Amazon Linux 2023, which will be their path forward, with maintenance support through 2027.

It’s time to begin the migration of servers running Linux 2 to Linux 2023, and I decided to use, Fourmilab’s blockchain server as the pathfinder, since it has the smallest number of external dependencies and complex system-dependent software requirements. Amazon doesn’t make it easy to migrate servers from one version of their Linux platform to the next. This time, they have complicated things further by moving from a CentOS-like upstream system to Fedora and changing their release cadence from incremental updates to grouped updates.

Given the rapt interest in my earlier “Roswell System Narrative” post (almost five regular readers!), I thought I’d chronicle the migration of this system from AWS Linux to to Linux 2023, creating a guide to which I can link when others ask for advice in passing through the same tree chipper.

Here we go!


2023 October 8

Began campaign to migrate Agora to Amazon Linux 2023.  The Getting
Started document is:

Made a backup AMI.
    Agora Backup 2023-10-08  ami-0292585a6324f6fb0
        /           snap-09a895039c55900ac
        /server     snap-0d939f4a884ba9c18
        /vault      snap-0e1ae02b85f83f4ee  (Encrypted)
The snapshots of /server and /vault from this AMI will be used to create
the corresponding file systems of the Linux 2023 system.

Made volumes from the /server and /vault snapshots above.  All created
by selecting the snapshot, then Actions/Create volume from snapshot.
Choose Availability Zone eu-central-1b.
    /server     snap-0d939f4a884ba9c18  vol-0535b25530efd1498  Agora server L2023
    /vault      snap-0e1ae02b85f83f4ee  vol-03633dd175f06efa4
For /vault, the ARN for the encryption key is:
and the KMS key is:

Created a new instance with:
    AMI:            Amazon Linux 2023 AMI 2023.2.20231002.0 x86_64 HVM kernel-6.1
Clicked "Launch instance from AMI":
    Instance type:  t3.medium
    Key pair name:  Agora
    Instance details:
        Network:    (default)
        Subnet:     eu-central-1b
        Auto-assign IPv6 IP: Enable
        Firewall: Existing security group, Agora, sg-06373ae6aba10e811
        All other: (default)
        Root    /dev/xvda   snap-0371fdb2a84a9fb84  8 Gb    No delete on termination
        (Will add volumes from snapshots after launch.  Cannot add at
        launch time.)
        Name:   Agora L2023
Selected Launch.

New instance was created as i-0f273900c3dd3a6e3 with:
    IPv4 public address:
    IPv6 address:           2a05:d014:d43:3101:f550:4e24:1c7f:c7f2
    /dev/xvda   vol-06308ea129026e41d       Agora root L2023

Made an /etc/hosts entry on Hayek:   ag2
to reduce the amount of typing in system configuration.

Changed root volume to not delete on termination with:
    cd ~/w/Agora_AWS
    #   No super
    aws ec2 modify-instance-attribute --instance-id i-0f273900c3dd3a6e3 --block-device-mappings file://map.json
where the file map.json contained:
        "DeviceName": "/dev/xvda",
        "Ebs": {
          "DeleteOnTermination": false
Confirmed the mode had changed in the instance page (had to refresh to
see it).

Logged in with:
    $ ssh -i Agora.pem ec2-user@ag2
    X11 forwarding request failed on channel 0
       ,     #_
       ~\_  ####_        Amazon Linux 2023
      ~~  \_#####\
      ~~     \###|
      ~~       \#/ ___
       ~~       V~' '->
        ~~~         /
          ~~._.   _/
             _/ _/

    sudo su
    yum update
which reported nothing to update.

uname -a reports:
    Linux 6.1.55-75.123.amzn2023.x86_64 #1 SMP PREEMPT_DYNAMIC Tue Sep 26 20:06:16 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Attached the /server volume created from the production Agora system.
Go to Volumes, select "Agora server L2023".  Actions/Attach volume.
Select instance Agora L2023 (i-0f273900c3dd3a6e3).  Select device
name "/dev/sdb" (same as on Agora).
Click Attach, and it reports successfully attached.

Back on the console, running as root:
    fsck -f /dev/sdb
    /dev/nvme1n1: recovering journal
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    Free blocks count wrong (229710002, counted=229268449).
    Fix? yes
    Free inodes count wrong (98042691, counted=98042796).
    Fix? yes to all
    /dev/nvme1n1: ***** FILE SYSTEM WAS MODIFIED *****
    /dev/nvme1n1: 261204/98304000 files (3.5% non-contiguous), 163947551/393216000 blocks

    fsck -f /dev/sdb
    fsck from util-linux 2.37.4
    e2fsck 1.46.5 (30-Dec-2021)
    Pass 1: Checking inodes, blocks, and sizes
    Pass 2: Checking directory structure
    Pass 3: Checking directory connectivity
    Pass 4: Checking reference counts
    Pass 5: Checking group summary information
    /dev/nvme1n1: 261204/98304000 files (3.5% non-contiguous), 163947551/393216000 blocks

Mounted file system with:
    mkdir /server
    mount /dev/sdb /server
and it looks OK.

Now let's try the same with the encrypted /vault file system.  This
ought to be fun.
Go to Volumes, select "Agora vault L2023".  Actions/Attach volume.
Select instance Agora L2023 (i-0f273900c3dd3a6e3).  Select device
name "/dev/sdc" (same as on Agora).
Click Attach, and it reports successfully attached.

Back on the console, running as root:
    fsck -f /dev/sdb
e2fsck 1.46.5 (30-Dec-2021)
/dev/nvme2n1: recovering journal
Pass 1: Checking inodes, blocks, and sizes
Pass 2: Checking directory structure
Pass 3: Checking directory connectivity
Pass 4: Checking reference counts
Pass 5: Checking group summary information
/dev/nvme2n1: 16/524288 files (0.0% non-contiguous), 75215/2097152 blocks

Mounted file system with:
    mkdir /vault
    mount /dev/sdc /vault
and it looks OK.  Whew!  That was a lot easier than I feared.  It looks
like once you specify the encryption key with the file system, all you
have to do is mount it.

Confirmed /dev/sdb and /dev/sdc mounts shown in the Storage tab of the
Instances page for Agora L2023.  The KMS Key ID is shown for the /vault
file system.

    /dev/sdb    /server     ext4    defaults        1   2
    /dev/sdc    /vault      ext4    defaults        1   2
to /etc/fstab to mount file system on reboot.

Rebooted to make sure they were re-mounted.  They were.

Migrated from the default ec2-user account to kelvin as follows:
    Edited /etc/passwd and /etc/shadow entries and added lines:
        passwd: kelvin:x:500:500:John Walker:/server/home/kelvin:/bin/bash
        shadow: kelvin:!!:17509:0:99999:7:::
    Edited /etc/group and /etc/gshadow entries and changed lines for
    ec2_user/500 to:
        group:  kelvin:x:500:
        gshadow: kelvin:!::
Confirmed user name and group shown correctly in:
    ls -l /server/home

Installed my ~/.ssh and ~/bin directories as well as other .config
files from those on SC.

Installed /usr/bin/super so I can get to root from my login.

Added to /etc/sudoers.d/90-cloud-init-users:
    # User rules for kelvin
    kelvin   ALL=(ALL) NOPASSWD:ALL
so sudo works from my login.

Changed host name in prompt in ~/.bash_profile to @ag23.

Now I can log in directly with:
    ssh ag2
using my own private key.

Backed up /root/.ssh/authorized_keys as authorized_keys_ORIG
and installed the copy from AG.  I can now ssh in directly as
root from Hayek.

Now it's time to take the big gulp and reboot, then see if I can still
get in to the system with my regular log in and private key.

It worked!

    yum install gtk3-devel
This installed 147 dependencies.

    yum install gcc
    yum install gcc-c++
    yum install intltool

This permitted building and installing Geany, which had been downloaded
into ~/linuxtools/geany-1.38 and I re-built with:
    make distclean
    make install
This installs in the /usr/local directory tree.  Since Geany is
not available as an AWS package, this locally-built version will
not be automatically updated by the YUM package manager
(although all of its dependencies will be).  But then Geany is a
stable package which changes only very slowly.

    yum install xauth
Logged out and logged back in again, which created an ~/.Xauthority
file.  Geany now works and can pop up a window on the machine from
which I logged in with SSH.

    yum install "perl(JSON)"
    yum install "perl(CGI)"
This is required by "credit", which is not presently working:
    Parameter validation failed:
    Invalid length for parameter Dimensions[0].Value, value: 0, valid min length: 1
    malformed JSON string, neither array, object, number, string or atom, at character offset 0 (before "(end of string)") at /server/home/kelvin/bin/ line 124.
Maybe we haven't been up long enough to initialise the data.  I'll try

The Motif emulator from AWS Linux 2 doesn't seem to be available on
2023, which torpedoes nedit below the water line.  We may have to just
bid farewell to it and move entirely to Geany.

Installed the following packages to support Bitcoin Core.
    yum install xcb-util
    yum install xcb-util-wm
    yum install xcb-util-image
    yum install xcb-util-keysyms
    yum install xcb-util-renderutil
    yum install libxkbcommon-x11

Updated the ~/.bitcoin/bitcoin.conf for the private and (ephemeral)
public IP addresses of AG2:

Let's try starting Bitcoin Core.  Note that this is the whacko patched
version to run with locally-built libraries on AWS Linux 2.
    bitcoin -server
OK, that didn't work because we haven't installed the whacko GLIBC in
/opt.  Let's try running the vanilla Bitcoin Core from Satoshi's
treasure chest.

Cazart!  It works.  Or, at least it loads, pops up its start window,
and goes into "Verifying blocks", which seems to proceed very slowly.
Is it, for some reason, re-verifying the whole blockchain?  That would
be interesting.  (Note that I brought over all of its files from AG.
Why would it be doing that?)

It's only using around 0.3% of the CPU.  So, whatever it's doing, it
isn't CPU bound doing it.  After about 5 minutes, just jumped to 16%
done verifying blocks.

Made a symbolic link:
    ln -s /server/home/kelvin /home/kelvin
This allows shell scripts to run in either the local Juno environment
or from an AWS /server file system.

And, finally, it came up and is catching up, from eight hours behind
the inexorably growing blockchain.

It's caught up, and it seems to have loaded the wallets from the
encrypted /vault file system with no problems.  Let's see if it can
handle CLI status requests.

    bitcoin-cli -getinfo
        Chain: main
        Blocks: 811290
        Headers: 811290
        Verification progress: 99.9996%
        Difficulty: 57321508229258.04

        Network: in 4, out 10, total 14
        Version: 250000
        Time offset (s): -6
        Proxies: n/a
        Min tx relay fee rate (BTC/kvB): 0.00001000

    bitcoin-cli -netinfo 4
        Bitcoin Core client v25.0.0 - server 70016/Satoshi:25.0.0/

        <->   type   net  mping   ping send recv  txn  blk  hb addrp addrl  age id address                                    version
         in         ipv4      6      6    4   10                              3 25                       70002/
         in         ipv4     20     20    4   74                              1 32                        70016/bitcoinj:0.16.2/Bitcoin Wallet:9.26/
         in         ipv4     29     29    4   64                              1 33                         70002/
         in         ipv4     57     57    4   54                              3 23                       70002/
        out   full  ipv4      5      6    1    7    0    2  .   1005          7  3                           70016/Satoshi:25.0.0/
        out   full  ipv6      7      7    2    6    0           1002          2 27 [2001:7c0:2310:0:f816:3eff:fe52:6059]:8333 70016/Satoshi:22.0.0/
        out  block  ipv4     16     16   10   10    *              .          2 30                       70016/Satoshi:0.21.1/
        out   full  ipv4     28     28    1    1    0    4  .   1004          6  6                        70016/Satoshi:22.0.0/
        out   full  ipv4     93     93    0    5    0           1002          3 26                           70015/Satoshi:0.20.1/
        out   full  ipv4    114    114    1    1    0           1004          6  8                         70016/Satoshi:22.0.0/
        out   full  ipv4    117    117    4    5    0           1001          3 19                         70016/Satoshi:0.21.1/
        out   full  ipv4    160    160    8    3    0           1007          5 11                         70016/Satoshi:24.0.0/
        out   full  ipv4    225    226    3   13    0           1001          4 12                         70016/Satoshi:0.21.0/
        out  block  ipv4    265    265  104  104    *              .          1 31                        70016/Satoshi:22.0.0/
                             ms     ms  sec  sec  min  min                  min

                 ipv4    ipv6   total   block
        in          4       0       4
        out         9       1      10       2
        total      13       1      14

        Local addresses
        2a05:d014:d43:3101:f550:4e24:1c7f:c7f2     port   8333    score      1

Looks good, and we're receiving blocks from the network and serving
them to others.

There is a great deal that remains to be done, but this is much farther
than I expected to get in the first assault on the summit, so I'm going
to call it a night and see how it runs overnight.

Amazon Linux sounds like a whole new style of pursuing vendor lock-in.


Their update schedule will keep users on a treadmill which deters them from leaving but, on the other hand, the distribution is explicitly based upon Fedora Core, which is a freely available Linux distribution which you can run on any other cloud provider or on your own hardware. Their earlier free Linux products were vaguely based upon CentOS, but differed in ways that could be a gotcha if you tried to move elsewhere.

You don’t have to use the Amazon Linux AMI on AWS—a variety of other distributions are available such as Red Hat and Ubuntu, but you have to pay for some of them.


“What doesn’t kill you, mutates and tries again.”


2023 October 9

There were no problems running overnight.  We are currently serving
42 connections to the Bitcoin network, 32 in and 10 out.

    yum install texinfo
    yum install gmp-devel mpfr-devel libmpc-devel
    yum install patchelf

Now we segue into getting Fourmilab Blockchain Tools running on this
configuration.  Start by installing the Perl and Python pre-requisites.

Installed the following Perl modules required for Fourmilab Blockchain
Tools which are available via yum.

    yum install "perl(LWP)"
    yum install "perl(LWP::Protocol::https)"
    yum install "perl(Term::ReadKey)"

The following non-core module was already installed on the system or
as a dependency of those above:

Configured Perl for installation of modules from CPAN by
logging in as "root":
    su - root
    yum install perl-CPAN
    perl -MCPAN -e "shell"
then performed:
        install Bundle::CPAN
        reload cpan
reconfigured, followed by:
        o conf commit
We will always install Perl modules from the root account for
system-wide access.

    yum install openssl-devel
a prerequisite for Perl's Crypt::OpenSSL::AES.

Installed the following Perl modules via CPAN:
    Bitcoin::Crypto::Key::Public    (installed by above)
    Crypt::Digest::Keccak256        (installed by above)
    Crypt::Random::Seed             (installed by above)
    Digest::SHA3                    (installed by above)

Installed the following Python modules used by the Blockchain Tools
validate_wallet program.
    yum install python3-devel
    yum install python3-pip

    pip3 install base58
    pip3 install coincurve
    pip3 install cryptos
    pip3 install pysha3

Ran a test of the audit report generator with:
    cd ~/crontab/audit
    perl -verbose -shuffle -sort -zero 1audit.csv
It worked correctly.

    yum install cronie
This is the user-side crontab tools, which used to be installed
out of the box on Amazon Linux 2.

Installed cron job (in my account) for audit report.
    17 4 * * Mon /home/kelvin/crontab/audit 2>&1 | Mail -s "Audit report"

Next we undertake the steps to permit this server to send mail which
(more or less) runs the gauntlet of spam filters on its way to my

    yum install mailx
    yum install sendmail-cf

Backed up in /etc/mail:
    cp -p sendmail.cf_2023-10-09
    cp -p sendmail.mc_2023-10-09

In /etc/mail, did:
and verified a new had been generated which differed only in
header comments (generation time) from the backup.

Installed the following code in after the
"DAEMON_OPTIONS(`Port=smtp,Addr=, Name=MTA')dnl" line.
dnl #
dnl # The following are required to send mail to mail servers such as
dnl # Gmail with restrictive verification policies.  The CLIENT_OPTIONS
dnl # line causes an IPv6 connection, if accepted by the server, to be made
dnl # with the private IP address assigned to the instance.  Note that if
dnl # you kill and restart the instance and a new IP address is assigned,
dnl # you must update this statement.  You cannot specify the Elastic IP
dnl # (fixed, public) address in this statement--it doesn't work.
dnl # The next three lines masquerade the sending domain from the
dnl # AWS internal name to, which is necessary so that the
dnl # SPF record for the domain can be used to verify that the sending
dnl # IP address (the Elastic IP, this time) is valid.

Rebuilt with:
and restarted:
    systemctl restart sendmail

Mail sent from this system now arrives at my mail address, albeit
sometimes dropped by Gmail into the Spam bucket.

Made a ~/crontab/audit/audit_test job to test the audit
report on a small set of addresses.  Ran from cron with:
    15 14 * * * /home/kelvin/crontab/audit/audit_test 2>&1 | Mail -s "Audit report test"
(change time as appropriate to something in the near future).

Installed ~/.bash_profile from the existing Agora.  This is required to
run cron jobs that manually assume my identity.

For some screwball reason, crond.service wasn't running:
    systemctl status crond.service
        * crond.service - Command Scheduler
        Loaded: loaded (/usr/lib/systemd/system/crond.service; enabled; preset: enabled)
        Active: inactive (dead)
I enabled and started it with:
    systemctl enable crond.service
    systemctl start crond.service
and status now reports it running.

Tried the cron job test again.  This time it worked.  We'll need to
check the cron service to see if it starts correctly after the next

Fixed the shell scripts in ~/bin to run bitcoin, bitcoind, and bitcoin-cli
to not specify the local library path since the current versions work
with installed system libraries.

In the AWS EC2 console, selected the following Elastic IP address:
    Name:               Agora
    Allocated IPv4:
    Allocation ID:      eipalloc-0e70d67b0b3630c49
    Associated Instance: i-006f9abf48e689af6
    Private IP address:
and performed Actions/Disassociate Elastic IP address.

Selected Elastic IP address Agora and Actions/Associate Elastic IP 
    Name:               Agora
    Resource type:      Instance
    Instance:           i-0f273900c3dd3a6e3 (Agora L2023)
    Private IP address:
    Reassociation:      No

The Instances item for Agora L2023 now shows the Elastic IP address as
its public IPv4 address.  I can SSH to it after clearing the changed
host key warning.

Stopped bitcoind on the previous Agora instance and "shutdown -h now"
halted it.  After about two minutes, its status changed to "Stopped".

Rebooted Linux 2023 instance with "shutdown -r now".

After the reboot, restarted bitcoin.  It said it was 28 hours behind
the blockchain.  I have no idea what that is all about.

Updated the DNS AAAA record for to
2a05:d014:d43:3101:f550:4e24:1c7f:c7f2.  Since we transferred the
IPv4 elastic IP address of to the new instance, it remains
correct.  Similarly, the SPF record for requires no change.

We can ping6 from AWS using the new IPv6 address.

Synchronisation with the blockchain caught up and we're serving network
connections again.

Verified that crond.service is running after the reboot.

Added a:
    @reboot /home/kelvin/bin/bitcoind
statement to my crontab.  This is supposed to be run after each reboot
under my account.  And, after I corrected a typo in the shell script it
runs, it did!  After a reboot, bitcoind starts automatically and I can
control it with bitcoin-cli.

At this point, I proclaim that the migration is complete, and that the
production Agora is running on AWS Linux 2023.  Renamed the previous
instance "Agora (Linux 2 Legacy)" and this instance just "Agora".  The
Linux 2 Legacy instance is stopped, so it's no longer running up CPU
charges, but we're still paying for the instance and attached storage.
I will terminate it and release the storage as soon as I'm confident
we're here to stay on Linux 2023.

Deleted the "ag2" entry in /etc/hosts on Hayek.  All we need now is
"ag" or "agora".

Our "credit" utility, which monitors CPU credit available to an
instance, failed because AWS silently broke the previously-documented
way to obtain the instance ID on which a program was running.  You
used to use a query to http://instance-data/latest/meta-data/instance-id
but that now silently returns nothing.  Apparently, now you're supposed
to use:
    ec2-metadata -i
which returns a string like:
    instance-id: i-0f273900c3dd3a6e3
from which you dig out the instance-id.  I made the requisite changes
in ~/bin/ and now "credit" works.

Had to re-introduce ag to Hayek while running as root in order to
update the host key to that of the new instance.

2023 October 10

In ~/webdev/Blockchain/BackupWallet, changed:
to the internal DNS name of the new server.

    yum install libdb-utils
This provides db_dump, a utility for dumping Berkeley databases.

The invocation of blockchain-cli via a shell script in ~/bin ran
squarely into argument quoting Hell when BCBU attempted to pass a
wallet pass phrase containing spaces to unlock the wallet.  I
modified the ~/files/kelvin/webdev/Blockchain/BackupWallet script
to directly call the binary bitcoin-cli in its distribution lair
(via symbolic links which allow easy swapping in of new versions).
If we need to play games with library overrides sometime in the
future, this will have to be revisited.

Modified the hayek:~/w/Blockchain/ArchiveWallet script to specify
the location of the backup directory as situated on the new
Agora server.

And finally, after a couple of hours of pulling teeth, backups are
working again.

After the adventure with passing arguments containing spaces through
a shell script, I discovered:$*-and-$@
that bash has a gimmick which allows this to be accomplished in a
relatively straightforward manner.  If you use the special macro idiom:
within a shell script, it expands to all of the arguments of the
script with each surrounded by double quotes.  This preserves quoted
strings containing spaces as single arguments to commands invoked by
the script.

I modified the ~/bin/bitcoin-cli script to invoke the binary with:
    /home/kelvin/bin/bitcoin-core/bin/bitcoin-cli "$@"
and now arguments such as, for example, quoted pass phrases with
embedded spaces can be passed through the script.

This, in turn, allows the ~/webdev/Blockchain/BackupWallet script to
call blockchain-cli using the regular shell script intermediary
without problems with pass phrases.

After more than 19 hours of uptime, Bitcoin Core is serving 124
connections, 114 in and 9 out, both IPv4 and IPv6.

Added the -daemon option to the invocation of bitcoind in
~/bin/bitcoind.  This causes it to detach from the console and, more
importantly, not blither about what it's up to.

Now that we're up and running on the new instance, it's time to start
a backup cycle for it.  Since this is a new instance and volumes,
the saved snapshots for the old instance cannot be used as the
basis for this one, so we'll have to start with a new set.  This will
create a bump in storage charges, but it will soon revert to the
status quo ante when we delete the backups and volumes for the old
AWS Linux 2 instance.  To get a perfectly clean starting point for the
snapshots to follow, I'm making this image in reboot mode, where the
server is taken down to mirror the storage to snapshot.

Here is the base backup.  Halted bitcoind, then made image with reboot.
    Agora Backup 2023-10-10  ami-0315f2288ada87539
        /           snap-04fac8cf720677262
        /server     snap-0919b3e942ca908f8
        /vault      snap-0061885b42300fbcb  (Encrypted)

Well, that was awfully fast!  I wonder if it's somehow pulling in the
snapshots from which the volumes were created as the base for this
snapshot.  I didn't think it could do that, but there's no way it could
have made a copy of the 680 Gb /server file system from scratch
so quickly.

After the reboot, bitcoind restarted correctly with -daemon mode
specified.  We now just see the server process:
    /home/kelvin/bin/bitcoin-core/bin/bitcoind -server -daemon
in a "ps -ef".

2023 October 16

Mail from cron jobs was not being delivered because Sendmail had not
been set to start automatically at boot time.  I set:
    systemctl enable sendmail
    systemctl start sendmail
I'll confirm it starts automatically after the next boot.

After I started Sendmail, the weekly audit report, which ran normally
at the scheduled time, was delivered with no problems.  It had been
sitting in the queue waiting for Sendmail to deliver it.

Renamed instances and volumes to reflect completion of the migration.
    i-006f9abf48e689af6     Agora (Linux 2 Legacy)
        vol-08d93e687bc1fc2e6   Agora L2 root
        vol-041ca2cdef844eee2   Agora L2 server
        vol-012e08a09ad8cb7bf   Agora L2 vault
    i-0f273900c3dd3a6e3     Agora
        vol-06308ea129026e41d   Agora root
        vol-0535b25530efd1498   Agora server
        vol-03633dd175f06efa4   Agora vault

2023 October 17

AWS have announced that an incremental update version of AWS Linux
has been released.  You'll be notified of this when you log in, or
you can check any time with:
    dnf check-release-update
      A newer release of "Amazon Linux" is available.

      Available Versions:

      Version 2023.2.20231011:
        Run the following command to upgrade to 2023.2.20231011:

          dnf upgrade --releasever=2023.2.20231011

        Release notes:

The way Linux 2023 handles updates is described in:

The key change from previous AWS Linux distributions is that you will
never receive updates to individual packages of the most recent version
you have installed.  This is intended to keep clusters of systems which
were all built from the same AMI completely consistent in terms of
AWS-supported packages.  To install updates, you have to point dnf
(the successor of yum) to the new release version and update using it
as the source.  This will install all updated packages betwen the
version you're running and the designated one.

To check which packages will be updated, use:
    dnf check-update --releasever=2023.2.20231011
In this case I get:
    Amazon Linux 2023 repository
    Amazon Linux 2023 Kernel Livepatch repository

These look pretty inconsequential, but since the goal is to test the
update procedure, we'll install them anyway.

First, make a AMI/Snapshot backup before the update.
    Agora Backup 2023-10-17  ami-069b4b65b9f89d540
        /           snap-0b003d349dfcc39a4
        /server     snap-0daebc9621ea07da0
        /vault      snap-02cfbfbe28e0602f3 (Encrypted)

Out of an abundance of caution, stopped the Bitcoin Core server before
applying the updates.
    bitcoin-cli stop

    dnf update --releasever=2023.2.20231011
From this point on, it works essentially like a yum update in the
previous Linux 2 AMI.

    shutdown -r now

System came up quickly.  Bitcoin Core started normally.

We are now reported as up to date:
    dnf check-update
    Last metadata expiration check: 0:23:20 ago on Tue Oct 17 18:50:40 2023.
    dnf update
does nothing.

Based on this experience, the new update procedure seems pretty benign
and more likely to avoid inconsistencies in system configuration than
the previous package-incremental process.

John, thank you for educating us on the latest developments.
Amazon must have changed the link to the referenced article:


Apparently, Discourse and/or the browser (I am using Chrome on the Xubuntu Linux desktop) chops off the ends of long lines in <pre> HTML blocks at the right side of the window rather than wrapping them. When I view the source code for the post, the URL is complete and correct, but in the displayed post, it’s truncated after “cloud-optimized-linux” with no way to see the rest of the URL.

I know of no work-around for this.

The link you posted in your comment is to the document I intended to cite.


The successor of yum is do not finish?

1 Like

They say it’s an acronym for “Dandified YUM”, but methinks “do not finish” is more appropriate given the depth of the rabbit hole its web of dependencies can send you down when all you wanted to do is add some simple prerequisite for something you’re trying to build.

Perhaps, in response to the epidemic of obesity in the West, they decided “do not finish” was a better message to send to sedentary software developers than “yum”.


2023 October 23

Converted the following volumes from gp2 to gp3 storage type, which
reduces cost and separates size from throughput.  To do this, on the
Volumes page, select the volume then Actions/Modify volume and then
set the volume type to gp3.

        Volume ID              Name         IOPS     Throughput
    vol-03633dd175f06efa4   Agora vault     3000        125
    vol-0535b25530efd1498   Agora server    3000        125

You can modify the volume while it is attached and in use.  Its status
changes to "In-use - modifying", "In-use optimizing", and then after
about five minutes (for the /vault volume), to "In-use".  After this it
shows the new settings. The time required for the optimizing step
depends on the size of the volume: for the 1.5 Tb server partition, it
took around half an hour.

For the /server volume, I reduced the IOPS and Throughput which had
been guessed from its gp2 size, to the minimums (and defaults) for
gp3, which should save a bit of money and be more than sufficient for
this server's workload.

Deleted the ~/.nedit directory, as we have retired nedit.

It’s a good thing I retired and became a slacker. I’m so old I can remember when this stuff was a lot of fun sprinkled with the obligatory frustration and adult language. Today it takes me a couple years to get around to fixing the RPi 2 that quit running the Unifi controller after an OS update. All the access points are still working…


2023 November 2

We have been running on the AWS Linux 2023 platform since 2023-10-09
with no problems, so there's no reason to keep the legacy Linux 2
instance and file systems around.  Since the /server file system on
this instance is 1.5 Tb, it costs quite a bit sitting idle on the

Here's what we're cleaning up.
    Instance:   i-006f9abf48e689af6 (Agora (Linux 2 Legacy))
        /       vol-08d93e687bc1fc2e6 Agora L2 root     8 Gb
        /server vol-041ca2cdef844eee2 Agora L2 server   1500 Gb
        /vault  vol-012e08a09ad8cb7bf Agora L2 vault    8 Gb

Detached all storage volumes from the instance.

Instance panel now shows the instance with no storage attached.
Selected the instance and performed Instance state/Terminate instance.
Instance status changed to Shutting-down, then Terminated.

On the Volumes panel, selected each of the volumes in turn (all now
show as "Available", indicating they are bound to no instance) and
performed Actions/Delete volume.

Checked back on the Instances panel after a decent interval and the
terminated instance was now gone.

2023 November 10

Made a AMI/Snapshot backup before the update.
    Agora Backup 2023-11-10  ami-02a6238d1a123da7d
        /           snap-00375a5210507ba08
        /server     snap-0f02d288eaee0e4ec
        /vault      snap-02cfbfbe28e0602f3 (Encrypted)

Applied updates to the most recent Linux 2023 version (see the
2023-10-25 entry for AWS for details).
    dnf check-update --releasever=2023.2.20231030
    dnf upgrade --releasever=2023.2.20231030
Updated 38 packages, including a new kernel.


We are now running on kernel 6.1.56-82.125.amzn2023.x86_64.

Now, dnf check-release-update produces no output.

Halted Bitcoin Core:
    bitcoin-cli stop

Downloaded and installed Bitcoin Core v25.1.0 in:
Changed the symbolic link in ~/bin to point to the new version.

Started the GUI version and verified that it's running normally.

Shut down the GUI version and started the daemon.  Verified from
bitcoin-cli that it is running.

Confirmed that Bitcoin Core is running and serving requests.

2023 November 14

To see the log of CROND execution, use:
You will see CMD/CMDEND pairs for every job run.

To list the sendmail mail queue, use:

Sendmail somehow managed to get stuck, leaving the audit report stuck
in the sendmail queue, where it showed up in mailq as:
    3AD5n9Au113311    19437 Mon Nov 13 05:49 <
                     (host map: lookup (
I sent a test message and it too was immediately stuck in the sendmail
queue as well.

Just to see what happened, I tried:
    super systemctl restart sendmail
and after it started, it immediately delivered both of my messages in
the queue.  Included in those messages was an E-mail sent to my local
address by Sendmail, as follows:
    Subject: Warning: could not send message for past 4 hours
    The original message was received at Mon, 13 Nov 2023 05:49:09 GMT
    from localhost []
       ----- Transcript of session follows -----
    451 ip-172-31-19-99...ompute.internal: Name server timeout
    451 ip-172-31-19-99...ompute.internal: Name server timeout
    451 Name server timeout
    Warning: message still undelivered after 4 hours
    Will keep trying until message is 5 days old
    451 ip-172-31-19-99...ompute.internal: Name server timeout
Now, this just seems crazy.  The name it's trying to look up:
is Agora's own internal DNS name within AWS!  Why on Earth should it
encounter a name server timeout?  I tried a ping of that name from
Agora and it worked fine, both name server resolution and ping of the
internal IP address.

Sending mail from this cron job has worked with no problem since the
system was brought up on AWS Linux 2023.  This is the first failure.
At the moment, this is a complete mystery.