2023 October 19
Decided to set aside the assault on MariaDB (which is used only by
Movable Type for Fourmilog and the original Scanalyzer) and proceed
with bringing up other site infrastructure.
dnf install netpbm
dnf install ImageMagick-devel
dnf install ImageMagick-perl
dnf install perl-CPAN
Configured Perl CPAN installer with:
perl -MCPAN -e "shell"
then performed:
o conf init pushy_https
Do you want to turn the pushy_https behaviour on? [yes] yes
o conf commit
exit
perl -MCPAN -e "shell"
then performed:
install Term::ReadLine::Perl
install Bundle::CPAN
reload cpan
reconfigured, followed by:
o conf commit
exit
Installed Perl module packages:
dnf install \
> perl-Compress-Zlib \ # Already installed
> perl-Convert-ASN1 \
> perl-DateManip \
> perl-DBD-MySQL \
> perl-DBI \ # Already installed
> perl-HTML-Parser \ # Already installed
> perl-HTML-Tagset \ # Installed by perl-HTML-Parser
> perl-libwww-perl \
> perl-libxml-perl \
> perl-SGMLSpm \
> perl-URI \ # Already installed
> perl-XML-Dumper \
> perl-XML-NamespaceSupport \
> perl-XML-Parser \ # Installed by perl-libxml-perl
> perl-XML-SAX
Installed prerequisites for Perl packages:
dnf install gd # Already installed
dnf install gd-devel
Installed dependencies for assorted Web services:
dnf install ncurses-devel
dnf install perl-GD
dnf install libxml2-devel # Already installed
dnf install openssl-devel
dnf install flex
dnf install mailx
dnf install aspell
dnf install gnuplot
# dnf install rdist # Removed. Use rsync instead
dnf install pigz
dnf install sendmail-cf
dnf install gcc-gfortran
Installed support for HTTPS access:
dnf install mod_ssl
Deleted ~/bin/nedit. The OpenMotif libraries it requires are no longer
supported by standard AWS, and the pain in building them from scratch
isn't worth it to run this vintage editor.
Installed Perl modules accessible with dnf:
dnf install "perl(LWP)" # Already installed
dnf install "perl(GD)" # Already installed
dnf install "perl(Digest::SHA1)" # Already installed
dnf install "perl(Time::Local)" # Already installed
dnf install "perl(XML::LibXML)"
dnf install "perl(Sys::Syslog)" # Already installed
dnf install "perl(CGI)" # Already installed
dnf install "perl(Getopt::Long)" # Already installed
dnf install "perl(Encode)" # Already installed
dnf install "perl(Digest::MD5)" # Already installed
Installed Perl modules accessible only through CPAN:
perl -MCPAN -e "shell"
install Crypt::OpenSSL::AES
install Crypt::CBC
install Crypt::SSLeay # Not clear we need this any more
# due to changes in LWP.
At this point, I believe we have all of the dependencies
installed.
Copied over the existing:
/etc/httpd
conf/httpd.conf
conf.d/
fourmilab_0.conf
fourmilab_Aliases.conf
fourmilab_Hosting.conf
ssl.conf
renaming pre-existing files _ORIGINAL. Restarted HTTPD.
HTTPD crashed because the certificate files cited in ssl.conf
aren't present. We have something of a chicken and egg problem
because we haven't yet installed Let's Encrypt, which we can't
do until we're running under the domain name for which we're
issuing the certificate.
The installation procedure for Certbot has changed substantially:
https://certbot.eff.org/instructions?ws=apache&os=pip
We start with:
dnf install python3 # Already installed
dnf install augeas-libs
Install Python virtual environment:
python3 -m venv /opt/certbot/
/opt/certbot/bin/pip install --upgrade pip
Install Certbot in virtual environment:
/opt/certbot/bin/pip install certbot certbot-apache
Link certbot in virtual environment to /usr/bin:
ln -s /opt/certbot/bin/certbot /usr/bin/certbot
Run certbot in certificate-only mode:
certbot certonly --apache
And now the chicken starts pecking us from within the egg:
Error while running apachectl configtest.
AH00526: Syntax error on line 129 of /etc/httpd/conf.d/ssl.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/fourmilab.ch/fullchain.pem' does not exist or is empty
Error while running systemctl restart httpd.
Job for httpd.service failed because the control process exited with error code.
See "systemctl status httpd.service" and "journalctl -xeu httpd.service" for details.
The apache plugin is not working; there may be problems with your existing configuration.
Apache won't start because the letsencrypt certificate directory is
missing, and certbot won't run because it can't restart Apache.
To try to Band-Aid this to permit further testing, I just bodily copied
over the entire /etc/letsencrypt directory from the production server
and installed it here.
cp -pRv /o/etc/letsencrypt /etc
Now I can:
systemctl start httpd
and it starts successfully, reporting:
Server configured, listening on: port 443, port 80
I can still access static site content via:
http://aws2/
Attempting:
https://aws2/
puts up a warning:
NET::ERR_CERT_COMMON_NAME_INVALID
which is correct since we're not running within the domain for which
the certificate was issued. If I click through the warning, I can
then get to the site.
We still require a symbolic link from the cgi-bin
directory in /var/www. I added:
super
cd /var/www
mv cgi-bin cgi-bin.ORIG
ln -s /server/bin/httpd/cgi-bin cgi-bin
and now Earth and Moon Viewer works. These symbolic links
are really tacky, and were only intended as a stopgap when
we were bringing up the site. We should modify the Apache
configuration files to point directly to the directories
in /server, eliminating the need for them. But I'll defer
that until we're running stably on the new platform.
After copying over the /etc/letsencrypt directory tree, I can run:
super
certbot renew
and it runs normally and reports:
The following certificates are not due for renewal yet:
/etc/letsencrypt/live/fourmilab.ch/fullchain.pem expires on 2024-01-13 (skipped)
No renewals were attempted.
It appears everything is properly configured for Certbot.
Festival (text to speech) failed because it was linked to an obsolete
shared library. I rebuilt it with:
cd /server/src/festival-2.4
cd speech-tools
./configure
make
cd ../festival
./configure
make
The CGI scripts run these programs directly from the bin directories
within this source build tree. After the rebuild, the:
https://aws2/cgi-bin/SayJulian
https://aws2/cgi-bin/SayTime
utilities worked.
For SayQuote, installed:
super
perl -MCPAN -eshell
install Finance::YahooQuote
Now SayQuote works mechanically through Festival, but doesn't include
the actual quotes because of another gratuitous change by Yahoo. But
this fails the same way on the production server, so it's not a
migration issue.
Tested cursorily to verify there aren't problems which
require recompiling CGI binaries or Perl library problems.
Earth and Moon Viewer
Solar System Live
Your Sky (Map, Horizon, and Telescope)
Bombcalc
Installed:
dnf install cronie
This is the user-side crontab tools, which used to be installed
out of the box on Amazon Linux 2.
Installed my crontab.
Installed root crontab.
Trying to start local servers, bacula and waisserver failed due to
referencing obsolete shared libraries.
Rebuilt waisserver:
cd /server/src/wais/freeWAIS-sf-2.2.14_Fourmilab_a
./BuildFourmilab
The build fails with dozens of build errors.
/usr/bin/ld: ../lib/libwais.a(stoplist.o):(.bss+0x28): multiple definition of `use_both_stoplist'; waisindex.o:(.bss+0x10): first defined here
/usr/bin/ld: ../lib/libwais.a(irhash.o):(.bss+0x18): multiple definition of `use_both_stoplist'; waisindex.o:(.bss+0x10): first defined here
Patched:
/server/src/wais/freeWAIS-sf-2.2.14_Fourmilab_a/freeWAIS-sf-2.2.14/lib/ir/stoplist.h
to declare: extern boolean use_both_stoplist;
and added to:
/server/src/wais/freeWAIS-sf-2.2.14_Fourmilab_a/freeWAIS-sf-2.2.14/lib/ir/stoplist.c
declaration of: boolean use_both_stoplist = true;
Now all of the modules built successfully. Tried:
/server/init/wais start
and it says the server started successfully.
Test of WAIS search in the Internal Revenue Code database failed with:
/var/www/cgi-bin/TaxSearch: dump() must be written as CORE::dump()
as of Perl 5.30 at /var/www/cgi-bin/TaxSearch line 2221.,
referer: https://aws2/ustax/TaxSearch.html
Fixed that and now it dies with:
Can't locate getopts.pl in @INC at /var/www/cgi-bin/TaxSearch line 2003
Well, it turns out that starting with Perl 5.26, the current directory
is no longer on the module include search path. I added a statement to
the start of /server/bin/httpd/cgi-bin/TaxSearch:
use lib '/server/bin/httpd/cgi-bin';
and now the thing can find its getopts.pl and the search works.
Of course, the same two fixes have to be made to ~/cgi/T8Search as
well. After the patches, it also works.
The same Perl library search was what torpedoed ~/cgi/HackDiet below
the water line. I added a statement:
use lib "/server/bin/httpd/cgi-bin";
before the first reference to an HDiet:: module and it's now working.
Attempted to rebuild the Bacula file daemon to fix the library
incompatibility in the binary we copied from the production system:
cd /server/src/bacula/bacula-5.2.10
./BuildFourmilabClient
This collapsed in a hideous efflorescence of error messages compiling
the file src/lib/crypto.c, with G++'s trademark incomprehensibility
such as:
error: expected constructor, destructor, or type conversion before 'IMPLEMENT_STACK_OF'
error: field 'ctx' has incomplete type 'EVP_MD_CTX' {aka 'evp_md_ctx_st'}
Now, this was very puzzling, since a few days ago I had built a Bacula
file daemon for the Scanalyst server from precisely the same source code
with no problems at all. I eventually resorted to saving transcripts of
the build on both the Scanalyst and aws2 sites and diffing them, and
discovered the clue:
195,196c201
< checking for OpenSSL... yes
< checking for EVP_PKEY_encrypt_old in -lcrypto... yes
---
> checking for OpenSSL... no
So, on aws2 we have OpenSSL installed, since it's required to build
several Perl modules we use for various things, but on the Scanalyst
site, where most of the functionality is embedded in the Docker image
of Discourse, we had no reason to install it. The mere presence of
OpenSSL on the system, however, was enough to cause Bacula to try to
compile features in the File Daemon to support encrypting the
transmission of backups, and the code to handle this wasn't up to date
with the incessant changes in the OpenSSL programming interface in the
interest of purity of essence and other goals which transcend mere
compatibility, stability, and protection of users' investment. The fix
was to add the option:
--without-openssl
to the ./configure run in:
/server/src/bacula/bacula-5.2.10/BuildFourmilabClient
which excludes OpenSSL from the build even if happens to be installed
on the system performing the build. With this specified, it worked
just fine, installed, and now I'm able to start the Bacula file
daemon with no problems.
HotBits requests work, so the HotBits proxy server doesn't
need to be rebuilt. But, just to prevent library incompatibilities in
the future and be sure we can, I rebuilt it anyway with no problems.
Started local servers with:
/server/init/servers start
They all appear to be running.
Rebuilt Webalizer, after verifying this was the current version, with:
cd /server/src/webalizer/webalizer-2.23-08
./BuildFourmilab
./InstallFourmilab
and now it runs. Note that this will have to be re-done on the
production /server once we cut over. I'm doing it now to be sure we
have everything installed we need to rebuild it. Deleted the
now-obsolete webaizer/webalizer-2.21-02 from /server/src and
server/bin.
Ran a webalizer job for the main site and a hosted site to
confirm they work.
/server/bin/webalizer/current/DailyUpdate
/server/pub/hosting/fondation.lignieres.org/statistics/UpdateStats
Both jobs ran OK.
Rebuilt units:
cd /server/src/units/units-2.19
./BuildFourmilab
to fix a shared library incompatibility.
Rebuilt Your Sky:
cd /server/src/yoursky/yoursky-2.6
./BuildFourmilab
to verify library compatibility and buy time until the next torpedo.
Rebuilt UnCGI:
cd /server/src/uncgi/uncgi-1.11
./BuildFourmilab
to verify library compatibility and buy time until the next torpedo.
Rebuilt Terranova:
cd /server/src/terranova/terranova-2.1
./BuildFourmilab
Tested with:
/server/cron/TerraNova
Terranova worked, but the job failed with:
pnmtopng: error while loading shared libraries: libpng15.so.15: cannot open shared object file: No such file or directory
...shared library Hell again. It turns out this failed because of a
shared library problem in the local copy of Netpbm we maintain due to
submerged magnetic mines in earlier distribution versions of NetPBM.
Let's rebuild *that*:
cd /server/src/netpbm/netpbm-10.73.20
./BuildFourmilab
# Note that you'll have to answer a lot of questions here. Most
# can accept defaults, but you should specify static linking to
# avoid library path Hell when these are run from CGI tasks or
# cron jobs, and you should install in the corresponding bin
# directory, in this case /server/bin/netpbm/netpbm-10.73.20.
And now, the Terranova cron job runs to completion and the output is
correct.
Deleted the obsolete:
/server/src/netpbm-10.35.97
/server/bin/netpbm-10.35.97
directories.
Installed the system:
dnf install netpbm-progs
The "netpbm" package installs only libraries, not the executables
in /usr/bin. This also installs Ghostscript.
In the quest for library compatibility, attempted to rebuild:
cd /server/src/barcode
./BuildFourmilab
which crashed due to the latest GNU fad in anti-compatibility munitions,
banning multiple C exports of the same name being treated like Fortran
COMMON BLOCKs and being mapped to the same memory location. This has
been a feature of C since the early 1970s, but now it has been
proscribed by the high priests of sacred wildebeest, and all must now
comply with their fat-headed fatwas. So, I patched:
/server/src/barcode/barcode-0.99
barcode.h Declare "streaming" as extern
library.c Declare exported "int streaming"
and, Bob's your (creepy) uncle, it builds and works. I verified that:
http://aws2/cgi-bin/ISBNquest?isbn=9780804139298&delim=-&assoc=fourmilabwwwfour&asite=www.amazon.com
can generate a bar code for the book it's investigating.
Changed permissions:
chown kelvin:wheel /server/bin/httpd/cgi-bin
Something (probably tar extracts) keeps setting back to root:root, which
is intensely irritating.
Rebuilt Earth and Moon Viewer:
cd /server/src/earthview/earthview-3.0
./BuildFourmilab
This was in the interest of library compatibility.
Rebuilt Solar System Live:
cd /server/src/solar/solar-2.4
./BuildFourmilab
It works.
Deleted the following directories containing obsolete versions of
server components imported from the production server.
/server/src/earthview
earthview-2.8
earthview-2.7
earthview-2.6
earthview-2.5
earthview-2.4
/server/src/hotbits
hotbits-3.11
hotbits-3.9
hotbits-3.8
hotbits-3.7
hotbits-3.6
hotbits-3.5
hotbits-3.4
hotbits-3.3
/server/bin/netpbm
netpbm-10.35.97
/server/src/solar
solar-2.3
I shall defer the next assault on the citadel of MariaDB until "no earlier than" to-morrow.