Booko V3, the refresh

Booko’s last big visual refresh was way back in 2014-ish. Some of the code used to generate that look has been slowly approaching end of life, and nothing motivates me like the impending doom of actual end of life. CSS and Javascript have come a long way in the years since that refresh and one of the brightest stars to emerge is Tailwind CSS. If you’re involved or interested in learning frontend programming, I can’t recommend it enough. It’s been as transformative to me personally as Ruby on Rails was way back when. Another new technology to arrive in the last few years has been a Javascript framework called StimulusJS – built by the same guys who write Ruby on Rails.

Overhauled Profile page. Lists are paginated by Name.

Booko v2 was mostly a frontend update – moving from the Programmer-forced-to-do-design aesthetic to a modern web designer look. It was a great transformation and I think it has stood the test of time nicely.

Access the new profile page easily from the new user menu

The original plan was to just attempt a one-to-one migration from Booko v2 to v3 – just remove the impending doom of end-of-life. That seemed like enough to bite off while learning the vagaries of CSS. However, once I started to get the hang of TailwindCSS & StimulusJS, I had a shot at fixing some of the more egregious UX problems, as I see them.

I’m particularly pleased with the new system for adding & removing products to lists. Once the dropdown is open, just select which lists the book should appear on. You’ll also see an Alert icon. Clicking on this will let you set alerts for that product. Products with a red alert icon have an active alert.

Much improved list management. Quickly & Easily add books to any list.

You’ll find that new system on your list page and on the product page – you can now easily see if a product is already on a list.

Access the new list management from product pages

One feature which has been dropped is lists for non-logged in users. The code behind this feature was some of the first Rails code I wrote back in the late noughties. Certainly not some of my best work. However I did enjoy deleting it all.

New Login Page

I hope you like these changes. Feel free to reach out if something is broken or changed in a way you love or can’t stand. Cheers! ✌️

Setting Thin up behind Nginx & with Lets Encrypt certificates

Part of Booko’s infrastructure includes a Sinatra application served by Thin. This application takes CSV / XML / JSONL files containing product pricing data, loads it into PostgreSQL and presents it via an API.

Thin is perfectly capable of being internet facing, but I have a preference for reverse proxying it behind Nginx. There are plenty of ways to deploy an app like this, but here’s a very standard approach.

Create a Systemd Service

Running applications as daemons is straight forward. Systemd will ensure that it’s started up at boot time and will be restarted it on failure.

[Unit]
Description=CSV To API Service
After=syslog.target

[Service]
Type=simple
User=deploy

Environment=MM_ENV=production

WorkingDirectory=/var/www/c2a/
ExecStart=/home/deploy/.rbenv/bin/rbenv exec bundle exec thin start -e production -p 1234 -a 127.0.0.1

Restart=on-failure
SyslogIdentifier=c2a

[Install]
WantedBy=multi-user.target

This will start up your app at boot, and restart it if it fails. The app will log to syslog and will be tagged as c2a in the syslog file. I’ve set an environment variable MM_ENV in this script, and the app will run as the user deploy.

Note that the app listens on 127.0.0.1 and will not be accessible on the internet.

Create an Nginx server

Next up, we’ll create an Nginx host. First, we’ll create a host which responds to the correct name but listens only on port 80. This is needed for Lets Encrypt to verify we’re in control of the domain name.

server {
   listen 80;
   server_name c2a.booko.info;
   root /var/www/c2a;

   location '/.well-known/acme-challenge' {
     default_type "text/plain";
   }

   location / {
     return 301 https://$host$request_uri;
   } 
 
   access_log  logs/c2a.log combined;
   error_log   logs/c2a.error;
 }

This will respond to the specify requests that Lets Encryption will make looking for the challenge file. Any other request will be redirected to https and will currently fail.

Now to get an SSL Certificate from Lets Encrypt.

sudo certbot certonly -d c2a.booko.info --webroot

When asked for the webroot, we enter it as in the Nginx configuration : /var/www/c2a

Certbot will store your certificate in /etc/letsencrypt/live/<domain> and we can refer to them in the Nginx configuration. Next, we can add the HTTPS server configuration:

server {
   listen 443 ssl http2;

   server_name c2a.booko.info;
 
   ssl                  on;
   ssl_certificate      /etc/letsencrypt/live/c2a.booko.info/fullchain.pem;
   ssl_certificate_key  /etc/letsencrypt/live/c2a.booko.info/privkey.pem; 

   location / {
     auth_basic "Booko's CSV to API service";
     auth_basic_user_file /var/www/c2a/etc/c2a_auth;
     proxy_pass http://127.0.0.1:8081;
   }
 
   access_log  logs/c2a.log combined;
   error_log   logs/c2a.error;
 }

You can see I’ve added one extra component here – basic authentication. Basic Auth for a single client via HTTPS is straight forward way of restrict access. You can generate the auth file with htpasswd -c <file> <user> and you’ll be prompted to enter a password.

And that’s it, we’re done!

Update: Webpacker v5, -= Coffeescript

Today I’ve updated webpacker to v5 and have started to remove Coffeescript. To remove Coffeescript, remove it from your Gemfile, then find app -iname '*.coffee' and convert each file back into Javascript. You can plonk the coffeescript into the Try CoffeeScript web app. Once you’ve done and are moving the changes to production, it’s important to clear out your rails cache with rake tmp:cache:clear

If you’re using Uglifer to compress Javascript and you’re using ES6, you’ll need to tell Uglifier to use “Harmony” mode:

config.assets.js_compressor = Uglifier.new(harmony: true)

Alternatively switch to Terser – you’ll need to set it up as a JS compressor:

Sprockets.register_compressor 'application/javascript', :terser, Terser::Compressor
Rails.application.configure do
  config.assets.js_compressor = :terser
end

Note To Past Self: Use Certbot

Free, ubiquitous SSL certificates as provided by Let’s Encrypt have helped make the Internet a safer place by ensuring your personal details, passwords, internet searches and even which URL on any site you visit are un-snoopable.

Booko’s used SSL for most of its time on the Internet. In the bad old days, it was a tedious process to apply for and receive SSL certificates often involving using OpenSSL on the command line. Not only do you need to remember to renew your certificates in time, you need to remember to renew them early enough that you can relearn the skills needed to renew your certificates.

Not long after Let’s Encrypt appeared, I wrote some Ruby scripts which provided some amount of automation using the http-01 challenge and later, once I’d moved domains over to DNSimple, DNS-01 challenge.

These were definitely a step up from OpenSSL, but I still needed to run them every 90 days and to remember which script used DNS and which used HTTP. Somewhat less tedious.

I’m not sure why I didn’t use Certbot earlier, but now I’ve bitten the bullet, I’ve automated all the things. The simplest approach for me, is the HTTP-01 challenge. Booko use Nginx for most of the services it needs and the easiest way to make it all work was to use the certonly approach with a specified webroot. In your Nginx port 80 server stanza, add a location section such as:

    location '/.well-known/acme-challenge' {
      default_type "text/plain";
      root        /var/www/lets;
    }

When Let’s Encrypt attempts to validate the file it provides, it hits that well-known path on your web server. Once that’s in place, run certbot as root:

certbot certonly -d booko.info -d www.booko.info -d booko.com.au -d www.booko.com.au

When asked, set your webroot value to the value specified above: /var/www/lets in this case.

If all goes smoothly, you’ll find your certificates and keys in /etc/letsencrypt/live/<your domain>/ directory. Update your SSL configuration to point to these files.

Running certbot renew does what it says – renews the certificates provided they’re within 30 day of expiring. The version of certbot I use also adds a cronjob to /etc/cron.d/. You can list all your cron jobs with this command: systemctl list-timers – that list should include a certbot renew job. Now delete that calendar entry reminding you to renew your certificates!

New Features

Lists are a great way of organising your books on Booko, but sometimes it’s hard to remember if you’ve already added a book to a list. Now, when you’re looking at a book and you’re wondering if it’s already on a list, click the “Add to a list” button and you’ll find your answer.

View a list of lists this book is already on

New Filters for price list. Have a shop you’re not a fan of? Now you can filter them out of Booko’s price table. Edit them at the filters page.

New Filters Management page on Booko

Some pages on Booko have a lot of covers art to display and often the page extends beyond the bottom of the window. To make pages load faster, it’s possible to tell the browser to not load images which aren’t yet visible on in the window. As you scroll down, the browser will load the images just before they appear – this is called lazy loading. I’ve started adding the “loading=lazy” attribute to suitable images so that images are loaded lazily. Safari doesn’t support this yet, but it is just ignored so it shouldn’t cause any issues. Firefox and Google’s Chrome do support it and they’ll only load cover art images if they’re visible in the windows. You’ll find it in use on the PreOrder and Most Clicked pages.

Bugs fixes

Fixed an awfully old bug related to the migration from Oracle’s MySQL to the excellent PostgreSQL. The bug stopped the creation of new lists, randomly. When you create a new list, Postgres will generate a new ID for the list and by default, the ID it chooses start from 1, then increment up from there. Postgres keeps track of these sequences and hands out the next largest value by default. When we migrated, we already have thousands and thousands of lists and Postgres should have been set to start the new sequence from the largest existing list ID. However it wasn’t. So sometimes new lists were assigned an existing ID, which caused the list creation to fail. The fix for this problem is simple:

SELECT setval('lists_id_seq', (SELECT MAX(id) FROM lists)+1);

This sets the value of the sequence to the maximum ID used for lists, plus one.

Yahoo! EOLs OpenID 2, migrates to OpenID Connect

Early in July, I got some bug report emails about Booko’s login via Yahoo! no longer working. A quick investigation confirmed it.

Yahoo!'s OpenID EOL message

Unfortunately, I missed that announcement and so migrating to Yahoo’s OIDC (OpenID Connect) was at the top of my todo list.

OIDC is an identity service which runs on top of OAuth 2.0. Yahoo’s migration document provides clear instructions on how to do this.

First up, let’s use the OAuth2 Ruby gem and get an OAuth client to use. In real code, you might pass in a ‘service’ argument, for say, Google or any other OIDC provider.

def yahoo_client
  client_id = Rails.application.credentials.yahoo_client_id
  client_secret = Rails.application.credentials.yahoo_client_secret

  site = 'https://api.login.yahoo.com'
  token_url = '/oauth2/get_token'
  authorize_url = '/oauth2/request_auth'
  state = session[:state] ||= SecureRandom.hex

  OAuth2::Client.new(
              client_id, client_secret,
              site: site,
              authorize_url: authorize_url, 
              token_url: token_url, 
              state: state)
  
end

In your controller, you’ll need an action to perform an OAuth login. As part of the redirect, you need to provide a URL that the user will be redirected back to.

def oauth_login
  client = get_yahoo_client
  scope = 'openid'
  response_type = 'code'

  yahoo_url = client.auth_code.authorize_url(
                   redirect_uri: 'https://booko.info/process_oauth',
                   scope: scope, 
                   nonce: session[:state])

  redirect_to yahoo_url, status: 303

When a user hits the “Login via Yahoo!” button on your site, they’ll need to hit this action. The action builds an OAuth client and then redirects the user over to Yahoo! to sign in and will ask if they want to authenticate to your site and maybe hand over their email address. Yahoo! will then send the user back to the redirect_url you passed into the OAuth client.

Web Server Updates

A perfectly miserable rainy day of stage 3 lock downs here in 3055. Perfect time for some boring updates.

  • Ubuntu from 16.0.4 -> 18.0.4
    • via do-release-upgrade
  • Nginx 1.16.1 -> 1.18.0
  • Passenger from 6.0.4 -> 6.0.5

Ruby required recompilation due to shared library updates.

export NV='2.6.6'; sudo apt install libjemalloc-dev libpng-dev libjpeg-dev; cd ~/.rbenv && git pull && cd plugins/ruby-build/ && git pull   && cd && RUBY_CONFIGURE_OPTS=--with-jemalloc rbenv install ${NV} ; rbenv shell ${NV} &&  gem install bundler; cd ~/booko; bundle install

Passenger upgrade fixed a compilation error when building Nginx 1.18.0.

sudo service nginx stop; passenger-install-nginx-module --auto --prefix=/opt/nginx --nginx-source-dir=/home/deploy/source/nginx-1.18.0 --languages ruby --extra-configure-flags="--with-http_realip_module  --with-http_v2_module"

Next time I build web servers I’ll be migrating to Phusion’s APT repository versions, rather than compiling them.