How To Use Basecamp’s Kamal With AWS and GitHub

What we’ve used before?

What is Kamal and how it works?

Kamal + Amazon AWS possible schemas

AWS offers a plethora of networking options to organize your application infrastructure. The image below illustrates one of the most commonly used schema that can be used.

Our domain is hosted on Route 53 in this network schema, pointing to an AWS ALB. All traffic is routed to the ALB, where SSL termination occurs. The HTTP traffic is directed to one or several EC2 instances managed with Kamal. Traefik on these instances routes the traffic to the Rails container, ensuring efficient request processing and optimal performance.

How to deploy your web application with Kamal?

• You can deploy any application with Kamal. It doesn’t matter what language or framework you use.
• The only requirement for Kamal to work is a Dockerfile.
• Kamal can be used to build and deploy from your local machine or from any CI/CD service.

Prepare application to run in a container

# This first and main rule that we follow in our Dockerfile is to keep it
# without hardcoded dependencies and versions as much as possible.
# This file is an ideal – it has zero.

# It accepts selected Ruby version as an argument.
ARG RUBY_VERSION

# Create new stage from ruby-slim image with selected Ruby version.
FROM ruby:$RUBY_VERSION-slim as base

# We are Rails developers, so we want to keep our app under /rails directory
WORKDIR /rails

# Here is some Ruby magic and configuration options
ENV BUNDLE_DEPLOYMENT="1" \
  BUNDLE_PATH="/usr/local/bundle" \
  BUNDLE_WITHOUT="development" \
  HOME=/rails

# Throw-away build stage to reduce size of final image
FROM base as build

# Install dependencies that are needed only during build stage
RUN apt-get update -qq && \
  apt-get install -y build-essential curl libpq-dev nodejs git

# Install JavaScript dependencies
ARG NODE_VERSION
ARG YARN_VERSION
ENV PATH=/usr/local/node/bin:$PATH
RUN curl -sL https://github.com/nodenv/node-build/archive/master.tar.gz | tar xz -C /tmp/ && \
    /tmp/node-build-master/bin/node-build "${NODE_VERSION}" /usr/local/node && \
    npm install -g yarn@$YARN_VERSION && \
    npm install -g mjml && \
    rm -rf /tmp/node-build-master

# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git

# Install node modules
COPY package.json yarn.lock ./
RUN yarn install --frozen-lockfile

# Copy application code
COPY . .

ARG RAILS_ENV

# If you are already on Rails 7.1+,
# then you will be able to use new SECRET_KEY_BASE_DUMMY=1 variable,
# however below is a workaround for Rails 7.0 and below.
RUN SECRET_KEY_BASE=1 DATABASE_URL=postgresql://dummy@localhost/dummy RAILS_ENV=$RAILS_ENV \
  bundle exec rake assets:precompile

# Start again from base image to throw away anything that is not needed
# from build stage.
FROM base

# Install dependencies that we needed during application run.
# We use PostgresSQL for almost all our projects,
# this is why we install `postgresql-client`.
# Also almost all web applications have image upload and convert functionality,
# so you need to install `imagemagick`.
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y libvips postgresql-client imagemagick tzdata curl && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Run and own the application files as a non-root user for security
RUN useradd rails --create-home --shell /bin/bash

# Copy built artifacts: node, gems, application
COPY --from=build --chown=rails:rails /usr/local/node /usr/local/node
COPY --from=build --chown=rails:rails /usr/local/bundle /usr/local/bundle
COPY --from=build --chown=rails:rails /rails /rails
RUN chown rails:rails /rails

USER rails:rails

# Entrypoint prepares the database.
ENTRYPOINT ["/rails/bin/docker-entrypoint"]

# Start the server by default, this can be overwritten at runtime
EXPOSE 3000
CMD ["./bin/rails", "server"]

#!/bin/bash -e

# If running the rails server then create or migrate existing database
if [ "${*}" == "./bin/rails server" ]; then
  ./bin/rails db:prepare
fi

exec "${@}"

# lib/middleware/assume_ssl.rb
#
class AssumeSSL
  def initialize(app)
    @app = app
  end

  def call(env)
    env['HTTPS'] = 'on'
    env['HTTP_X_FORWARDED_PORT'] = 443
    env['HTTP_X_FORWARDED_PROTO'] = 'https'
    env['rack.url_scheme'] = 'https'

    @app.call(env)
  end
end

# config/application.rb
#
Dir[Rails.root.join('lib/middleware/**/*.{rb}')].sort
  .each { |file| require file }

# config/environments/production.rb
#
Rails.application.config.middleware.insert_before(0, AssumeSSL)

Add Kamal to your application

# frozen_string_literal: true

source 'https://rubygems.org'

gem 'kamal', '~> 1.0.0'

BUNDLE_GEMFILE=kamal/Gemfile bundle install

BUNDLE_GEMFILE=kamal/Gemfile bundle binstub kamal --path ../bin

./bin/kamal init

# Name of your application. Used to uniquely configure containers.
service: onetribe

# Name of the container image.
image: onetribe

# Deploy to these servers.
servers:
  web:
    - "//web IP//"
  sidekiq:
    cmd: bin/sidekiq
    hosts:
      - "//sidekiq IP//"

# Use a different ssh user than root
ssh:
  user: www

# Credentials for your image host.
registry:
  # Specify the registry server, if you're not using Docker Hub
  server: AMAZON_ID.dkr.ecr.us-east-1.amazonaws.com
  username: AWS

  # Always use an access token rather than real password when possible.
  password:
    - KAMAL_REGISTRY_PASSWORD

# Container builder setup.
builder:
  args:
    RUBY_VERSION: 3.2.2
    RAILS_ENV: production
		NODE_VERSION: 18.12.0
		YARN_VERSION: 1.22.19
  multiarch: false
  cache:
    type: registry
    image: onetribe-build-cache
    options: mode=max,image-manifest=true,oci-mediatypes=true

# Container run setup
env:
  clear:
    RAILS_LOG_TO_STDOUT: 1
    RAILS_SERVE_STATIC_FILES: 1
    RAILS_ENV: production
  secret:
    - RAILS_MASTER_KEY
    - DATABASE_URL
    - REDIS_URL

# Configure a custom healthcheck (default is /up on port 3000)
healthcheck:
  path: /health

Prepare AWS EC2 instance(s) and AWS ALB

We will go with Ubuntu 22.04.

After the instance has been created don’t forget to allocate and associate the Elastic IP address with it. Put the newly obtained IP address to your Kamal config into servers section. Once the instance is up and running, connect to it with the default user (usually ubuntu) and install Docker to run our containers. You can read more about this in Docker official manual. Long story short, you should install docker-ce and docker-ce-cli.

ubuntu@ip-172-31-86-216:~$ sudo useradd -d /srv/www -m -s /bin/bash --groups docker www

www@ip-172-31-86-216:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES
www@ip-172-31-86-216:~$

The instance is ready. Time to configure ALB. Create new Application Load Balancer and configure two listeners on it. I find it easier to create a balancer with just one listener and configure everything when balancer already exists.

The key point is that we redirect all HTTP/80 traffic to HTTPS with 301 response code. For HTTPS configuration is also very clear: we forward all HTTPS traffic that comes from onetribe.team and *.onetribe.team to kamal target group (that includes the previously created EC2 instance) through HTTP 80 port. This is where SSL termination happens.

That covers the basics of the AWS setup. If you need additional instances to handle Sidekiq or to serve web requests, you can simply duplicate the existing EC2 instance.

GitHub Actions scenario

name: Kamal

on:
  push:
    branches:
    - master

jobs:
  spec:
    uses: ./.github/workflows/specs.yml
  lint:
    uses: ./.github/workflows/lint_code.yml

  build_and_deploy:
    needs: [spec, lint]
    runs-on: ubuntu-latest
    timeout-minutes: 20
    outputs:
      image: ${{ steps.build.outputs.image }}
    env:
      RAILS_ENV: production
    steps:
      - uses: actions/checkout@v3
        with:
          ref: ${{ github.event.workflow_run.head_branch }}
      - uses: webfactory/ssh-agent@v0.7.0
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - uses: ruby/setup-ruby@v1
        env:
          BUNDLE_GEMFILE: ./kamal/Gemfile
        with:
          ruby-version: 3.2.2
          bundler-cache: true
      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2
        with:
          driver-opts: image=moby/buildkit:master
      - name: Configure AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id    : ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region           : us-east-1
          mask-aws-account-id  : 'no'
      - name: Login to Amazon ECR
        id  : login-ecr
        uses: aws-actions/amazon-ecr-login@v1
      - name: Kamal Envify
        id  : kamal-envify
        env :
          KAMAL_REGISTRY_PASSWORD: ${{ steps.login-ecr.outputs.docker_password_<YOUR_AWS_ACCOUNT_ID>_dkr_ecr_<YOUR_AWS_REGION>_amazonaws_com }}
          DATABASE_URL: ${{ secrets.DATABASE_URL }}
          REDIS_URL: ${{ secrets.REDIS_URL }}
          RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }}
          DOCKER_BUILDKIT: 1
          BUNDLE_GEMFILE: ./kamal/Gemfile
        run: |
          ./bin/kamal envify

      - name: Kamal Deploy
        id: kamal-deploy
        run: |
          ./bin/kamal deploy

- name: Set up Docker Buildx
  uses: docker/setup-buildx-action@v2
    with:
      driver-opts: image=moby/buildkit:master

- name: Kamal Envify
  id  : kamal-envify
  env :
    KAMAL_REGISTRY_PASSWORD: ${{ steps.login-ecr.outputs.docker_password_<YOUR_AWS_ACCOUNT_ID>_dkr_ecr_<YOUR_AWS_REGION>_amazonaws_com }}
    DATABASE_URL: ${{ secrets.DATABASE_URL }}
    REDIS_URL: ${{ secrets.REDIS_URL }}
    RAILS_MASTER_KEY: ${{ secrets.RAILS_MASTER_KEY }}
    DOCKER_BUILDKIT: 1
    BUNDLE_GEMFILE: ./kamal/Gemfile
  run: |
    ./bin/kamal envify

- name: Kamal Deploy
  id: kamal-deploy
  run: |
    ./bin/kamal deploy

After all, preparations have been done; we are ready to deploy our app. After pushing code to the master branch our Kamal workflow will start automatically.

As you can see, the entire process of Docker build and deploy was completed in 1 and half minutes. This is quite impressive because I’ve used a EC2 medium instance to run the application and GitHub standard runners to build it.

Additional tweaks and Traefik configuration

With STDOUT logging you can access your application container logs from your host machine with docker logs <container_id> --follow, you will probably see something like this:

An alternative approach for accessing application logs is by using the ./bin/kamal app logs command directly from your development machine. This method offers a more developer-friendly experience.

The ./bin/kamal app logs command provides convenient options to enhance log analysis. For instance, you can use the grep command to search for specific log entries, and filter logs by timestamp, hosts, or roles. To explore the available options, you can use the --help flag, which provides a comprehensive list of command options and their functionalities.

It's essential to consider that we are utilizing Docker, which provides a variety of logging drivers, including integration with AWS CloudWatch.

logging:
  driver: awslogs
  options:
    awslogs-region: us-east-1
    awslogs-group: /application
    awslogs-create-group: true
    tag: "{{.Name}}-{{.ID}}"

To enable awslogs on your host machine, you must provide AWS credentials. This can be achieved by installing awscli and configuring it from the root user.

root@app:~# apt install awscli
root@app:~# aws configure

traefik:
  args:
    log: true
    log.level: ERROR
    accesslog: true
    accesslog.format: json
    accesslog.filters.minduration: 10ms

Conclusions