scripts

Organizational and maintenance scripts for Find Work repositories

This repository is heavily based on twolfson/twolfson.com-scripts.

Background

We maintain all of our service in a single VM via Vagrant. We use vagrant-lxc to get container performance while maintaining the ease of using a VM.

We choose to use Vagrant over Docker for a few reasons:

• Vagrant shares more common knowledge with a production server (i.e. use ssh)
• We are more comfortable with VMs and don't want to drain time debugging Docker
  • This includes edge cases like architectural structure (e.g. placing PostgreSQl in same or separate containers, minimizing downtime on container restart)

Getting Started

To get our server running, perform the following steps:

• Install Vagrant via its download page
  • https://www.vagrantup.com/downloads.html
• Run the following shell commands

# Clone our repository
git clone git@github.com:findworkco/scripts.git scripts
cd scripts

# Install our plugin dependencies
# https://github.com/fgrehm/vagrant-lxc/tree/v1.2.1#requirements
sudo apt-get install lxc redir

# Install our Vagrant plugin
vagrant plugin install vagrant-lxc

# Set up `sudoers` bindings for `vagrant-lxc` (avoids `sudo` for `vagrant up`/`vagrant ssh`/etc)
# https://github.com/fgrehm/vagrant-lxc/blob/v1.2.1/lib/vagrant-lxc/command/sudoers.rb
vagrant lxc sudoers

# Decrypt our configuration
CONFIG_COPY_ONLY=TRUE bin/decrypt-config.sh
# To decrypt our secrets (e.g. production db password), use
#   CONFIG_COPY_ONLY=FALSE bin/decrypt-config.sh

# Start our Vagrant instance
vagrant up

# SSH into the machine
vagrant ssh

Documentation

Ports

We manage our port reservations via docs/ports.md.

If you are registering a new service, please update the document via a pull request.

File structure

This repository has the following file structure:

• .bundle/ - Configuration for Bundler (used for managing Ruby gems)
• bin/ - Contains our executable files (e.g. deployment scripts)
• data/ - Contains static files used during provisioning
  • This starts at / as if it were the root of a file system
  • For multiple environment projects, it's good to have a data/{{env}} for each setup (e.g. data/development, data/production)
• src/ - Container for our bootstrapping scripts
• test/ - Container for our test files
• README.md - Documentation for this repository
• Vagrantfile - Configuration for Vagrant

Provisioning a new server

To provision a new server via Digital Ocean, follow the steps below:

• If we don't have a Digital Ocean SSH key pair yet, then generate one
  • https://help.github.com/articles/generating-ssh-keys/
• Create a new Ubuntu based droplet with our SSH key (14.04 x64)
• Add public key to data/home/ubuntu/.ssh/authorized_keys so we can ssh into the ubuntu user
  • Digital Ocean's SSH key will initially be registered to root user but we dislike having direct SSH access into a root user
• Once droplet has started, set up our ~/.ssh/config on the local machine

# Replace `digital-my-server` with a better name
# Replace 127.0.0.1 with droplet's public IP
Host digital-my-server
    User root
    HostName 127.0.0.1

• Install our SSL certificates

  # Install certbot for NGINX
  # DEV: We run all of this by hand as this only works for 1 server so far. We'll need to solve multiple server soon
  # Taken from: https://www.digitalocean.com/community/tutorials/how-to-secure-nginx-with-let-s-encrypt-on-ubuntu-16-04
  # Verified in: https://certbot.eff.org/lets-encrypt/ubuntutyakkety-nginx
  sudo add-apt-repository ppa:certbot/certbot
  sudo apt-get update
  sudo apt-get install python-certbot-nginx
  
  # Obtain our certificates for NGINX and explicitly our domain
  sudo certbot --nginx -d findwork.co -d www.findwork.co
  # Email address: todd@findwork.co
  # Terms of Service: Agree
  # Share email with EFF: Yes
  # No redirect/redirect: No redirect (already heavily configured)
  # Files are stored at /etc/letsencrypt/live/findwork.co/fullchain.pem
  
  # Outside of SSH, verify the certificate looks good
  # https://www.ssllabs.com/ssltest/analyze.html?d=findwork.co
  
  # Verify renewal works as expected
  sudo certbot renew --dry-run

• Install our Diffie-Hellman group to the server
  • bin/install-nginx-data-remote.sh digital-my-server --dhparam path/to/dhparam.pem
  • If you are trying to get a replica working (e.g. don't have these certificates), then self-signed certificates and a dhparam.pem can be generated via the openssl commands in bin/bootstrap-vagrant.sh
• Install our PGP private key to the server
  • bin/install-pgp-data-remote.sh digital-my-server --secret-key path/to/private.rsa
  • If you don't have the private.rsa file on hand, it can be dumped via
    • Find full fingerprint of key we want to export
      • gpg --fingerprint
      • Fingerprint will be 740D DBFA... in Key fingerprint = 740D DBFA...
    • Extract private key to file
      • gpg --export-secret-keys --armor {{fingerprint}} > private.rsa
      • --armor exports a human-friendly ASCII format instead of binary
  • If you are trying to get a replica working (e.g. don't have these certificates), then a key can be generated via these instructions
    • https://gist.github.com/twolfson/01d515258eef8bdbda4f#setting-up-sops-with-pgp
• Install our server configuration
  • bin/install-config-data-remote.sh digital-my-server
  • If you are trying to get a replica working (e.g. don't have these certificates), then use CONFIG_COPY_ONLY=TRUE to prevent SOPS errors
• Bootstrap our server
  • bin/bootstrap-remote.sh digital-my-server
• Update ~/.ssh/config to use User ubuntu instead of User root
  • During the bootstrap process, we intentionally lock our root access via ssh for security
• Run our tests on the server
  • bin/test-remote.sh digital-my-server
• Add swap space to the server (necessary for memory-intensive installs)
  • https://www.digitalocean.com/community/tutorials/how-to-add-swap-on-ubuntu-14-04
  • On a 1GB Digial Ocean instance, we use:
    • sudo dd if=/dev/zero of=/swapfile bs=512M count=4
    • sudo chmod 600 /swapfile
    • sudo mkswap /swapfile
    • sudo nano /etc/fstab
    • sudo sysctl vm.swappiness=10
    • sudo nano /etc/sysctl.conf
    • sudo sysctl vm.vfs_cache_pressure=50
    • sudo nano /etc/sysctl.conf
• Install Librato via https://metrics.librato.com/integrations
  • Instead of using curl | sudo bash. Download the script via wget, verify it looks good via less, and run it via sudo ./{{script-name}}

Updating a server configuration

We reuse our provisioning script for managing server state. As a result, we can reuse it for updates:

bin/bootstrap-remote.sh digital-my-server

# If we need to use a non-master ref, then pass it as a second parameter
# bin/bootstrap-remote.com.sh digital-my-server dev/new.feature

Deploying a service

To deploy a service, use its respective bin/deploy-*.sh script. Here's an example with findworkco/app:

bin/deploy-app.sh digital-my-server

# If we need to deploy a non-master ref, then pass as a second parameter
# bin/deploy-app.sh digital-my-server dev/new.feature

Editing secrets

We maintain a set of secrets (e.g. passwords) for provisioning in production in config/static-secrets.enc.yml. To edit these files locally, perform the following steps:

• Install SOPS' dependencies as specified by https://github.com/mozilla/sops/tree/0494bc41911bc6e050ddd8a5da2bbb071a79a5b7#up-and-running-in-60-seconds
• Install our consistent patched SOPS version
  • pip install --upgrade sops==1.15
• Ask a coworker for the findworkco/scripts PGP private key
  • We assume you will receive it as private.rsa
  • For coworkers, see the Provisioning a new server section for dump commands
• Install the findworkco/scripts PGP private key to GPG
  • gpg --import private.rsa
• Edit the SOPS file
  • bin/edit-secrets.sh

If you would like to learn more about PGP and SOPS, @twolfson has prepared this document:

https://gist.github.com/twolfson/01d515258eef8bdbda4f

Security

We try to keep our services as secure as possible via the following means:

• Restricting shell access for SSH users
• Preventing password only authentication for SSH
• Restricting permissions on sensitive files (e.g. SSL certificates, NGINX configurations)

Patched major CVE's

We have gone out of our way to patch the following CVE's:

• Shellshock - Patched by upgrading bash (default is fine on Ubuntu 14.04)
  • http://krebsonsecurity.com/2014/09/shellshock-bug-spells-trouble-for-web-security/
• Heartbleed - Patched by upgrading NGINX (default is fine on Ubuntu 14.04)
  • http://heartbleed.com/
• POODLE - Patched by restricting SSL methods used by NGINX
  • https://access.redhat.com/articles/1232123
  • https://cipherli.st/
• Logjam - Patched by using new Diffie-Hellman group
  • https://weakdh.org/sysadmin.html

Testing

We use [Serverspec][] for testing local and remote servers. This is a Ruby gem so you will need it installed to run our tests:

# Install bundler to manage gems for local directory
gem install bundler

# Install dependencies for this repo
bundle install

# Run our tests
./test.sh

To make iterating on our test suite faster, we have set up SKIP_LINT and SKIP_PROVISION environment variables. This skips running linting and vagrant provision in our tests:

# Skip both linting and provisioning
SKIP_LINT=TRUE SKIP_PROVISION=TRUE ./test.sh

Validating production

To run our test suite against a production machine, we can use the bin/test-remote.sh script.

bin/test-remote.sh digital-my-server

Copyright

All rights reserved, Shoulders of Titans LLC