morph.io: A scraping platform

• A Heroku lookalike system for Scrapers
• All code and collaboration through GitHub
• Write your scrapers in Ruby, Python, PHP, Perl or JavaScript (NodeJS, PhantomJS)
• Simple API to grab data
• Schedule scrapers or run manually
• Process isolation via Docker
• Email alerts for broken scrapers

Dependencies

Ruby, Docker, MySQL, SQLite 3, Redis, mitmproxy. (See below for more details about installing Docker)

Provisioning using Ansible

An ansible playbook is provided to provision:

• staging on a local Vagrant VM
• production

Read the provisioning README for further details.

Requirements

• A supported version of Ubuntu LTS, macOS/X or MS Windows is required.
• 8 GB of memory is the minimum, 16 GB is recommended
• SSD Disk

Coding Standards

• Must pass RuboCop (as configured in .rubocop.yml) to encourage good practices
• Must pass RSpec tests – please write specs for any new code
• Be nice to other developers and future you:
  • Clear, descriptive naming - code should tell you what it's doing
  • Add comments to explain WHY – when it's not obvious from the code itself
  • Keep methods small – ideally one paragraph, definitely under one screen page
  • Split large files – keep files <= 400 lines for readability and maintainability
    • See spec/models/scraper_spec.rb as an example split file
    • Consider splitting off concerns and domain logic into separate files when other files get too fat

To run style and coding checks

bundle exec rubocop

To check for security updates

Either check Dependabot alerts for the main branch, taking note of when the last check was run, or run manually on current branch:

bundle exec ruby-audit
bundle exec bundle-audit

Support for Developers / CI

Docker compose is used to provide redis, elasticsearch and mysql services as required for dev and CI (use SERVICES to specify which services to start if you don't want them all). You can customise the docker compose setup by copying docker-compose.override.yml-example to docker-compose.override.yml.

vagrant is used to provide a local staging environment to test ansible provisioning and capistrano app deployment.

If you don't want to set up ruby on your local host and/or have a different enough docker / mysql / redis version then cd /vagrant within a vagrant ssh session to work on the files mapped in from the project root.

Installing Docker

Install either a supported version of Docker Engine for Ubuntu Linux or Docker Desktop for macOS/X or MS Windows which includes Docker Engine.

On Linux, Your user account should be able to manipulate Docker (just add your user to the docker group).

Developing on macOS (Apple silicon)

Native development on the macOS host is not supported: Gemfile.lock only locks Linux platforms, and sorbet-static has no build for current macOS versions (see ADR 0006). Instead, develop inside the containerised dev environment, which runs the Ruby containers as linux/amd64 under Rosetta.

1. Install Docker Desktop and switch on "Use Rosetta for x86/amd64 emulation on Apple Silicon" in its settings.

2. Do the initial configuration below (config/database.yml, .env).

3. Open the repository in the devcontainer, either with "Reopen in Container" in an editor that supports devcontainers, or from a terminal with the devcontainer CLI:

   make devcontainer-up
   make devcontainer-shell
   

This starts the web, worker and faye containers plus MySQL, Redis and Elasticsearch with persistent volumes (the same stack as make docker-up), with the Rails server on http://localhost:3000.

Your local SSH agent is forwarded into the container, so Capistrano deployments run from a devcontainer shell without copying keys into the container:

bundle exec cap staging deploy

To run scrapers locally (including the :docker specs), pull the scraper base images and create the scraper network first, from the host:

make dev-scraper-images
make dev-scraper-network

Because everything runs under emulation, expect containers to be noticeably slower than on amd64 hardware.

Installing Vagrant

Install VirtualBox or other supported virtualization provider.

Then install Vagrant

Make targets

Various make targets have been added for developer convenience when developing on the local host. To see the current list of targets with descriptions, run

make help

Morph needs various services to run. We've made things easier for development by using docker to run Elasticsearch and the other services.

make services-up

To stop the services use

make services-down

To get a bash shell in the running web container if you are using the full docker environment:

docker compose exec web bash -i

To run commands in a temporary container rather than the currently running container, use instead

docker compose run web --rm -it bash -i

Initial configuration of morph development environment

cp config/database.yml.example config/database.yml
cp env-example .env
cp env-staging-example .env.staging # if needed
cp env-staging-example .env.vagrant # if needed

Edit

• config/database.yml with your database settings
• .env with your local environment and vagrant development settings
• .env.staging with staging environment settings that differ from .env
• .env.vagrant with settings for provisioning vagrant that differ from .env

Install gem requirements by running the following in the web container:

bundle install

Repositories

User-facing:

• openaustralia/morph - Main application
• openaustralia/morph-cli - Command-line morph.io tool
• openaustralia/scraperwiki-python - Fork of scraperwiki/scraperwiki-python updated to use morph.io naming conventions
• openaustralia/scraperwiki-ruby - Fork of scraperwiki/scraperwiki-ruby updated to use morph.io naming conventions

Docker images:

• openaustralia/buildstep - Base image for running scrapers in containers

Note - morph builds a docker image using these buildstep images combined with the config files from the scraper to build a separate docker image for each scraper with all the dependencies ready to go.

Tunnel GitHub webhook traffic back to your local development machine

We use "ngrok" a tool that makes tunnelling internet traffic to a local development machine easy. First download ngrok if you don't have it already. Then,

make share-web
# runs: ngrok http 3000

Make note of the ngrok forwarding url (*.ngrok-free.dev).

Creating Github Application

You'll need to create an application on GitHub So that morph.io can talk to GitHub. We've pre-filled most of the important fields for a few different configurations below:

• Create GitHub application on your personal account for use in development, port 3000
• Create GitHub application on your personal account for use in production
• On the OpenAustralia foundation, under Settings / Developer settings / GitHub Apps you should see the application already created using:
  • Create GitHub application on the openaustralia organization for use in production

You will need to add and change a few values manually:

• Disable "Expire user authorization tokens"
• Select "Any Account" if you are demoing with a team
• Add extra callback urls:
  • http://0.0.0.0:3000/users/auth/github/callback # if you click on the url puma lists on start up
  • /users/auth/github/callback
  • Change the port for the local urls if you are not using the default port 3000 for the rails app
• Add an image - you can use the standard logo at app/assets/images/logo.png (you can add this after the app is created)
• If the webhooks are active and being used in production (currently not the case) then you'll also need to:
  • add a "Webhook secret" for security.
  • add a "Webhook URL" - the ngrok url with /github/webhook on the end

For staging servers, you will need to add the callback url.

• User settings switch context to Open Australia (if you have sufficient permissions), otherwise you will need to use an app under your personal or an organisation you have permission for.
• Navigate to Developer Settings
• Under GitHub Apps you should have a Morph.io app
• Click Edit and scroll down to the "Identifying and authorizing users" section
• Click "Add Callback URL" if your url is not already present, eg:
  • https://morph-staging.thesite.info/users/auth/github/callback

Next you'll need to fill in some values in the .env file which come from the GitHub App that you've just created.

• GITHUB_APP_ID - Look for "App ID" near the top of the page. This should be an integer
• GITHUB_APP_NAME - Look for "Public link". The name is what appears after "https://github.com/settings/apps/". It's essentially a url happy version of the name you gave the app.
• GITHUB_APP_CLIENT_ID - Look for "Client ID" near the top of the page.
• GITHUB_APP_CLIENT_SECRET - Go to "Generate a new client secret".
• GITHUB_APP_INSTALLED_BY - A user that has installed the app (used by tests)

Also, a private key for the GitHub app is needed. This can be generated by clicking the "Generate a private key" button and will be automatically downloaded. Move and rename it to config/morph-github-app.private-key.pem.

For the staging server it will need to be copied to: deploy@morph-staging:/var/www/shared/config/morph-github-app.private-key.pem And it should have permission 0600 and be owned by deploy user. You may have to add write perms to this file first using chmod +w /var/www/shared/config/morph-github-app.private-key.pem, if the staging server is a clone of production as otherwise you won't be able to update it.

Now setup the databases:

bundle exec dotenv rake db:setup

Now you can start the server

bundle exec dotenv foreman start

and point your browser at http://127.0.0.1:3000

To get started, log in with GitHub. There is a simple admin interface accessible at http://127.0.0.1:3000/admin. To access this, run the following to give your account admin rights:

bundle exec rake app:promote_to_admin

Testing

See TESTING.md for automated and manual testing instructions.

Guard Livereload

We use Guard and Livereload so that whenever you edit a view in development the web page gets automatically reloaded. It's a massive time saver when you're doing design or lots of work in the view. To make it work run

bundle exec guard

Guard will also run tests when files change. See TESTING.md for how to control which tests are run.

Mail in development

By default in development mails are sent to Mailcatcher. To install

gem install mailcatcher

Rubymine specific advice

Disable Spring in RubyMine:

Under Run → Edit Configurations → your RSpec configuration, set Environment variable: DISABLE_SPRING=1

Deploying to production

This section will not be relevant to most people. It will however be relevant if you're deploying to a production server.

To deploy morph.io to production, normally you'll just want to deploy using Capistrano:

cap production deploy

Read the provisioning README for details of how to provision from updated ansible playbooks.

Copyright & License

Copyright OpenAustralia Foundation Limited. Licensed under the Affero GPL. See LICENSE file for more details.