Deploying

Note

This applies to the PostgreSQL rewrite, which has not yet been fully released.

Self-hosting mnbot isn't too difficult.

Questions or comments about these instructions! Let me know!

Basic instructions

Tracker, IRC bot, dashboard

Environment

mnbot needs a PostgreSQL database (version 18 or higher) for the tracker.

Outside of docker (recommended for dev)

See .env-example for a list of required environment variables, and the Dockerfile for a list of dependencies. (The list will probably be copied here at some point.)

In Docker (recommended for prod)

Copy .env-example to .env and make changes as necessary.

Creating a database

If using Docker, use docker compose run script .... Otherwise, run python3 -m tracker.scripts ... from the parent directory.

The particular subcommand you will want is create. You will also want to specify the --tries argument which will be the default number of tries for each page (1 is useful for quick failure feedback during development; in production, mnbot uses 3 to allow up to 3 non-fatal errors before giving up).

Creating a pipeline

The particular subcommand you will want is add_pipeline <name>. You can also specify --matchonly to create a matchonly pipeline (where jobs with no tag specified will not be claimed). Tags are added via IRC, not with this script.

IMPORTANT: It is imperative that each client (EVEN ONES ON THE SAME SERVER) use a different name. The server and IRC bot both assume that this is the case.

Scaling up: In future, it will be possible to make one client run multiple jobs in parallel. The code is there, but it has not yet been audited for race conditions.

Client

Because of all the dependencies, it is recommended to run this in Docker, even during development. This means you will have to docker compose build && docker compose up whenever you make a change.

Copy .env-example to .env and make changes as necessary. You should not need to modify docker-compose.yml, it is self-contained.

(Note: the tracker URL needs basic authentication to register with the server. This can be done like ws://pipeline_id:password@hostname.)

The client currently uses a RethinkDB database to store warcprox's deduplication database. (At some point this may be changed.) You can access the admin UI on port 4186 if needed. Deleting records is fine (mnbot does it after 7 days) but if you modify them to point to the wrong record, you'll make warcprox write incorrect revisit records. That's bad.

Before spinning up

Try docker compose run chrome_test /bin/chromium.

Chrome may spit out a bunch of dbus warnings, but if the VNC output shows that it starts up, you can skip this next section.

If Chrome fails to start with Operation not permitted or Permission denied, check dmesg for conflicts with AppArmor. If AppArmor is complaining, especially if it's about the usage of userns_create, awesome! Read on.

Fixing conflicts with AppArmor

AppArmor sometimes is configured to disable userspace namespacing. Chromium relies on namespacing for sandboxing. (If you're worried about disabling AppArmor protections, remember that the whole reason we're doing this is so the untrusted code can be sandboxed away.)

Many distros offer an existing configuration for Chromium. On my system, these are stored in /etc/apparmor.d, and the one we want is called chromium. In case this isn't on your system, either find an existing profile or create a new one similar to this one stored on my system:

abi <abi/4.0>,
include <tunables/global>

profile chromium flags=(unconfined) {
  userns,

  # Site-specific additions and overrides. See local/README for details.
  include if exists <local/chromium>
}

...and either globally install it in your AppArmor directory or load it temporarily with apparmor_parser -r -W /path/to/your_profile. If globally installing it, make sure to reload the configuration; on my system sudo systemctl reload apparmor.service works.)

(For systems that don't support ABI 4.0, like Debian bookworm, click here for one that worked for me.)

abi <abi/3.0>,
include <tunables/global>

profile chromium flags=(unconfined) {
}

Then change the value of APPARMOR_PROFILE in .env. (Changing chromium to something else to avoid conflicting with another profile is fine as long as whatever is in .env reflects the correct name.)

Target

Warning

The target has a few known issues that make it difficult to run unattended. The current plan is to move to an rsync target, so to avoid possibly introducing data loss bugs when the code is going to be replaced anyway, the target is not seeing any development.

The target is in target. Copy .env-example to .env and make changes as necessary. Build times are slow because Rust, so for debugging, you'll probably want to do it without the container.