Skip to content

CICD.md

Latest commit

 

History

History
361 lines (240 loc) · 12.8 KB

CICD.md

File metadata and controls

361 lines (240 loc) · 12.8 KB

CI/CD Pipeline Documentation

This document describes the CI/CD pipelines for the Open Data Ensemble (ODE) monorepo.

Overview

The ODE monorepo uses GitHub Actions for continuous integration and deployment. Each project has its own pipeline that triggers only when relevant files change.

Pipelines

Synkronus Docker Build & Publish

Workflow File: .github/workflows/synkronus-docker.yml

Triggers

  • Push to main: Builds and publishes release images
  • Push to dev: Builds and publishes pre-release images
  • Push to feature branches: Builds and publishes branch-specific images
  • Pull Requests: Builds but does not publish (validation only)
  • Manual Dispatch: Allows manual triggering with optional version tag

Path Filters

The workflow only runs when files in these paths change:

  • synkronus/** - Any file in the Synkronus project
  • .github/workflows/synkronus-docker.yml - The workflow itself

Image Registry

Images are published to GitHub Container Registry (GHCR):

  • Registry: ghcr.io
  • Image: ghcr.io/opendataensemble/synkronus

Tagging Strategy

Image tags are computed by docker/metadata-action. The highest-priority tag per event is also used as the primary tag in manifest verification.

Event Tags produced Published?
Release published, not prerelease v{X.Y.Z}, v{X.Y}, v{X}, latest Yes
Release published, is prerelease v{X.Y.Z}-{pre}, latest-pre-release Yes
Push → main main, sha-{short} Yes
Push → dev dev, sha-{short} Yes
Push → other branch {branch-name}, sha-{short} Yes
workflow_dispatch sha-{short} Yes
Pull request pr-{number} No (build only)

Moving pointer tags:

  • latest — always points to the most recent non-prerelease GitHub Release. Never moved by branch pushes.
  • latest-pre-release — always points to the most recent prerelease GitHub Release (e.g. -alpha.N, -rc.N).
  • main / dev — track the tip of the respective branch.

workflow_dispatch intentionally produces only sha-{short} so that manual runs cannot accidentally reassign latest, main, dev, or any other pointer tag.

Build Features

  • Multi-platform: Builds for linux/amd64 and linux/arm64 using Buildah
  • Attestation: Generates SLSA build provenance and pushes it to GHCR
  • Metadata: Includes OCI-compliant labels (title, description, vendor, source, revision)
  • Verification: After push, the manifest list and per-arch layers are pulled to confirm correctness

Permissions Required

The workflow requires these permissions:

  • contents: read - To checkout the repository
  • packages: write - To publish to GHCR
  • id-token: write - For OIDC-based build provenance attestation
  • attestations: write - To push attestation records to GHCR

Secrets Used

  • GITHUB_TOKEN - Automatically provided by GitHub Actions

Formulus Android Build

Workflow File: .github/workflows/formulus-android.yml

Purpose

Builds Android APK for the Formulus React Native application, and builds/consumes Formplayer assets in a single, two‑job workflow.

Triggers

  • Push to main/dev (formulus or formulus-formplayer changes): Builds Formplayer assets and then a release APK using those assets
  • Pull Requests (formulus or formulus-formplayer changes): Builds Formplayer assets and then a debug APK for validation
  • Release: Publishes APK to GitHub Release

Path Filters

The workflow runs when files in these paths change:

  • formulus/** - Any file in the formulus project
  • formulus-formplayer/** - Any file in the formulus-formplayer project
  • packages/tokens/** - Shared design tokens and build inputs
  • .github/workflows/formulus-android.yml - The workflow itself

Asset Handling

The workflow intelligently handles formplayer assets using two jobs:

  1. build-formplayer-assets job:

    • Builds @ode/tokens
    • Builds Formplayer assets using npm run build:rn in formulus-formplayer
    • Uploads the built assets from formulus/android/app/src/main/assets/formplayer_dist/ as a GitHub Actions artifact

  2. build-android job (depends on assets job):

    • Downloads the Formplayer assets artifact into formulus/android/app/src/main/assets/formplayer_dist/
    • Runs npm run vendor:notifee in formulus/ to clone the pinned invertase/notifee commit into third_party/notifee (gitignored; required for Gradle :notifee_core)
    • Builds the Android APK (debug for PRs, release for main/dev/release events)

Formplayer assets are not committed to git and are ignored via .gitignore. CI builds always use the assets artifact produced in the same workflow run, ensuring a single, consistent source of truth for each build.

Build Types

  • Pull Requests: Debug APK (unsigned)
  • Push to main/dev: Release APK (signed with secrets)
  • Release: Release APK published to GitHub Release

Secrets Required

  • FORMULUS_RELEASE_KEYSTORE_B64 - Base64 encoded keystore file
  • FORMULUS_RELEASE_STORE_PASSWORD - Keystore password
  • FORMULUS_RELEASE_KEY_ALIAS - Key alias
  • FORMULUS_RELEASE_KEY_PASSWORD - Key password

Workflow Integration

Formplayer asset building and Android APK building are now handled within the same workflow:

Formplayer or Formulus Changes → build-formplayer-assets job → build-android job (consumes artifact) → APK artifact / Release upload

This ensures:

  • No duplicate cross-workflow wiring
  • A single workflow owns both asset and APK builds
  • Each APK is built against the exact assets produced in the same run
  • Formplayer build outputs do not pollute git history

SBOM (CycloneDX) on releases

Workflow: .github/workflows/sbom-release.yml

  • Runs when a GitHub Release is published and uploads *.cdx.json files to that release (alongside other assets such as the Formulus APK from formulus-android.yml).
  • Manual test: Actions → SBOM (CycloneDX) → Run workflow; download the cyclonedx-sbom artifact.

Local generation (requires Node + Go):

node scripts/sbom/generate-sboms.mjs --out sbom-dist

Using Published Images

Pull Latest Stable Release

Points to the most recent non-prerelease GitHub Release.

docker pull ghcr.io/opendataensemble/synkronus:latest

Pull Latest Pre-release

Points to the most recent prerelease GitHub Release (e.g. -alpha.N, -rc.N).

docker pull ghcr.io/opendataensemble/synkronus:latest-pre-release

Pull Specific Version

docker pull ghcr.io/opendataensemble/synkronus:v1.0.0

Pull Development Build

Tracks the tip of the dev branch (rebuilt on every push to dev).

docker pull ghcr.io/opendataensemble/synkronus:dev

Pull Main Branch Build

Tracks the tip of the main branch between releases.

docker pull ghcr.io/opendataensemble/synkronus:main

Pull Feature Branch Build

docker pull ghcr.io/opendataensemble/synkronus:feature-xyz

Release Process

Versioned images are produced by publishing a GitHub Release (the workflow listens for release: [published]).

  1. Go to Releases → Draft a new release
  2. Create a new tag following semver, prefixed with v:
    • Stable release: v1.0.0
    • Pre-release: v1.0.0-rc.1, v1.0.1-alpha.7, etc.
  3. Select the target commit (typically tip of main for stable, tip of dev for pre-release)
  4. Tick Set as a pre-release for alpha/beta/rc tags
  5. Click Publish release

This will create:

Release kind Tags produced
Stable (v1.0.0) v1.0.0, v1.0, v1, latest
Pre-release (v1.0.0-rc.1) v1.0.0-rc.1, latest-pre-release

Note: workflow_dispatch is available for manual runs but intentionally does not create any pointer tags (only sha-{short}) — it is for debugging the build pipeline, not for publishing releases.

Image Visibility

By default, GHCR packages inherit the repository's visibility:

  • Public repositories → Public images (no authentication needed)
  • Private repositories → Private images (authentication required)

Authenticating with GHCR

For private images:

echo $GITHUB_TOKEN | docker login ghcr.io -u USERNAME --password-stdin

Monitoring Builds

View Workflow Runs

  1. Go to the Actions tab in GitHub
  2. Select Synkronus Docker Build & Publish
  3. View recent runs and their status

View Published Images

  1. Go to the repository main page
  2. Click Packages in the right sidebar
  3. Select synkronus
  4. View all published tags and their details

Troubleshooting

Build Fails on Push

  1. Check the Actions tab for error logs
  2. Common issues:
    • Dockerfile syntax errors
    • Missing dependencies in build context
    • Network issues during dependency download

Image Not Published

  1. Verify the branch name matches the workflow triggers
  2. Check that the workflow has packages: write permission
  3. Ensure the push event (not PR) triggered the workflow

Cannot Pull Image

  1. Verify the image tag exists in GHCR
  2. For private repos, ensure you're authenticated
  3. Check image name spelling: ghcr.io/opendataensemble/synkronus

Best Practices

For Developers

  1. Test locally first: Build and test Docker images locally before pushing
  2. Use feature branches: Create feature branches for experimental changes
  3. Review build logs: Check Actions logs even for successful builds
  4. Tag releases properly: Use semantic versioning for releases

For Deployments

  1. Pin versions in production: Use specific version tags (v{X.Y.Z}), not latest
  2. Staging/QA upcoming releases: Use latest-pre-release to track the most recent prerelease (alpha/rc)
  3. Bleeding-edge integration: Use dev for builds from the tip of the dev branch
  4. Monitor image sizes: Keep images lean for faster deployments
  5. Use health checks: Always configure health checks in deployments

Formplayer Asset Synchronization

How It Works

The automated asset build process ensures Formulus Android builds always have matching Formplayer assets:

  1. Developer makes changes to formulus-formplayer
  2. Opens/updates PR (or pushes to main/dev) → the Formulus Android workflow:
    • Runs the build-formplayer-assets job to build Formplayer assets and upload them as an artifact
    • Runs the build-android job, which downloads the artifact and builds the APK

Benefits

  • No manual work: Assets are automatically built and passed between jobs via artifacts
  • No conflicts: Assets are not committed to git, avoiding noisy diffs and merge issues
  • Always consistent: Each Android build uses the assets built in that same workflow run
  • Clean repository: Built assets live only in CI artifacts and local workspaces, not in version control

Local Development

For local development, you can manually build and copy assets:

cd formulus-formplayer
npm run build:rn

This will:

  1. Build the formplayer web app
  2. Clean existing assets in formulus
  3. Copy new assets to formulus/android/app/src/main/assets/formplayer_dist/

The build:rn script automatically handles cleaning, so no need to run clean-rn-assets separately.

ODE Desktop (Tauri)

Workflow file: .github/workflows/ode-desktop.yml

Triggers

  • Pull requests and pushes to main / dev when files under desktop/** change (or the workflow file itself)
  • Manual dispatch

Path filters

Runs only when desktop/** or .github/workflows/ode-desktop.yml changes.

What it runs

From desktop/: pnpm lint, pnpm format:check, pnpm test, pnpm typecheck, pnpm codegen:synk-client, then fails if desktop/src/generated drifts from the regenerated OpenAPI client. From desktop/src-tauri/: cargo fmt --check, cargo clippy -D warnings, cargo test.

Formplayer embed

After building formplayer (npm run build in formulus-formplayer), copy its build/ into the desktop app with pnpm copy:formplayer from desktop/ (see desktop/README.md). Copied assets are gitignored under desktop/public/formplayer_dist/.

Future Enhancements

Potential improvements to the CI/CD pipeline:

  • Automated formplayer asset synchronization
  • Add automated testing before build
  • Implement security scanning (Trivy, Snyk)
  • Add deployment to staging environment
  • Create release notes automation
  • Add Slack/Discord notifications
  • Implement rollback mechanisms
  • Add performance benchmarking

Related Documentation