diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 15196ba6..0ccebf40 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -88,6 +88,12 @@ - **Document relationships**: Specify all foreign keys and associations (hasMany, belongsTo, etc.) - **Explain patterns**: If using special patterns (STI, polymorphism, etc.), document the reasoning +## Running the Manager Locally + +- **`make dev`**: Sets up and starts the Manager (`create-a-container`) against local SQLite — creates `.env`, installs deps, runs migrations, builds the client, and serves at `http://localhost:3000`. Use this for Manager web app changes. +- **Full Docker stack**: Use `docker compose up -d` when you need Proxmox, provisioning, DNS, or reverse-proxy behavior. +- **Single source of truth**: See [Development Workflow](../mie-opensource-landing/docs/developers/development-workflow.md). Do not duplicate these steps elsewhere — link to that doc instead. + ## Working with GitHub Actions Workflows ### Development Philosophy diff --git a/.gitignore b/.gitignore index 45adf944..aa828224 100644 --- a/.gitignore +++ b/.gitignore @@ -1,3 +1,6 @@ node_modules .env .tmp-verify/ +data/ +*.sqlite +.playwright-mcp/ diff --git a/Makefile b/Makefile index 40d4aeb3..b75e6d74 100644 --- a/Makefile +++ b/Makefile @@ -1,15 +1,31 @@ -.PHONY: install install-create-container install-pull-config install-docs help +.PHONY: install install-create-container install-pull-config install-docs dev help help: @echo "opensource-server installation" @echo "" @echo "Available targets:" + @echo " make dev - Set up and start create-a-container locally (SQLite)" @echo " make install - Install all components" @echo " make install-create-container - Install create-a-container web application" @echo " make install-pull-config - Install pull-config system" @echo " make install-docs - Install documentation server" @echo "" +# Local development: sets up create-a-container with SQLite and starts the server. +# Creates .env if missing, installs deps, runs migrations, builds the client, then starts. +dev: + @if [ ! -f create-a-container/.env ]; then \ + echo "Creating create-a-container/.env with SQLite defaults..."; \ + printf 'DATABASE_DIALECT=sqlite\nSQLITE_STORAGE=./dev.sqlite\nSESSION_SECRET=%s\nNODE_ENV=development\n' \ + $$(node -e "process.stdout.write(require('crypto').randomBytes(32).toString('hex'))") \ + > create-a-container/.env; \ + fi + cd create-a-container && npm install + cd create-a-container && npm run db:migrate + cd create-a-container/client && npm install && node_modules/.bin/vite build + @echo "Starting server at http://localhost:3000 ..." + cd create-a-container && node server.js + install: install-create-container install-pull-config install-docs SYSTEMD_DIR := create-a-container/systemd diff --git a/README.md b/README.md index d5d246b1..955d1a31 100644 --- a/README.md +++ b/README.md @@ -9,6 +9,7 @@ Full documentation lives in [`mie-opensource-landing/docs/`](mie-opensource-land | If you want to... | Read | |---|---| | Install and operate a production deployment | [Installation Guide](mie-opensource-landing/docs/admins/installation.md) | +| Run just the Manager web app locally (`make dev`, SQLite) | [Run the Manager Locally](mie-opensource-landing/docs/developers/development-workflow.md#run-the-manager-locally-make-dev) | | Run the full stack locally to develop or contribute | [Development Workflow](mie-opensource-landing/docs/developers/development-workflow.md) | | Use a deployed cluster as an end user (create containers, etc.) | [User Getting Started](mie-opensource-landing/docs/users/getting-started.md) | | Contribute changes | [Contributing](mie-opensource-landing/docs/developers/contributing.md) | diff --git a/create-a-container/.gitignore b/create-a-container/.gitignore index d83eef34..3500acfa 100644 --- a/create-a-container/.gitignore +++ b/create-a-container/.gitignore @@ -1,3 +1,4 @@ .env node_modules data/ +*.sqlite diff --git a/create-a-container/migrations/20260603000000-remove-node-id-unique-from-containers.js b/create-a-container/migrations/20260603000000-remove-node-id-unique-from-containers.js new file mode 100644 index 00000000..d449d19f --- /dev/null +++ b/create-a-container/migrations/20260603000000-remove-node-id-unique-from-containers.js @@ -0,0 +1,86 @@ +'use strict'; + +/** + * SQLite's changeColumn recreates the table and accidentally applied a column- + * level UNIQUE on Containers.nodeId. A node (Proxmox host) can own many + * containers, so the constraint is wrong. The correct uniqueness guarantee is + * already enforced by the (nodeId, containerId) composite index. + * + * Postgres was never affected because its ALTER COLUMN does not rebuild tables. + * This migration is a no-op on Postgres (removeConstraint on a non-existent + * constraint will throw, so we guard). + */ +module.exports = { + async up(queryInterface, Sequelize) { + const dialect = queryInterface.sequelize.getDialect(); + + if (dialect === 'sqlite') { + // SQLite requires a full table rebuild to drop a column-level constraint. + await queryInterface.sequelize.transaction(async (t) => { + // 1. Create replacement table without the UNIQUE on nodeId + await queryInterface.sequelize.query( + `CREATE TABLE "Containers_new" AS SELECT * FROM "Containers" WHERE 0`, + { transaction: t }, + ); + await queryInterface.sequelize.query(`DROP TABLE "Containers_new"`, { transaction: t }); + + // Recreate properly using Sequelize's createTable so the model sync + // matches. Easier: use raw DDL mirroring the current schema minus UNIQUE. + const [[{ sql }]] = await queryInterface.sequelize.query( + `SELECT sql FROM sqlite_master WHERE type='table' AND name='Containers'`, + { transaction: t }, + ); + + // Replace the column-level UNIQUE on nodeId + const fixed = sql.replace( + /`nodeId` INTEGER NOT NULL UNIQUE/, + '`nodeId` INTEGER NOT NULL', + ); + + if (fixed === sql) { + // Constraint not present — nothing to do + return; + } + + const newName = 'Containers_new'; + await queryInterface.sequelize.query( + fixed.replace(/CREATE TABLE "Containers"/, `CREATE TABLE "${newName}"`), + { transaction: t }, + ); + await queryInterface.sequelize.query( + `INSERT INTO "${newName}" SELECT * FROM "Containers"`, + { transaction: t }, + ); + await queryInterface.sequelize.query(`DROP TABLE "Containers"`, { transaction: t }); + await queryInterface.sequelize.query( + `ALTER TABLE "${newName}" RENAME TO "Containers"`, + { transaction: t }, + ); + + // Recreate the indexes that were on the old table + await queryInterface.sequelize.query( + `CREATE UNIQUE INDEX IF NOT EXISTS "containers_node_id_container_id_unique" ON "Containers" ("nodeId", "containerId")`, + { transaction: t }, + ); + await queryInterface.sequelize.query( + `CREATE UNIQUE INDEX IF NOT EXISTS "containers_site_hostname_unique_idx" ON "Containers" ("siteId", "hostname")`, + { transaction: t }, + ); + await queryInterface.sequelize.query( + `CREATE UNIQUE INDEX IF NOT EXISTS "containers_site_ipv4_unique_idx" ON "Containers" ("siteId", "ipv4Address")`, + { transaction: t }, + ); + await queryInterface.sequelize.query( + `CREATE UNIQUE INDEX IF NOT EXISTS "containers_site_mac_unique_idx" ON "Containers" ("siteId", "macAddress")`, + { transaction: t }, + ); + }); + } + // Postgres: nodeId was never accidentally made UNIQUE — nothing to do. + }, + + async down(queryInterface, Sequelize) { + // Intentionally left empty — restoring the erroneous constraint would be + // counter-productive. + }, +}; diff --git a/create-a-container/package.json b/create-a-container/package.json index 5d13da6d..a77dec86 100644 --- a/create-a-container/package.json +++ b/create-a-container/package.json @@ -8,7 +8,7 @@ "scripts": { "dev": "concurrently -n server,client -c blue,green \"nodemon server.js\" \"npm --prefix client run build:watch\"", "dev:server": "nodemon server.js", - "db:migrate": "sequelize db:migrate && sequelize db:seed:all", + "db:migrate": "sequelize-cli db:migrate && sequelize-cli db:seed:all", "job-runner": "node job-runner.js", "client:install": "npm --prefix client install", "client:dev": "npm --prefix client run dev", diff --git a/create-a-container/routers/api/v1/auth.js b/create-a-container/routers/api/v1/auth.js index 0be58d3e..13cb98fe 100644 --- a/create-a-container/routers/api/v1/auth.js +++ b/create-a-container/routers/api/v1/auth.js @@ -19,6 +19,8 @@ const { User, Group, Setting, + Site, + Node, ExternalDomain, PasswordResetToken, InviteToken, @@ -68,6 +70,29 @@ if (process.env.NODE_ENV !== 'production') { const isAdmin = req.body?.role === 'admin'; const uid = isAdmin ? 'dev-admin' : 'dev-user'; + // Ensure a localhost site exists to work with. + const [site] = await Site.findOrCreate({ + where: { name: 'localhost' }, + defaults: { internalDomain: 'localhost', externalIp: '127.0.0.1' }, + }); + + // Ensure a local dev node exists so containers can be created without + // a real Proxmox host. `apiUrl: 'local'` marks it as the mock node; + // container creation short-circuits provisioning for it in dev. + await Node.findOrCreate({ + where: { siteId: site.id, apiUrl: 'local' }, + defaults: { name: 'local', tokenId: 'local', secret: 'local', ipv4Address: '127.0.0.1' }, + }); + + // Ensure a default external domain exists so HTTP services (added + // automatically by image templates) have a domain to bind to. Without + // this the new-container form is unsubmittable for any image that + // exposes an HTTP port. + await ExternalDomain.findOrCreate({ + where: { name: 'localhost' }, + defaults: { siteId: site.id }, + }); + let user = await User.findOne({ where: { uid }, include: [{ association: 'groups' }], diff --git a/create-a-container/routers/api/v1/containers.js b/create-a-container/routers/api/v1/containers.js index ca4ff762..90e6f612 100644 --- a/create-a-container/routers/api/v1/containers.js +++ b/create-a-container/routers/api/v1/containers.js @@ -4,6 +4,7 @@ */ const express = require('express'); +const crypto = require('crypto'); const { Container, Service, @@ -347,6 +348,30 @@ router.post( } } + // Local dev: short-circuit Proxmox provisioning. The on-demand `local` + // node (apiUrl === 'local') has no real hypervisor, so mark the container + // running immediately with placeholder VMID/MAC/IP. This is the hook point + // where a real Docker-based local provisioner can plug in later. + if (process.env.NODE_ENV !== 'production' && node.apiUrl === 'local') { + const fakeVmid = 9000 + container.id; + const hex = () => crypto.randomBytes(1)[0].toString(16).padStart(2, '0').toUpperCase(); + await container.update( + { + containerId: fakeVmid, + macAddress: `BC:24:11:${hex()}:${hex()}:${hex()}`, + ipv4Address: `10.0.0.${100 + (container.id % 150)}`, + status: 'running', + }, + { transaction: t }, + ); + await t.commit(); + return created(res, { + containerId: container.id, + hostname: container.hostname, + status: 'running', + }); + } + const job = await Job.create( { command: `node bin/create-container.js --container-id=${container.id}`, diff --git a/mie-opensource-landing/docs/developers/development-workflow.md b/mie-opensource-landing/docs/developers/development-workflow.md index 9541aa88..8b1d9dd7 100644 --- a/mie-opensource-landing/docs/developers/development-workflow.md +++ b/mie-opensource-landing/docs/developers/development-workflow.md @@ -1,12 +1,39 @@ # Development Workflow +There are two ways to run the Manager locally: + +- **Manager only (`make dev`)** — fastest loop for working on the Manager web app. SQLite, no Docker, no Proxmox. Start here for UI/API changes. +- **Full stack (Docker Compose)** — Proxmox, Manager, docs, and bootstrap together. Use when you need real container provisioning, DNS, or reverse-proxy behavior. + +## Run the Manager Locally (`make dev`) + +From the repository root: + +```bash +make dev +``` + +This sets up and starts the Manager (`create-a-container`) against a local SQLite database: + +1. Creates `create-a-container/.env` with SQLite defaults and a random `SESSION_SECRET` (only if missing). +2. Installs Manager and client dependencies. +3. Runs database migrations. +4. Builds the React client. +5. Starts the server at . + +In development (`NODE_ENV=development`) the login page exposes **dev login** buttons that create `dev-user` / `dev-admin` on demand and ensure a `localhost` site exists — no seeding required. + +The SQLite database (`create-a-container/dev.sqlite`) is gitignored; delete it to reset local state. + +## Run the Full Stack (Docker Compose) + The entire stack — Proxmox, Manager, docs, and bootstrap — runs locally inside Docker. -## Prerequisites +### Prerequisites - [Docker Desktop](https://www.docker.com/products/docker-desktop/) -## Start the Stack +### Start the Stack From the repository root: @@ -25,7 +52,7 @@ This brings up: | `zensical` | Rebuilds these docs on file changes | | Bootstrap (one-shot) | Configures the Manager container to use the virtualized Proxmox | -## Manager Image Selection +### Manager Image Selection By default the compose stack deploys `ghcr.io/mieweb/opensource-server/manager:latest`. Override the tag by setting `MANAGER_TAG` in your environment or in a `.env` file alongside `compose.yml`: @@ -41,7 +68,7 @@ The template download step checks your **local** Docker images first, so a local - Delete the cached tar file from the `local` volume, or - Recreate the volume (e.g., `docker compose down -v`). -## Persistent State and Full Reset +### Persistent State and Full Reset Proxmox state is persisted across container creation and destruction in two named volumes, so `docker compose down` followed by `docker compose up` keeps @@ -62,7 +89,7 @@ This deletes the cached template tarball, all stored images/volumes, and the cluster state, so the next `docker compose up` re-downloads the Manager template and re-runs the bootstrap from scratch. -## Endpoints +### Endpoints | URL | Description | |---|---| @@ -71,7 +98,7 @@ and re-runs the bootstrap from scratch. | `https://localhost` | Documentation site | | `https://manager.localhost` | Manager Web UI | -## Credentials and Shell Access +### Credentials and Shell Access **Proxmox Web UI:** `root` / `root` @@ -93,7 +120,7 @@ docker compose exec -it proxmox pct enter 100 docker compose exec -it proxmox pct exec 100 -- sudo -u postgres psql cluster_manager ``` -## Live Reload +### Live Reload The local git repository is mounted **read-only** into `/opt/opensource-server` on the Manager container, so source changes on the host are visible immediately. @@ -105,7 +132,7 @@ The local git repository is mounted **read-only** into `/opt/opensource-server` | Job runner | Restart manually | | Database migrations | Run manually in the proper server context (see below) | -### Frontend Rebuilds +#### Frontend Rebuilds The Manager serves the compiled React app from `create-a-container/client/dist`; it does **not** run the Vite dev server. Because the repository is mounted **read-only** into the Manager container, Vite can't write build output from there. Instead, the dedicated `client` service mounts the repository **read-write** and runs `vite build --watch`, rebuilding `client/dist` on the host whenever the client source changes. @@ -115,7 +142,7 @@ The Manager container's read-only bind mount still reflects those host changes, The `client` service performs an initial build before Proxmox starts serving (the `proxmox` service waits for it to become healthy), so a bundle always exists even on a clean checkout where `client/dist` isn't present yet. -### Run Database Migrations +#### Run Database Migrations ```bash docker compose exec proxmox pct exec 100 -- \