Skip to main content

CI/CD Deployment Guide

Last Updated: 2026-01-25
Audience: Developers setting up automated deployment to production

Overview​

This guide covers the automated CI/CD deployment pipeline for ARC // OS to a Linux server using Docker. The deployment workflow builds Docker images, pushes them to GitHub Container Registry, and deploys to the production server at arc.corgicy.com.

Architecture​

Services Deployed​

  1. Marketing Website - Static marketing site (root /)
  2. Main Application - React web app (at /app)
  3. Backend API - Fastify server (at /api)
  4. Documentation Site - Docusaurus docs (at /docs)
  5. PostgreSQL Database - Data persistence
  6. Traefik Reverse Proxy - SSL termination (Let's Encrypt) and routing; only service that publishes host ports 80 and 443. All HTTP/HTTPS enters here; Traefik routes internally to file-server and server.
  7. File server (nginx) - Single image built in CI with web app, marketing, and docs static assets; serves /app, /marketing, /docs; no ports published (internal only; reached via Traefik on arcos-network).

Important: On the host, nothing else (e.g. host nginx or Apache) should bind to 80/443. If something else is on 80, you will see "Welcome to nginx!" or connection issues; see Port 80 shows "Welcome to nginx!". 8. Backup Service - Automated backups (on-demand)

Deployment Flow​

GitHub Push → CI Tests → Build Images → Push to Registry → Deploy to Server

Accessing the app​

Once deployed, use these URLs (replace with your domain if different):

URLWhat
https://arcos.corgicy.com/Marketing / landing page
https://arcos.corgicy.com/appMain web app (login, dashboard, etc.)
https://arcos.corgicy.com/docs/Documentation (Docusaurus)
https://arcos.corgicy.com/apiBackend API
https://arcos.corgicy.com/mcpClaude MCP connector
https://arcos.corgicy.com/uploadsUser uploads (images, files; served by backend)

Docs are served at https://arcos.corgicy.com/docs/ from the file-server image (Docusaurus static build baked in CI). /docs is routed by Traefik to file-server.

HTTP is redirected to HTTPS. If you see "TRAEFIK DEFAULT CERT", see Let's Encrypt / ACME below.

Prerequisites​

Server Requirements​

  • Linux server (Ubuntu 20.04+ recommended)
  • Docker 20.10+ installed
  • Docker Compose 2.0+ installed
  • SSH access configured
  • Domain configured (arc.corgicy.com)
  • SSL certificates (Let's Encrypt recommended)

GitHub Secrets Required​

Configure these secrets in GitHub repository settings:

  1. DEPLOY_SSH_KEY - Private SSH key for server access
  2. DEPLOY_HOST - Server hostname or IP address
  3. DEPLOY_USER - SSH username (e.g., deploy or ubuntu)

Server Setup​

  1. Create deployment directory:

    sudo mkdir -p /opt/arcos/{docker,ssl,env}
    sudo chown $USER:$USER /opt/arcos -R

  2. Create environment file (source of truth; deploy adds image tags into env/.env):

    cd /opt/arcos
    # Copy from repo example, or create from scratch (see env/.env.example)
    cp /path/to/arcos/env/.env.example .env
    nano .env   # fill POSTGRES_PASSWORD, JWT_SECRET, CORS_ORIGIN, etc. Do NOT add SERVER_IMAGE or FILE_SERVER_IMAGE
    chmod 600 .env

    Deploy uses only /opt/arcos/.env as source: it overwrites env/.env with the contents of .env (stripping any image vars) plus SERVER_IMAGE and FILE_SERVER_IMAGE. You never edit env/.env by hand; you maintain only .env. If .env is missing on the server, the deploy step fails with a clear error.

    Migrating: If you had env/.env.prod or a bloated env/.env, create /opt/arcos/.env once with your config (no SERVER_IMAGE or FILE_SERVER_IMAGE). Example: grep -v '^SERVER_IMAGE=\|^FILE_SERVER_IMAGE=\|^WEB_IMAGE=\|^MARKETING_IMAGE=\|^DOCS_IMAGE=' env/.env | head -n 25 > .env (keep one block). Then deploy will keep writing a clean env/.env from .env + image tags.

  3. Setup SSL certificates:

    # Install Certbot
    sudo apt-get update
    sudo apt-get install certbot

    # Generate certificates
    sudo certbot certonly --standalone -d arc.corgicy.com

    # Copy certificates to deployment directory
    sudo cp /etc/letsencrypt/live/arc.corgicy.com/fullchain.pem /opt/arcos/ssl/
    sudo cp /etc/letsencrypt/live/arc.corgicy.com/privkey.pem /opt/arcos/ssl/
    sudo chown $USER:$USER /opt/arcos/ssl -R

  4. Setup automated certificate renewal:

    # Create renewal script
    sudo cat > /etc/cron.monthly/renew-arcos-ssl << 'EOF'
    #!/bin/bash
    certbot renew --quiet
    cp /etc/letsencrypt/live/arc.corgicy.com/fullchain.pem /opt/arcos/ssl/
    cp /etc/letsencrypt/live/arc.corgicy.com/privkey.pem /opt/arcos/ssl/
    docker compose --env-file /opt/arcos/env/.env -f /opt/arcos/docker-compose.yml restart traefik
    EOF
    sudo chmod +x /etc/cron.monthly/renew-arcos-ssl

GitHub Actions Workflow​

Workflow File​

The deployment workflow is located at .github/workflows/deploy.yml.

Trigger Conditions​

  • Automatic: Pushes to main branch (ignores markdown and docs-only changes)
  • Manual: Can be triggered manually via GitHub Actions UI

Workflow Steps​

  1. Build Docker Images

    • Builds server, web, marketing, and docs images
    • Pushes to GitHub Container Registry
    • Tags images with commit SHA and latest

  2. Deploy to Server

    • Connects to server via SSH
    • Copies docker-compose and nginx config
    • Pulls latest images
    • Runs database migrations
    • Restarts services
    • Verifies deployment

Image Tags​

Images are tagged as:

  • ghcr.io/corgicy/arcos-server:<commit-sha>
  • ghcr.io/corgicy/arcos-web:<commit-sha>
  • ghcr.io/corgicy/arcos-marketing:<commit-sha>
  • ghcr.io/corgicy/arcos-docs:<commit-sha>

Latest images are also tagged with :latest.

Production Docker Compose​

Configuration File​

Production configuration is in docker-compose.yml; Traefik and file-server use profile prod (deploy runs with --profile prod).

Key Features​

  • Pre-built images - Uses images from GitHub Container Registry
  • Environment variables - Loaded from env/.env (generated from /opt/arcos/.env + image tags by deploy)
  • Health checks - All services have health checks
  • Restart policies - Services restart automatically
  • Network isolation - Services on dedicated network
  • Volume persistence - Data persisted across restarts

Service Configuration​

Database:

  • .env has only POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB (no DATABASE_URL). Compose builds the connection URL from those when needed. Set DATABASE_URL in .env only if the password contains +, /, @, #, or % (URL-encode it).
  • Persistent volume for data
  • Health checks for dependency management
  • Automatic restart

Backend:

  • Health check endpoint
  • Volume for file uploads
  • Environment variables from env/.env

Frontend Services:

  • Nginx serving static files
  • Volume mounts for build artifacts
  • No build step (uses pre-built images)

Nginx:

  • SSL/TLS termination
  • Reverse proxy for API
  • Rate limiting
  • Security headers
  • SPA routing for all frontend services

Deployment Process​

Automatic Deployment​

  1. Push to main branch
  2. CI tests run (must pass)
  3. Images built and pushed
  4. Deployment triggered
  5. Services updated on server

Manual Deployment​

  1. Go to Actions tab in GitHub
  2. Select Deploy to Production workflow
  3. Click Run workflow
  4. Select branch (usually main)
  5. Click Run workflow

Deployment Steps (Automated)​

  1. SSH to server
  2. Copy configuration files
  3. Update environment with image tags
  4. Pull latest images
  5. Run database migrations
  6. Restart services
  7. Verify health checks

Post-Deployment​

Verification Checklist​

After deployment, verify:

  1. Marketing site - https://arc.corgicy.com/
  2. Main app - https://arc.corgicy.com/app
  3. Documentation - https://arc.corgicy.com/docs
  4. API health - https://arc.corgicy.com/api/health
  5. Backend health - https://arc.corgicy.com/health

Database Migrations​

Migrations run automatically during deployment. To run manually:

ssh user@arc.corgicy.com
cd /opt/arcos
docker compose --env-file env/.env -f docker-compose.yml run --rm server pnpm prisma migrate deploy

Database Seeding​

Initial data is seeded automatically during each deploy (after db push). The seed script is idempotent: it skips records that already exist, so running it multiple times does not create duplicates.

To run the full seed manually (e.g. to add new global data or refresh demo user):

docker compose --env-file env/.env -f docker-compose.yml run --rm server pnpm prisma:seed

Gamification-only seed:

docker compose --env-file env/.env -f docker-compose.yml run --rm server pnpm prisma:seed:gamification

Rate limiting​

Effective limits are set by server env vars (per time window). Put them in /opt/arcos/.env; deploy copies them into env/.env and docker-compose passes them to the server container:

  • RATE_LIMIT_MAX – API requests per window (default 1000)
  • RATE_LIMIT_TIME_WINDOW – Window in ms (default 900000 = 15 min)
  • RATE_LIMIT_AUTH_MAX – Auth endpoint requests per window (default 100)
  • RATE_LIMIT_AUTH_TIME_WINDOW – Auth window in ms
  • RATE_LIMIT_PUBLIC_API_MAX / RATE_LIMIT_PUBLIC_API_TIME_WINDOW – Public API key limits

Traefik also applies a per-second ceiling (in docker/traefik-dynamic.yml). It is set high (1000/2000 for API, 200/400 for auth) so it does not throttle before the server; the server env vars above are the real limit.

Rollback Procedure​

Quick Rollback​

  1. SSH to server:

    ssh user@arc.corgicy.com
    cd /opt/arcos

  2. Rollback image tags (edit generated env/.env; next deploy will overwrite):

    cd /opt/arcos
    nano env/.env   # set SERVER_IMAGE and FILE_SERVER_IMAGE to previous SHA

  3. Pull and restart:

    docker compose --env-file env/.env -f docker-compose.yml pull
    docker compose --env-file env/.env -f docker-compose.yml --profile prod up -d

Database Migration Rollback​

If a migration causes issues:

# Connect to database
docker compose --env-file env/.env -f docker-compose.yml exec db psql -U arcos -d arcos

# Manually rollback migration (see Prisma migration files)
# Or restore from backup

Monitoring​

Health Checks​

All services expose health check endpoints:

  • Backend: /health and /api/health
  • Nginx: Serves health checks from backend

Logs​

View logs for services:

# All services
docker compose --env-file env/.env -f docker-compose.yml logs -f

# Specific service
docker compose --env-file env/.env -f docker-compose.yml logs -f server
docker compose --env-file env/.env -f docker-compose.yml logs -f file-server

Service Status​

docker compose --env-file env/.env -f docker-compose.yml ps

Troubleshooting​

Deployment Fails​

  1. Check GitHub Actions logs - See which step failed
  2. SSH to server - Verify server is accessible
  3. Check Docker - Ensure Docker is running on server
  4. Verify secrets - Ensure GitHub secrets are correct

Services Not Starting​

  1. Always run compose from /opt/arcos with the env file (otherwise you'll see "POSTGRES_PASSWORD variable is not set" and services may not start):

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml logs

  2. Check logs:

    docker compose --env-file env/.env -f docker-compose.yml logs

  3. Verify environment variables:

    cat /opt/arcos/env/.env

  4. Check health:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod ps

SSL Certificate Issues​

  1. Verify certificates exist:

    ls -la /opt/arcos/ssl/

  2. Renew certificates:
    With Traefik + Let's Encrypt, certificates are renewed automatically. If you use external certs, copy them into the Traefik volume and restart:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod restart traefik

Let's Encrypt / Traefik default cert​

If the browser shows "TRAEFIK DEFAULT CERT" or "Your connection is not private" / ERR_CERT_AUTHORITY_INVALID, Traefik is serving its self-signed cert because ACME hasn't issued one yet or the HTTP challenge failed.

  1. Ensure port 80 is reachable from the internet — Let's Encrypt validates via http://arcos.corgicy.com/.well-known/acme-challenge/.... Our config redirects HTTP→HTTPS except for /.well-known/, so the challenge can succeed. If something else (e.g. host nginx) is on port 80, Traefik won't receive the challenge; see Port 80 shows "Welcome to nginx!".

  2. Check Traefik logs for ACME errors:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod logs traefik

    Look for lines like "level=error" and "acme" or "certificate".

  3. ACME email — In docker/traefik.yml the resolver uses email: admin@corgicy.com. To use a different address, edit that file (and redeploy the docker/ files to the server) or set ACME_EMAIL in .env if your Traefik version supports it.

  4. Restart Traefik after any config change so it retries certificate issuance:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod restart traefik

  5. Cert is Let's Encrypt but browser still says "not private" — Often the server isn't sending the full chain (intermediate missing), or the client doesn't trust the R12 intermediate yet. We set preferredChain: "ISRG Root X1" so Traefik requests a chain that may be more widely trusted. To force re-issuance with the preferred chain, on the server clear ACME storage and restart Traefik:

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml --profile prod stop traefik
    docker run --rm -v arcos_letsencrypt_data:/data alpine sh -c "rm -f /data/acme.json"
    docker compose --env-file env/.env -f docker-compose.yml --profile prod start traefik

    Wait 1–2 minutes, then reload the site. To verify the chain from your machine: openssl s_client -connect arcos.corgicy.com:443 -servername arcos.corgicy.com </dev/null 2>/dev/null | openssl x509 -noout -text and check the issuer chain.

Port 80 shows "Welcome to nginx!" (Traefik not receiving traffic)​

All traffic should hit Traefik on ports 80/443; Traefik then redirects HTTP→HTTPS and routes to the file-server (nginx) or API. If you see the default nginx page on http://arcos.corgicy.com, something else is bound to port 80 on the host.

  1. See what is listening on 80 and 443:

    sudo ss -tlnp | grep -E ':80 |:443 '
    # or: sudo lsof -i :80 -i :443

    Only the Traefik container should show for 80 and 443 (e.g. docker-proxy or the traefik process).

  2. If host nginx (or another process) is on 80: stop it so Traefik can bind:

    sudo systemctl stop nginx
    sudo systemctl disable nginx

  3. Ensure the arcos stack is up and Traefik has 80/443:

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml --profile prod ps

    Check that arcos_traefik_prod shows 0.0.0.0:80->80/tcp and 0.0.0.0:443->443/tcp.

  4. Expected flow: http://arcos.corgicy.com → Traefik (80) → redirect to https://arcos.corgicy.com → Traefik (443) → routes to file-server or server.

Backend server not running (only db in docker ps -a)​

  1. Backup service is under profile backup so up -d does not try to build it on the server (no build context there). Main services (server, file-server, traefik, worker) should start.

  2. Start the stack manually (always from /opt/arcos with env/.env; use --profile prod for Traefik and file-server):

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml --profile prod up -d --remove-orphans
    docker compose --env-file env/.env -f docker-compose.yml --profile prod ps -a

  3. If server exits immediately: check docker compose --env-file env/.env -f docker-compose.yml logs server. If you see "password authentication failed for user arcos", the DB was initialized with a different password; see Database Connection Issues below.

Database Connection Issues​

  1. Verify database is running:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod ps db

  2. Check connection: .env has POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_DB; compose builds DATABASE_URL from those. If you set DATABASE_URL in .env (e.g. for special chars in password), it will be in env/.env after deploy:

    grep -E 'POSTGRES_|DATABASE_URL' /opt/arcos/env/.env

  3. Test connection:

    docker compose --env-file env/.env -f docker-compose.yml exec server pnpm prisma db push --dry-run

  4. If you see "password authentication failed for user arcos": The DB was initialized with a different POSTGRES_PASSWORD than in your current .env. Either:

    • Recreate the DB (data loss): cd /opt/arcos && docker compose --env-file env/.env -f docker-compose.yml --profile prod down && docker volume rm arcos_postgres_data (volume name may vary; check docker volume ls). Then up -d --profile prod again so Postgres initializes with the password from .env.
    • Or change the password inside Postgres to match .env: docker exec arcos_db_prod psql -U arcos -d arcos -c "ALTER USER arcos PASSWORD 'your-password-from-env';" (use the value from POSTGRES_PASSWORD in .env).

Table does not exist (P2021 / "The table `public.users` does not exist")​

The database is empty or the schema was never applied.

  1. Try migrations first (if the project has migration files in the image):

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml run --rm server pnpm prisma migrate deploy

  2. If you see "No migration found in prisma/migrations" — migrations are not in the repo (they are gitignored), so the image has no migration files. Create the schema from schema.prisma with db push:

    cd /opt/arcos
    docker compose --env-file env/.env -f docker-compose.yml run --rm server pnpm prisma db push

    This creates all tables (users, etc.) from the current schema. Then restart:

    docker compose --env-file env/.env -f docker-compose.yml --profile prod restart server worker

  3. Long term: To use migrate deploy in production, add migrations to the repo: create an initial migration locally (pnpm prisma migrate dev --name init), remove server/prisma/migrations/ from .gitignore, and commit the migrations folder.

Security Considerations​

SSH Key Security​

  • Use dedicated deploy user with limited permissions
  • Restrict SSH key to deployment commands only
  • Rotate keys regularly

Environment Variables​

  • Never commit env/.env to repository
  • Use strong passwords for database
  • Rotate JWT_SECRET periodically
  • Limit file permissions: 
    chmod 600 /opt/arcos/env/.env

Container Security​

  • Run as non-root where possible
  • Keep images updated (security patches)
  • Use specific image tags (not latest in production)
  • Limit container resources (CPU, memory)

Best Practices​

  1. Test deployments - Test in staging before production
  2. Monitor deployments - Watch logs during deployment
  3. Backup before deploy - Always backup database before major updates
  4. Gradual rollout - Consider blue-green deployment for zero downtime
  5. Document changes - Keep deployment notes for each release

Future Enhancements​

  • Blue-green deployment - Zero-downtime deployments
  • Health check automation - Automatic rollback on health check failure
  • Deployment notifications - Slack/Discord notifications
  • Staging environment - Separate staging deployment
  • Database migration testing - Test migrations before production
  • Performance monitoring - Integrate monitoring tools

Additional Resources​