Skip to content

Self-Hosting

This page is for maintainers. It explains how the whole system fits together and exactly how to bring it up on a fresh Mac (e.g. after buying a new machine).

1. Big picture

The orchestrator (testlab-test-orchestrator, a Spring Boot app) receives test requests over HTTP (POST /api/v1/startTest), runs them on the physical devices, builds a report, uploads it to Google Drive and emails the result.

Test devices are connected by USB to the Mac. There are two execution worlds:

Platform How it runs Where the orchestrator runs
Android / Flutter / React Native adb + flutter Docker container
iOS xcodebuild (Xcode) macOS host (cannot be containerized)

So in normal operation there are two orchestrator instances:

  • A Docker instance for Android/Flutter/RN (IOS_ENABLED=false).
  • A host-native instance for iOS (IOS_ENABLED=true), run directly on macOS.
GitLab CI / clients ──HTTP──► orchestrator
                                  │
   Android/Flutter/RN  ──────────►│ (Docker)  ──adb over TCP──► host adb server ──USB──► phones
   iOS (xcuitest)      ──────────►│ (host)    ──xcodebuild────────────────────────────► iPhones

2. Why Docker can't see USB (and how we solve it)

Docker Desktop on macOS runs containers inside a hidden Linux VM. That VM does not get the Mac's USB devices — there is no USB passthrough on macOS.

We don't need USB inside the container. adb is split into a server (owns the USB devices) and a client (sends commands); they talk over TCP. So:

  • The adb server runs on the Mac (it has USB access).
  • The container only runs the adb client, which connects to the Mac's server over the network. Device data flows: container → TCP → host adb server → USB → phone.

The Mac's adb server only listens on 127.0.0.1, which the container can't reach, so a tiny forwarder (socat) listens on :5038 and relays to 127.0.0.1:5037. The container reaches the Mac via the special name host.docker.internal.

amd64 / Rosetta: Google ships adb/aapt for Linux as x86_64 only. On Apple Silicon the image therefore runs as linux/amd64 (via Rosetta) — set with platform: linux/amd64 in docker-compose.yml.

Why not iOS in Docker: iOS testing needs Xcode, which only runs on macOS. A Linux container can't run Xcode, so iOS stays on the host regardless of USB.

3. New-Mac setup (step by step)

3.1 Base tools (Homebrew)

# Homebrew (if not present): https://brew.sh
brew install --cask docker          # Docker Desktop
brew install --cask temurin@21      # JDK 21 (host-native orchestrator / builds)
brew install socat                  # adb TCP bridge
brew install xctesthtmlreport       # iOS xcuitest HTML report (host)
brew install cocoapods              # iOS Flutter builds (host)
  • Android SDK + adb: install Android Studio (or brew install --cask android-commandlinetools) and platform-tools. adb ends up at /opt/homebrew/bin/adb (symlink) or ~/Library/Android/sdk/platform-tools/adb.
  • Flutter: install the Flutter SDK (host-side only needed for iOS Flutter; the Docker image bundles its own Flutter for Android Flutter tests).
  • Xcode: install from the App Store, then xcode-select --install and open it once to accept the license. Required for iOS.

3.2 Docker Desktop settings

  • Settings → General → Start Docker Desktop when you log in ✅
  • Settings → General → Use Rosetta for x86/amd64 emulation ✅ (Apple Silicon)
  • Settings → Resources → File sharing: ensure the repo path is shared.

3.3 macOS auto-login

So the adb bridge (a launchd agent) and Docker start after a reboot without someone logging in manually:

  • System Settings → Users & Groups → Automatically log in as the testlab user.

3.4 Connect & authorize devices

  • Plug in the devices over USB. On each device accept "Trust this computer" (tick "Always allow").
  • Enable Developer options and USB debugging. On some devices also enable "Disable permission monitoring" (Developer options) for screenshot tests.
  • For iOS devices: enable Developer Mode (Settings → Privacy & Security → Developer Mode → reboot) and keep them unlocked during runs.
  • Verify: adb devices -l shows each Android device as device (not unauthorized).

4. The adb bridge (launchd, foolproof)

This keeps the adb server up and exposes it to Docker on :5038, surviving reboots and device unplug/replug. Files live in the orchestrator repo under scripts/ and deploy/.

# 1. Install the bridge script OUTSIDE ~/Desktop (macOS TCP/TCC blocks launchd
#    from executing files in Desktop/Documents/Downloads).
mkdir -p /Users/Shared/testlab
cp scripts/adb-bridge.sh /Users/Shared/testlab/adb-bridge.sh
chmod +x /Users/Shared/testlab/adb-bridge.sh

# 2. Install and load the LaunchAgent (auto-start + auto-restart).
cp deploy/ee.taltech.testlab.adb-bridge.plist ~/Library/LaunchAgents/
launchctl load -w ~/Library/LaunchAgents/ee.taltech.testlab.adb-bridge.plist

# 3. Verify
launchctl list | grep adb-bridge          # should show a PID and status 0
lsof -nP -iTCP:5038 -sTCP:LISTEN           # socat should be LISTENing

What it does: a background loop runs adb start-server every 20s (idempotent; restarts the server if anything kills it or a device is replugged), and socat forwards :5038 → 127.0.0.1:5037. KeepAlive/RunAtLoad in the plist restart it on crash and at login.

5. Orchestrator in Docker (Android / Flutter / RN)

In testlab-test-orchestrator/:

# Secrets: place the Google Drive OAuth client at secrets/credentials.json.
# (The container writes its StoredCredential token cache next to it, so the
#  secrets/ mount must be read-write.)

export GIT_TOKEN=glpat-xxxxxxxxxxxxxxxx     # token used to clone student repos
docker compose up -d --build

Key points (already configured in docker-compose.yml):

  • platform: linux/amd64 (Rosetta).
  • ADB_SERVER_SOCKET=tcp:host.docker.internal:5038 → the adb bridge.
  • IOS_ENABLED=false → rejects iOS jobs cleanly.
  • Volumes: ./output (reports/screenshots) and ./secrets (read-write).
  • restart: unless-stopped → comes back with Docker after a reboot.

Verify:

docker compose exec orchestrator adb devices   # should list the host's devices
curl -s -X POST http://localhost:8080/api/v1/startTest \
  -F platform=android -F reportType=screenshot \
  -F apk=@app-debug.apk -F email=you@taltech.ee -F repositoryName=test

See testlab-test-orchestrator/DOCKER.md for more detail.

5b. Docker Desktop container grouping

All lab containers are presented as one group named testlab in Docker Desktop, instead of one group per repository.

How it works. Docker Desktop groups containers solely by the com.docker.compose.project label, and the view is flat (no nested sub-groups). So every lab stack uses the same Compose project name:

  • Compose-based stacks set name: testlab at the top of their docker-compose.yml (testlab-proxy, testlab-android-device-portal, testlab-test-orchestrator).
  • The plain docker run sites add the label explicitly in their CI: docker run ... --label com.docker.compose.project=testlab --label com.docker.compose.service=<name> ... (testlab-documentation, testlab-projects-documentation, testlab-results-portal).

Readable sub-grouping by naming. Because Docker Desktop sorts services alphabetically within the group, the multi-service stacks use prefixed service names so related containers cluster together (the closest thing to nesting that Docker Desktop allows):

testlab
├─ android-device-portal-rethinkdb
├─ android-device-portal-stf
├─ proxy-certbot
├─ proxy-cron
├─ proxy-nginx
├─ proxy-oauth2
├─ testlab-documentation
├─ testlab-projects-documentation
├─ testlab-results-portal
└─ testlab-test-orchestrator

Network aliases keep things working after a rename. Two services are reached by their original name on the Docker network, so when they were renamed they kept that name as an alias:

Service (renamed) Network alias Used by
android-device-portal-rethinkdb rethinkdb STF → tcp://rethinkdb:28015
proxy-oauth2 oauth2-proxy nginx → proxy_pass http://oauth2-proxy:4180

Data was preserved across the project rename by: pinning the RethinkDB volume to its original name (rethinkdb-data: { name: testlab-android-device-portal_rethinkdb-data }), and keeping the proxy's nginx_conf / letsencrypt_certs volumes external: true.

Trade-offs of one shared project (this is a cosmetic grouping, not isolation):

  • docker compose down in any stack targets the whole testlab project — to stop a single service use docker compose stop <service> / docker compose rm <service>.
  • --remove-orphans is intentionally omitted from the CI up commands; with a shared project it would delete the other stacks' containers. A plain up logs harmless "orphan containers" warnings.

Applying a service rename (one-time, manual). Because all stacks share the project, you must remove the old-named containers by name and bring the renamed ones up — never docker compose down (it would hit the whole group):

# example: after renaming proxy services in docker-compose.yml
docker rm -f testlab-nginx-1 oauth2-proxy ...        # remove old-named containers
cd testlab-proxy && docker compose up -d              # creates the new-named ones

A normal CI redeploy alone cannot switch a running stack to new names (the old containers still hold the ports) — do this manual step once, after which CI works normally.

6. iOS (host-native)

iOS runs directly on macOS. Run a second orchestrator instance on the host (./gradlew bootRun, IOS_ENABLED=true is the default) and route platform=ios jobs to it. Requirements:

  • Xcode + signing under the lab team (DEVELOPMENT_TEAM=VTBHR6ZFS7). The team has a wildcard "iOS Team Provisioning Profile: ", so any bundle id signs — do not* force PRODUCT_BUNDLE_IDENTIFIER (it makes the app and UI-test runner collide and breaks signing).
  • xctesthtmlreport (HTML report) and the iOS device tools (idevicescreenshot, etc.) installed on the host.
  • Example iOS apps need a shared scheme that tests only the UI-test target (the unit-test/logic target can't run on a device) and a deployment target low enough for the test devices (e.g. 16.2).

Listing iOS devices from the Docker container

Problem: the container is Linux and can't run xctrace (a macOS/Xcode tool), so GET /api/v1/devices/ios would return an empty list straight from the container — there's no Xcode there.

Solution (same idea as adb: the host produces the data, the container reads it):

  1. The launchd bridge script on the host (scripts/adb-bridge.sh) runs xcrun xctrace list devices every ~20s and writes the output to /Users/Shared/testlab/xctrace-devices.txt. The write is atomic (write a .tmp file, then mv) so the container never reads a half-written file.
  2. That directory is mounted into the container read-only as /app/shared.
  3. The container sets IOS_DEVICES_FILE=/app/shared/xctrace-devices.txt. IOSDeviceBridge reads that file instead of running xctrace (when the variable is set) and parses names/uuids the same way.
  4. The host-native iOS instance does not set IOS_DEVICES_FILE, so it queries xctrace directly.
Mac: xctrace ──(every 20s)──► /Users/Shared/testlab/xctrace-devices.txt
                                      │ (volume, read-only)
                            Container: /app/shared/xctrace-devices.txt
                                      │
                            GET /api/v1/devices/ios  ──►  ["TalTech's iPhone 8", ...]

Result: GET /api/v1/devices/ios returns iOS device data (including names) from the container. Only online (USB-connected) devices are shown; offline ones are excluded.

Note

This only provides the device list. iOS tests still cannot run in the container (that needs Xcode on the Mac). The list is also cached (@Cacheable), so changes (a newly connected device) appear after the cache refreshes / the app restarts.

7. After a reboot — what should happen automatically

  1. Mac auto-logs-in to testlab.
  2. The launchd agent starts the adb server + socat bridge (:5038).
  3. Docker Desktop starts (login setting) and brings the orchestrator container back up (restart: unless-stopped).
  4. Devices reconnect over USB; the adb server picks them up automatically.

No manual steps. If something is off, see Troubleshooting.

8. Troubleshooting

Symptom Cause / Fix
Container adb devices empty Bridge down: launchctl list \| grep adb-bridge, check /tmp/adb-bridge.log; ensure socat listens on :5038.
Operation not permitted running the bridge script Script is under ~/Desktop (macOS TCC). Install it to /Users/Shared/testlab/.
qemu-x86_64: Could not open ld-linux... Image is arm64; Android tools are x86_64. Ensure platform: linux/amd64 + Rosetta.
Failed to delete: output ... Device or resource busy output/ is a mounted volume; the app must clear its contents, not the dir (already fixed).
/app/secrets: Read-only file system (Drive) Mount secrets/ read-write (Google writes its token cache there).
iOS: Logic Testing Unavailable Device locked/unavailable or Developer Mode off; also use a shared scheme testing only the UI-test target.
iOS: missing provisioning profile (...mobileprovision not found) Don't override PRODUCT_BUNDLE_IDENTIFIER; let the wildcard team profile sign (set only DEVELOPMENT_TEAM).
iOS: xchtmlreport: command not found brew install xctesthtmlreport.
Device unauthorized Accept "Trust this computer" on the device.
Android SDK not found in CI The GitLab runner is a macOS shell executor; write local.properties with sdk.dir (see service docs).
Feedback