Skip to content

Enterprise Features Manual Test Plan

Document Status: Active
Last Updated: 2026-02-25
Related Issues: #338, #299, CHAOS-512, CHAOS-516

This document defines step-by-step manual test procedures for validating enterprise features across SaaS and self-hosted deployments.


Table of Contents

  1. Test Environment Setup
  2. License Validation
  3. RBAC Enforcement
  4. SSO Authentication
  5. Audit Logging
  6. Retention Policies
  7. IP Allowlisting
  8. Limit Enforcement
  9. Admin Portal
  10. Deployment-Specific Tests
  11. Execution Workflow and Batches (CHAOS-516)

11. Execution Workflow and Batches (CHAOS-516)

This section converts the full manual plan into executable batches. Existing section test cases remain unchanged and are executed through the batch workflow below.

11.1 Phase Order

  • Phase A - Foundation: Batch A1
  • Phase B - Access and Identity: Batch B1, Batch B2
  • Phase C - Enterprise Controls: Batch C1, Batch C2, Batch C3
  • Phase D - Limits and Operations: Batch D1, Batch D2

11.2 Batch Catalog

Batch Scope (Existing Sections) Required Prerequisites Owner Evidence Template ID Exit Criteria
A1 Section 1 + Section 2 + Section 10.1-10.3 Env vars set, fixtures created, deployment target selected QA Lead EV-BATCH-A1 Licensing and deployment baseline validated
B1 Section 3 + Section 9.1-9.3 A1 complete, owner/admin/editor/viewer accounts active QA + Security EV-BATCH-B1 RBAC and admin surface behavior validated
B2 Section 4 B1 complete, IdP sandbox available, enterprise entitlement active QA + IAM Owner EV-BATCH-B2 SAML/OIDC flows and SSO edge cases validated
C1 Section 5 B1 complete, enterprise entitlement active QA + Compliance EV-BATCH-C1 Audit generation/query/detail/gating validated
C2 Section 6 C1 complete, retention test dataset seeded QA + Data Governance EV-BATCH-C2 Retention lifecycle behavior validated
C3 Section 7 B1 complete, test CIDRs and request replay setup available QA + Security EV-BATCH-C3 Allowlist CRUD and enforcement validated
D1 Section 8 A1 complete, tier-specific org fixtures prepared QA + Platform EV-BATCH-D1 Tier limits and rate behaviors validated
D2 Section 9.4 + regression sweep across Sections 2-9 B2/C1/C2/C3/D1 complete QA Lead EV-BATCH-D2 Credential flow and final regression signoff complete

11.3 Batch Prerequisite Checklist

Checkpoint Description Owner Evidence Required Status
PR-1 Test environment initialized (Section 1.1-1.4) QA Lead Environment export, startup logs, health checks [ ]
PR-2 Tier fixtures and account roles prepared QA + Platform Entitlements output + user/org seed logs [ ]
PR-3 IdP integrations available for SAML/OIDC IAM Owner Provider metadata and callback endpoint verification [ ]
PR-4 Audit/retention datasets seeded for delete/query checks Compliance Record counts before execution [ ]
PR-5 Network paths available for allowlist scenarios Security Source IP inventory and test route capture [ ]

11.4 Evidence Capture Template

Use one record per executed batch and attach it to the issue comment or test run note.

Field Value
Template ID EV-BATCH-<BATCH-ID>
Linear Issue CHAOS-516
Parent Issue CHAOS-512
Batch ID
Sections Covered
Owner
Support Roles
Environment SaaS / Self-Hosted / Both
Build/Commit Ref
Start Time (UTC)
End Time (UTC)
Result Pass / Fail / Blocked
Blocked By CHAOS-176, CHAOS-184, CHAOS-185, CHAOS-136, or none
Evidence Links API responses, logs, screenshots, exports
Notes / Deviations

11.5 Explicit Blocked-By Mapping

Blocked Area Affected Batches Blocked By (Linear) Dependency Reason Unblock Validation
SSO provider provisioning and login flows (Section 4) B2, D2 CHAOS-176 Upstream SSO plumbing and provider lifecycle must be stable before full manual execution Provider create/activate/login paths complete without known blocker tags
Audit event visibility and query fidelity (Section 5) C1, D2 CHAOS-184 Audit event schema/query behavior must match expected contract for manual validation Audit list/detail/filter endpoints return expected event payloads
Retention execution + legal hold path (Section 6) C2, D2 CHAOS-185 Retention policy execution and hold semantics require upstream completion Execute/delete/history behavior matches expected outcomes without bypasses
IP allowlist enforcement and edge behavior (Section 7) C3, D2 CHAOS-136 Allowlist middleware/control-path needs upstream fixes before deterministic testing Allowed/blocked decisions consistently match configured CIDRs

11.6 Batch Execution Tracker

Batch Planned Window Owner Result Evidence ID Blocked By Notes
A1 [ ] Pass [ ] Fail [ ] Blocked
B1 [ ] Pass [ ] Fail [ ] Blocked
B2 [ ] Pass [ ] Fail [ ] Blocked
C1 [ ] Pass [ ] Fail [ ] Blocked
C2 [ ] Pass [ ] Fail [ ] Blocked
C3 [ ] Pass [ ] Fail [ ] Blocked
D1 [ ] Pass [ ] Fail [ ] Blocked
D2 [ ] Pass [ ] Fail [ ] Blocked

1. Test Environment Setup

1.1 Required Environment Variables

SaaS (Primary) Configuration

# Core SaaS configuration
export DATABASE_URI="clickhouse://localhost:8123/default"
export POSTGRES_URI="postgresql+asyncpg://postgres:postgres@localhost:5432/devhealth"
export AUTH_SECRET="$(openssl rand -base64 32)"

# SSO test IdP (optional, for SSO tests)
export SSO_TEST_IDP_URL="https://idp.test.local"

Self-Hosted Testing Configuration

# License configuration (for self-hosted enterprise features)
export LICENSE_PUBLIC_KEY="<base64-encoded-ed25519-public-key>"
export LICENSE_KEY="<valid-license-key>"  # Generate from ops CLI (see below)

Note: SaaS deployments manage entitlements via Stripe webhooks directly to dev-health-ops (no license key needed). License keys are for self-hosted testing only.

1.2 Test License Keys

SaaS Entitlements: In SaaS deployments, entitlements are managed via Stripe webhooks sent directly to dev-health-ops at /api/v1/billing/webhooks/stripe. No license key environment variable is needed. The license key generation below is for self-hosted testing only.

Generate test licenses for each tier using the dev-health-ops CLI:

# Generate an Ed25519 keypair (one-time setup)
python -m dev_health_ops.cli admin licenses keygen

# Community tier (no license needed - default behavior)
unset LICENSE_KEY

# Team tier license
export LICENSE_KEY="$(python -m dev_health_ops.cli admin licenses create --tier team --org test-org --days 30)"

# Enterprise tier license
export LICENSE_KEY="$(python -m dev_health_ops.cli admin licenses create --tier enterprise --org test-org --days 30)"

# Expired license (for grace period testing)
export LICENSE_KEY="$(python -m dev_health_ops.cli admin licenses create --tier enterprise --org test-org --days -1)"

# Hard-expired license (past grace period)
export LICENSE_KEY="$(python -m dev_health_ops.cli admin licenses create --tier enterprise --org test-org --days -45)"

1.3 Test Data Setup

# Create admin user
python -m dev_health_ops.cli admin users create \
  --email admin@test.local \
  --password "TestPassword123!" \
  --superuser

# Create test organization
python -m dev_health_ops.cli admin orgs create \
  --name "Test Organization" \
  --owner-email admin@test.local \
  --tier enterprise

# Create additional test users
python -m dev_health_ops.cli admin users create \
  --email viewer@test.local \
  --password "ViewerPass123!"

python -m dev_health_ops.cli admin users create \
  --email editor@test.local \
  --password "EditorPass123!"

1.4 Start Services

Docker Compose:

cd dev-health-ops
docker compose up -d
alembic upgrade head
dev-hops api --db "$DATABASE_URI" --reload

Kubernetes:

helm install dev-health fullchaos/dev-health-platform \
  -f values-test.yaml \
  --namespace dev-health-test \
  --create-namespace


2. License Validation

2.1 Community Tier (No License)

Step Action Expected Result Pass/Fail
2.1.1 Unset LICENSE_KEY and restart API API starts successfully [ ]
2.1.2 Call GET /api/v1/entitlements Returns tier: "community" [ ]
2.1.3 Verify feature flags sso: false, audit_log: false, retention_policies: false [ ]
2.1.4 Verify limits users: 5, repos: 3, api_rate: 60 [ ]

2.2 Team Tier License

Step Action Expected Result Pass/Fail
2.2.1 Set valid Team license and restart API starts, logs "License validated" [ ]
2.2.2 Call GET /api/v1/entitlements Returns tier: "team" [ ]
2.2.3 Verify feature flags team_dashboard: true, sso: false, audit_log: false [ ]
2.2.4 Verify limits users: 25, repos: 20, api_rate: 300 [ ]

2.3 Enterprise Tier License

Step Action Expected Result Pass/Fail
2.3.1 Set valid Enterprise license and restart API starts, logs "License validated" [ ]
2.3.2 Call GET /api/v1/entitlements Returns tier: "enterprise" [ ]
2.3.3 Verify feature flags All enterprise features true [ ]
2.3.4 Verify limits All limits -1 (unlimited) [ ]

2.4 Grace Period Behavior

Step Action Expected Result Pass/Fail
2.4.1 Set expired license (within grace period) API starts with warning banner [ ]
2.4.2 Call GET /api/v1/entitlements in_grace_period: true, features still active [ ]
2.4.3 Check audit logs "license_grace_period_entered" event logged [ ]
2.4.4 Set hard-expired license API starts, downgrades to community [ ]
2.4.5 Call enterprise-gated endpoint Returns 402 Payment Required [ ]

2.5 Invalid License Handling

Step Action Expected Result Pass/Fail
2.5.1 Set malformed license key API starts, falls back to community [ ]
2.5.2 Set license with invalid signature API starts, falls back to community [ ]
2.5.3 Set license with wrong issuer API starts, falls back to community [ ]
2.5.4 Check logs "License validation failed" with error details [ ]

3. RBAC Enforcement

3.1 Role Definitions

Role Permissions Test User
owner Full control, billing, org deletion admin@test.local
admin Manage users, settings, full data -
editor Modify teams, run syncs, write access editor@test.local
viewer Read-only analytics viewer@test.local

3.2 Permission Checks

Step Action Expected Result Pass/Fail
3.2.1 Login as viewer@test.local Login succeeds, JWT includes role: viewer [ ]
3.2.2 Call GET /api/v1/metrics Returns 200 OK with data [ ]
3.2.3 Call POST /api/v1/admin/settings Returns 403 Forbidden [ ]
3.2.4 Call POST /api/v1/admin/users Returns 403 Forbidden [ ]

3.3 Role Assignment

Step Action Expected Result Pass/Fail
3.3.1 Login as owner Login succeeds [ ]
3.3.2 Add viewer@test.local to org as editor POST /api/v1/admin/orgs/{id}/memberships succeeds [ ]
3.3.3 Login as viewer (now editor) JWT shows updated role [ ]
3.3.4 Test editor permissions Can run syncs, cannot manage users [ ]

3.4 Ownership Transfer

Step Action Expected Result Pass/Fail
3.4.1 Transfer ownership from admin to editor POST /api/v1/admin/orgs/{id}/transfer-ownership succeeds [ ]
3.4.2 Verify old owner demoted Previous owner now has admin role [ ]
3.4.3 Verify new owner promoted New user has owner role [ ]

4. SSO Authentication

Prerequisite: Enterprise license with sso: true feature enabled

4.1 SAML Provider Setup

Step Action Expected Result Pass/Fail
4.1.1 Call POST /api/v1/auth/sso/providers with SAML config Provider created, status: pending [ ]
4.1.2 Configure IdP with SP metadata IdP accepts metadata [ ]
4.1.3 Activate provider POST /sso/providers/{id}/activate returns active provider [ ]
4.1.4 List providers GET /api/v1/auth/sso/providers shows active provider [ ]

SAML Provider Create Payload:

{
  "name": "Test SAML IdP",
  "protocol": "saml",
  "saml_config": {
    "entity_id": "https://idp.test.local/metadata",
    "sso_url": "https://idp.test.local/sso",
    "certificate": "-----BEGIN CERTIFICATE-----\n...\n-----END CERTIFICATE-----",
    "attribute_mapping": {
      "email": "email",
      "full_name": "displayName"
    }
  },
  "auto_provision_users": true,
  "default_role": "viewer"
}

4.2 SAML Login Flow

Step Action Expected Result Pass/Fail
4.2.1 Initiate SAML login POST /api/v1/auth/sso/saml/init returns redirect URL [ ]
4.2.2 Authenticate at IdP User authenticates successfully [ ]
4.2.3 Process SAML callback POST /api/v1/auth/sso/saml/acs returns JWT [ ]
4.2.4 Verify user created New user exists with SSO link [ ]
4.2.5 Verify JWT claims JWT contains org_id, role, email [ ]

4.3 OIDC Provider Setup

Step Action Expected Result Pass/Fail
4.3.1 Call POST /api/v1/auth/sso/providers with OIDC config Provider created [ ]
4.3.2 Verify discovery endpoint fetched Provider has token_endpoint, userinfo_endpoint [ ]
4.3.3 Activate provider Provider status becomes active [ ]

OIDC Provider Create Payload:

{
  "name": "Test OIDC IdP",
  "protocol": "oidc",
  "oidc_config": {
    "client_id": "dev-health-client",
    "client_secret": "test-secret",
    "issuer": "https://oidc.test.local",
    "scopes": ["openid", "email", "profile"]
  },
  "auto_provision_users": true,
  "default_role": "viewer"
}

4.4 OIDC Login Flow

Step Action Expected Result Pass/Fail
4.4.1 Initiate OIDC login POST /api/v1/auth/sso/oidc/init returns auth URL [ ]
4.4.2 Authenticate at IdP User authenticates, grants consent [ ]
4.4.3 Process OIDC callback POST /api/v1/auth/sso/oidc/callback with code [ ]
4.4.4 Verify token exchange Access token obtained, user info fetched [ ]
4.4.5 Verify JWT issued User logged in, JWT returned [ ]

4.5 SSO Edge Cases

Step Action Expected Result Pass/Fail
4.5.1 SSO login without enterprise license Returns 402 Payment Required [ ]
4.5.2 SAML response with invalid signature Returns 401, login fails [ ]
4.5.3 OIDC callback with invalid state Returns 400, login fails [ ]
4.5.4 User from disallowed domain Returns 403 if domain filtering enabled [ ]
4.5.5 SSO provider disabled Login redirects to standard auth [ ]

5. Audit Logging

Prerequisite: Enterprise license with audit_log: true feature enabled

5.1 Audit Event Generation

Step Action Expected Result Pass/Fail
5.1.1 User logs in Audit event: user.login [ ]
5.1.2 User created Audit event: user.created [ ]
5.1.3 Setting changed Audit event: setting.updated with changes [ ]
5.1.4 SSO provider configured Audit event: sso_provider.created [ ]
5.1.5 License validated Audit event: license_validated [ ]

5.2 Audit Log Queries

Step Action Expected Result Pass/Fail
5.2.1 List all logs GET /api/v1/admin/audit-logs returns paginated results [ ]
5.2.2 Filter by user ?user_id=xxx returns only that user's actions [ ]
5.2.3 Filter by action ?action=user.login returns login events [ ]
5.2.4 Filter by date range ?start_date=...&end_date=... works [ ]
5.2.5 Filter by resource ?resource_type=user&resource_id=xxx works [ ]

5.3 Audit Log Details

Step Action Expected Result Pass/Fail
5.3.1 Get single audit log GET /api/v1/admin/audit-logs/{id} returns full details [ ]
5.3.2 Verify changes captured changes field shows before/after values [ ]
5.3.3 Verify request metadata IP address, user agent captured [ ]
5.3.4 Get resource history GET /api/v1/admin/audit-logs/resource/{type}/{id} works [ ]
5.3.5 Get user activity GET /api/v1/admin/audit-logs/user/{id} works [ ]

5.4 Audit Feature Gating

Step Action Expected Result Pass/Fail
5.4.1 Query audit logs without enterprise license Returns 402 Payment Required [ ]
5.4.2 Events still logged internally Logs contain audit entries [ ]
5.4.3 Upgrade to enterprise Audit history becomes accessible [ ]

6. Retention Policies

Prerequisite: Enterprise license with retention_policies: true feature enabled

6.1 Policy Management

Step Action Expected Result Pass/Fail
6.1.1 List available resource types GET /api/v1/admin/retention-policies/resource-types [ ]
6.1.2 Create retention policy POST /api/v1/admin/retention-policies succeeds [ ]
6.1.3 List policies GET /api/v1/admin/retention-policies shows new policy [ ]
6.1.4 Update policy PATCH /api/v1/admin/retention-policies/{id} updates retention_days [ ]
6.1.5 Deactivate policy Set is_active: false, policy stops running [ ]

Create Policy Payload:

{
  "resource_type": "audit_log",
  "retention_days": 90,
  "description": "Keep audit logs for 90 days"
}

6.2 Policy Execution

Step Action Expected Result Pass/Fail
6.2.1 Manually execute policy POST /api/v1/admin/retention-policies/{id}/execute [ ]
6.2.2 Verify records deleted Response shows deleted_count [ ]
6.2.3 Verify policy updated last_run_at, last_run_deleted_count updated [ ]
6.2.4 Delete policy DELETE /api/v1/admin/retention-policies/{id} [ ]
Step Action Expected Result Pass/Fail
6.3.1 Create legal hold Policy execution skips affected data [ ]
6.3.2 Release legal hold Data becomes eligible for deletion [ ]

7. IP Allowlisting

Prerequisite: Enterprise license with ip_allowlist: true feature enabled

7.1 Allowlist Management

Step Action Expected Result Pass/Fail
7.1.1 Add IP range POST /api/v1/admin/ip-allowlist with CIDR [ ]
7.1.2 List entries GET /api/v1/admin/ip-allowlist shows entries [ ]
7.1.3 Update entry PATCH /api/v1/admin/ip-allowlist/{id} works [ ]
7.1.4 Deactivate entry Set is_active: false [ ]
7.1.5 Delete entry DELETE /api/v1/admin/ip-allowlist/{id} [ ]

Add IP Range Payload:

{
  "ip_range": "192.168.1.0/24",
  "description": "Office network"
}

7.2 IP Checking

Step Action Expected Result Pass/Fail
7.2.1 Check allowed IP POST /api/v1/admin/ip-allowlist/check returns allowed: true [ ]
7.2.2 Check blocked IP Returns allowed: false [ ]
7.2.3 Check with expired entry Expired entries not considered [ ]

8. Limit Enforcement

8.1 User Limits

Step Action Expected Result Pass/Fail
8.1.1 Community tier: add 6th user Returns 402 with limit_exceeded [ ]
8.1.2 Team tier: add 26th user Returns 402 with limit_exceeded [ ]
8.1.3 Enterprise tier: unlimited users User creation succeeds [ ]
8.1.4 Check audit log limit_exceeded event logged [ ]

8.2 Repository Limits

Step Action Expected Result Pass/Fail
8.2.1 Community tier: sync 4th repo Returns 402 with limit_exceeded [ ]
8.2.2 Team tier: sync 21st repo Returns 402 with limit_exceeded [ ]
8.2.3 Enterprise tier: unlimited repos Sync succeeds [ ]

8.3 API Rate Limits

Step Action Expected Result Pass/Fail
8.3.1 Community tier: exceed 60 req/min Returns 429 Too Many Requests [ ]
8.3.2 Team tier: exceed 300 req/min Returns 429 [ ]
8.3.3 Enterprise tier: no rate limit All requests succeed [ ]

9. Admin Portal

9.1 User Management

Step Action Expected Result Pass/Fail
9.1.1 List users GET /api/v1/admin/users returns all org users [ ]
9.1.2 Create user POST /api/v1/admin/users creates user [ ]
9.1.3 Update user PATCH /api/v1/admin/users/{id} updates user [ ]
9.1.4 Set password POST /api/v1/admin/users/{id}/set-password works [ ]
9.1.5 Deactivate user User cannot log in [ ]

9.2 Organization Management

Step Action Expected Result Pass/Fail
9.2.1 Get org details GET /api/v1/admin/orgs/{id} returns org [ ]
9.2.2 Update org PATCH /api/v1/admin/orgs/{id} updates settings [ ]
9.2.3 List memberships GET /api/v1/admin/orgs/{id}/memberships works [ ]
9.2.4 Add member POST /api/v1/admin/orgs/{id}/memberships works [ ]
9.2.5 Remove member DELETE /api/v1/admin/memberships/{id} works [ ]

9.3 Settings Management

Step Action Expected Result Pass/Fail
9.3.1 List setting categories GET /api/v1/admin/settings/categories [ ]
9.3.2 Get settings by category GET /api/v1/admin/settings/{category} [ ]
9.3.3 Set a setting PUT /api/v1/admin/settings/{category}/{key} [ ]
9.3.4 Set encrypted setting Setting stored encrypted, displayed as [ENCRYPTED] [ ]
9.3.5 Delete setting DELETE /api/v1/admin/settings/{category}/{key} [ ]

9.4 Integration Credentials

Step Action Expected Result Pass/Fail
9.4.1 Add GitHub credential POST /api/v1/admin/credentials stores token [ ]
9.4.2 Test connection POST /api/v1/admin/credentials/{id}/test validates token [ ]
9.4.3 List credentials Secrets masked in response [ ]
9.4.4 Update credential Token updated, old token invalidated [ ]
9.4.5 Delete credential Credential removed [ ]

10. Deployment-Specific Tests

SaaS (Primary): Sections 10.1 (Docker Compose) and 10.2 (Kubernetes) cover the primary SaaS deployment paths. Section 10.3 covers self-hosted enterprise deployments.

10.1 Docker Compose

Step Action Expected Result Pass/Fail
10.1.1 Fresh install docker compose up -d starts all services [ ]
10.1.2 Migrations run alembic upgrade head completes [ ]
10.1.3 API accessible curl http://localhost:8000/health returns OK [ ]
10.1.4 Web accessible curl http://localhost:3000 returns HTML [ ]
10.1.5 Grafana accessible curl http://localhost:3001 returns login page [ ]
10.1.6 License applied Restart with LICENSE_KEY env var works [ ]
10.1.7 Data persists Restart containers, data still present [ ]

10.2 Kubernetes (Helm)

Step Action Expected Result Pass/Fail
10.2.1 Helm install helm install completes without errors [ ]
10.2.2 Pods running All pods reach Ready state [ ]
10.2.3 Ingress configured External URL accessible [ ]
10.2.4 License via secret Create Kubernetes secret, pods pick up license [ ]
10.2.5 Rolling update helm upgrade completes without downtime [ ]
10.2.6 Horizontal scaling Increase replicas, load balanced [ ]

10.3 Self-Hosted Enterprise

Step Action Expected Result Pass/Fail
10.3.1 Offline installation No internet required after images pulled [ ]
10.3.2 License offline validation Public key embedded, no phone-home [ ]
10.3.3 Custom database Connect to external PostgreSQL/ClickHouse [ ]
10.3.4 TLS termination Works behind reverse proxy with TLS [ ]

Test Result Summary

Section Tests Passed Failed Blocked
License Validation 16
RBAC Enforcement 11
SSO Authentication 20
Audit Logging 14
Retention Policies 9
IP Allowlisting 7
Limit Enforcement 10
Admin Portal 19
Deployment Tests 13
Total 119

Pass/Fail Criteria

  • Pass: All steps complete successfully with expected results
  • Fail: Any step produces unexpected result or error
  • Blocked: Cannot execute due to missing dependency or environment issue

Overall Pass Criteria

  • All critical path tests (licensing, auth, RBAC) must pass
  • No security-related tests may fail
  • At least 95% of total tests must pass