Skip to main content

Test Framework Documentation

This document describes the unit testing framework setup for the Wegent project.

Overview​

The project includes comprehensive unit testing support across all modules:

  • Backend (FastAPI): pytest + pytest-asyncio + pytest-cov + pytest-mock
  • Executor (AI Agent Engine): pytest + pytest-mock + pytest-asyncio
  • Executor Manager (Task Management): pytest + pytest-mock + pytest-cov
  • Shared (Utilities): pytest + pytest-cov
  • Frontend (Next.js + React 19): Jest + @testing-library/react

Current Test Coverage​

Backend (backend/)​

  • βœ… Core security: Authentication, JWT tokens, password hashing
  • βœ… Configuration management
  • βœ… Exception handling
  • βœ… User service and models
  • βœ… GitHub repository provider
  • ⏳ API endpoints (placeholder directory exists)

Executor (executor/)​

  • βœ… Agent factory
  • βœ… Base agent classes
  • βœ… Mocked AI client interactions (Anthropic, OpenAI)

Executor Manager (executor_manager/)​

  • βœ… Base executor classes
  • βœ… Task dispatcher
  • βœ… Docker executor and utilities
  • βœ… Docker constants and configuration

Shared (shared/)​

  • βœ… Cryptography utilities
  • βœ… Sensitive data masking (tokens, API keys, etc.)

Frontend (frontend/)​

  • ⏳ Component tests (basic setup in place)
  • ⏳ Hook tests
  • ⏳ Utility tests

Test Coverage Goals​

  • Target: 40-60% code coverage initially
  • Priority: Core business logic and critical paths
  • Strategy: Incremental coverage improvement

Backend Tests​

Running Tests​

cd backend
pytest                          # Run all tests
pytest tests/core/             # Run core tests only
pytest --cov=app               # Run with coverage report
pytest -v                      # Verbose output
pytest -k test_security        # Run specific test pattern
pytest -m unit                 # Run only unit tests
pytest -m integration          # Run only integration tests

Test Structure​

backend/tests/
β”œβ”€β”€ conftest.py              # Global test fixtures
β”œβ”€β”€ core/                    # Core infrastructure tests
β”‚   β”œβ”€β”€ test_security.py     # Authentication & JWT tests
β”‚   β”œβ”€β”€ test_config.py       # Configuration tests
β”‚   └── test_exceptions.py   # Exception handler tests
β”œβ”€β”€ services/                # Service layer tests
β”‚   └── test_user_service.py # User service tests
β”œβ”€β”€ models/                  # Data model tests
β”‚   └── test_user_model.py   # User model tests
β”œβ”€β”€ repository/              # Repository integration tests
β”‚   └── test_github_provider.py
└── api/                     # API endpoint tests (placeholder)

Test Configuration​

The backend uses pytest.ini for configuration with the following settings:

[pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts =
    -v
    --strict-markers
    --cov=app
    --cov-report=term-missing
    --cov-report=html
    --cov-report=xml
asyncio_mode = auto
markers =
    unit: Unit tests
    integration: Integration tests
    slow: Slow running tests

Key Fixtures​

  • test_db: SQLite in-memory database session (function scope)
  • test_settings: Test settings with overridden values
  • test_user: Test user instance
  • test_admin_user: Test admin user instance
  • test_inactive_user: Inactive test user instance
  • test_token: Valid JWT token for test user
  • test_admin_token: Valid JWT token for admin user
  • test_client: FastAPI test client with database override
  • mock_redis: Mocked Redis client

Executor Tests​

Running Tests​

cd executor
pytest tests/ --cov=agents

Test Structure​

executor/tests/
β”œβ”€β”€ conftest.py              # Executor-specific fixtures
└── agents/                  # Agent tests

Key Fixtures​

  • mock_anthropic_client: Mocked Anthropic API client for testing Claude models
  • mock_openai_client: Mocked OpenAI API client for testing GPT models
  • mock_callback_client: Mocked callback HTTP client for agent responses
  • suppress_resource_warnings: Session-scoped fixture to suppress ResourceWarning messages
  • cleanup_logging: Session-scoped fixture to clean up logging handlers and prevent daemon thread errors

Executor Manager Tests​

Running Tests​

cd executor_manager
pytest tests/ --cov=executors

Key Fixtures​

  • mock_docker_client: Mocked Docker SDK client for container operations
  • mock_executor_config: Mock executor configuration with image, CPU, memory, and network settings

Test Structure​

executor_manager/tests/
β”œβ”€β”€ conftest.py              # Executor manager fixtures
└── executors/               # Executor tests
    β”œβ”€β”€ test_base.py
    β”œβ”€β”€ test_dispatcher.py
    β”œβ”€β”€ test_docker_executor.py
    β”œβ”€β”€ test_docker_utils.py
    └── test_docker_constants.py

Shared Tests​

Running Tests​

cd shared
pytest tests/ --cov=utils

Test Structure​

shared/tests/
└── utils/
    β”œβ”€β”€ test_crypto.py               # Encryption/decryption tests
    └── test_sensitive_data_masker.py # Sensitive data masking tests

Key Features Tested​

  • Cryptography: Encryption and decryption of sensitive data (Git tokens, API keys)
  • Data Masking: Automatic masking of sensitive information in logs and outputs
    • GitHub tokens (github_pat_*)
    • Anthropic API keys (sk-ant-api03-*)
    • OpenAI API keys
    • Generic API keys and secrets
    • File path protection (no false positives)
    • URL protection (no false positives)

Frontend Tests​

Running Tests​

cd Wegent
pnpm --dir frontend test                  # Run all tests
pnpm --dir frontend run test:watch        # Watch mode
pnpm --dir frontend run test:coverage     # With coverage report

Test Structure​

frontend/src/__tests__/
β”œβ”€β”€ utils/                   # Utility function tests
β”œβ”€β”€ hooks/                   # React hooks tests
└── components/              # Component tests

Continuous Integration​

GitHub Actions Workflow​

The test and cache workflows run automatically as follows:

  • .github/workflows/test.yml responds to PRs targeting main and merge-queue checks.
  • .github/workflows/ci-cache-warmup.yml prewarms shared caches after related dependencies, source, or CI configuration enter main.

CI first runs .github/scripts/classify-ci-changes.sh to select module jobs from the changed paths. Dependency relationships are included. For example, a shared/ change runs the Backend, Executor Manager, Knowledge Engine, Shared, and CLI suites, while a packages/chat-core/ change runs both Frontend and Wework checks. Changes to the test workflows or the classifier itself select every module so CI orchestration changes receive full validation.

Apply the ci:all label to a PR to bypass path classification and force every module test, the platform E2E suite, Wework browser and desktop E2E, and the Wework macOS memory gate. This is useful when changed paths might not capture the full blast radius of a change. All three test workflows respond to label events; without ci:all, test selection still follows the changed paths. ci:memory continues to trigger only the Wework macOS memory gate.

Feature-branch pushes do not run a duplicate CI suite; the pull_request event validates the branch after a PR is opened. A newer commit to the same PR or to main cancels the older in-progress run. test-summary and lint-summary always appear and verify that every selected job actually succeeded. Skipped, unrelated modules do not make the summary fail.

CI Cache Ownership​

Large caches are prewarmed and owned by main. Pull requests and merge-queue runs may restore the default-branch cache, but they do not save another copy:

  • Linux workspace node_modules has one shared cache keyed by the Node version and pnpm-lock.yaml. Tests, Lint, platform E2E, and Wework E2E reuse it.
  • uv caches downloads and build results only, not project .venv directories. Python 3.10, 3.11, and 3.12 each have one shared cache whose key includes the dependency lockfiles, and CI pruning runs before save.
  • Playwright browsers, the Next.js build cache, the Claude Code CLI, and desktop Cargo targets use explicit restore/save steps; only refs/heads/main saves. The Claude Code CLI uses a repository package-lock.json to pin the complete dependency graph and package integrity.
  • Rust unit tests, the Windows check, release and snapshot binaries, and the macOS memory gate use sccache. The main warmup runs the same desktop --build-only flow on macos-14; non-main jobs access the shared compiler cache in read-only mode.
  • Wework Desktop Core E2E retains its main-owned Cargo target cache because several desktop jobs must reuse the same complete binary output.
  • Platform E2E, Release, and Snapshot Docker BuildKit caches live in corresponding GHCR build-cache tags instead of consuming GitHub Actions dependency cache storage.
  • Executor and Tauri system dependency installers first check packages already present on the runner. Only main writes the APT download cache; PRs restore it read-only.

.github/workflows/ci-cache-warmup.yml runs after related source, dependency lockfile, or cache implementation changes enter main, and rejects manual warmups from non-main refs. It downloads dependencies or compiles cache entries without repeating tests already validated by the merge queue. Do not add large automatically saved PR or merge-group caches. Use actions/cache/restore plus a main-only actions/cache/save, or reuse the existing shared action.

Writing GitHub Actions Outputs​

Every line written to $GITHUB_OUTPUT must use the name=value format. Do not write the substituted output of a command that can emit progress or diagnostic messages directly to that file. Extract the needed value first, then write the output. For example, to resolve the Playwright version:

version="$(pnpm exec playwright --version | sed -n 's/^Version //p')"
echo "version=$version" >> "$GITHUB_OUTPUT"

This prevents package-manager supply-chain checks or other auxiliary logs from invalidating the workflow output format.

Workflow Jobs​

Regular PR checks must cover module tests; E2E jobs do not replace them:

  1. test-backend runs the Backend suite with Python 3.11 and uv run pytest.
  2. test-executor runs cargo fmt --check, cargo test --all-features, and Clippy for the Rust Executor. It places CARGO_TARGET_DIR under the runner's temporary directory to avoid shared build contamination and workspace disk pressure.
  3. test-executor-manager, test-shared, and test-knowledge-engine run their Python suites with uv run pytest.
  4. test-frontend runs the Chat Core and Frontend unit tests.
  5. test-wework runs the Wework unit suite with Vitest and generates coverage.
  6. test-wegent-cli and test-wegent-cli-integration run CLI unit tests and integration tests backed by MySQL, Redis, and a real Backend, respectively.
  7. test-summary always runs and depends on every job above. It must fail when any module test job fails.

E2E coverage lives in dedicated workflows: .github/workflows/e2e-tests.yml covers product end-to-end flows, while .github/workflows/wework-e2e.yml covers Wework flows. They add cross-module verification and do not replace the module tests in .github/workflows/test.yml. Platform E2E runs only when Backend, Frontend, Executor, Executor Manager, Shared, Chat Shell, Chat Core, Docker, or related build configuration changes, so a Wework-only PR no longer starts an unrelated platform E2E suite. Draft PRs skip expensive E2E and run it after they become ready for review.

The full platform E2E suite runs daily at 02:00 UTC, and the full Wework E2E suite runs daily at 04:00 UTC. Scheduled runs use a different concurrency group from PR and merge-queue runs, so they do not cancel each other.

Coverage Reports​

Coverage reports are automatically uploaded to Codecov (if configured).

Mocking Strategy​

External APIs​

  • GitHub/GitLab/Gitee: Mock with httpx-mock or pytest-mock
  • Anthropic/OpenAI: Mock SDK clients
  • Redis: Use fakeredis or mock

Database​

  • Test DB: SQLite in-memory database
  • Isolation: Each test gets a fresh transaction
  • Cleanup: Automatic rollback after each test

Docker​

  • Mock docker.from_env() and container operations

Best Practices​

Writing Tests​

  1. One assertion per test: Each test should verify one specific behavior
  2. Descriptive names: Use clear, descriptive test function names that explain what is being tested
  3. AAA pattern: Arrange, Act, Assert - structure your tests clearly
  4. Mock external dependencies: Never call real external services (APIs, databases, etc.)
  5. Use fixtures: Share common test setup via fixtures to reduce duplication
  6. Test edge cases: Include tests for error conditions, boundary values, and unusual inputs
  7. Keep tests independent: Each test should be able to run independently without relying on other tests

Security Testing Best Practices​

The project includes comprehensive security testing examples in backend/tests/core/test_security.py:

  • Password hashing and verification (bcrypt)
  • JWT token creation and validation
  • Token expiration handling
  • User authentication with valid/invalid credentials
  • Inactive user detection
  • Role-based access control (admin vs regular users)

Example test pattern for security features:

@pytest.mark.unit
class TestPasswordHashing:
    """Test password hashing and verification functions"""

    def test_verify_password_with_correct_password(self):
        """Test password verification with correct password"""
        password = "testpassword123"
        hashed = get_password_hash(password)
        assert verify_password(password, hashed) is True

    def test_verify_password_with_incorrect_password(self):
        """Test password verification with incorrect password"""
        password = "testpassword123"
        hashed = get_password_hash(password)
        assert verify_password("wrongpassword", hashed) is False

Test Organization​

@pytest.mark.unit
class TestFeatureName:
    """Test feature description"""

    def test_success_case(self):
        """Test successful operation"""
        # Arrange
        data = {"key": "value"}

        # Act
        result = function_under_test(data)

        # Assert
        assert result == expected_value

    def test_error_case(self):
        """Test error handling"""
        with pytest.raises(ExpectedException):
            function_under_test(invalid_data)

Using Test Markers​

Test markers help categorize and selectively run tests:

# Run only unit tests
pytest -m unit

# Run only integration tests
pytest -m integration

# Run slow tests
pytest -m slow

# Skip slow tests
pytest -m "not slow"

Async Tests​

@pytest.mark.asyncio
async def test_async_function():
    """Test asynchronous function"""
    result = await async_function()
    assert result is not None

The backend's pytest.ini has asyncio_mode = auto which automatically detects and runs async tests.

Adding New Tests​

Backend​

  1. Create test file in appropriate tests/ subdirectory (e.g., tests/services/test_new_service.py)
  2. Import necessary fixtures from conftest.py
  3. Use @pytest.mark.unit or @pytest.mark.integration to categorize tests
  4. Follow the AAA (Arrange-Act-Assert) pattern
  5. Write test classes and methods with descriptive names
  6. Run tests locally before committing: pytest tests/ -v
  7. Ensure coverage is maintained or improved: pytest --cov=app --cov-report=term-missing

Frontend​

  1. Create test file in src/__tests__/ matching source structure
  2. Use @testing-library/react for component tests
  3. Mock API calls and external dependencies
  4. Ensure tests pass with pnpm --filter wecode-ai-assistant test

Debugging Tests​

Backend​

# Run specific test with verbose output
pytest tests/core/test_security.py::TestPasswordHashing::test_verify_password_with_correct_password -v

# Drop into debugger on failure
pytest --pdb

# Show print statements
pytest -s

Frontend​

# Run tests in watch mode
pnpm --dir frontend run test:watch

# Debug specific test file
pnpm --dir frontend test -- src/__tests__/utils/test_example.test.ts

Configuration Files​

Backend​

  • backend/pytest.ini: pytest configuration with coverage settings and test markers
    • Enables verbose output, strict markers, and automatic async mode
    • Configures coverage reports in terminal, HTML, and XML formats
    • Defines custom markers: unit, integration, slow

Executor/Executor Manager/Shared​

  • pytest.ini: Module-specific pytest configuration
  • Similar setup to backend but with module-specific coverage targets

Frontend​

  • frontend/jest.config.ts: Jest configuration
  • frontend/jest.setup.js: Test environment setup

Future Improvements​

  • Increase coverage to 70-80%
  • Add integration tests for API endpoints (currently placeholder)
  • Add E2E tests for critical user flows
  • Performance/load testing
  • Mutation testing with mutmut
  • Add more frontend component tests
  • Implement database migration tests
  • Add tests for WebSocket connections and real-time features

Troubleshooting​

Common Issues​

Import errors in tests:

  • Ensure you're running pytest from the correct directory
  • Check that modules are installed: uv sync

Database errors:

  • Tests use SQLite in-memory DB, no setup needed
  • Check that fixtures are imported correctly

Frontend test failures:

  • Ensure Node.js 20+ is installed
  • Run pnpm install --frozen-lockfile from the repository root to install exact dependency versions
  • Clear Jest cache: pnpm --dir frontend exec jest --clearCache

Resources​