Testing Private Repository Access¶

This document explains how to verify that the GitHub and GitLab connectors work correctly with private repositories and projects.

Overview¶

The connectors fully support private repositories when provided with tokens that have the appropriate permissions:

GitHub: Requires repo scope for private repository access
GitLab: Requires read_api and read_repository scopes for private project access

Quick Test¶

Run the example script to verify private repository access:

# Set up GitHub credentials
export GITHUB_TOKEN=ghp_your_token_with_repo_scope
export GITHUB_PRIVATE_REPO=your-username/your-private-repo

# Set up GitLab credentials
export GITLAB_TOKEN=glpat_your_token
export GITLAB_PRIVATE_PROJECT=your-group/your-private-project

# Run the test
python examples/private_repo_example.py

Detailed Testing¶

GitHub Private Repository Testing¶

1. Create a GitHub Token¶

Go to https://github.com/settings/tokens
Click "Generate new token" → "Generate new token (classic)"
Give it a descriptive name: "MergeStat Private Repo Test"
Select scopes:
✅ repo (Full control of private repositories)
✅ read:org (optional, for organization repos)
Click "Generate token" and copy it immediately

2. Set Environment Variables¶

# Your GitHub token (with 'repo' scope)
export GITHUB_TOKEN=ghp_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

# Your private repository (format: owner/repo)
export GITHUB_PRIVATE_REPO=myusername/my-private-repo

3. Run Tests¶

Option A: Run the example script

python examples/private_repo_example.py

Option B: Run integration tests

pytest tests/test_private_repo_access.py::TestGitHubPrivateRepoAccess -v

Option C: Test with actual data sync

python cli.py sync git --provider github \
  --db "postgresql+asyncpg://localhost:5432/mergestat" \
  --auth "$GITHUB_TOKEN" \
  --owner myusername \
  --repo my-private-repo

4. Verify Success¶

The test should:

✅ Successfully list the private repository
✅ Fetch repository statistics
✅ Get contributors list
✅ Access pull requests
✅ Check rate limit status

GitLab Private Project Testing¶

1. Create a GitLab Token¶

Go to your GitLab instance Settings → Access Tokens
Create a new token:
Name: "MergeStat Private Project Test"
Scopes:
- ✅ read_api (Read API access)
- ✅ read_repository (Read repository content)
Click "Create personal access token" and copy it

2. Set Environment Variables¶

# Your GitLab token (with required scopes)
export GITLAB_TOKEN=glpat-xxxxxxxxxxxxxxxxxxxx

# Your private project (format: group/project or numeric ID)
export GITLAB_PRIVATE_PROJECT=mygroup/my-private-project
# OR
export GITLAB_PRIVATE_PROJECT=12345

# Optional: Custom GitLab instance URL
export GITLAB_URL=https://gitlab.example.com

3. Run Tests¶

Option A: Run the example script

python examples/private_repo_example.py

Option B: Run integration tests

pytest tests/test_private_repo_access.py::TestGitLabPrivateProjectAccess -v

Option C: Test with actual data sync

python cli.py sync git --provider gitlab \
  --db "postgresql+asyncpg://localhost:5432/mergestat" \
  --auth "$GITLAB_TOKEN" \
  --project-id 12345

4. Verify Success¶

The test should:

✅ Successfully access the private project
✅ Fetch project statistics
✅ Get contributors list
✅ Access merge requests
✅ List accessible projects (including private)

Troubleshooting¶

GitHub Issues¶

Problem: 404 Not Found error

Cause: Token doesn't have access to the repository or lacks repo scope
Solution:
Verify the repository exists and you have access
Check token has repo scope at https://github.com/settings/tokens
Generate a new token with correct scopes if needed

Problem: 401 Unauthorized error

Cause: Invalid or expired token
Solution: Generate a new token

Problem: Rate limit exceeded

Cause: Too many API requests
Solution: Wait for rate limit to reset or use a different token

GitLab Issues¶

Problem: 404 Not Found or 401 Unauthorized error

Cause: Token doesn't have access to the project or lacks required scopes
Solution:
Verify the project exists and you have access
Check token has read_api and read_repository scopes
Generate a new token with correct scopes if needed

Problem: Project ID vs Project Path

Cause: Using project path when numeric ID is expected or vice versa
Solution:
For cli.py sync git --provider gitlab: Use numeric project ID
For connector methods: Can use either project name or ID

Problem: Self-hosted GitLab connection issues

Cause: Incorrect URL or network issues
Solution: Verify GITLAB_URL is correct and accessible

Verifying Token Scopes¶

GitHub Token Verification¶

from connectors import GitHubConnector

token = "ghp_your_token"
connector = GitHubConnector(token=token)

# If this works, token is valid
rate_limit = connector.get_rate_limit()
print(f"Token is valid. Rate limit: {rate_limit['remaining']}/{rate_limit['limit']}")

# If you can list private repos, token has 'repo' scope
repos = connector.list_repositories(max_repos=5)
print(f"Can access {len(repos)} repositories")

connector.close()

GitLab Token Verification¶

from connectors import GitLabConnector

token = "glpat_your_token"
connector = GitLabConnector(url="https://gitlab.com", private_token=token)

# If this works, token is valid
try:
    projects = connector.list_projects(max_projects=5)
    print(f"Token is valid. Can access {len(projects)} projects")
except Exception as e:
    print(f"Token validation failed: {e}")

connector.close()

Best Practices¶

Never commit tokens to version control
Use environment variables
Add tokens to .gitignore
Use secret management tools in production
Use minimal required scopes
GitHub: repo for private repos, no scope needed for public
GitLab: read_api + read_repository for private projects
Rotate tokens regularly
Generate new tokens periodically
Revoke old tokens when no longer needed
Test with both public and private repositories
Ensure your setup works for both cases
Verify error handling for insufficient permissions
Monitor rate limits
Check rate limit status regularly
Implement backoff strategies for production use

Integration Tests¶

Run the full test suite:

# Run all private repo tests
pytest tests/test_private_repo_access.py -v

# Run specific test classes
pytest tests/test_private_repo_access.py::TestGitHubPrivateRepoAccess -v
pytest tests/test_private_repo_access.py::TestGitLabPrivateProjectAccess -v

# Skip integration tests in CI/CD
export SKIP_INTEGRATION_TESTS=1
pytest tests/test_private_repo_access.py -v

CI/CD Considerations¶

When running tests in CI/CD pipelines:

Store tokens as secrets
GitHub Actions: Use repository secrets
GitLab CI: Use CI/CD variables (masked)
Other CI: Use secure secret storage
Skip integration tests by default

# GitHub Actions example
- name: Run tests
  env:
    SKIP_INTEGRATION_TESTS: 1
  run: pytest

Optional: Run integration tests with secrets

# Only run on main branch or with specific label
- name: Run integration tests
  if: github.ref == 'refs/heads/main'
  env:
    GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
    GITHUB_PRIVATE_REPO: ${{ secrets.TEST_PRIVATE_REPO }}
  run: pytest tests/test_private_repo_access.py -v

Support¶

If you encounter issues:

Check this documentation for troubleshooting steps
Verify your token scopes and permissions
Review the connector documentation in connectors/README.md
Check example scripts in examples/ directory
Review test files for usage patterns