test: Initialize load testing framework
This commit is contained in:
Chris Coutinho
@@ -0,0 +1,506 @@
|
||||
# OAuth Multi-User Load Testing Framework
|
||||
|
||||
Comprehensive multi-user benchmarking system for testing OAuth-authenticated Nextcloud MCP server with realistic collaborative workflows.
|
||||
|
||||
## Quick Start
|
||||
|
||||
```bash
|
||||
# 1. Ensure docker-compose is running
|
||||
docker-compose up -d
|
||||
|
||||
# 2. Run a benchmark with 2 users for 30 seconds
|
||||
uv run python -m tests.load.oauth_benchmark --users 2 --duration 30
|
||||
|
||||
# 3. Clean up test users (IMPORTANT - always run after benchmark)
|
||||
uv run python -m tests.load.cleanup_loadtest_users
|
||||
|
||||
# Optional: Verify cleanup
|
||||
uv run python -m tests.load.cleanup_loadtest_users --dry-run
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
This framework extends the basic load testing infrastructure to support:
|
||||
- **Multiple OAuth-authenticated users** running concurrently
|
||||
- **Coordinated workflows** spanning multiple users (sharing, collaboration, permissions)
|
||||
- **Per-user metrics** tracking individual user performance
|
||||
- **Workflow-specific metrics** measuring cross-user operation latencies
|
||||
- **Realistic scenarios** mimicking actual user collaboration patterns
|
||||
- **Concurrent user creation** - all users created and authenticated in parallel for fast setup
|
||||
|
||||
## Architecture
|
||||
|
||||
### Components
|
||||
|
||||
```
|
||||
tests/load/
|
||||
├── oauth_pool.py # OAuth user pool management
|
||||
├── oauth_workloads.py # Multi-user workflow definitions
|
||||
├── oauth_metrics.py # Enhanced metrics collection
|
||||
├── oauth_benchmark.py # Main CLI entry point
|
||||
└── README_OAUTH.md # This file
|
||||
```
|
||||
|
||||
### Key Classes
|
||||
|
||||
**OAuthUserPool** (`oauth_pool.py`)
|
||||
- Manages N OAuth-authenticated users
|
||||
- Handles token acquisition and storage
|
||||
- Creates and manages MCP sessions per user
|
||||
- Tracks per-user operation statistics
|
||||
|
||||
**UserSessionWrapper** (`oauth_pool.py`)
|
||||
- Wraps MCP ClientSession for a specific user
|
||||
- Automatic operation tracking
|
||||
- Convenient tool/resource access methods
|
||||
|
||||
**Workflow** (`oauth_workloads.py`)
|
||||
- Base class for multi-user coordinated workflows
|
||||
- Step-by-step execution with timing
|
||||
- Comprehensive error handling and reporting
|
||||
|
||||
**OAuthBenchmarkMetrics** (`oauth_metrics.py`)
|
||||
- Per-user operation counts and latencies
|
||||
- Workflow completion rates and timings
|
||||
- Baseline operation statistics
|
||||
- Detailed reporting and JSON export
|
||||
|
||||
## Available Workflows
|
||||
|
||||
### 1. NoteShareWorkflow
|
||||
**Scenario**: Alice creates a note and shares it with Bob, who then reads it.
|
||||
|
||||
**Steps**:
|
||||
1. User A creates a note
|
||||
2. User A shares note with User B (read-only permissions)
|
||||
3. User B lists their shared notes (measures propagation delay)
|
||||
4. User B reads the shared note
|
||||
|
||||
**Metrics**: Creation latency, share propagation time, read latency
|
||||
|
||||
### 2. CollaborativeEditWorkflow
|
||||
**Scenario**: Multiple users concurrently edit the same note.
|
||||
|
||||
**Steps**:
|
||||
1. Owner creates a note
|
||||
2. All users read the note simultaneously
|
||||
3. All users append content concurrently
|
||||
4. Owner verifies final state
|
||||
|
||||
**Metrics**: Concurrent read latency, concurrent write conflicts, final state consistency
|
||||
|
||||
### 3. FileShareAndDownloadWorkflow
|
||||
**Scenario**: Alice uploads a file, shares it with Bob, who then downloads it.
|
||||
|
||||
**Steps**:
|
||||
1. User A creates a file via WebDAV
|
||||
2. User A shares file with User B (read-only)
|
||||
3. User B lists their shares
|
||||
4. User B downloads the file
|
||||
|
||||
**Metrics**: Upload latency, share creation, download latency
|
||||
|
||||
### 4. MixedOAuthWorkload
|
||||
**Distribution**:
|
||||
- 50% Baseline operations (individual user CRUD)
|
||||
- 30% Note sharing workflows
|
||||
- 15% Collaborative editing workflows
|
||||
- 5% File sharing workflows
|
||||
|
||||
## Usage
|
||||
|
||||
### Basic Usage
|
||||
|
||||
```bash
|
||||
# 4 users, 60-second test with mixed workload
|
||||
uv run python -m tests.load.oauth_benchmark --users 4 --duration 60
|
||||
|
||||
# 10 users, 5-minute test
|
||||
uv run python -m tests.load.oauth_benchmark -u 10 -d 300
|
||||
|
||||
# Export results to JSON
|
||||
uv run python -m tests.load.oauth_benchmark -u 5 -d 120 --output results.json
|
||||
```
|
||||
|
||||
### Advanced Options
|
||||
|
||||
```bash
|
||||
# Sharing-focused workload
|
||||
uv run python -m tests.load.oauth_benchmark --workload sharing -u 8 -d 180
|
||||
|
||||
# Collaborative editing workload
|
||||
uv run python -m tests.load.oauth_benchmark --workload collaboration -u 6 -d 120
|
||||
|
||||
# Baseline operations only (no workflows)
|
||||
uv run python -m tests.load.oauth_benchmark --workload baseline -u 10 -d 60
|
||||
|
||||
# Verbose logging for debugging
|
||||
uv run python -m tests.load.oauth_benchmark -u 2 -d 30 --verbose
|
||||
```
|
||||
|
||||
### CLI Options
|
||||
|
||||
| Option | Short | Default | Description |
|
||||
|--------|-------|---------|-------------|
|
||||
| `--users` | `-u` | 2 | Number of concurrent users (max 4 with default config) |
|
||||
| `--duration` | `-d` | 30.0 | Test duration in seconds |
|
||||
| `--warmup` | `-w` | 5.0 | Warmup period before metrics collection (seconds) |
|
||||
| `--url` | | `http://127.0.0.1:8001/mcp` | MCP OAuth server URL |
|
||||
| `--output` | `-o` | None | JSON output file path |
|
||||
| `--workload` | | `mixed` | Workload type: mixed, sharing, collaboration, baseline |
|
||||
| `--verbose` | `-v` | False | Enable verbose logging |
|
||||
|
||||
## Default Test Users
|
||||
|
||||
The framework includes 4 pre-configured test users:
|
||||
|
||||
| Username | Display Name | Groups | Role |
|
||||
|----------|--------------|--------|------|
|
||||
| alice | Alice Anderson | owners | Owner - full permissions |
|
||||
| bob | Bob Brown | viewers | Viewer - read-only |
|
||||
| charlie | Charlie Chen | editors | Editor - read/write |
|
||||
| diana | Diana Davis | (none) | No special permissions |
|
||||
|
||||
## Metrics Output
|
||||
|
||||
### Console Report
|
||||
|
||||
```
|
||||
================================================================================
|
||||
OAUTH MULTI-USER BENCHMARK RESULTS
|
||||
================================================================================
|
||||
|
||||
Duration: 120.45s
|
||||
Total Users: 4
|
||||
Total Workflows Executed: 247
|
||||
Total Baseline Operations: 531
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
WORKFLOW STATISTICS
|
||||
--------------------------------------------------------------------------------
|
||||
Workflow Total Success Rate P50 P95
|
||||
--------------------------------------------------------------------------------
|
||||
note_share 89 87 97.8% 0.2341s 0.4782s
|
||||
collaborative_edit 52 48 92.3% 0.5123s 0.9234s
|
||||
file_share 23 23 100.0% 0.3456s 0.6123s
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
PER-USER STATISTICS
|
||||
--------------------------------------------------------------------------------
|
||||
User Total Ops Success Errors Rate P50
|
||||
--------------------------------------------------------------------------------
|
||||
alice 234 229 5 97.9% 0.2456s
|
||||
bob 198 195 3 98.5% 0.2123s
|
||||
charlie 187 183 4 97.9% 0.2345s
|
||||
diana 159 157 2 98.7% 0.2234s
|
||||
|
||||
--------------------------------------------------------------------------------
|
||||
BASELINE OPERATIONS
|
||||
--------------------------------------------------------------------------------
|
||||
Total Operations: 531
|
||||
Success Rate: 98.1%
|
||||
Latency: min=0.0234s, p50=0.1234s, p95=0.3456s, max=0.8123s
|
||||
================================================================================
|
||||
```
|
||||
|
||||
### JSON Export
|
||||
|
||||
```json
|
||||
{
|
||||
"summary": {
|
||||
"duration": 120.45,
|
||||
"total_workflows": 247,
|
||||
"total_baseline_ops": 531,
|
||||
"total_users": 4
|
||||
},
|
||||
"workflows": {
|
||||
"note_share": {
|
||||
"total_executions": 89,
|
||||
"successful_executions": 87,
|
||||
"failed_executions": 2,
|
||||
"success_rate": 97.8,
|
||||
"latency": {
|
||||
"min": 0.1234,
|
||||
"max": 0.8765,
|
||||
"mean": 0.2891,
|
||||
"median": 0.2341,
|
||||
"p90": 0.4123,
|
||||
"p95": 0.4782,
|
||||
"p99": 0.7234
|
||||
},
|
||||
"step_latencies": {
|
||||
"create_note": {...},
|
||||
"share_note": {...},
|
||||
"list_shared_with_me": {...},
|
||||
"read_shared_note": {...}
|
||||
}
|
||||
}
|
||||
},
|
||||
"users": {
|
||||
"alice": {
|
||||
"total_operations": 234,
|
||||
"successful_operations": 229,
|
||||
"failed_operations": 5,
|
||||
"success_rate": 97.9,
|
||||
"latency": {...},
|
||||
"operations_breakdown": {...},
|
||||
"errors_breakdown": {...}
|
||||
}
|
||||
},
|
||||
"baseline": {...}
|
||||
}
|
||||
```
|
||||
|
||||
## Implementation Status
|
||||
|
||||
### ✅ Completed Components
|
||||
|
||||
**Framework:**
|
||||
- OAuth user pool management with dynamic user creation
|
||||
- User session wrappers with automatic tracking
|
||||
- Workflow base classes and framework
|
||||
- 3 example workflows (note share, collaborative edit, file share)
|
||||
- Enhanced metrics with per-user and workflow tracking
|
||||
- CLI interface with multiple workload options
|
||||
- Comprehensive reporting (console + JSON)
|
||||
|
||||
**OAuth Integration:**
|
||||
- ✅ Playwright browser automation for OAuth login
|
||||
- ✅ OAuth callback server for auth code capture
|
||||
- ✅ Token exchange with OIDC provider
|
||||
- ✅ OAuth token injection into MCP sessions via Authorization headers
|
||||
- ✅ Cancel scope error handling for reliable cleanup
|
||||
- ✅ Dynamic user creation and deletion via Nextcloud Users API
|
||||
|
||||
**Implementation Details:**
|
||||
The benchmark now successfully:
|
||||
1. Creates Nextcloud users dynamically with unique passwords
|
||||
2. Acquires OAuth tokens via automated Playwright browser flows
|
||||
3. Creates MCP client sessions with proper `Authorization: Bearer {token}` headers
|
||||
4. Executes coordinated multi-user workflows
|
||||
5. Tracks per-user and per-workflow metrics
|
||||
6. Provides standalone cleanup utility for test users
|
||||
|
||||
**Key Fix (oauth_pool.py:163-164)**:
|
||||
```python
|
||||
# Pass OAuth token as Authorization header
|
||||
headers = {"Authorization": f"Bearer {profile.token}"}
|
||||
streamable_context = streamablehttp_client(mcp_url, headers=headers)
|
||||
```
|
||||
|
||||
## Creating Custom Workflows
|
||||
|
||||
### Example: Permission Escalation Workflow
|
||||
|
||||
```python
|
||||
class PermissionEscalationWorkflow(Workflow):
|
||||
"""Test sharing permission changes."""
|
||||
|
||||
def __init__(self):
|
||||
super().__init__("permission_escalation")
|
||||
|
||||
async def execute(self, users: list[UserSessionWrapper]) -> WorkflowResult:
|
||||
self.start_time = time.time()
|
||||
|
||||
if len(users) < 2:
|
||||
return self._finish(False, error="Requires 2+ users")
|
||||
|
||||
owner, collaborator = users[0], users[1]
|
||||
|
||||
# Step 1: Owner creates note
|
||||
create_result = await self._execute_step(
|
||||
"create_note",
|
||||
owner,
|
||||
lambda: owner.call_tool("nc_notes_create_note", {...})
|
||||
)
|
||||
|
||||
# Step 2: Share read-only
|
||||
await self._execute_step(
|
||||
"share_readonly",
|
||||
owner,
|
||||
lambda: owner.call_tool("nc_share_create", {
|
||||
"permissions": 1 # Read-only
|
||||
})
|
||||
)
|
||||
|
||||
# Step 3: Upgrade to edit permissions
|
||||
await self._execute_step(
|
||||
"upgrade_permissions",
|
||||
owner,
|
||||
lambda: owner.call_tool("nc_share_update", {
|
||||
"permissions": 15 # Read+update+create+delete
|
||||
})
|
||||
)
|
||||
|
||||
# Step 4: Collaborator edits
|
||||
await self._execute_step(
|
||||
"collaborator_edit",
|
||||
collaborator,
|
||||
lambda: collaborator.call_tool("nc_notes_update_note", {...})
|
||||
)
|
||||
|
||||
return self._finish(success=True)
|
||||
```
|
||||
|
||||
### Registering Custom Workflows
|
||||
|
||||
```python
|
||||
# In oauth_workloads.py
|
||||
class MixedOAuthWorkload:
|
||||
def __init__(self, users: list[UserSessionWrapper]):
|
||||
self.users = users
|
||||
self.workflows = {
|
||||
"note_share": NoteShareWorkflow(),
|
||||
"collaborative_edit": CollaborativeEditWorkflow(),
|
||||
"file_share": FileShareAndDownloadWorkflow(),
|
||||
"permission_escalation": PermissionEscalationWorkflow(), # Add your workflow
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Expectations
|
||||
|
||||
### Baseline Performance (basic auth, from existing benchmarks)
|
||||
- **Throughput**: 50-200 RPS for mixed workload
|
||||
- **Latency**: p50 <100ms, p95 <500ms, p99 <1000ms
|
||||
|
||||
### OAuth Multi-User Expectations
|
||||
- **Lower throughput**: ~30-60% of baseline due to:
|
||||
- OAuth token validation overhead
|
||||
- Cross-user synchronization delays
|
||||
- Workflow coordination overhead
|
||||
- **Higher p99 latency**: Due to workflow step dependencies
|
||||
- **Focus**: End-to-end workflow completion time more important than raw RPS
|
||||
|
||||
### Common Bottlenecks
|
||||
1. **OAuth token validation**: Per-request overhead
|
||||
2. **Share propagation**: Time for shares to become visible to recipients
|
||||
3. **Concurrent edit conflicts**: ETags and conflict resolution
|
||||
4. **Permission checks**: Cross-user access validation
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Start Small**: Begin with 2-3 users to validate workflows
|
||||
2. **Monitor Errors**: Watch for permission errors and conflicts
|
||||
3. **Adjust Delays**: Tune sleep delays between operations based on server response
|
||||
4. **Profile Workflows**: Use step latencies to identify bottlenecks
|
||||
5. **Export Results**: Always export to JSON for historical comparison
|
||||
|
||||
## Performance Optimizations
|
||||
|
||||
### Concurrent User Creation
|
||||
|
||||
The benchmark creates and authenticates users **concurrently** for maximum performance:
|
||||
|
||||
**Step 5: User Creation & OAuth Authentication**
|
||||
- All N users are created in parallel using `asyncio.gather()`
|
||||
- Each user runs through the full OAuth flow simultaneously
|
||||
- Multiple Playwright browser contexts operate independently
|
||||
|
||||
**Step 6: MCP Session Creation**
|
||||
- All user sessions are created concurrently
|
||||
- OAuth tokens passed as Authorization headers to each session
|
||||
|
||||
**Performance Impact:**
|
||||
- **Sequential** (old): ~10-12s per user → 40-48s for 4 users
|
||||
- **Concurrent** (new): ~12-15s total for 4 users (3-4x speedup!)
|
||||
|
||||
Example output showing concurrent execution:
|
||||
```
|
||||
Step 5/6: Creating 4 users and acquiring OAuth tokens...
|
||||
(Running concurrently for faster setup)
|
||||
|
||||
[1/4] Creating user 'loadtest_user_1'...
|
||||
[2/4] Creating user 'loadtest_user_2'...
|
||||
[3/4] Creating user 'loadtest_user_3'...
|
||||
[4/4] Creating user 'loadtest_user_4'...
|
||||
✓ User 'loadtest_user_4' authenticated
|
||||
✓ User 'loadtest_user_2' authenticated
|
||||
✓ User 'loadtest_user_1' authenticated
|
||||
✓ User 'loadtest_user_3' authenticated
|
||||
|
||||
✓ Successfully created and authenticated 4 users
|
||||
```
|
||||
|
||||
**Implementation** (oauth_benchmark.py:402-437):
|
||||
```python
|
||||
# Create tasks for all users
|
||||
tasks = [
|
||||
create_user_task(i, browser, callback_server.auth_states)
|
||||
for i in range(num_users)
|
||||
]
|
||||
# Run all concurrently
|
||||
results = await asyncio.gather(*tasks, return_exceptions=True)
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
**Important**: Due to asyncio scoping issues with the MCP client library, automatic cleanup in the benchmark's finally block may not execute reliably. Always use the cleanup utility after running benchmarks.
|
||||
|
||||
### Cleanup Utility (Recommended)
|
||||
|
||||
Use the cleanup utility to remove test users:
|
||||
|
||||
```bash
|
||||
# Dry run - see what would be deleted
|
||||
uv run python -m tests.load.cleanup_loadtest_users --dry-run
|
||||
|
||||
# Delete all loadtest users
|
||||
uv run python -m tests.load.cleanup_loadtest_users
|
||||
|
||||
# Delete users with custom prefix
|
||||
uv run python -m tests.load.cleanup_loadtest_users --prefix mytest
|
||||
```
|
||||
|
||||
### Disable Automatic Cleanup
|
||||
|
||||
To keep test users after the benchmark for inspection:
|
||||
|
||||
```bash
|
||||
uv run python -m tests.load.oauth_benchmark --users 2 --no-cleanup
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Leftover Test Users
|
||||
**Symptom**: Test users remain in Nextcloud after benchmark crashes
|
||||
|
||||
**Solution**: Run the cleanup utility:
|
||||
```bash
|
||||
uv run python -m tests.load.cleanup_loadtest_users
|
||||
```
|
||||
|
||||
### "User X not in pool" Error
|
||||
- Ensure user count doesn't exceed configured limits
|
||||
- Check that user creation succeeded in previous steps
|
||||
|
||||
### High Error Rates
|
||||
- Increase delay between operations (`await asyncio.sleep()` in worker)
|
||||
- Check OAuth token validity
|
||||
- Verify MCP OAuth server is running and accessible (port 8001)
|
||||
- Rebuild mcp-oauth container after code changes: `docker-compose up --build -d mcp-oauth`
|
||||
|
||||
### Workflows Failing
|
||||
- Check step-by-step latencies to identify failing steps
|
||||
- Verify users have correct permissions
|
||||
- Review server logs for errors
|
||||
|
||||
### MCP Session Creation Fails (401 Unauthorized)
|
||||
**Solution**: This issue has been fixed! OAuth tokens are now properly passed as Authorization headers when creating MCP sessions.
|
||||
|
||||
If you still see 401 errors:
|
||||
- Rebuild the mcp-oauth container: `docker-compose up --build -d mcp-oauth`
|
||||
- Verify OAuth tokens are being acquired successfully in verbose mode
|
||||
- Check that the token hasn't expired (use shorter test durations during troubleshooting)
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
- [x] Dynamic user creation (beyond 4 default users) - **COMPLETED**
|
||||
- [x] OAuth token injection for MCP sessions - **COMPLETED**
|
||||
- [x] Cancel scope error handling - **COMPLETED**
|
||||
- [x] Concurrent user creation and authentication - **COMPLETED** (3-4x speedup!)
|
||||
- [ ] Workflow templates for common patterns
|
||||
- [ ] Real-time dashboard for live monitoring
|
||||
- [ ] Historical comparison and regression detection
|
||||
- [ ] Load ramping (gradual user increase)
|
||||
- [ ] Geographic distribution simulation (latency injection)
|
||||
- [ ] Improve cleanup reliability in finally block
|
||||
Reference in New Issue
Block a user