admin/mcp-nextcloud
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

test: Update oauth benchmark tests

Browse Source
This commit is contained in:
Chris Coutinho
2025-10-18 22:02:25 +02:00
parent 644c59bf78
commit 371d0c93a5
2 changed files with 68 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/load/README_OAUTH.md
+61 -33
View File
@@ -142,24 +142,35 @@ uv run python -m tests.load.oauth_benchmark -u 2 -d 30 --verbose
| Option | Short | Default | Description | | Option | Short | Default | Description |
|--------|-------|---------|-------------| |--------|-------|---------|-------------|
| `--users` | `-u` | 2 | Number of concurrent users (max 4 with default config) | | `--users` | `-u` | 2 | Number of concurrent users (dynamically created) |
| `--duration` | `-d` | 30.0 | Test duration in seconds | | `--duration` | `-d` | 30.0 | Test duration in seconds |
| `--warmup` | `-w` | 5.0 | Warmup period before metrics collection (seconds) | | `--warmup` | `-w` | 5.0 | Warmup period before metrics collection (seconds) |
| `--url` | | `http://127.0.0.1:8001/mcp` | MCP OAuth server URL | | `--url` | | `http://127.0.0.1:8001/mcp` | MCP OAuth server URL |
| `--output` | `-o` | None | JSON output file path | | `--output` | `-o` | None | JSON output file path |
| `--workload` | | `mixed` | Workload type: mixed, sharing, collaboration, baseline | | `--workload` | | `mixed` | Workload type: mixed, sharing, collaboration, baseline |
| `--user-prefix` | | `loadtest` | Prefix for dynamically created usernames |
| `--cleanup/--no-cleanup` | | `cleanup` | Delete created users after benchmark |
| `--browser` | | `chromium` | Playwright browser: firefox, chromium, webkit |
| `--headed` | | False | Run browser in headed mode (visible window) |
| `--verbose` | `-v` | False | Enable verbose logging | | `--verbose` | `-v` | False | Enable verbose logging |
## Default Test Users ## Test User Creation
The framework includes 4 pre-configured test users: The framework **dynamically creates test users** on-demand with OAuth authentication:
| Username | Display Name | Groups | Role | - **Naming**: Users are created with the pattern `{prefix}_user_{n}` (default: `loadtest_user_1`, `loadtest_user_2`, etc.)
|----------|--------------|--------|------| - **Customization**: Use `--user-prefix` to change the prefix (e.g., `--user-prefix mytest` → `mytest_user_1`)
| alice | Alice Anderson | owners | Owner - full permissions | - **Scalability**: No limit on user count - create as many concurrent users as your system can handle
| bob | Bob Brown | viewers | Viewer - read-only | - **Credentials**: Each user gets a randomly generated secure password
| charlie | Charlie Chen | editors | Editor - read/write | - **OAuth Tokens**: All users authenticate via automated OAuth flow using Playwright
| diana | Diana Davis | (none) | No special permissions | - **Cleanup**: Users are automatically deleted after the benchmark (disable with `--no-cleanup`)
**Example**: Running `--users 5` creates:
- `loadtest_user_1` (Display: Load Test User 1, Email: loadtest_user_1@benchmark.local)
- `loadtest_user_2` (Display: Load Test User 2, Email: loadtest_user_2@benchmark.local)
- `loadtest_user_3` (Display: Load Test User 3, Email: loadtest_user_3@benchmark.local)
- `loadtest_user_4` (Display: Load Test User 4, Email: loadtest_user_4@benchmark.local)
- `loadtest_user_5` (Display: Load Test User 5, Email: loadtest_user_5@benchmark.local)
## Metrics Output ## Metrics Output
@@ -171,34 +182,35 @@ OAUTH MULTI-USER BENCHMARK RESULTS
================================================================================ ================================================================================
Duration: 120.45s Duration: 120.45s
Total Users: 4 Total Users: 5
Total Workflows Executed: 247 Total Workflows Executed: 312
Total Baseline Operations: 531 Total Baseline Operations: 678
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
WORKFLOW STATISTICS WORKFLOW STATISTICS
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Workflow Total Success Rate P50 P95 Workflow Total Success Rate P50 P95
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
note_share 89 87 97.8% 0.2341s 0.4782s note_share 112 109 97.3% 0.2341s 0.4782s
collaborative_edit 52 48 92.3% 0.5123s 0.9234s collaborative_edit 65 61 93.8% 0.5123s 0.9234s
file_share 23 23 100.0% 0.3456s 0.6123s file_share 29 29 100.0% 0.3456s 0.6123s
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
PER-USER STATISTICS PER-USER STATISTICS
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
User Total Ops Success Errors Rate P50 User Total Ops Success Errors Rate P50
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
alice 234 229 5 97.9% 0.2456s loadtest_user_1 289 283 6 97.9% 0.2456s
bob 198 195 3 98.5% 0.2123s loadtest_user_2 245 241 4 98.4% 0.2123s
charlie 187 183 4 97.9% 0.2345s loadtest_user_3 231 226 5 97.8% 0.2345s
diana 159 157 2 98.7% 0.2234s loadtest_user_4 198 195 3 98.5% 0.2234s
loadtest_user_5 187 184 3 98.4% 0.2189s
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
BASELINE OPERATIONS BASELINE OPERATIONS
-------------------------------------------------------------------------------- --------------------------------------------------------------------------------
Total Operations: 531 Total Operations: 678
Success Rate: 98.1% Success Rate: 98.2%
Latency: min=0.0234s, p50=0.1234s, p95=0.3456s, max=0.8123s Latency: min=0.0234s, p50=0.1234s, p95=0.3456s, max=0.8123s
================================================================================ ================================================================================
``` ```
@@ -209,16 +221,16 @@ Latency: min=0.0234s, p50=0.1234s, p95=0.3456s, max=0.8123s
{ {
"summary": { "summary": {
"duration": 120.45, "duration": 120.45,
"total_workflows": 247, "total_workflows": 312,
"total_baseline_ops": 531, "total_baseline_ops": 678,
"total_users": 4 "total_users": 5
}, },
"workflows": { "workflows": {
"note_share": { "note_share": {
"total_executions": 89, "total_executions": 112,
"successful_executions": 87, "successful_executions": 109,
"failed_executions": 2, "failed_executions": 3,
"success_rate": 97.8, "success_rate": 97.3,
"latency": { "latency": {
"min": 0.1234, "min": 0.1234,
"max": 0.8765, "max": 0.8765,
@@ -237,15 +249,19 @@ Latency: min=0.0234s, p50=0.1234s, p95=0.3456s, max=0.8123s
} }
}, },
"users": { "users": {
"alice": { "loadtest_user_1": {
"total_operations": 234, "total_operations": 289,
"successful_operations": 229, "successful_operations": 283,
"failed_operations": 5, "failed_operations": 6,
"success_rate": 97.9, "success_rate": 97.9,
"latency": {...}, "latency": {...},
"operations_breakdown": {...}, "operations_breakdown": {...},
"errors_breakdown": {...} "errors_breakdown": {...}
} },
"loadtest_user_2": {...},
"loadtest_user_3": {...},
"loadtest_user_4": {...},
"loadtest_user_5": {...}
}, },
"baseline": {...} "baseline": {...}
} }
@@ -473,6 +489,18 @@ uv run python -m tests.load.cleanup_loadtest_users
- Ensure user count doesn't exceed configured limits - Ensure user count doesn't exceed configured limits
- Check that user creation succeeded in previous steps - Check that user creation succeeded in previous steps
### CancelledError During Benchmark
**Symptom**: Error message like `'CancelledError' object has no attribute 'username'` appears in logs
**Cause**: Async task cancellation during benchmark shutdown or errors can cause race conditions in error handling
**Solution**: This has been mitigated with defensive error handling. The worker now:
- Catches `asyncio.CancelledError` specifically before general exceptions
- Logs cancellation gracefully without attempting to access potentially invalid state
- Re-raises the exception to allow proper cleanup chain
If you still see this error, it's likely harmless and occurs during shutdown. The benchmark results should still be valid.
### High Error Rates ### High Error Rates
- Increase delay between operations (`await asyncio.sleep()` in worker) - Increase delay between operations (`await asyncio.sleep()` in worker)
- Check OAuth token validity - Check OAuth token validity
tests/load/oauth_benchmark.py
+7
View File
@@ -263,6 +263,13 @@ async def oauth_benchmark_worker(
f"Worker for {user_wrapper.username} completed {operation_count} operations" f"Worker for {user_wrapper.username} completed {operation_count} operations"
) )
except asyncio.CancelledError:
# Handle task cancellation gracefully (e.g., during benchmark shutdown)
logger.info(
f"Worker for {user_wrapper.username} was cancelled "
f"(completed {operation_count} operations)"
)
raise # Re-raise to allow proper cleanup
except Exception as e: except Exception as e:
logger.error(f"Worker {user_wrapper.username} error: {e}", exc_info=True) logger.error(f"Worker {user_wrapper.username} error: {e}", exc_info=True)