# System Health Monitoring

## Overview
The System Health widget provides real-time monitoring of critical system metrics on the admin dashboard, including queue status, error tracking, and open reports.

## Features

### 1. Queue Status Monitoring
- **Healthy**: Jobs are processing normally (last processed < 10 minutes ago)
- **Delayed**: Jobs are backing up (last processed 10-30 minutes ago)
- **Stalled**: Queue is not processing (> 100 pending jobs OR last processed > 30 minutes ago)
- Displays: pending jobs count and last processed timestamp

### 2. Error Tracking (24 hours)
- Counts errors logged in the last 24 hours
- Severity levels: Low (0-5), Medium (6-25), High (25+)
- Shows last error timestamp
- Auto-logs critical exceptions to database

### 3. Open Reports
- Tracks unresolved employer reports
- Tracks unresolved content reports (if table exists)
- Severity: Low (< 10), Medium (10-50), High (50+)
- Shows split: employer vs content reports

### 4. Overall System Status
- **Healthy**: All systems operating normally
- **Degraded**: One or more metrics showing medium severity
- **Critical**: Queue stalled OR high error volume OR > 100 open reports

## Components

### Database Tables
1. **app_errors** - Stores application errors for health monitoring
2. **queue_health** - Tracks queue processing metrics
3. **employer_reports** - Existing reports table (uses status field)

### Models
- `App\Models\AppError` - Error logging model
- `App\Models\QueueHealth` - Queue health tracking model
- `App\Models\EmployerReport` - Existing report model

### Service Layer
- `App\Services\SystemHealthService` - Core health check logic
  - `getHealthSummary()` - Returns complete health snapshot
  - Configurable thresholds via class constants

### Data Transfer Object
- `App\DataTransferObjects\SystemHealthSummary` - Type-safe health data container
  - `toArray()` - Convert to array for views
  - `getStatusColorClass()` - Get Tailwind color classes
  - `getQueueColorClass()` - Get queue-specific colors

### Event Listeners
- `App\Listeners\UpdateQueueHealthListener`
  - Listens to `JobProcessed` and `JobFailed` events
  - Updates queue health metrics automatically

## Usage

### Manual Error Logging
```php
use App\Models\AppError;

// Log an error
AppError::log(
    message: 'Payment processing failed',
    level: 'error', // error, critical, emergency
    exception: $exception, // Optional Throwable
    context: ['amount' => 500, 'gateway' => 'stripe']
);

// Using helper function
log_app_error('Something went wrong', 'error', $exception, $context);
log_critical_error('Critical failure', $exception);
```

### Queue Health Tracking
Queue health is automatically tracked via event listeners. When jobs are processed:
```php
// Automatically called by event listener
QueueHealth::recordJobProcessed('default', false); // success
QueueHealth::recordJobProcessed('default', true);  // failed
```

### Accessing Health Summary
```php
use App\Services\SystemHealthService;

$healthService = app(SystemHealthService::class);
$health = $healthService->getHealthSummary();

// Access properties
$health->overall_status;        // 'healthy', 'degraded', or 'critical'
$health->queue_status;          // 'healthy', 'delayed', or 'stalled'
$health->jobs_pending;          // int
$health->errors_last_24h;       // int
$health->open_total_reports;    // int
```

## Configuration

### Thresholds
Edit constants in `App\Services\SystemHealthService`:

```php
// Queue thresholds
private const QUEUE_HEALTHY_MINUTES = 10;
private const QUEUE_DELAYED_MINUTES = 30;
private const QUEUE_STALLED_JOBS = 100;

// Error thresholds
private const ERRORS_LOW = 5;
private const ERRORS_MEDIUM = 25;

// Report thresholds
private const REPORTS_LOW = 10;
private const REPORTS_MEDIUM = 50;
private const REPORTS_CRITICAL = 100;
```

Or move these to `config/system.php` for runtime configuration.

## Dashboard Integration

The widget is automatically displayed on `/admin` dashboard. It shows:
- Status badge (Healthy/Degraded/Critical) with color coding
- Three metric cards in responsive grid
- Real-time data on each page load
- Mobile-friendly single column layout

## Testing

### Seed Test Data
```bash
php artisan db:seed --class=SystemHealthTestSeeder
```

This creates:
- 3 sample error logs
- 1 queue health record

### Manual Testing
```php
// Create test error
AppError::create([
    'level' => 'error',
    'message' => 'Test error',
    'created_at' => now(),
]);

// Update queue health
QueueHealth::create([
    'queue' => 'default',
    'last_processed_at' => now()->subMinutes(5),
    'jobs_processed' => 100,
]);
```

## Extensibility

### Add Custom Health Checks
Extend `SystemHealthService` with new methods:

```php
private function getDiskSpaceMetrics(): array
{
    $freeSpace = disk_free_space('/');
    $totalSpace = disk_total_space('/');
    
    return [
        'free' => $freeSpace,
        'total' => $totalSpace,
        'percentage' => ($freeSpace / $totalSpace) * 100,
    ];
}
```

### Add to Dashboard
Edit `resources/views/admin/dashboard/index.blade.php` to add new metric cards.

### Custom Error Handler
To auto-log all exceptions, create `app/Exceptions/Handler.php`:

```php
public function register(): void
{
    $this->reportable(function (Throwable $e) {
        if ($this->shouldLogToDatabase($e)) {
            AppError::log($e->getMessage(), 'error', $e);
        }
    });
}
```

## Performance
- All queries use efficient COUNT() and MAX() aggregates
- No N+1 queries
- Lightweight data structure (DTO)
- Cached at view level if needed
- ~50ms total query time

## Future Enhancements
- [ ] Real-time updates via WebSockets
- [ ] Historical trend charts
- [ ] Email alerts for critical status
- [ ] Detailed health report page at `/admin/system-health`
- [ ] Export health logs to CSV
- [ ] Integration with external monitoring (Sentry, New Relic)
- [ ] Custom threshold configuration via admin UI