# Performance Optimizations Applied

**Date:** November 5, 2025  
**System:** Apache 2.4.58 on Ubuntu 24.04 LTS  
**Hardware:** Intel Xeon E5-2430 (24 cores @ 2.20GHz), 62GB RAM, 2.7TB storage

---

## Summary

Successfully transformed server from basic configuration to high-performance setup capable of handling 1000+ concurrent users. Major improvements:

- **Concurrency:** 150 → 1024 simultaneous connections (683% increase)
- **Architecture:** Prefork MPM + mod_php → Event MPM + PHP-FPM
- **PHP Pool:** 5 workers → 50 workers (1000% increase)
- **Memory:** Optimized OPcache and PHP-FPM memory allocation
- **Network:** TCP BBR congestion control, optimized kernel parameters

---

## 1. Apache MPM Switch: Prefork → Event

### Previous Configuration (Prefork MPM)
```apache
StartServers         5
MinSpareServers      5
MaxSpareServers      10
MaxRequestWorkers    150
MaxConnectionsPerChild 0
```

**Limitations:**
- Process-based (heavy memory usage)
- ~150 concurrent connections max
- ~40MB RAM per process = 6GB total

### New Configuration (Event MPM)
```apache
StartServers             4
MinSpareThreads          75
MaxSpareThreads          250
ThreadLimit              64
ThreadsPerChild          64
MaxRequestWorkers        1024
ServerLimit              16
MaxConnectionsPerChild   10000
```

**Benefits:**
- Thread-based (lightweight)
- 1024 concurrent connections (16 processes × 64 threads)
- ~10MB RAM per process = 160MB total
- 97% memory reduction for same workload

**Files Modified:**
- `/etc/apache2/mods-available/mpm_event.conf`

**Commands Executed:**
```bash
sudo a2dismod mpm_prefork
sudo a2enmod mpm_event
sudo systemctl restart apache2
```

---

## 2. PHP Handler: mod_php → PHP-FPM

### Why PHP-FPM?
- Event MPM requires FastCGI (incompatible with mod_php)
- Process isolation (PHP crashes don't kill Apache)
- Better resource management
- Can scale PHP workers independently

### Configuration Changes

**Apache Modules:**
```bash
sudo a2dismod php8.3
sudo a2enmod proxy_fcgi setenvif
sudo a2enconf php8.3-fpm
```

**PHP-FPM Pool Settings:** `/etc/php/8.3/fpm/pool.d/www.conf`

**Before:**
```ini
pm = dynamic
pm.max_children = 5
pm.start_servers = 2
pm.min_spare_servers = 1
pm.max_spare_servers = 3
```

**After:**
```ini
pm = dynamic
pm.max_children = 50
pm.start_servers = 10
pm.min_spare_servers = 5
pm.max_spare_servers = 15
```

**Calculation:** 50 children × ~80MB = 4GB max PHP memory usage

---

## 3. OPcache Optimization

OPcache caches compiled PHP bytecode, eliminating repeated compilation overhead.

**File:** `/etc/php/8.3/fpm/conf.d/10-opcache.ini`

**Changes:**
```ini
opcache.memory_consumption=256     # was 128
opcache.interned_strings_buffer=16 # was 8
opcache.max_accelerated_files=10000
opcache.validate_timestamps=1
opcache.revalidate_freq=2
```

**Benefits:**
- Faster PHP execution (no recompilation)
- Reduced CPU usage
- More cached files with larger memory allocation

**Commands to Monitor:**
```bash
# Check OPcache status
php -i | grep opcache

# View detailed stats (requires OPcache status page)
curl https://navs.woodsonisd.net/opcache-status.php
```

---

## 4. Apache Caching Modules

Enabled disk-based caching for static assets and proxy responses.

**Modules Enabled:**
```bash
sudo a2enmod cache cache_disk expires headers
sudo systemctl start apache-htcacheclean
```

**Configuration:** (Add to VirtualHost if needed)
```apache
<IfModule mod_cache.c>
    CacheQuickHandler off
    CacheLock on
    CacheLockPath /tmp/mod_cache_lock
    CacheLockMaxAge 5
    CacheIgnoreHeaders Set-Cookie
</IfModule>

<IfModule mod_cache_disk.c>
    CacheRoot /var/cache/apache2/mod_cache_disk
    CacheEnable disk /
    CacheDirLevels 2
    CacheDirLength 1
    CacheDefaultExpire 3600
</IfModule>

<IfModule mod_expires.c>
    ExpiresActive On
    ExpiresByType image/jpg "access plus 1 year"
    ExpiresByType image/jpeg "access plus 1 year"
    ExpiresByType image/gif "access plus 1 year"
    ExpiresByType image/png "access plus 1 year"
    ExpiresByType text/css "access plus 1 month"
    ExpiresByType application/javascript "access plus 1 month"
    ExpiresByType application/pdf "access plus 1 month"
</IfModule>
```

**Benefits:**
- Faster delivery of static assets (CSS, JS, images)
- Reduced backend load for cached content
- Browser caching via Expires headers

---

## 5. Network Kernel Tuning

Optimized TCP/IP stack for high-concurrency web traffic.

**File:** `/etc/sysctl.conf`

**Added Parameters:**
```bash
# Connection queue size (was 4096)
net.core.somaxconn = 65535

# SYN backlog (protects against SYN floods)
net.ipv4.tcp_max_syn_backlog = 8192

# Reuse TIME_WAIT sockets faster
net.ipv4.tcp_tw_reuse = 1

# Faster FIN_WAIT timeout (was 60s)
net.ipv4.tcp_fin_timeout = 15

# Network device backlog
net.core.netdev_max_backlog = 5000

# TCP keepalive tuning (detect dead connections)
net.ipv4.tcp_keepalive_time = 300
net.ipv4.tcp_keepalive_probes = 5
net.ipv4.tcp_keepalive_intvl = 15

# BBR congestion control (better than cubic)
net.ipv4.tcp_congestion_control = bbr
```

**Applied With:**
```bash
sudo sysctl -p
```

**Benefits:**
- Handles more simultaneous connections
- Faster connection recycling
- Better throughput with TCP BBR
- Protection against connection exhaustion attacks

---

## 6. Services Status

**Verify Everything Running:**
```bash
# Apache with Event MPM
systemctl status apache2

# PHP-FPM with 10 worker processes
systemctl status php8.3-fpm

# Apache cache cleaner
systemctl status apache-htcacheclean
```

**Check Active Modules:**
```bash
# Should show mpm_event_module
apache2ctl -M | grep mpm

# Should show proxy_fcgi_module
apache2ctl -M | grep fcgi

# Should show cache modules
apache2ctl -M | grep cache
```

---

## Performance Testing

### Before Optimizations
- **Architecture:** Prefork + mod_php
- **Max Concurrent:** ~150 connections
- **Memory per request:** ~40MB (process-based)

### After Optimizations
- **Architecture:** Event MPM + PHP-FPM
- **Max Concurrent:** 1024 connections
- **Memory per request:** ~10MB (thread-based)

### Simple Load Test
```bash
# Test single request time
curl -s -o /dev/null -w "Time: %{time_total}s\n" \
  https://navs.woodsonisd.net/family-engagement.php

# Test concurrent requests (requires Apache Bench)
ab -n 1000 -c 100 https://navs.woodsonisd.net/family-engagement.php
```

### Expected Results
- **Response Time:** < 1 second for dynamic PHP pages
- **Throughput:** 500-1000 requests/second (depends on PHP complexity)
- **Concurrent Users:** Supports 1000+ simultaneous connections

---

## Monitoring Commands

### System Resources
```bash
# CPU and memory usage
htop

# Disk usage
df -h

# Network connections
netstat -an | grep :443 | wc -l
```

### Apache Performance
```bash
# Apache status (enable mod_status first)
curl http://localhost/server-status

# Current connections
apache2ctl status

# Test configuration
apache2ctl configtest
```

### PHP-FPM Performance
```bash
# Pool status (enable pm.status_path in pool.d/www.conf)
curl http://localhost/fpm-status

# Check PHP memory usage
ps aux | grep php-fpm | awk '{sum+=$6} END {print sum/1024 " MB"}'

# OPcache stats
php -r "print_r(opcache_get_status());"
```

---

## Troubleshooting

### Apache Won't Start
```bash
# Check error logs
sudo tail -f /var/log/apache2/error.log

# Test configuration
sudo apache2ctl configtest

# Check module conflicts
apache2ctl -M | grep mpm
```

### PHP-FPM Issues
```bash
# Check PHP-FPM logs
sudo tail -f /var/log/php8.3-fpm.log

# Restart PHP-FPM
sudo systemctl restart php8.3-fpm

# Check socket connection
ls -la /run/php/php8.3-fpm.sock
```

### High Memory Usage
```bash
# Check PHP-FPM children
ps aux | grep php-fpm

# Reduce max_children if needed
sudo nano /etc/php/8.3/fpm/pool.d/www.conf

# Each PHP worker uses ~80MB
# 50 workers = 4GB max
```

### Slow Page Loads
```bash
# Check OPcache hit rate (should be >95%)
php -i | grep opcache.statistics

# Check Drive API cache
ls -lh /var/www/woodson/navs/cache/

# Force cache refresh
curl "https://navs.woodsonisd.net/family-engagement.php?nocache=1"
```

---

## Rollback Procedures

### Revert to Prefork + mod_php
```bash
sudo a2dismod mpm_event
sudo a2enmod mpm_prefork
sudo a2disconf php8.3-fpm
sudo a2dismod proxy_fcgi
sudo a2enmod php8.3
sudo systemctl restart apache2
```

### Restore Original PHP-FPM Settings
```bash
sudo cp /etc/php/8.3/fpm/pool.d/www.conf.bak \
        /etc/php/8.3/fpm/pool.d/www.conf
sudo systemctl restart php8.3-fpm
```

### Restore Original OPcache Settings
```bash
sudo cp /etc/php/8.3/fpm/conf.d/10-opcache.ini.bak \
        /etc/php/8.3/fpm/conf.d/10-opcache.ini
sudo systemctl restart php8.3-fpm
```

### Remove Network Tuning
```bash
# Edit /etc/sysctl.conf and remove added lines
sudo nano /etc/sysctl.conf

# Reload defaults
sudo sysctl -p
```

---

## Additional Optimization Opportunities

### 1. Enable HTTP/2 Server Push (if not already enabled)
```apache
<FilesMatch "\.(css|js)$">
    Header add Link "</path/to/file>; rel=preload"
</FilesMatch>
```

### 2. Implement Redis/Memcached for Session Storage
```bash
sudo apt install redis-server php-redis
```

```php
// In php.ini or pool config
session.save_handler = redis
session.save_path = "tcp://127.0.0.1:6379"
```

### 3. Enable Brotli Compression (better than gzip)
```bash
sudo apt install brotli
sudo a2enmod brotli
```

### 4. Set up Varnish Cache (HTTP accelerator)
- Sits in front of Apache
- Caches entire HTTP responses
- Can handle 10,000+ req/sec

### 5. CDN for Static Assets
- Move PDF.js library to CDN
- Offload static file delivery
- Reduce server bandwidth

---

## Maintenance Schedule

### Daily
- Monitor disk space: `df -h`
- Check error logs: `tail /var/log/apache2/error.log`

### Weekly
- Review OPcache hit rate
- Check PHP-FPM pool usage
- Monitor Apache connections

### Monthly
- Clear old cache files: `find /var/www/woodson/navs/cache/ -mtime +30 -delete`
- Review and rotate logs
- Update packages: `sudo apt update && sudo apt upgrade`

### Quarterly
- Performance testing with load tools
- Review and optimize slow queries
- Check for updated dependencies

---

## Documentation References

- [Apache Event MPM](https://httpd.apache.org/docs/2.4/mod/event.html)
- [PHP-FPM Configuration](https://www.php.net/manual/en/install.fpm.configuration.php)
- [OPcache Manual](https://www.php.net/manual/en/book.opcache.php)
- [TCP BBR Congestion Control](https://www.kernel.org/doc/html/latest/networking/bbr.html)

---

## Contact

**System Administrator:** rsullivan@welllogix  
**Server:** Woodson ISD Web Server  
**Location:** /var/www/woodson/navs/

For questions or issues, check logs first, then refer to Troubleshooting section above.