# textiq-doc-extraction - Technical Specification: Redis Authentication

**Author:** TextIQ
**Date:** 2025-12-04
**Project Level:** Simple (Quick Flow - Single Story)
**Change Type:** Security Enhancement
**Development Context:** Brownfield - Adding authentication to existing Redis configuration

---

## Context

### Available Documents

**PRD:** docs/prd.md - Complete product requirements
**Architecture:** docs/architecture.md - System architecture with Redis + Celery configuration
**Epics:** docs/epics.md - Full epic breakdown for textiq-doc-extraction

### Project Stack

**Runtime:** Python 3.11+
**Framework:** FastAPI 0.122
**Task Queue:** Celery 5.3+ with Redis 7+ backend
**Database:** PostgreSQL 18
**Vector Store:** Milvus 2.4
**Containerization:** Docker Compose for local development

**Current Redis Configuration:**
- Redis 7 running in Docker container (dsol-redis)
- No authentication enabled
- Used for: Celery broker/backend, rate limiting, progress tracking
- Connection string: `redis://localhost:6379` (no password)

### Existing Codebase Structure

**Key Redis Integration Points:**
- `src/core/config.py` - Settings with `redis_url` property
- `src/workers/celery_app.py` - Celery configuration using `settings.redis_url`
- `src/api/dependencies.py` - Redis client creation for API routes
- `src/services/rate_limiter.py` - Rate limiting using Redis
- `src/services/health.py` - Redis health checks
- `docker-compose.yml` - Redis container configuration
- `.env.example` - Environment variable templates

---

## The Change

### Problem Statement

The current Redis implementation has no authentication configured. The Redis URL is `redis://localhost:6379` without password or database selection. This creates security vulnerabilities:

1. **No Access Control:** Any process with network access can connect to Redis
2. **Missing Database Isolation:** All keys stored in database 0 (default)
3. **Production Risk:** Configuration not production-ready (Redis should always be password-protected)
4. **NFR Violation:** Conflicts with NFR11 (secure communication) and NFR12 (secure credential storage)

This is a security gap that must be addressed before production deployment.

### Proposed Solution

Add Redis password authentication and database selection to the Redis URL configuration:

**Current URL format:**
```
redis://localhost:6379
```

**New URL format with authentication:**
```
redis://:password@localhost:6379/0
```

**Implementation approach:**
1. Add `REDIS_PASSWORD` and `REDIS_DB` environment variables to configuration
2. Update `Settings` class to build authenticated Redis URL
3. Configure Redis container with password (`requirepass` directive)
4. Update all Redis client instantiations to use authenticated URL
5. Update health check to authenticate properly
6. Backward compatibility: Empty password = no auth (for local dev without Docker)

### Scope

**In Scope:**
- Add `REDIS_PASSWORD` and `REDIS_DB` environment variables
- Build authenticated Redis URL in `Settings` class
- Configure Docker Compose Redis with password
- Update `.env.example` with new variables
- Verify health check works with authentication
- Update documentation (README.md)

**Out of Scope:**
- Redis TLS/SSL encryption (can be added later if needed)
- Redis Sentinel or cluster configuration
- Redis ACL (Access Control Lists) - password auth is sufficient for MVP
- Redis Sentinel configuration in Celery (noted in config but not actively used)

---

## Implementation Details

### Source Tree Changes

All file paths are exact, specifying CREATE, MODIFY, or DELETE actions:

**MODIFY** - `.env.example` (lines 14-15)
- Add `REDIS_PASSWORD` environment variable after `REDIS_URL`
- Add `REDIS_DB` environment variable (default: 0)

**MODIFY** - `src/core/config.py` (lines 13, add new properties after line 13)
- Add `redis_password: str` field (default: empty string)
- Add `redis_db: int` field (default: 0)
- Modify `redis_url` property to build authenticated URL from components

**MODIFY** - `docker-compose.yml` (lines 21-33)
- Add Redis `command` with `--requirepass` directive
- Add environment variable `REDIS_PASSWORD` from `.env`
- Update healthcheck to use `-a` auth flag

**MODIFY** - `README.md` (Development Setup section)
- Document new Redis environment variables
- Update setup instructions to include password configuration

**NO CHANGES NEEDED:**
- `src/workers/celery_app.py` - Already uses `settings.redis_url`
- `src/api/dependencies.py` - Already uses `settings.redis_url`
- `src/services/rate_limiter.py` - Receives Redis client, no URL handling
- `src/services/health.py` - Receives Redis client, no URL handling
- All task files - Use Celery app configuration

### Technical Approach

**Configuration Strategy:**
- Use Pydantic Settings to validate and build Redis URL
- Support both authenticated and unauthenticated modes
- Environment variables override defaults
- Redis URL format follows standard: `redis://[username][:password]@host:port/db`

**URL Building Logic:**
```python
@property
def redis_url(self) -> str:
    """Build Redis URL with optional authentication and database selection."""
    if self.redis_password:
        # Authenticated URL: redis://:password@host:port/db
        return f"redis://:{self.redis_password}@{self.redis_host}:{self.redis_port}/{self.redis_db}"
    else:
        # Unauthenticated URL (local dev): redis://host:port/db
        return f"redis://{self.redis_host}:{self.redis_port}/{self.redis_db}"
```

**Why this approach:**
- Backward compatible (empty password = no auth)
- Single source of truth (`redis_url` property)
- No code changes needed in consumers (they already use `settings.redis_url`)
- Standard Redis URL format understood by redis-py and Celery

### Existing Patterns to Follow

**Configuration patterns from `src/core/config.py`:**
- Use Pydantic `BaseSettings` for environment variable loading
- Provide sensible defaults for development
- Use properties for computed values (e.g., `sync_database_url`)
- Document each setting with type hints

**Docker Compose patterns:**
- Store sensitive values in environment variables
- Use `.env` file for local configuration
- Configure healthchecks for all services
- Use named volumes for persistence

**Code style conventions:**
- Python 3.11+ type hints
- Docstrings for all public methods
- Follow existing formatting (Black, Ruff)

### Integration Points

**Internal Modules:**
- `src.core.config` → Provides `settings.redis_url` to all modules
- `src.workers.celery_app` → Uses Redis URL for broker and backend
- `src.api.dependencies` → Creates Redis client for API routes
- `src.services.rate_limiter` → Receives Redis client instance
- `src.services.health` → Receives Redis client instance

**External Services:**
- Redis 7 container → Configured with `--requirepass` command
- Celery workers → Connect using authenticated Redis URL

**Configuration Flow:**
```
.env file → Settings class → redis_url property → Celery/Redis clients
```

---

## Development Context

### Relevant Existing Code

**Configuration loading (src/core/config.py:6-36):**
- `Settings` class with `model_config` for `.env` loading
- Existing `redis_url` field (will be converted to computed property)
- Pattern: `database_url` with computed `sync_database_url` property

**Docker Compose service (docker-compose.yml:21-33):**
- Redis 7 image with volume and healthcheck
- Current healthcheck: `["CMD", "redis-cli", "ping"]`
- No authentication configured

### Dependencies

**Framework/Libraries:**
- redis-py (via `redis>=5.0` in pyproject.toml) - Handles Redis URL parsing
- Celery 5.3+ - Understands Redis URL format with password
- pydantic-settings 2.0 - Environment variable validation

**Internal Modules:**
- `src.core.config.Settings` - Configuration management
- `src.api.dependencies.get_redis_client()` - Redis client factory

### Configuration Changes

**Environment Variables (add to .env.example and .env):**
```bash
# Redis Authentication
REDIS_PASSWORD=your_secure_password_here
REDIS_DB=0
```

**Docker Compose (modify redis service):**
```yaml
redis:
  image: redis:7
  container_name: dsol-redis
  command: sh -c 'redis-server $${REDIS_PASSWORD:+--requirepass "$$REDIS_PASSWORD"}'
  environment:
    - REDIS_PASSWORD=${REDIS_PASSWORD:-}
  ports:
    - "6379:6379"
  volumes:
    - redis_data:/data
  healthcheck:
    test: ["CMD", "sh", "-c", "redis-cli $${REDIS_PASSWORD:+-a \"$$REDIS_PASSWORD\"} ping | grep PONG"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 5s
```

**Note:** The syntax `${VAR:+value}` means "if VAR is set and non-empty, use value". This conditionally adds `--requirepass` only when password is configured.

### Existing Conventions (Brownfield)

**Code Style:**
- Type hints on all function signatures
- Docstrings in Google/NumPy style
- Properties for computed configuration values
- Environment variables in UPPER_SNAKE_CASE

**Configuration Patterns:**
- Pydantic Settings with `.env` file
- Default values for local development
- Required values have no defaults (will raise validation error)
- Sensitive values never logged

**Docker Patterns:**
- Environment variable substitution with defaults: `${VAR:-default}`
- Healthchecks for all services
- Named volumes with project prefix

### Test Framework & Standards

**Framework:** pytest 8.0+ with pytest-asyncio 0.23+
**Location:** tests/ directory
**Naming:** test_*.py files, test_* functions
**Async:** Use @pytest.mark.asyncio for async tests

**Test Coverage Requirements:**
- Unit tests for Settings property (redis_url building)
- Integration test for Redis connection with auth
- Health check test with authenticated Redis

---

## Implementation Stack

**Python:** 3.11+
**Configuration:** pydantic-settings 2.0
**Redis Client:** redis 5.0+ (async support)
**Task Queue:** Celery 5.3+ with Redis backend
**Container:** Docker Compose
**Testing:** pytest 8.0+, pytest-asyncio 0.23+

---

## Technical Details

### Redis URL Format Specification

Standard Redis URL format (RFC 3986):
```
redis://[username][:password]@host:port/db
```

**Examples:**
```bash
# No auth (local dev):
redis://localhost:6379/0

# With password:
redis://:mypassword@localhost:6379/0

# With username and password (Redis 6+ ACL):
redis://default:mypassword@localhost:6379/0

# textiq-doc-extraction will use (password only, no username):
redis://:${REDIS_PASSWORD}@localhost:6379/0
```

### Password Security

**Storage:**
- Password stored in `.env` file (not committed to git)
- `.env` listed in .gitignore
- `.env.example` contains placeholder only

**Access:**
- Loaded via pydantic-settings (validated at startup)
- Never logged in application logs
- Used only to build Redis connection URL

**Production:**
- Use secret management service (AWS Secrets Manager, HashiCorp Vault, etc.)
- Set environment variables via deployment config (K8s secrets, etc.)
- Rotate passwords regularly

### Database Selection

Redis supports 16 databases (0-15) by default. textiq-doc-extraction uses database 0 for all operations:
- Celery broker: db 0
- Celery result backend: db 0
- Rate limiting: db 0
- Progress tracking: db 0

**Future consideration:** Separate databases for different concerns:
- db 0: Celery tasks
- db 1: Rate limiting
- db 2: Progress tracking

For MVP, single database is sufficient.

### Backward Compatibility

**Empty password support:**
If `REDIS_PASSWORD` is not set or empty, Redis URL is built without auth:
```python
redis://localhost:6379/0  # No auth
```

This allows developers to run Redis without password in local development without Docker.

**Migration path:**
1. Update configuration files (this story)
2. Update documentation
3. Developers update `.env` files
4. No code changes needed in application logic

---

## Development Setup

### Prerequisites
- Docker & Docker Compose
- Python 3.11+
- uv (Python package manager)

### Setup Commands

```bash
# 1. Clone and navigate to project
cd /Users/nguyen/Work/textiq-doc-extraction

# 2. Update .env file with Redis password
cp .env.example .env
# Edit .env and set REDIS_PASSWORD=your_secure_password_here

# 3. Restart Redis container with authentication
docker-compose down redis
docker-compose up -d redis

# 4. Verify Redis authentication works
docker exec -it dsol-redis redis-cli -a your_secure_password_here ping
# Should return: PONG

# 5. No code reinstall needed (configuration only)

# 6. Restart API and workers to pick up new config
# (if running via Docker, restart containers)
# (if running locally, restart processes)

# 7. Run tests
uv run pytest tests/services/test_health.py -v
```

---

## Implementation Guide

### Setup Steps

1. **Update environment configuration**
   - Add `REDIS_PASSWORD` to `.env.example` with placeholder
   - Add `REDIS_DB` to `.env.example` (default: 0)
   - Update developer `.env` files with actual password

2. **Modify Settings class**
   - Add `redis_password` and `redis_db` fields
   - Convert `redis_url` from field to property
   - Implement URL building logic with conditional auth

3. **Configure Docker Redis**
   - Add `command` with `--requirepass` directive
   - Update healthcheck with `-a` auth flag
   - Add environment variable for password

4. **Update documentation**
   - Document new environment variables in README
   - Add setup instructions for Redis authentication

### Implementation Steps

**Step 1: Update .env.example**
Location: `.env.example`
Action: Add Redis authentication variables after line 14

```bash
# ===================
# Redis Configuration
# ===================
# Redis connection URL for task queue and rate limiting
REDIS_URL=redis://localhost:6379

# Redis password (leave empty for no authentication in local dev)
REDIS_PASSWORD=your_secure_password_here

# Redis database number (0-15, default: 0)
REDIS_DB=0
```

**Step 2: Modify Settings class**
Location: `src/core/config.py`
Action: Replace `redis_url` field with computed property

```python
class Settings(BaseSettings):
    """Application settings loaded from environment variables."""

    # Database
    database_url: str = "postgresql+asyncpg://dsol:dsol_dev_password@localhost:5432/dsol"

    # Redis
    redis_host: str = "localhost"
    redis_port: int = 6379
    redis_password: str = ""  # Empty string = no auth
    redis_db: int = 0

    # ... rest of settings ...

    @property
    def redis_url(self) -> str:
        """Build Redis URL with optional authentication and database selection.

        Returns:
            Redis connection URL in format:
            - With auth: redis://:password@host:port/db
            - Without auth: redis://host:port/db
        """
        if self.redis_password:
            return f"redis://:{self.redis_password}@{self.redis_host}:{self.redis_port}/{self.redis_db}"
        else:
            return f"redis://{self.redis_host}:{self.redis_port}/{self.redis_db}"

    @property
    def sync_database_url(self) -> str:
        """Return synchronous database URL for Alembic migrations."""
        return self.database_url.replace("+asyncpg", "+psycopg2")
```

**Step 3: Update Docker Compose**
Location: `docker-compose.yml`
Action: Modify redis service (lines 21-33)

```yaml
  redis:
    image: redis:7
    container_name: dsol-redis
    command: sh -c 'redis-server $${REDIS_PASSWORD:+--requirepass "$$REDIS_PASSWORD"}'
    environment:
      - REDIS_PASSWORD=${REDIS_PASSWORD:-}
    ports:
      - "6379:6379"
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "sh", "-c", "redis-cli $${REDIS_PASSWORD:+-a \"$$REDIS_PASSWORD\"} ping | grep PONG"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 5s
```

**Step 4: Update README.md**
Location: `README.md`
Action: Add Redis authentication documentation in "Environment Configuration" section

```markdown
### Redis Configuration

- `REDIS_PASSWORD`: Password for Redis authentication (optional, leave empty for no auth)
- `REDIS_DB`: Redis database number (0-15, default: 0)

For production deployments, always set a strong `REDIS_PASSWORD`.
```

### Testing Strategy

**Unit Tests:**
- Test Settings.redis_url property with password
- Test Settings.redis_url property without password
- Test Settings.redis_url property with custom database
- Verify URL format matches Redis URL specification

**Integration Tests:**
- Test Redis connection with authentication
- Test Celery task dispatch with authenticated Redis
- Test rate limiter with authenticated Redis
- Test health check with authenticated Redis

**Manual Verification:**
```bash
# 1. Verify Redis requires password
docker exec -it dsol-redis redis-cli ping
# Should return: (error) NOAUTH Authentication required.

# 2. Verify Redis accepts password
docker exec -it dsol-redis redis-cli -a your_password ping
# Should return: PONG

# 3. Verify health check endpoint
curl http://localhost:8000/health
# Should return: {"status": "healthy", "dependencies": {"redis": "healthy", ...}}

# 4. Verify Celery connects successfully
# Check worker logs for successful broker connection
```

### Acceptance Criteria

**Given** the Redis password is configured in environment
**When** the application starts
**Then** all Redis connections use authenticated URL
**And** health check reports Redis as healthy
**And** Celery workers connect successfully
**And** rate limiting works correctly

**Given** the Redis password is empty
**When** the application starts in local development
**Then** Redis connections work without authentication
**And** backward compatibility is maintained

**Given** a developer updates their .env file with password
**When** they restart Docker services
**Then** Redis requires authentication
**And** all services connect successfully with password

---

## Developer Resources

### File Paths Reference

**Configuration Files:**
- `.env.example` - Environment variable template
- `src/core/config.py` - Settings class with redis_url property
- `docker-compose.yml` - Redis service configuration

**Documentation Files:**
- `README.md` - Development setup instructions

**Test Files:**
- `tests/services/test_health.py` - Health check tests (verify Redis auth)
- `tests/conftest.py` - Test fixtures (Redis client setup)

### Key Code Locations

**Settings class:** `src/core/config.py:6-46`
- `redis_url` property to be added after line 13
- Uses `redis_host`, `redis_port`, `redis_password`, `redis_db` fields

**Redis service:** `docker-compose.yml:21-33`
- Add `command` with `--requirepass`
- Update healthcheck with auth flag

**Redis client factory:** `src/api/dependencies.py`
- Already uses `settings.redis_url` (no changes needed)

**Celery configuration:** `src/workers/celery_app.py:56-66`
- Already uses `settings.redis_url` for broker and backend (no changes needed)

### Testing Locations

**Unit tests:** `tests/core/` (create `test_config.py`)
**Integration tests:** `tests/services/test_health.py` (Redis health check)
**Manual tests:** Use `docker exec` and `curl` commands above

### Documentation to Update

- `README.md` - Add Redis authentication configuration section
- `.env.example` - Add REDIS_PASSWORD and REDIS_DB variables
- This tech-spec serves as implementation reference

---

## UX/UI Considerations

No UI/UX impact - backend/infrastructure change only. This is a configuration and security enhancement that does not affect the API surface or user-facing functionality.

---

## Testing Approach

### Test Framework

**Framework:** pytest 8.0+ with pytest-asyncio 0.23+
**Location:** tests/ directory
**Conventions:** test_*.py files, test_* functions

### Test Strategy

**1. Unit Tests (tests/core/test_config.py):**

```python
def test_redis_url_with_password():
    """Test redis_url property builds authenticated URL."""
    settings = Settings(
        redis_host="localhost",
        redis_port=6379,
        redis_password="testpass",
        redis_db=0
    )
    assert settings.redis_url == "redis://:testpass@localhost:6379/0"

def test_redis_url_without_password():
    """Test redis_url property builds unauthenticated URL."""
    settings = Settings(
        redis_host="localhost",
        redis_port=6379,
        redis_password="",
        redis_db=0
    )
    assert settings.redis_url == "redis://localhost:6379/0"

def test_redis_url_custom_database():
    """Test redis_url property with custom database."""
    settings = Settings(
        redis_host="localhost",
        redis_port=6379,
        redis_password="testpass",
        redis_db=2
    )
    assert settings.redis_url == "redis://:testpass@localhost:6379/2"
```

**2. Integration Tests (tests/services/test_health.py):**

```python
@pytest.mark.asyncio
async def test_redis_health_check_with_auth(redis_client):
    """Test Redis health check with authentication."""
    result = await check_redis(redis_client)
    assert result.status == "healthy"
    assert result.latency_ms < 300  # Under timeout threshold
```

**3. Manual Tests:**
- Verify Docker Compose starts with authentication
- Verify health endpoint reports Redis as healthy
- Verify Celery workers connect successfully
- Verify rate limiting still works

### Coverage Target

- Unit test coverage: 100% (simple property logic)
- Integration coverage: Critical paths (health check, Celery connection)

---

## Deployment Strategy

### Deployment Steps

This is a configuration change, not a code deployment. Rolling update process:

1. **Stage 1: Update configuration files**
   - Merge PR with updated `.env.example` and code changes
   - Do NOT restart services yet

2. **Stage 2: Configure secrets in deployment environment**
   - Set `REDIS_PASSWORD` in Kubernetes secrets / AWS Parameter Store / etc.
   - Verify environment variables are accessible to pods/containers

3. **Stage 3: Update Redis configuration**
   - Update Redis deployment with `requirepass` directive
   - Rolling restart Redis (brief downtime acceptable for MVP)

4. **Stage 4: Rolling restart application services**
   - Restart API pods/containers (will pick up new config from environment)
   - Restart Celery workers (will reconnect with authentication)
   - Monitor logs for successful Redis connections

5. **Stage 5: Verify**
   - Check health endpoint: all dependencies healthy
   - Verify Celery tasks processing successfully
   - Verify rate limiting functional

### Rollback Plan

If issues occur after deployment:

1. **Quick rollback (restore no-auth state):**
   - Remove `--requirepass` from Redis configuration
   - Restart Redis without authentication
   - Restart application services
   - All services revert to unauthenticated Redis connections

2. **Verify rollback:**
   - Health endpoint reports healthy
   - Celery tasks processing
   - API requests successful

3. **Debug and retry:**
   - Check environment variables set correctly
   - Verify Redis URL format in application logs
   - Test authentication manually with redis-cli

### Monitoring

**Key Metrics:**
- Redis connection errors (should be zero)
- Celery task processing rate (should remain stable)
- API response times (should be unaffected)
- Health check status (should be healthy)

**Log Messages to Watch:**
```
# Success indicators:
"redis_health_check" status="healthy"
"Celery worker started" broker="redis://:*****@localhost:6379/0"

# Failure indicators:
"redis_health_check" status="unhealthy" error="NOAUTH Authentication required"
"Celery worker failed to connect" error="Connection refused"
```

---

## Summary

This tech-spec addresses the security gap of unauthenticated Redis by adding password authentication and database selection. The implementation is straightforward:

1. Add environment variables for Redis password and database
2. Build authenticated Redis URL in Settings property
3. Configure Docker Compose Redis with password
4. No changes needed in consuming code (already uses `settings.redis_url`)

**Impact:**
- **Security:** ✅ Closes authentication vulnerability
- **Compatibility:** ✅ Backward compatible (empty password = no auth)
- **Complexity:** ✅ Minimal (configuration only, no logic changes)
- **Risk:** ✅ Low (can rollback by removing password)

**Estimated Effort:** 1-2 hours (Simple change, single story)

---

_This tech-spec follows the BMad Method Quick Flow template for simple security enhancements._