# Story 1.4: Structured Logging Setup

Status: done

## Story

As a **developer**,
I want **structured logging configured throughout the application**,
so that **I can debug issues and track request flows**.

## Acceptance Criteria

1. **AC1:** Logs are output in structured JSON format with these fields:
   - `timestamp` (ISO 8601 UTC format)
   - `level` (DEBUG, INFO, WARNING, ERROR)
   - `event` (descriptive event name)
   - `request_id` (UUID for request correlation/tracing)
   - Relevant context fields

2. **AC2:** Log level is configurable via `LOG_LEVEL` environment variable:
   - Supports DEBUG, INFO, WARNING, ERROR
   - Default is INFO
   - Changes take effect on application restart

3. **AC3:** Request/response logging middleware captures:
   - `method` (GET, POST, etc.)
   - `path` (request URL path)
   - `status_code` (HTTP response status)
   - `duration_ms` (request processing time in milliseconds)
   - `request_id` (propagated from request context)

4. **AC4:** Sensitive content is NOT logged:
   - File contents from uploads
   - Query answers and retrieved content
   - API keys (masked or excluded)
   - Any JSONB content fields

5. **AC5:** Request ID middleware is implemented:
   - Generates UUID for each incoming request
   - Available via request state for use in handlers
   - Included in all log entries for that request
   - Returned in `X-Request-ID` response header

6. **AC6:** Log format is environment-aware:
   - JSON format in production (LOG_LEVEL != DEBUG)
   - Pretty console format in development (LOG_LEVEL == DEBUG)

7. **AC7:** All existing modules use structured logging:
   - `src/api/main.py` lifespan logs use structlog
   - `src/api/middleware.py` exception handlers use structlog
   - All log calls use event-based format (not string formatting)

## Tasks / Subtasks

- [x] **Task 1: Install and configure structlog** (AC: 1, 6)
  - [x] Add `structlog` to dependencies if not present
  - [x] Create `src/core/logging.py` module
  - [x] Implement `configure_logging()` function
  - [x] Configure JSON processor for production
  - [x] Configure pretty console processor for development
  - [x] Add standard processors: timestamp, level, event

- [x] **Task 2: Implement request ID middleware** (AC: 5)
  - [x] Create request ID middleware in `src/api/middleware.py`
  - [x] Generate UUID for each request
  - [x] Store in request.state for handler access
  - [x] Add `X-Request-ID` header to responses
  - [x] Bind request_id to structlog context

- [x] **Task 3: Implement request logging middleware** (AC: 3, 4)
  - [x] Create request/response logging middleware
  - [x] Log request start with method, path, request_id
  - [x] Log request completion with status_code, duration_ms
  - [x] Exclude sensitive paths or headers from logging
  - [x] Add to middleware stack in main.py

- [x] **Task 4: Create sensitive data filter** (AC: 4)
  - [x] Implement structlog processor to filter sensitive fields
  - [x] Filter keys: `content`, `answer`, `api_key`, `password`, `secret`
  - [x] Replace sensitive values with `[REDACTED]`
  - [x] Add to processor chain

- [x] **Task 5: Update existing modules to use structlog** (AC: 7)
  - [x] Update `src/api/main.py` lifespan logging
  - [x] Update `src/api/middleware.py` exception handler logging
  - [x] Ensure all logs use `logger.info("event_name", key=value)` format
  - [x] Call `configure_logging()` at app startup

- [x] **Task 6: Verify logging configuration** (AC: 1, 2, 3, 6)
  - [x] Start server and make test requests
  - [x] Verify JSON output format in production mode
  - [x] Verify pretty output in debug mode
  - [x] Confirm request_id appears in all related logs
  - [x] Test LOG_LEVEL changes affect output

## Dev Notes

### Architecture Patterns and Constraints

- **Logging Library:** structlog ^24.0
- **Log Format:** JSON in production, pretty console in development
- **Request Tracing:** UUID-based request_id middleware
- **Sensitive Data:** Must be filtered before logging

### Project Structure Notes

From Story 1.3, the following exist and need modification:
- `src/api/main.py` - FastAPI app (add logging configuration call)
- `src/api/middleware.py` - Middleware module (add request ID and logging middleware)
- `src/core/config.py` - Settings (LOG_LEVEL already exists)

Files to create in this story:
- `src/core/logging.py` - Structlog configuration module

### Learnings from Previous Story

**From Story 1-3-fastapi-application-shell (Status: done)**

- **Middleware Pattern**: `add_cors_middleware()` and `add_exception_handlers()` functions established
- **Lifespan Management**: `@asynccontextmanager` lifespan in main.py handles startup/shutdown
- **Standard Logging**: Currently using Python's `logging` module - needs migration to structlog
- **Package Manager**: Project uses `uv` - use `uv add structlog` if needed

[Source: docs/sprint-artifacts/1-3-fastapi-application-shell.md#Dev-Agent-Record]

### References

- [Source: docs/sprint-artifacts/tech-spec-epic-1.md#Observability] - Required log fields specification
- [Source: docs/sprint-artifacts/tech-spec-epic-1.md#Security] - NFR14 no sensitive logging
- [Source: docs/architecture.md#Logging-Strategy] - structlog configuration pattern
- [Source: docs/epics.md#Story-1.4] - Original story definition

## Dev Agent Record

### Context Reference

N/A - Story context will be generated by story-context workflow

### Agent Model Used

Claude Opus 4.5 (claude-opus-4-5-20251101)

### Debug Log References

- Verified JSON log output: `{"version": "1.0.0", "event": "api_starting", "level": "info", ...}`
- Request logging confirmed with method, path, status_code, duration_ms, request_id
- X-Request-ID header verified in response: `x-request-id: 22f9543e-f4d8-429c-a087-f9d50e1a989c`

### Completion Notes List

- Created `src/core/logging.py` with structlog configuration, sensitive data filter, environment-aware formatting
- Implemented `RequestIdMiddleware` generating UUID, binding to structlog context, adding X-Request-ID header
- Implemented `RequestLoggingMiddleware` logging request_started and request_completed events with timing
- Added `filter_sensitive_data` processor filtering content, answer, api_key, password, secret fields
- Updated `src/api/main.py` to call `configure_logging()` and use structlog for lifespan events
- Updated `src/api/middleware.py` exception handlers to use structlog with request_id context

### File List

| Status | File Path |
|--------|-----------|
| NEW | src/core/logging.py |
| MODIFIED | src/api/main.py |
| MODIFIED | src/api/middleware.py |