# Story 1.1: Airflow Docker Infrastructure Setup

**Epic:** Airflow Infrastructure & Parallel Deployment
**Status:** Done
**Priority:** High
**Estimate:** 4 hours

---

## User Story

As a **DevOps engineer**, I want to **deploy Apache Airflow infrastructure via Docker Compose** so that **we can run DAG-based workflows alongside the existing Celery system**.

---

## Acceptance Criteria

- [x] Airflow webserver container running and accessible at port 8081
- [x] Airflow scheduler container running
- [x] Airflow metadata database initialized (separate PostgreSQL DB or schema)
- [x] Airflow connections configured for SeaweedFS, PostgreSQL, Milvus, Azure OpenAI
- [x] Environment variables properly configured via `.env`
- [x] Health check endpoint returning healthy status
- [x] Docker Compose file integrates with existing `docker-compose.yml`

---

## Technical Details

### Files to Create/Modify

| File | Action | Description |
|------|--------|-------------|
| `docker/docker-compose.airflow.yml` | Create | Airflow services definition |
| `.env` | Modify | Add Airflow environment variables |
| `scripts/setup_airflow_connections.py` | Create | Connection setup script |
| `docker-compose.yml` | Modify | Add include for airflow compose |

### Docker Compose Configuration

```yaml
# docker/docker-compose.airflow.yml
services:
  airflow-webserver:
    image: apache/airflow:3.0.6-python3.11
    environment:
      - AIRFLOW__CORE__EXECUTOR=LocalExecutor
      - AIRFLOW__DATABASE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@postgres:5432/airflow
    ports:
      - "8081:8080"
    volumes:
      - ./dags:/opt/airflow/dags
      - ./plugins:/opt/airflow/plugins
    depends_on:
      - postgres
      - redis

  airflow-scheduler:
    image: apache/airflow:3.0.6-python3.11
    command: scheduler
    volumes:
      - ./dags:/opt/airflow/dags
      - ./plugins:/opt/airflow/plugins
```

### Airflow Connections Required

| Connection ID | Type | Purpose |
|---------------|------|---------|
| `seaweedfs_default` | S3 | SeaweedFS object storage |
| `postgres_default` | Postgres | Document/record storage |
| `milvus_default` | Generic | Vector database |
| `azure_openai_default` | HTTP | Embedding generation |

---

## Tasks

- [x] Create `docker/docker-compose.airflow.yml` with webserver and scheduler
- [x] Add Airflow variables to `docker/.env`
- [x] Create Airflow database in PostgreSQL (or separate container)
- [x] Create `scripts/setup_airflow_connections.py` for connection bootstrap
- [x] Test Airflow webserver startup and accessibility
- [x] Verify scheduler is running and can process tasks
- [x] Document startup procedure

---

## Dependencies

- PostgreSQL running (existing)
- Redis running (existing)
- Docker and Docker Compose installed

---

## Testing

```bash
# Start Airflow services
docker-compose -f docker/docker-compose.airflow.yml up -d

# Verify webserver is accessible
curl http://localhost:8081/health

# Check scheduler logs
docker-compose -f docker/docker-compose.airflow.yml logs airflow-scheduler

# Run connection setup
python scripts/setup_airflow_connections.py
```

---

## Notes

- Airflow will use LocalExecutor for simplicity (can upgrade to CeleryExecutor later)
- Separate Airflow metadata DB to avoid conflicts with application data
- Initial deployment is for development/testing only

---

## File List

| File | Action | Description |
|------|--------|-------------|
| `docker/docker-compose.airflow.yml` | Created | Airflow services (webserver, scheduler, init, cli) |
| `.env` | Modified | Added Airflow environment variables |
| `docker/airflow-entrypoint.sh` | Created | Auto-initialization entrypoint script |
| `docker/init-db.sh` | Modified | Added Airflow database and user creation |
| `docker/docker-compose.yml` | Modified | Added postgres/redis to textiq-network |
| `scripts/setup_airflow_connections.py` | Created | Script to configure Airflow connections |
| `airflow/dags/.gitkeep` | Created | DAGs directory placeholder |
| `airflow/plugins/.gitkeep` | Created | Plugins directory placeholder |
| `airflow/config/.gitkeep` | Created | Config directory placeholder |
| `airflow/.gitignore` | Created | Ignore logs directory |
| `docker/README-airflow.md` | Created | Startup and usage documentation |

---

## Dev Agent Record

### Implementation Notes

- Used Apache Airflow 3.0.6 with Python 3.11 (stable release as per tech spec)
- Configured LocalExecutor for simplicity (single-node development)
- Created separate `airflow` database user and database in PostgreSQL
- Added all services to `textiq-network` for inter-container communication
- Airflow webserver exposed on port 8081 to avoid conflicts with other services
- Created comprehensive connection setup script for all TextIQ services
- Used Docker Compose profiles for init and CLI containers

### Verification Results

All tests passed:
- Webserver accessible at http://localhost:8081
- Health endpoint returns healthy status for metadatabase and scheduler
- Admin user created successfully (admin/admin)

---

## Change Log

| Date | Change |
|------|--------|
| 2025-12-08 | Initial implementation of Airflow Docker infrastructure |
| 2025-12-08 | Fixed entrypoint script to prepend 'airflow' to commands |
| 2025-12-08 | Verified all acceptance criteria - story complete |