# Airflow Docker Setup

This document describes how to run Apache Airflow alongside the existing TextIQ infrastructure.

## Prerequisites

- Docker and Docker Compose installed
- Base services running (PostgreSQL, Redis)

## Quick Start

### 1. Start Base Infrastructure

```bash
cd docker

# Start PostgreSQL and Redis first (creates the textiq-network)
docker compose -f docker-compose.yml up -d postgres redis
```

### 2. Start Airflow Services

```bash
# Start webserver and scheduler (auto-initializes on first run)
docker compose -f docker-compose.airflow.yml up -d
```

The entrypoint script automatically:
- Waits for PostgreSQL to be ready
- Runs database migrations
- Creates admin user if it doesn't exist (default: admin/admin)

**Note:** For manual initialization (useful for debugging), you can still run:
```bash
docker compose -f docker-compose.airflow.yml --profile init up airflow-init
```

### 3. Verify Services

```bash
# Check health endpoint
curl http://localhost:8081/health

# View logs
docker compose -f docker-compose.airflow.yml logs -f
```

### 4. Access Airflow UI

Open http://localhost:8081 in your browser.

Default credentials:
- Username: `admin`
- Password: `admin`

## Configuration

### Environment Variables

Edit `docker/.env.airflow` to configure:

- `AIRFLOW_UID`: Linux user ID for Airflow containers
- `_AIRFLOW_WWW_USER_USERNAME`: Web UI admin username
- `_AIRFLOW_WWW_USER_PASSWORD`: Web UI admin password
- Service connection details (PostgreSQL, Milvus, Redis, SeaweedFS, Azure OpenAI)

### Airflow Connections

Run the connection setup script after Airflow is running:

```bash
docker compose -f docker-compose.airflow.yml exec airflow-webserver \
  python /opt/airflow/src/../scripts/setup_airflow_connections.py
```

This configures connections to:
- `postgres_default`: TextIQ PostgreSQL database
- `milvus_default`: Milvus vector database
- `redis_default`: Redis cache
- `seaweedfs_default`: SeaweedFS S3-compatible object storage
- `azure_openai_default`: Azure OpenAI API

## DAG Development

Place DAG files in the `airflow/dags/` directory. They will be automatically picked up by the scheduler.

```
airflow/
├── dags/          # DAG definitions
├── plugins/       # Custom plugins
├── logs/          # Execution logs (gitignored)
└── config/        # Additional configuration
```

## Stopping Services

```bash
# Stop Airflow only
docker compose -f docker-compose.airflow.yml down

# Stop all services including base infrastructure
docker compose -f docker-compose.yml down
docker compose -f docker-compose.airflow.yml down
```

## Troubleshooting

### Database Connection Issues

Ensure PostgreSQL is running and the `airflow` database exists:

```bash
docker exec -it textiq-postgres psql -U textiq -c "\l"
```

### Permission Issues

If you see permission errors, set your user ID:

```bash
echo "AIRFLOW_UID=$(id -u)" >> docker/.env.airflow
```

### Reset Airflow

To completely reset Airflow:

```bash
docker compose -f docker-compose.airflow.yml down -v
docker compose -f docker-compose.airflow.yml --profile init up airflow-init
docker compose -f docker-compose.airflow.yml up -d
```