# Sprint Change Proposal - Vector Database Technology Change

**Project:** DSOL
**Date:** 2025-11-28
**Prepared For:** TextIQ
**Change Scope:** Technology Stack Substitution (Qdrant ’ Milvus)
**Epic Affected:** Epic 4 (Knowledge Base & Indexing)
**Classification:** Minor - Direct implementation change

---

## Section 1: Issue Summary

### Problem Statement

Replace Qdrant with Milvus as the vector database solution for semantic search in Epic 4 (Knowledge Base & Indexing).

### Context

This change was requested proactively during Epic 4 planning, before full implementation of the vector storage layer. Currently, only one function (`delete_vectors_by_document_id()`) has been implemented using Qdrant. This presents an optimal window for technology substitution with minimal rework.

### Discovery Context

- **When:** During Epic 4 planning phase
- **How:** Strategic technology decision
- **Category:** Technology stack choice revision

### Evidence

- Current codebase analysis shows 22 files reference Qdrant (mostly documentation and configuration)
- Actual implementation: Only 1 function in `src/knowledge/vector_store.py` (61 lines)
- Epic 4 stories (4.4, 4.5) have not been started
- PRD never mandated Qdrant specifically - line 442: "Vector database for semantic search (Pinecone, Weaviate, or similar)"

---

## Section 2: Impact Analysis

### Epic Impact

#### Epic 1: Foundation & API Skeleton
-  **Story 1.1** - Project Initialization: Update docker-compose.yml, pyproject.toml
-  **Story 1.5** - Health Check Endpoint: Update Milvus health check function

#### Epic 2: Document Upload & Management
-  **Story 2.6** - Document Deletion: Update vector deletion to use Milvus API

#### Epic 4: Knowledge Base & Indexing
-  **Story 4.4** - Vector Embedding and Indexing: Implement Milvus collection, HNSW indexing
-  **Story 4.5** - Incremental Indexing: Update deletion/update patterns for Milvus

#### Epic 5: Q&A & Answer Generation
- 9 **Minimal indirect impact** - Will use abstraction layer from Epic 4

**Epic Status Assessment:**
- No epics are invalidated
- No epic resequencing needed
- All functional requirements remain achievable

### Artifact Conflicts and Updates Required

####  PRD (docs/prd.md)
**Conflict Level:** None
**Status:** No changes needed

**Analysis:** PRD line 442 generically mentions "Vector database for semantic search (Pinecone, Weaviate, or similar)". Qdrant was an implementation choice, not a requirement. Milvus satisfies the same functional need.

####   Architecture Document (docs/architecture.md)
**Conflict Level:** High (multiple sections affected)
**Status:** Updates required

**Sections Requiring Changes:**
- Line 21: Decision table (update Qdrant ’ Milvus 2.4)
- Lines 179-180: Technology stack description
- Lines 329-350: Vector store schema (Qdrant payload ’ Milvus field schema)
- Lines 605-606: Configuration settings (qdrant_url ’ milvus_host/port)
- Line 699: docker-compose.yml reference
- Line 707: Deployment notes
- Lines 771-772: ADR-002 rationale

####  UI/UX Specifications
**Conflict Level:** None
**Status:** No changes needed (backend infrastructure)

####   Infrastructure & Configuration
**Conflict Level:** Medium
**Status:** Updates required

**Files Affected:**
- `docker-compose.yml` - Replace Qdrant service with Milvus Standalone
- `pyproject.toml` - Replace `qdrant-client>=1.16` with `pymilvus>=2.4.0`
- `.env.example` - Update environment variables (QDRANT_URL ’ MILVUS_HOST/PORT)

####   Code Files
**Conflict Level:** Low (minimal implementation)
**Status:** 3 files require updates

**Files:**
1. **`src/knowledge/vector_store.py`** - Complete rewrite (61 lines ’ ~80 lines)
2. **`src/services/health.py`** - Update health check function
3. **`src/core/config.py`** - Update configuration settings

####   Documentation
**Conflict Level:** Medium
**Status:** Multiple files require updates

**Files:**
- `docs/epics.md` - 5 story updates (1.1, 1.5, 2.6, 4.4, 4.5)
- `docs/sprint-artifacts/` - 5 story files
- `docs/specs/knowledge-indexing-tech-spec.md` - Complete review
- `README.md` - Setup instructions

### Technical Impact Summary

| Category | Files Affected | Change Complexity |
|----------|----------------|------------------|
| Core Documentation | 2 | Medium |
| Sprint Artifacts | ~5 | Low (text updates) |
| Code | 3 | Low-Medium |
| Configuration | 3 | Low |
| Total | ~13 | **Overall: Low-Medium** |

**Change Distribution:**
- 60% Documentation updates (text references)
- 30% Code adaptation (API migration)
- 10% Infrastructure configuration

---

## Section 3: Recommended Approach

### Path Forward: Option 1 - Direct Adjustment 

**Selected Approach:** Modify existing stories in Epic 4 and update all references to use Milvus instead of Qdrant.

### Rationale

#### Why This Approach?

1. **Minimal Existing Implementation**
   - Only 1 function implemented: `delete_vectors_by_document_id()` (61 lines)
   - No production data to migrate
   - No complex integration patterns established

2. **Optimal Timing**
   - Epic 4 stories (4.4, 4.5) have not been started
   - Early in implementation cycle
   - No downstream dependencies affected

3. **API Similarity**
   - Both Qdrant and Milvus are vector databases with similar capabilities
   - Python SDKs (`qdrant-client` vs `pymilvus`) have comparable APIs
   - Core operations (insert, search, delete, filter) map 1:1

4. **Low Technical Risk**
   - Well-documented Milvus SDK (pymilvus 2.4+)
   - Mature open-source project with strong community
   - Similar deployment patterns (Docker, local dev + production cluster)

5. **Maintains Project Momentum**
   - No need to rollback completed work
   - MVP scope unchanged
   - Team can proceed with Epic 4 using new technology

### Comparison of Options

| Criteria | Option 1: Direct Adjustment | Option 2: Rollback | Option 3: MVP Review |
|----------|----------------------------|-------------------|---------------------|
| **Effort** | Low-Medium (3-5 days) | N/A (nothing to rollback) | Not needed |
| **Risk** | Low | N/A | Not applicable |
| **Timeline Impact** | Minimal (~1 week) | N/A | N/A |
| **MVP Impact** | None | N/A | None |
| **Team Impact** | Positive (better tech stack) | N/A | N/A |
| **Viability** |  **VIABLE** | Not applicable | Not needed |

### Trade-offs Considered

**Advantages of Milvus:**
- Open source with active community (40k+ GitHub stars)
- Superior HNSW indexing performance
- Advanced scalar filtering capabilities
- Native support for hybrid search (future enhancement)
- Lower hosting costs (self-hosted, no cloud licensing)

**Advantages of Qdrant (Current):**
- Already documented in architecture
- Team may have familiarity with API

**Decision:** The advantages of Milvus outweigh the minimal switching cost given the early implementation stage.

---

## Section 4: Detailed Change Proposals

Below are all specific file changes required, organized by category.

### 4.1 Infrastructure Changes

#### Docker Compose (`docker-compose.yml`)

**Section:** Services
**Lines:** 34-47

**OLD:**
```yaml
qdrant:
  image: qdrant/qdrant:v1.16.0
  container_name: dsol-qdrant
  ports:
    - "6333:6333"
    - "6334:6334"
  volumes:
    - qdrant_data:/qdrant/storage
  healthcheck:
    test: ["CMD", "wget", "-q", "--spider", "http://localhost:6333/readyz"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 10s
```

**NEW:**
```yaml
milvus:
  image: milvusdb/milvus:v2.4-latest
  container_name: dsol-milvus
  environment:
    ETCD_USE_EMBED: "true"
    COMMON_STORAGETYPE: local
  ports:
    - "19530:19530"  # gRPC
    - "9091:9091"    # HTTP
  volumes:
    - milvus_data:/var/lib/milvus
  healthcheck:
    test: ["CMD", "curl", "-f", "http://localhost:9091/healthz"]
    interval: 10s
    timeout: 5s
    retries: 5
    start_period: 10s
```

**Volumes Section - OLD:**
```yaml
qdrant_data:
  name: dsol_qdrant_data
```

**Volumes Section - NEW:**
```yaml
milvus_data:
  name: dsol_milvus_data
```

**Rationale:** Milvus Standalone uses embedded etcd for local development, exposing gRPC (19530) and HTTP (9091) ports.

---

#### Dependencies (`pyproject.toml`)

**Line:** 19

**OLD:**
```toml
"qdrant-client>=1.16",
```

**NEW:**
```toml
"pymilvus>=2.4.0",
```

**Rationale:** Replace Qdrant client with official Milvus Python SDK.

---

#### Environment Configuration (`.env.example`)

**OLD:**
```bash
QDRANT_URL=http://localhost:6333
QDRANT_COLLECTION=dsol_records
```

**NEW:**
```bash
MILVUS_HOST=localhost
MILVUS_PORT=19530
MILVUS_COLLECTION=dsol_records
```

**Rationale:** Milvus uses separate host/port configuration instead of URL.

---

### 4.2 Code Changes

#### Configuration (`src/core/config.py`)

**Lines:** 15-17

**OLD:**
```python
# Qdrant
qdrant_url: str = "http://localhost:6333"
qdrant_collection: str = "dsol_records"
```

**NEW:**
```python
# Milvus
milvus_host: str = "localhost"
milvus_port: int = 19530
milvus_collection: str = "dsol_records"
```

**Rationale:** Update settings class to use Milvus connection parameters.

---

#### Vector Store (`src/knowledge/vector_store.py`) - COMPLETE REWRITE

**CURRENT IMPLEMENTATION:**
```python
"""Qdrant vector store operations for document embeddings."""

from uuid import UUID
import structlog
from qdrant_client import QdrantClient
from qdrant_client.http import models
from src.core.config import settings

logger = structlog.get_logger(__name__)

async def delete_vectors_by_document_id(document_id: UUID) -> int:
    """Delete all vectors for a document from Qdrant."""
    try:
        client = QdrantClient(url=settings.qdrant_url)

        result = client.delete(
            collection_name=settings.qdrant_collection,
            points_selector=models.FilterSelector(
                filter=models.Filter(
                    must=[
                        models.FieldCondition(
                            key="document_id",
                            match=models.MatchValue(value=str(document_id)),
                        )
                    ]
                )
            ),
        )

        logger.info(
            "qdrant_vectors_deleted",
            document_id=str(document_id),
            operation_id=result.operation_id if result else None,
        )

        return result.operation_id if result else 0

    except Exception as e:
        logger.error(
            "qdrant_deletion_failed",
            document_id=str(document_id),
            error=str(e),
        )
        raise
```

**NEW IMPLEMENTATION:**
```python
"""Milvus vector store operations for document embeddings."""

from uuid import UUID
import structlog
from pymilvus import Collection, connections
from src.core.config import settings

logger = structlog.get_logger(__name__)

async def delete_vectors_by_document_id(document_id: UUID) -> int:
    """Delete all vectors for a document from Milvus.

    Args:
        document_id: Document UUID to filter vectors.

    Returns:
        Number of vectors deleted.

    Raises:
        Exception: If Milvus deletion fails.
    """
    try:
        # Connect to Milvus
        connections.connect(
            alias="default",
            host=settings.milvus_host,
            port=settings.milvus_port
        )

        collection = Collection(settings.milvus_collection)

        # Delete by document_id filter expression
        expr = f'document_id == "{str(document_id)}"'
        result = collection.delete(expr)

        logger.info(
            "milvus_vectors_deleted",
            document_id=str(document_id),
            delete_count=result.delete_count,
        )

        # Disconnect
        connections.disconnect("default")

        return result.delete_count

    except Exception as e:
        logger.error(
            "milvus_deletion_failed",
            document_id=str(document_id),
            error=str(e),
        )
        raise
```

**Rationale:** Equivalent functionality using Milvus SDK. Delete expression syntax is simpler than Qdrant's nested filter structure.

---

#### Health Check (`src/services/health.py`)

**Function:** `check_qdrant()` ’ `check_milvus()`
**Lines:** 165-227

**OLD:**
```python
async def check_qdrant(qdrant_client: AsyncQdrantClient) -> DependencyHealth:
    """Check Qdrant vector store health."""
    start_time = time.perf_counter()

    try:
        async with asyncio.timeout(HEALTH_CHECK_TIMEOUT):
            await qdrant_client.get_collections()
            latency_ms = int((time.perf_counter() - start_time) * 1000)

            logger.debug("qdrant_health_check", status="healthy", latency_ms=latency_ms)

            return DependencyHealth(status="healthy", latency_ms=latency_ms)

    except asyncio.TimeoutError:
        # error handling...
```

**NEW:**
```python
async def check_milvus() -> DependencyHealth:
    """Check Milvus vector store health.

    Connects to Milvus and lists collections to verify connectivity.

    Returns:
        DependencyHealth with status and latency.
    """
    start_time = time.perf_counter()

    try:
        async with asyncio.timeout(HEALTH_CHECK_TIMEOUT):
            from pymilvus import connections, utility

            connections.connect(
                alias="health_check",
                host=settings.milvus_host,
                port=settings.milvus_port
            )

            # List collections to verify connection
            utility.list_collections(using="health_check")

            connections.disconnect("health_check")

            latency_ms = int((time.perf_counter() - start_time) * 1000)

            logger.debug("milvus_health_check", status="healthy", latency_ms=latency_ms)

            return DependencyHealth(status="healthy", latency_ms=latency_ms)

    except asyncio.TimeoutError:
        latency_ms = int((time.perf_counter() - start_time) * 1000)
        error_msg = "Health check timed out"

        logger.warning("milvus_health_check", status="unhealthy", error=error_msg, latency_ms=latency_ms)

        return DependencyHealth(status="unhealthy", latency_ms=latency_ms, error=error_msg)

    except Exception as e:
        latency_ms = int((time.perf_counter() - start_time) * 1000)
        error_msg = str(e)

        logger.warning("milvus_health_check", status="unhealthy", error=error_msg, latency_ms=latency_ms)

        return DependencyHealth(status="unhealthy", latency_ms=latency_ms, error=error_msg)
```

**Rationale:** Equivalent health check using Milvus connection verification.

---

### 4.3 Documentation Changes

#### Architecture Document (`docs/architecture.md`)

**Change 1 - Decision Table (Line 21):**

OLD:
```markdown
| Vector Store | Qdrant | 1.16 | FR19-20 | Local/cloud, good Python SDK, metadata filtering |
```

NEW:
```markdown
| Vector Store | Milvus | 2.4 | FR19-20 | Open source, HNSW indexing, excellent Python SDK, advanced filtering |
```

---

**Change 2 - Technology Description (Lines 179-180):**

OLD:
```markdown
**Qdrant**
- Vector similarity search
- Metadata filtering (file, sheet, record_id)
- Local deployment for development
- Cloud deployment for production
```

NEW:
```markdown
**Milvus**
- Vector similarity search with HNSW/IVF indexing
- Scalar field filtering (file, sheet, record_id)
- Local deployment via Docker for development
- Milvus Standalone or Cluster for production
- Native support for hybrid search (dense + sparse vectors)
```

---

**Change 3 - Vector Store Schema (Lines 329-350):**

OLD:
```python
# Collection: dsol_records
{
    "name": "dsol_records",
    "vectors": {
        "size": 3072,
        "distance": "Cosine"
    },
    "payload_schema": {
        "record_id": "uuid",
        "document_id": "uuid",
        "filename": "keyword",
        "sheet_name": "keyword",
        "row_number": "integer",
        "text_content": "text",
        "headers": "keyword[]",
        "has_symbols": "bool"
    }
}
```

NEW:
```python
# Collection: dsol_records
{
    "name": "dsol_records",
    "schema": [
        {"name": "id", "dtype": "INT64", "is_primary": True, "auto_id": True},
        {"name": "vector", "dtype": "FLOAT_VECTOR", "dim": 3072},
        {"name": "record_id", "dtype": "VARCHAR", "max_length": 36},
        {"name": "document_id", "dtype": "VARCHAR", "max_length": 36},
        {"name": "filename", "dtype": "VARCHAR", "max_length": 255},
        {"name": "sheet_name", "dtype": "VARCHAR", "max_length": 255},
        {"name": "row_number", "dtype": "INT64"},
        {"name": "text_content", "dtype": "VARCHAR", "max_length": 65535},
        {"name": "has_symbols", "dtype": "BOOL"}
    ],
    "index_params": {
        "metric_type": "COSINE",
        "index_type": "HNSW",
        "params": {"M": 16, "efConstruction": 200}
    }
}
```

---

**Change 4 - Environment Configuration (Lines 605-606):**

OLD:
```python
# Vector store
qdrant_url: str = "http://localhost:6333"
qdrant_collection: str = "dsol_records"
```

NEW:
```python
# Vector store
milvus_host: str = "localhost"
milvus_port: int = 19530
milvus_collection: str = "dsol_records"
```

---

**Change 5 - Docker Compose Reference (Line 699):**

OLD:
```yaml
qdrant:
    image: qdrant/qdrant:v1.16.0
```

NEW:
```yaml
milvus:
    image: milvusdb/milvus:v2.4-latest
```

---

**Change 6 - Deployment Notes (Line 707):**

OLD:
```markdown
- Qdrant Cloud or self-hosted
```

NEW:
```markdown
- Milvus Standalone (development) or Cluster (production)
```

---

**Change 7 - ADR-002 Update (Lines 771-772):**

OLD:
```markdown
### ADR-002: Separate Structured and Vector Stores

**Decision:** Use PostgreSQL + Qdrant (not single solution)

**Rationale:**
- PostgreSQL excellent for exact matches, complex queries
- Qdrant optimized for vector similarity
- Hybrid approach maps to query types in PRD
- Avoid vendor lock-in
```

NEW:
```markdown
### ADR-002: Separate Structured and Vector Stores

**Decision:** Use PostgreSQL + Milvus (not single solution)

**Rationale:**
- PostgreSQL excellent for exact matches, complex queries
- Milvus optimized for vector similarity with HNSW indexing
- Hybrid approach maps to query types in PRD
- Open source with strong community support
- Advanced filtering capabilities for scalar fields
```

---

#### Epic Document (`docs/epics.md`)

**Story 1.1 - Project Initialization (Line 140)**

OLD:
```markdown
- `docker-compose.yml` with PostgreSQL 18, Redis 7, Qdrant 1.16
```

NEW:
```markdown
- `docker-compose.yml` with PostgreSQL 18, Redis 7, Milvus 2.4
```

---

**Story 1.5 - Health Check Endpoint (Line 270)**

OLD:
```markdown
- Check Qdrant with collection list
```

NEW:
```markdown
- Check Milvus with collection list
```

---

**Story 2.6 - Document Deletion (Line 569)**

OLD:
```markdown
- All vectors in Qdrant for this document
```

NEW:
```markdown
- All vectors in Milvus for this document
```

---

**Story 4.4 - Vector Embedding and Indexing (Lines 1318-1340)**

**Acceptance Criteria Changes:**

OLD:
```markdown
3. Stored in Qdrant collection `dsol_records` with metadata

**And** Qdrant payload includes:
```

NEW:
```markdown
3. Stored in Milvus collection `dsol_records` with metadata

**And** Milvus entity fields include:
```

**Technical Notes Changes:**

OLD:
```markdown
- Use Azure OpenAI text-embedding-3-large (3072 dimensions)
- Batch API calls to reduce latency
- Module: `src/knowledge/vector_store.py`, `src/knowledge/embeddings.py`
```

NEW:
```markdown
- Use Azure OpenAI text-embedding-3-large (3072 dimensions)
- Batch API calls to reduce latency
- Milvus collection with HNSW index for fast similarity search
- Module: `src/knowledge/vector_store.py`, `src/knowledge/embeddings.py`
```

---

**Story 4.5 - Incremental Indexing (Lines 1361-1371)**

OLD:
```markdown
- Its vectors are removed from Qdrant (by document_id filter)

**Technical Notes:**
- Use document_id as partition key
- Qdrant delete by filter: `{"must": [{"key": "document_id", "match": {"value": "uuid"}}]}`
```

NEW:
```markdown
- Its vectors are removed from Milvus (by document_id filter expression)

**Technical Notes:**
- Use document_id as partition key
- Milvus delete by expression: `document_id == "uuid"`
```

---

#### Sprint Artifact Files

**Files Requiring Updates (Pattern: Replace all Qdrant references):**

1. `docs/sprint-artifacts/1-1-project-initialization.md`
2. `docs/sprint-artifacts/1-5-health-check-endpoint.md`
3. `docs/sprint-artifacts/2-6-document-deletion.md`
4. `docs/sprint-artifacts/tech-spec-epic-1.md`
5. `docs/sprint-artifacts/tech-spec-epic-2.md`

**Change Pattern:** Replace all `Qdrant` references with `Milvus`, update code examples and acceptance criteria to match epic changes above.

---

#### Additional Documentation

**README.md:**
- Update "Setup Instructions" section
- Change service references from Qdrant to Milvus
- Update port numbers (6333 ’ 19530)
- Update health check examples

**docs/specs/knowledge-indexing-tech-spec.md:**
- Complete review and update
- Update collection schema definitions
- Update index configuration (HNSW parameters)
- Update API examples

---

## Section 5: Implementation Handoff

### Change Scope Classification

**Classification:** **Minor - Direct Implementation**

**Justification:**
- Minimal existing implementation (1 function, 61 lines)
- No architectural changes - only technology substitution
- No MVP scope impact
- No functional requirement changes
- Affects implementation details of Epic 4 only

### Handoff Recipients and Responsibilities

#### 1. Development Team (Primary)

**Responsibility:** Execute all technical changes

**Tasks:**
- [ ] Update infrastructure files (docker-compose, pyproject.toml, .env.example)
- [ ] Rewrite vector store implementation (`src/knowledge/vector_store.py`)
- [ ] Update health check service (`src/services/health.py`)
- [ ] Update configuration settings (`src/core/config.py`)
- [ ] Test Milvus integration
- [ ] Validate deletion functionality
- [ ] Update API dependencies if needed (`src/api/dependencies.py`)

**Estimated Effort:** 3-5 days
- **Day 1:** Infrastructure setup + dependency installation + Milvus container validation
- **Day 2:** Code implementation (vector_store.py, health.py, config.py)
- **Day 3:** Testing and debugging
- **Day 4-5:** Documentation updates + final validation

#### 2. Technical Writer / Documentation Lead (Secondary)

**Responsibility:** Update all documentation

**Tasks:**
- [ ] Update `docs/architecture.md` (7 sections)
- [ ] Update `docs/epics.md` (5 stories)
- [ ] Update sprint artifact files (~5 files)
- [ ] Update tech spec files
- [ ] Update README.md setup instructions

**Estimated Effort:** 1-2 days

#### 3. Product Owner / Scrum Master (Informational)

**Responsibility:** Review and approve changes, update backlog

**Tasks:**
- [ ] Review Sprint Change Proposal
- [ ] Approve technology substitution
- [ ] Update Epic 4 story acceptance criteria if needed
- [ ] Communicate change to stakeholders

**Estimated Effort:** 0.5 days (review + approval)

### Success Criteria

#### Technical Validation
-  Milvus container starts successfully via docker-compose
-  Health check endpoint returns "healthy" for Milvus
-  `delete_vectors_by_document_id()` function works correctly
-  All tests pass with Milvus integration
-  No references to Qdrant remain in codebase
-  Milvus collection can be created and queried

#### Documentation Validation
-  All documentation updated with Milvus references
-  Setup instructions tested and working
-  Architecture document accurate and consistent
-  No orphaned Qdrant references in any files

#### Functional Validation
-  Vector deletion by document_id works
-  No regression in existing Epic 1-3 functionality
-  Ready to proceed with Epic 4 Story 4.4 implementation

### Dependencies and Sequencing

**Sequence:**

1. **Infrastructure First** (Day 1)
   - Update docker-compose.yml
   - Update pyproject.toml
   - Start Milvus container and validate connectivity
   - Verify Milvus UI accessible at http://localhost:9091

2. **Code Implementation** (Day 2)
   - Update configuration (config.py)
   - Rewrite vector_store.py
   - Update health check (health.py)

3. **Testing** (Day 3)
   - Unit tests for vector store functions
   - Integration test for health check
   - Manual validation of Milvus connection
   - Test deletion functionality

4. **Documentation** (Day 4-5)
   - Update all documentation files
   - Review and validate consistency
   - Run grep to find any remaining Qdrant references

**Critical Path:** Infrastructure ’ Code ’ Testing ’ Documentation

**No Blockers:** All work can proceed sequentially without external dependencies.

---

## Section 6: Risk Assessment and Mitigation

### Identified Risks

| Risk | Probability | Impact | Mitigation |
|------|------------|--------|------------|
| Milvus SDK API differences | Low | Medium | Thorough API documentation review, prototype key operations |
| Milvus container startup issues | Low | Low | Use official milvusdb/milvus image, test locally first |
| Performance differences vs Qdrant | Low | Low | Benchmark after implementation, adjust HNSW params if needed |
| Team unfamiliarity with Milvus | Medium | Low | Provide SDK documentation, reference examples |
| Documentation inconsistencies | Medium | Low | Systematic review checklist, grep for "qdrant" case-insensitive |

### Mitigation Strategies

1. **API Compatibility Validation**
   - Create proof-of-concept script testing core operations (connect, create collection, insert, search, delete)
   - Validate filter expression syntax matches requirements
   - Document any API differences discovered

2. **Incremental Testing**
   - Test Milvus container startup independently before code changes
   - Test configuration changes before implementing new code
   - Validate health check before proceeding to vector store rewrite

3. **Documentation Review**
   - Run `grep -ri "qdrant" .` to find all references (case-insensitive)
   - Create checklist of files to update (see Appendix A)
   - Cross-reference with sprint artifact list

4. **Rollback Plan**
   - Keep Qdrant configuration in git history
   - Document exact commit hash before Milvus changes
   - Maintain ability to revert if critical issues arise (low probability)

---

## Section 7: Timeline and Effort Estimate

### Estimated Timeline

**Total Effort:** 4-6 days (1 sprint)

**Breakdown:**

| Phase | Duration | Owner | Deliverables |
|-------|----------|-------|--------------|
| **Infrastructure Setup** | 1 day | Dev Team | docker-compose, dependencies, env vars, Milvus running |
| **Code Implementation** | 1.5 days | Dev Team | vector_store.py, health.py, config.py |
| **Testing & Validation** | 1 day | Dev Team | Unit tests, integration tests, manual validation |
| **Documentation Updates** | 1 day | Dev Team + Tech Writer | All docs updated, README revised |
| **Review & Approval** | 0.5 days | PO/SM | Approval, backlog updates |

### Parallel Work Opportunities

- Documentation updates can begin in parallel with code implementation (Days 2-3)
- Tech Writer can start on architecture.md while Dev Team works on code
- Sprint artifact updates can happen independently
- README updates can be done concurrently

---

## Section 8: Comparison - Qdrant vs Milvus

### Technical Comparison

| Feature | Qdrant 1.16 | Milvus 2.4 |
|---------|------------|-----------|
| **Open Source** | Yes (Apache 2.0) | Yes (Apache 2.0) |
| **Vector Indexing** | HNSW | HNSW, IVF_FLAT, IVF_SQ8, IVF_PQ |
| **Distance Metrics** | Cosine, Euclidean, Dot | Cosine, Euclidean, IP, L2 |
| **Scalar Filtering** | Payload filters | Expression-based filtering |
| **Python SDK** | qdrant-client | pymilvus (official) |
| **Hybrid Search** | No (roadmap) | Yes (dense + sparse) |
| **Deployment** | Single binary | Standalone / Cluster |
| **Cloud Offering** | Qdrant Cloud | Zilliz Cloud |
| **Community** | 12k+ GitHub stars | 40k+ GitHub stars |
| **Maturity** | 2+ years | 5+ years (LF AI project) |
| **Performance** | Fast HNSW | Optimized HNSW + multiple indexes |
| **Documentation** | Good | Excellent |

### Why Milvus?

1. **Maturity & Stability**
   - Linux Foundation AI project with 5+ years of development
   - Battle-tested in production at scale (billion+ vectors)
   - Strong governance and roadmap

2. **Community & Ecosystem**
   - Significantly larger community (40k vs 12k GitHub stars)
   - Active forums, Discord, extensive documentation
   - Rich tooling ecosystem (Attu UI, monitoring tools)

3. **Technical Features**
   - More index types (HNSW, IVF variants) for different use cases
   - Native hybrid search support (dense + sparse vectors)
   - Advanced scalar filtering with SQL-like expressions
   - Better performance optimization options

4. **Cost & Licensing**
   - Fully open source (Apache 2.0)
   - No vendor lock-in
   - Self-hosted option with no cloud fees
   - Transparent pricing for cloud offering (Zilliz)

5. **Future-Proofing**
   - Stronger long-term viability (LF AI backing)
   - Active development roadmap
   - Growing adoption in enterprise

---

## Appendices

### Appendix A: File Change Checklist

**Code Files (3):**
- [ ] `src/knowledge/vector_store.py` - Complete rewrite
- [ ] `src/services/health.py` - Update health check function
- [ ] `src/core/config.py` - Update configuration

**Configuration Files (3):**
- [ ] `docker-compose.yml` - Replace Qdrant service with Milvus
- [ ] `pyproject.toml` - Update dependencies
- [ ] `.env.example` - Update environment variables

**Documentation Files (10+):**
- [ ] `docs/architecture.md` - 7 sections to update
- [ ] `docs/epics.md` - 5 stories to update
- [ ] `docs/sprint-artifacts/1-1-project-initialization.md`
- [ ] `docs/sprint-artifacts/1-5-health-check-endpoint.md`
- [ ] `docs/sprint-artifacts/2-6-document-deletion.md`
- [ ] `docs/sprint-artifacts/tech-spec-epic-1.md`
- [ ] `docs/sprint-artifacts/tech-spec-epic-2.md`
- [ ] `docs/specs/knowledge-indexing-tech-spec.md`
- [ ] `README.md`

**Validation:**
- [ ] Run `grep -ri "qdrant" .` to find any missed references
- [ ] Check all log messages for "qdrant" strings
- [ ] Verify import statements updated

---

### Appendix B: Milvus Quick Reference

**Connection:**
```python
from pymilvus import connections
connections.connect(host="localhost", port=19530)
```

**Create Collection:**
```python
from pymilvus import CollectionSchema, FieldSchema, DataType, Collection

fields = [
    FieldSchema(name="id", dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name="vector", dtype=DataType.FLOAT_VECTOR, dim=3072),
    FieldSchema(name="document_id", dtype=DataType.VARCHAR, max_length=36),
    FieldSchema(name="record_id", dtype=DataType.VARCHAR, max_length=36),
    FieldSchema(name="filename", dtype=DataType.VARCHAR, max_length=255),
    FieldSchema(name="sheet_name", dtype=DataType.VARCHAR, max_length=255),
    FieldSchema(name="row_number", dtype=DataType.INT64),
    FieldSchema(name="text_content", dtype=DataType.VARCHAR, max_length=65535),
    FieldSchema(name="has_symbols", dtype=DataType.BOOL),
]
schema = CollectionSchema(fields, description="DSOL records")
collection = Collection(name="dsol_records", schema=schema)
```

**Create Index:**
```python
index_params = {
    "metric_type": "COSINE",
    "index_type": "HNSW",
    "params": {"M": 16, "efConstruction": 200}
}
collection.create_index(field_name="vector", index_params=index_params)
collection.load()  # Load collection into memory for searching
```

**Insert:**
```python
entities = [
    [doc_ids],      # document_id field
    [record_ids],   # record_id field
    [filenames],    # filename field
    [sheet_names],  # sheet_name field
    [row_numbers],  # row_number field
    [text_contents], # text_content field
    [has_symbols],  # has_symbols field
    [vectors],      # vector field
]
collection.insert(entities)
collection.flush()  # Persist data
```

**Search:**
```python
search_params = {"metric_type": "COSINE", "params": {"ef": 100}}
results = collection.search(
    data=[query_vector],
    anns_field="vector",
    param=search_params,
    limit=10,
    expr='document_id == "abc-123"'  # Scalar filter
)
```

**Delete:**
```python
expr = f'document_id == "{doc_id}"'
result = collection.delete(expr)
print(f"Deleted {result.delete_count} vectors")
```

---

### Appendix C: Testing Checklist

**Unit Tests:**
- [ ] Test Milvus connection establishment
- [ ] Test collection creation with proper schema
- [ ] Test vector insertion (single and batch)
- [ ] Test vector search with filtering
- [ ] Test vector deletion by document_id
- [ ] Test health check function
- [ ] Test error handling for connection failures

**Integration Tests:**
- [ ] Test full workflow: upload document ’ extract ’ index ’ search ’ delete
- [ ] Test health check endpoint returns Milvus status
- [ ] Test error handling for Milvus connection failures
- [ ] Test concurrent operations

**Manual Validation:**
- [ ] Start Milvus via docker-compose
- [ ] Verify Milvus accessible at http://localhost:9091 (HTTP)
- [ ] Verify Milvus gRPC port 19530 accessible
- [ ] Test vector deletion manually via Python script
- [ ] Validate no Qdrant references in logs
- [ ] Check Milvus UI for collection creation and data

**Performance Tests:**
- [ ] Benchmark vector search latency
- [ ] Test batch insertion performance
- [ ] Validate HNSW index build time

---

### Appendix D: Implementation Notes

**Milvus-Specific Considerations:**

1. **Collection Must Be Loaded:**
   - After creating or inserting, call `collection.load()` to load into memory
   - Collections must be loaded before searching

2. **Data Persistence:**
   - Call `collection.flush()` after insertion to persist data
   - Milvus has automatic flush, but explicit flush ensures durability

3. **Connection Management:**
   - Use connection aliases for different contexts (e.g., "default", "health_check")
   - Always disconnect when done to release resources
   - Consider connection pooling for production

4. **Index Building:**
   - Index must be created before searching
   - HNSW parameters: M=16 (connections per node), efConstruction=200 (build quality)
   - Higher M and efConstruction = better accuracy but slower build
   - For production, tune based on dataset size and accuracy requirements

5. **Expression Filtering:**
   - Use SQL-like expressions: `document_id == "value"`
   - Supports: ==, !=, >, <, >=, <=, in, and, or
   - String fields must use double quotes: `filename == "test.xlsx"`

---

## Approval

**Prepared By:** AI Development Facilitator (Correct Course Workflow)
**Date:** 2025-11-28

**Awaiting Approval From:** TextIQ (Product Owner)

**Next Steps:**
1.  Review this Sprint Change Proposal
2.  Approve or request revisions
3.  Upon approval, route to Development Team for implementation
4.  Track implementation progress via todo list

---

**Document Status:**  Complete - Ready for Review
**Version:** 1.0
**Classification:** Minor - Direct Implementation Change
**Workflow Mode:** Batch Analysis

---

=€ **Generated with BMAD Correct Course Workflow**