# Semantic Tagging Service

## Overview

The Semantic Tagging Service assigns standardized, retrievable tags to extracted document records using LLM-based classification. This enables precise content retrieval for specific document sections.

## Features

- **Config-Driven Taxonomy**: Tag definitions in `config/tag_taxonomy.yaml`
- **Automated Classification**: LLM assigns tags based on ISO 29148 and Diátaxis standards
- **Batch Processing**: Efficient LLM usage with configurable batch sizes
- **Error Resilience**: Graceful failure handling (empty tags on error)
- **Flexible Filtering**: Query by tag combinations (`contains_any`, `contains_all`)

## Tag Taxonomy (SoftDev Base Profile)

### Requirements (`req:`)
- `req:functional` - Functional requirements
- `req:non_functional` - Quality attribute requirements (NFRs)
- `req:constraint` - Design/business constraints
- `req:interface` - External interface requirements
- `req:data` - Data requirements

### Risk & Compliance (`risk:`)
- `risk:safety` - Safety-critical items
- `risk:security` - Security considerations
- `risk:assumption` - Project assumptions

### Information (`info:`)
- `info:background` - Background/context
- `info:scope` - Scope definitions

### Topics (`topic:`)
- `topic:{keyword}` - Domain topics (LLM-extracted)
- Examples: `topic:authentication`, `topic:billing`, `topic:api`

## Configuration

### Environment Variables

```bash
# Batch size for LLM tagging (default: 20)
TAG_BATCH_SIZE=20
```

### Taxonomy Configuration

Edit `config/tag_taxonomy.yaml` to add new tags:

```yaml
version: "1.0"
namespaces:
  custom:
    description: "Custom namespace"
    tags:
      - name: my_tag
        description: "My custom tag"
        examples: ["Example text"]
```

## API Usage

### Query with Tag Filtering

```http
POST /query
Content-Type: application/json

{
  "query": "What are the security requirements?",
  "tag_filters": {
    "contains_any": ["req:functional", "req:interface"],
    "contains_all": ["topic:security"]
  }
}
```

### Tag Filter Operators

- **`contains_any`**: Match records with ANY of the specified tags
- **`contains_all`**: Match records with ALL of the specified tags

## Architecture

### Pipeline Integration

```
1. DocumentConverter → HTML/JSON
2. SemanticChunker → Logical chunks
3. LlmHeaderDetector → Assign parent headers
4. RecordBuilder → Complete records
5. SemanticTagger → Assign tags ← NEW
6. Store → PostgreSQL + Milvus
```

### Components

- **TaxonomyLoader**: Loads `tag_taxonomy.yaml`
- **PromptBuilder**: Generates LLM prompts from taxonomy
- **BatchProcessor**: Groups records for efficient processing
- **SemanticTagger**: Main service coordinating tagging

### Data Storage

- **PostgreSQL**: `tags TEXT[]` column with GIN index
- **Milvus**: `tags` ARRAY field with INVERTED index

## Error Handling

The tagging service is designed for **fault tolerance**:

- **LLM Timeout/Rate Limit**: Tags left empty, processing continues
- **Invalid Response**: Warning logged, tags set to `[]`
- **Partial Batch Failure**: Successfully tagged records preserved
- **No Retry**: Fast pipeline execution prioritized

## Testing

Run unit tests:

```bash
pytest tests/extraction_v2/test_semantic_tagger.py -v
```

## Extending the Taxonomy

1. Edit `config/tag_taxonomy.yaml`
2. Add new namespace or tags
3. Restart the service (no code changes needed)

Example:

```yaml
performance:
  description: "Performance-related tags"
  tags:
    - name: latency
      description: "Latency requirements"
      examples: ["Response time under 100ms"]
```

## Performance

- **Batch Size**: Default 20 records/batch (configurable)
- **Cost Optimization**: Single LLM call per batch
- **Index Performance**: GIN (PostgreSQL) and INVERTED (Milvus) indexes for fast filtering

## Monitoring

Check logs for tagging metrics:

```
semantic_tagger_initialized: Tagging service started
batch_tagged: Batch N/M processed
tagging_complete: All records tagged
```

## Troubleshooting

### Tags Not Appearing

1. Check if `TAG_BATCH_SIZE` is set
2. Verify `config/tag_taxonomy.yaml` exists
3. Check logs for LLM errors

### Poor Tag Quality

1. Review taxonomy descriptions in YAML
2. Add more examples to tag definitions
3. Adjust batch size (smaller batches = more context per call)

### Performance Issues

1. Increase `TAG_BATCH_SIZE` to reduce API calls
2. Monitor LLM response times
3. Check Milvus index status

## References

- **Spec**: `docs/specs/extract_tag_function_spec.md`
- **Implementation**: `src/extraction_v2/semantic_tagger.py`
- **Tests**: `tests/extraction_v2/test_semantic_tagger.py`
- **Config**: `config/tag_taxonomy.yaml`