# Phase 5: Tag Governance & Promotion Workflow ## Overview Phase 5 implements the **Dynamic Tag Governance** system per Specification Section 4.4, enabling administrators to: - Identify trending dynamic tags - Promote popular dynamic tags to standard taxonomy - Track tag usage statistics - Maintain taxonomy evolution ## Architecture ### Components 1. **DynamicTagAnalyzer** - Analyzes tag usage across the database 2. **TagPromotionService** - Manages tag promotion lifecycle 3. **Admin API Endpoints** - RESTful interface for governance operations 4. **CLI Script** - Command-line tool for batch migrations --- ## Tag Lifecycle (Spec Section 4.4.1) ``` ┌─────────────────────────────────────────────────────────────┐ │ Stage 1: Dynamic (Wild West) │ │ - User creates topic:gdpr │ │ - Validation: Checks namespace only │ │ - Status: Valid Dynamic Tag │ └─────────────────┬───────────────────────────────────────────┘ │ Usage grows... ▼ ┌─────────────────────────────────────────────────────────────┐ │ Stage 2: Candidate (Analysis) │ │ - System: Weekly batch job identifies trending tags │ │ - Criteria: Usage > 50 blocks across 3+ projects │ │ - Output: Taxonomy Dashboard alert for Admin │ └─────────────────┬───────────────────────────────────────────┘ │ Admin reviews ▼ ┌─────────────────────────────────────────────────────────────┐ │ Stage 3: Promotion (Governance) │ │ - Action: Admin maps topic:gdpr → compliance:regulatory:gdpr│ │ - Decision: Add to standard or profile taxonomy │ │ - Configuration: Update standard_tags.json or fintech.json │ └─────────────────┬───────────────────────────────────────────┘ │ Execute ▼ ┌─────────────────────────────────────────────────────────────┐ │ Stage 4: Migration (Data Fix) │ │ - Batch Job: UPDATE all records with new tag name │ │ - Re-indexing: Trigger vector embedding update (future) │ │ - History: Add promotion entry to tag history │ └─────────────────────────────────────────────────────────────┘ ``` --- ## API Endpoints ### 1. GET /v1/admin/taxonomy/trending **Purpose:** Identify trending dynamic tags for promotion consideration **Query Parameters:** - `min_usage` (int, default: 50) - Minimum usage count - `min_documents` (int, default: 3) - Minimum document count - `namespace` (string, optional) - Filter by namespace (topic, entity, tech, domain) **Response:** ```json { "trending_tags": [ { "name": "topic:gdpr", "usage_count": 127, "document_count": 8, "verified_ratio": 0.85, "first_seen": "2026-01-15", "last_seen": "2026-01-22" } ], "total_count": 15, "filters_applied": { "min_usage": 50, "min_documents": 3, "namespace": "topic" } } ``` **Example:** ```bash curl "http://localhost:8000/v1/admin/taxonomy/trending?namespace=topic&min_usage=50" ``` --- ### 2. GET /v1/admin/taxonomy/stats **Purpose:** Get overall tag usage statistics **Response:** ```json { "total_tags": 1523, "by_namespace": { "req": 456, "topic": 789, "compliance": 112, "entity": 166 }, "by_status": { "verified": 1200, "pending": 323 }, "dynamic_tags": 789 } ``` **Example:** ```bash curl "http://localhost:8000/v1/admin/taxonomy/stats" ``` --- ### 3. POST /v1/admin/taxonomy/promote **Purpose:** Promote a dynamic tag to standard taxonomy **Request Body:** ```json { "old_tag": "topic:gdpr", "new_tag": "compliance:regulatory:gdpr", "description": "GDPR compliance requirements", "examples": [ "Must comply with GDPR Article 17", "GDPR data retention policy" ], "profile": null, "add_to_taxonomy": true } ``` **Parameters:** - `old_tag` (required) - Current dynamic tag - `new_tag` (required) - New canonical tag in standard namespace - `description` (required if add_to_taxonomy=true) - Tag description - `examples` (optional) - List of example usages - `profile` (optional) - Profile to add to (fintech, healthcare, automotive) - `add_to_taxonomy` (default: true) - Add to taxonomy config file **Response:** ```json { "success": true, "old_tag": "topic:gdpr", "new_tag": "compliance:regulatory:gdpr", "records_updated": 127, "added_to_taxonomy": true, "message": "Successfully promoted topic:gdpr to compliance:regulatory:gdpr. Updated 127 records. Added to taxonomy config." } ``` **Example:** ```bash curl -X POST "http://localhost:8000/v1/admin/taxonomy/promote" \ -H "Content-Type: application/json" \ -d '{ "old_tag": "topic:gdpr", "new_tag": "compliance:regulatory:gdpr", "description": "GDPR compliance requirements", "examples": ["Must comply with GDPR Article 17"], "add_to_taxonomy": true }' ``` --- ## CLI Script Usage ### List Trending Tags ```bash # List all trending tags (usage >= 50) python scripts/promote_dynamic_tag.py --list-trending # Filter by namespace python scripts/promote_dynamic_tag.py --list-trending --namespace topic # Custom threshold python scripts/promote_dynamic_tag.py --list-trending --min-usage 100 ``` **Output:** ``` Trending Tags (usage >= 50): ================================================================================ Tag Usage Docs Verified First/Last Seen -------------------------------------------------------------------------------- topic:gdpr 127 8 85.0% 2026-01-15 / 2026-01-22 topic:pci_dss 89 5 92.1% 2026-01-12 / 2026-01-20 entity:stripe 78 4 88.5% 2026-01-10 / 2026-01-22 Total: 3 trending tags ``` ### Promote a Tag ```bash # Promote to standard taxonomy python scripts/promote_dynamic_tag.py \ topic:gdpr \ compliance:regulatory:gdpr \ --description "GDPR compliance requirements" \ --examples "Must comply with GDPR Article 17" "GDPR data retention" # Promote to specific profile python scripts/promote_dynamic_tag.py \ topic:pci_dss \ compliance:pci_dss \ --description "PCI DSS compliance" \ --profile fintech # Database migration only (skip taxonomy file) python scripts/promote_dynamic_tag.py \ topic:authentication \ req:quality:security \ --no-taxonomy ``` **Output:** ``` Promoting tag: From: topic:gdpr To: compliance:regulatory:gdpr Step 1: Migrating database records... ✓ Successfully updated 127 records Step 2: Adding to taxonomy configuration... ✓ Added to config/taxonomy/standard_tags.json ✓ Tag promotion complete! topic:gdpr → compliance:regulatory:gdpr Records updated: 127 ``` --- ## Workflow Example ### Scenario: Promoting "topic:gdpr" to Standard Taxonomy **Step 1: Identify Trending Tags** Admin runs weekly review: ```bash curl "http://localhost:8000/v1/admin/taxonomy/trending?namespace=topic&min_usage=50" ``` Identifies `topic:gdpr` with 127 usages across 8 documents (85% verified ratio). **Step 2: Review Tag Quality** Admin checks tag usage in records: ```bash curl "http://localhost:8000/v1/blocks/{record_id}/tags" ``` Confirms tag is used correctly and consistently. **Step 3: Decide Canonical Mapping** Admin maps `topic:gdpr` → `compliance:regulatory:gdpr` based on taxonomy structure. **Step 4: Execute Promotion** ```bash curl -X POST "http://localhost:8000/v1/admin/taxonomy/promote" \ -H "Content-Type: application/json" \ -d '{ "old_tag": "topic:gdpr", "new_tag": "compliance:regulatory:gdpr", "description": "GDPR compliance requirements", "examples": ["Must comply with GDPR Article 17"], "add_to_taxonomy": true }' ``` **Step 5: Verify Migration** - Database: All 127 records updated with new tag name - History: Each tag has promotion entry added - Config: `compliance:regulatory:gdpr` added to `standard_tags.json` - Future uses: LLM now suggests `compliance:regulatory:gdpr` instead of `topic:gdpr` --- ## Database Migration Details The promotion performs an **atomic batch update** using SQL: ```sql UPDATE extracted_records SET tags = ( -- For each tag in the array SELECT jsonb_agg( CASE -- If tag matches old_tag, rename and add history WHEN tag->>'name' = 'topic:gdpr' THEN jsonb_set( jsonb_set(tag, '{name}', '"compliance:regulatory:gdpr"'), '{history}', tag->'history' || '[{"action": "promoted", "by": "admin", "ts": 1769067396, "old_value": "topic:gdpr"}]' ) -- Otherwise, keep unchanged ELSE tag END ) FROM jsonb_array_elements(tags) as tag ) WHERE EXISTS ( SELECT 1 FROM jsonb_array_elements(tags) as tag WHERE tag->>'name' = 'topic:gdpr' ); ``` **Result:** - Tag name updated - History entry added - All metadata preserved (confidence, status, source) - Transaction-safe (rollback on error) --- ## Taxonomy File Updates When `add_to_taxonomy: true`, the tag is added to the appropriate config file: ### Standard Taxonomy (`config/taxonomy/standard_tags.json`) ```json { "namespaces": { "compliance": { "tags": [ { "name": "regulatory:gdpr", "description": "GDPR compliance requirements", "examples": [ "Must comply with GDPR Article 17", "GDPR data retention policy" ] } ] } } } ``` ### Profile Taxonomy (`config/profiles/fintech.json`) ```json { "additional_namespaces": { "compliance": { "tags": [ { "name": "pci_dss", "description": "PCI DSS compliance", "examples": ["..."] } ] } } } ``` --- ## Best Practices ### 1. Review Before Promotion - Check verified_ratio (aim for > 80%) - Review actual usage in records - Confirm tag is semantically correct ### 2. Choose Canonical Names Carefully - Follow existing namespace patterns - Use descriptive, unambiguous names - Consider future taxonomy evolution ### 3. Run Weekly Analysis ```bash # Cron job example (every Monday 9 AM) 0 9 * * 1 python scripts/promote_dynamic_tag.py --list-trending --min-usage 50 > /var/log/trending_tags.log ``` ### 4. Document Decisions - Add detailed descriptions - Provide multiple examples - Update internal wiki ### 5. Test Before Production ```bash # Test on staging database first python scripts/promote_dynamic_tag.py topic:test compliance:test --no-taxonomy ``` --- ## Monitoring & Alerts Set up alerts for: - High trending tag count (> 20) - Low verified ratios (< 60%) - Rapid tag growth (usage spike) Dashboard metrics: ```bash curl "http://localhost:8000/v1/admin/taxonomy/stats" ``` --- ## Troubleshooting **Q: Tag promotion fails with "Tag not found"** - Run `--list-trending` to verify tag exists - Check exact spelling (case-sensitive) **Q: Cannot add to taxonomy file** - Check file permissions - Verify JSON syntax in target file - Ensure namespace exists in config **Q: High number of pending tags** - Review tagging quality with domain experts - Adjust confidence thresholds - Consider batch approval workflow --- ## Future Enhancements - [ ] Auto-suggest canonical mappings (ML-based) - [ ] Batch promotion (multiple tags at once) - [ ] Tag merge (combine similar tags) - [ ] Vector re-indexing trigger - [ ] Taxonomy version control integration - [ ] Rollback mechanism for promotions --- ## Related Documentation - [Specification: Extract Tag Function](../specs/extract_tag_function_spec.md) - [Phase 1: Data Model Migration](../migrations/) - [Phase 2: Enhanced Taxonomy](../config/taxonomy/) - [Phase 3: Inheritance & Confidence](../src/extraction_v2/) - [Phase 4: HITL APIs](../src/api/routes/tags.py)