---
title: 'Add Japanese Word Segmentation to Domain Term Extraction'
slug: 'japanese-word-segmentation-domain-extraction'
created: '2026-01-12'
status: 'completed'
stepsCompleted: [1, 2, 3, 4, 5, 6]
tech_stack: ['Python 3.11+', 'SudachiPy', 'Azure OpenAI', 'Airflow 3.0.6', 'structlog', 'pytest', 'httpx', 'uv']
files_to_modify: ['src/extraction_v2/domain_term_extractor.py', 'pyproject.toml', '.env.example']
files_to_create: ['tests/extraction_v2/test_japanese_segmentation.py']
code_patterns: ['lazy_initialization', 'graceful_fallback', 'fork_safe_clients', 'env_var_config', 'structlog_events']
test_patterns: ['pytest', 'unittest_mock', 'patch_decorator', 'patch_dict_os_environ', 'mock_llm_response']
confluence_id: '814681530'
confluence_parent: Architecture and Design
confluence_space: TEXTIQ
confluence_url: https://insight.fsoft.com.vn/conf/spaces/TEXTIQ/pages/814681530/Add+Japanese+Word+Segmentation+to+Domain+Term+Extraction
---
# Tech-Spec: Add Japanese Word Segmentation to Domain Term Extraction
**Created:** 2026-01-12
## Overview
### Problem Statement
LLMs struggle to detect domain terms in Japanese text because Japanese doesn't use spaces between words, making word boundary detection difficult. Current baseline extraction results in:
- Lower term extraction rates for Japanese documents
- Poor category classification (documents marked as "general" instead of specific domains like "finance" or "hr")
- Suboptimal query pre-filtering in RAG pipeline
Testing shows baseline extracts 14.5 avg terms/doc on real customer documents, with most Japanese docs incorrectly categorized as "general."
### Solution
Add SudachiPy word segmentation preprocessing to the `_llm_extract()` method in `DomainTermExtractor`. When Japanese text is detected, segment it with spaces before sending to the LLM for term extraction. This approach:
- Uses Mode C (long unit) segmentation to preserve compound words
- Implements lazy initialization for fork-safety in Airflow workers
- Includes graceful fallback to baseline if segmentation fails
- Controlled via `ENABLE_CJK_SEGMENTATION` environment variable
Validated improvement: +3.4% term extraction on real customer documents (15.0 vs 14.5 avg terms).
### Scope
**In Scope:**
- Modify `_llm_extract()` method in `src/extraction_v2/domain_term_extractor.py` to add segmentation preprocessing
- Add `_is_japanese()` helper method for language detection (checks for Hiragana, Katakana, Kanji)
- Add `_segment_japanese()` helper method with SudachiPy integration
- Add `_get_sudachi_tokenizer()` for lazy initialization of tokenizer
- Add `ENABLE_CJK_SEGMENTATION` environment variable (default: true)
- Add structured logging for segmentation events (success, failures, timings)
- Add `sudachipy>=0.6.0` and `sudachidict_core>=20240716` to `pyproject.toml`
- Document Docker/Airflow worker dependency installation
- Create new test file `tests/extraction_v2/test_japanese_segmentation.py` with comprehensive test coverage
- Write deployment procedures for staging and production phased rollout
- Define monitoring strategy and metrics collection
- Document rollback procedures and troubleshooting guide
**Out of Scope:**
- Modifying `_extract_by_rules()` method (segmentation only benefits LLM extraction path)
- Chinese or Korean segmentation support (future enhancement)
- Changes to `DomainTaxonomy` categories or term patterns
- Automated A/B testing infrastructure (manual validation phase only)
- Real-time metrics dashboard (use existing logging infrastructure)
- Sophisticated language detection to distinguish Japanese from Chinese (Kanji Unicode range U+4E00-U+9FFF overlaps with Chinese characters. Chinese documents may trigger Japanese detection but will fail segmentation gracefully and fall back to original text)
## Context for Development
### Codebase Patterns
**1. Lazy Initialization Pattern (Fork-Safe Clients)**
Location: `src/extraction_v2/domain_term_extractor.py:151-178`
Current implementation in `_get_client()`:
```python
def _get_client(self) -> AzureOpenAI:
"""Get or create Azure OpenAI client lazily.
Lazy initialization prevents creating HTTP clients before process forking,
which causes issues in Celery/Airflow workers.
"""
if not self._client_initialized:
import httpx # Import only when needed
# Create httpx client with fork-safe settings
http_client = httpx.Client(
http2=False, # Disable HTTP/2 to avoid fork issues
limits=httpx.Limits(
max_connections=1,
max_keepalive_connections=0,
),
timeout=httpx.Timeout(60.0),
)
self._client = AzureOpenAI(...)
self._client_initialized = True
return self._client
```
**Apply same pattern for SudachiPy tokenizer:**
- Add in `__init__()` (line ~149): `self._sudachi_tokenizer = None`, `self._sudachi_initialized = False`
- Create `_get_sudachi_tokenizer()` method following exact same structure
- Import sudachipy inside the method, not at module level
- Handle ImportError gracefully (library might not be installed in all environments)
**2. Environment Variable Configuration**
Location: `src/extraction_v2/domain_term_extractor.py:22-23`
Current pattern:
```python
# Environment configuration
MAX_TERMS_PER_CHUNK = int(os.getenv("DOMAIN_TERM_MAX_PER_CHUNK", "50"))
LLM_THRESHOLD = int(os.getenv("DOMAIN_TERM_LLM_THRESHOLD", "2"))
```
**Add new variable at line ~24:**
```python
ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"
```
**3. Graceful Fallback with Error Handling**
Location: `src/extraction_v2/domain_term_extractor.py:313-362` (`_llm_extract()` method)
Current error handling pattern:
```python
try:
# Main logic
...
return terms
except json.JSONDecodeError as e:
self.logger.error("llm_json_parse_error", error=str(e), result=result_text)
return []
except Exception as e:
self.logger.error("llm_extraction_failed", error=str(e))
return []
```
**Apply to segmentation:**
- Wrap segmentation in try/except
- Log specific error but continue with original text
- Never raise exceptions that would break the pipeline
- Return original text as fallback
**4. Structured Logging with structlog**
Location: Throughout `domain_term_extractor.py`, logger initialized at line 149
Current logging pattern:
```python
self.logger.info("using_llm_extraction", terms_found=len(terms), threshold=LLM_THRESHOLD)
self.logger.info("llm_extraction_success", terms_count=len(terms))
self.logger.error("llm_extraction_failed", error=str(e))
self.logger.warning("llm_returned_non_list", result=result_text)
```
**Add new log events:**
- `japanese_text_detected` - when `_is_japanese()` returns True
- `japanese_segmentation_success` - after successful segmentation (include char_count, token_count)
- `japanese_segmentation_failed` - on exception (include error, fallback action)
- `sudachi_init_failed` - if tokenizer initialization fails (include error)
- `cjk_segmentation_disabled` - if env var is false
**5. LLM Truncation Pattern**
Location: `src/extraction_v2/domain_term_extractor.py:325-327`
Current truncation:
```python
# Truncate content if too long (max ~2000 tokens)
if len(content) > 8000:
content = content[:8000] + "..."
```
**Segmentation should happen BEFORE truncation:**
- Detect Japanese → Segment → Truncate → Send to LLM
- This ensures segmented text benefits from full context before truncation
### Files to Reference
| File | Purpose | Key Sections |
| ---- | ------- | ------------ |
| `src/extraction_v2/domain_term_extractor.py` | Main file to modify | `__init__`: L135-149
`_get_client()`: L151-178 (lazy init pattern)
`_llm_extract()`: L313-362 (add segmentation here)
Env vars: L22-23 |
| `tests/extraction_v2/test_domain_term_extractor.py` | Unit test patterns | Mock pattern: L144-152
Env var pattern: L158
LLM response mock: L148-152 |
| `pyproject.toml` | Add dependencies | Dependencies: L8-39
Dev deps: L42-48
Pytest config: L102-107 |
| `.env.example` | Document env var | Add to L84-92 (API Configuration section) |
| `docker/Dockerfile.airflow` | Airflow worker setup | uv sync: L17-26
Validate deps install after adding to pyproject.toml |
| `src/core/config.py` | Pydantic settings | Reference only - env var uses `os.getenv()`, not Pydantic |
| `airflow/dags/tasks/processing_tasks.py` | Airflow integration | Import extractor: L386
**No changes needed** - uses extractor directly |
| `test_domain_extraction.py` | Manual test script | Integration test patterns for validation |
| `JAPANESE_SEGMENTATION_PROPOSAL.md` | Original proposal | Implementation examples, code snippets |
| `CUSTOMER_TEST_RESULTS_SUMMARY.md` | Validation data | Real customer test results: +3.4% improvement |
| `experiment_japanese_segmentation.py` | Experimental validation | Shows SudachiPy Mode C usage example |
### Code Structure & Implementation Points
**Class: DomainTermExtractor** (`src/extraction_v2/domain_term_extractor.py`)
```
Lines 132-363: DomainTermExtractor class
├── __init__() [L135-149]
│ ├── Current: self.taxonomy, self._client, self._client_initialized, self.logger
│ └── ADD: self._sudachi_tokenizer = None, self._sudachi_initialized = False
│
├── _get_client() [L151-178] - REFERENCE for lazy init pattern
│ └── Pattern to follow for _get_sudachi_tokenizer()
│
├── extract_from_chunk() [L180-241] - Main entry point
│ └── Calls _llm_extract() at line 227 when terms < threshold
│
├── _extract_by_rules() [L290-311] - NO CHANGES (regex works without segmentation)
│
└── _llm_extract() [L313-362] - PRIMARY MODIFICATION POINT
├── Line 323: Get client
├── Line 325-327: Truncate content (ADD segmentation BEFORE this)
├── Line 329: Format prompt
├── Line 331-339: Call Azure OpenAI
├── Line 344: Parse JSON
└── Line 357-362: Error handling
```
**Add 3 New Methods (insert after line 311, before `_llm_extract()`):**
1. `_get_sudachi_tokenizer()` - Lazy init SudachiPy tokenizer
2. `_is_japanese()` - Detect Japanese characters
3. `_segment_japanese()` - Segment Japanese text
**Modify `_llm_extract()` method:**
- Add check at line ~324 (after `client = self._get_client()`)
- If Japanese detected AND env var enabled → segment content
- Log segmentation events
- Apply graceful fallback if errors
**Environment Variables** (add at line ~24):
```python
ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"
```
### Technical Decisions
**Decision 1: SudachiPy Mode C (Long Unit)**
- **Rationale:** Testing showed Mode C preserves compound words better (e.g., "売上高" stays as one term)
- **Alternative Considered:** Mode A (short unit) - too granular, splits meaningful compounds
- **Implementation:** Use `tokenizer.Tokenizer.SplitMode.C` in segmentation
**Decision 2: Environment Variable Control**
- **Variable:** `ENABLE_CJK_SEGMENTATION` (default: "true")
- **Rationale:** Allows gradual rollout and easy rollback if issues arise
- **Values:** "true" enables, "false" or any other value disables
**Decision 3: Only Segment for LLM Path**
- **Rationale:** Rule-based extraction uses regex matching which already works with Japanese (no spaces needed)
- **Benefit:** Minimal code changes, focused improvement where it matters most
**Decision 4: Conservative Language Detection**
- **Method:** Check for Unicode ranges: Hiragana (U+3040-U+309F), Katakana (U+30A0-U+30FF), Kanji (U+4E00-U+9FFF)
- **Rationale:** Avoid false positives - only segment if clear Japanese characters present
- **Trade-off:** Won't segment mixed-script text (acceptable for initial implementation)
**Decision 5: Lazy Initialization for Fork-Safety**
- **Rationale:** Airflow workers use multiprocessing with fork - initializing clients before fork causes issues
- **Pattern:** Initialize tokenizer on first use, not in `__init__`
- **Benefit:** Consistent with existing `_get_client()` pattern (lines 151-178)
**Decision 6: Use pyproject.toml (Not requirements.txt)**
- **Finding:** Project uses `pyproject.toml` with `hatchling` build backend and `uv` package manager
- **Action:** Add dependencies to `pyproject.toml` dependencies list (lines 8-39)
- **Deployment:** Run `uv lock` after adding dependencies to update `uv.lock`
- **Docker:** `docker/Dockerfile.airflow` uses `uv sync` (line 26) - will automatically pick up new dependencies
**Decision 7: Document Environment Variable in .env.example**
- **Finding:** `.env.example` exists with well-organized sections
- **Action:** Add `ENABLE_CJK_SEGMENTATION=true` under "API Configuration" section (lines 84-92)
- **Format:** Follow existing pattern with comment explaining purpose
**Decision 8: Testing Strategy with unittest.mock**
- **Pattern:** Use `@patch("module.path.Class.method")` decorator (line 144 in test file)
- **Env Vars:** Use `with patch.dict("os.environ", {"KEY": "value"})` (line 158 in test file)
- **LLM Mocking:** Mock `chat.completions.create` return value (lines 148-152 in test file)
- **Rationale:** Consistent with existing test patterns, no external dependencies for unit tests
**Decision 9: No Changes to Airflow DAG**
- **Finding:** `extract_domain_terms_task` (line 372-533 in processing_tasks.py) simply imports and uses `DomainTermExtractor`
- **Conclusion:** Segmentation will automatically apply when Airflow calls the extractor
- **Benefit:** Zero deployment changes to DAG files
**Decision 10: Deployment Uses uv Package Manager**
- **Finding:** Dockerfile.airflow uses `uv` for dependency management (not pip)
- **Workflow:** Add to pyproject.toml → Run `uv lock` → Docker rebuild picks up changes
- **Benefit:** Fast, reproducible builds with locked dependencies
## Implementation Plan
**NOTE ON LINE NUMBERS:** Where specific line numbers are mentioned, they are approximate references based on the codebase at time of spec creation. Use the provided landmarks (method names, comments, code patterns) to locate the correct insertion points. Line numbers may drift as code evolves.
### Tasks
**Phase 1: Dependencies & Configuration**
- [x] Task 1: Add SudachiPy dependencies to pyproject.toml
- File: `pyproject.toml`
- Action: Add to `dependencies = [...]` list: `"sudachipy>=0.6.0"` and `"sudachidict_core>=20240716"`
- Notes: Insert alphabetically between existing packages (search for `dependencies = [` section)
- [x] Task 2: Update dependency lock file
- Command: `uv lock`
- Action: Run from project root to update `uv.lock` with new dependencies
- Notes: Verify lock file updated successfully (check for sudachipy entries)
- [x] Task 3: Add environment variable configuration
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Add near top of file after existing environment variables (search for "# Environment configuration" comment or LLM_THRESHOLD variable):
```python
ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"
```
- Notes: Follow existing pattern with other env vars. Exact line number may vary - place after LLM_THRESHOLD and before class definition.
- [x] Task 4: Document environment variable in .env.example
- File: `.env.example`
- Action: Add to "API Configuration" section (search for "# API Configuration" or Azure OpenAI settings):
```
# ===================
# Japanese Segmentation Configuration
# ===================
# Enable Japanese word segmentation for improved domain term extraction
# Set to 'false' to disable (useful for rollback or debugging)
ENABLE_CJK_SEGMENTATION=true
```
- Notes: Include helpful comment explaining purpose
**Phase 2: Core Implementation**
- [x] Task 5: Initialize segmentation state in __init__
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Add to `DomainTermExtractor.__init__()` method (end of method, after logger initialization):
```python
self._sudachi_tokenizer = None
self._sudachi_initialized = False
```
- Notes: Place after the line `self.logger = structlog.get_logger(__name__)` inside the `__init__` method
- [x] Task 6: Implement lazy tokenizer initialization method
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Add new method in `DomainTermExtractor` class after `_extract_by_rules()` method and before `_llm_extract()` method:
```python
def _get_sudachi_tokenizer(self):
"""Get or create SudachiPy tokenizer lazily.
Lazy initialization prevents creating tokenizer before process forking,
which causes issues in Celery/Airflow workers.
Returns:
Tokenizer instance or None if initialization fails
"""
if not self._sudachi_initialized:
try:
from sudachipy import tokenizer, dictionary
tokenizer_obj = dictionary.Dictionary().create()
self._sudachi_tokenizer = tokenizer_obj
self._sudachi_initialized = True
except ImportError:
self.logger.warning(
"sudachi_init_failed",
error="sudachipy not installed",
fallback="segmentation_disabled"
)
self._sudachi_initialized = True # Mark as attempted, don't retry every call
except Exception as e:
self.logger.error(
"sudachi_init_failed",
error=str(e),
fallback="segmentation_disabled"
)
self._sudachi_initialized = True # Mark as attempted, don't retry every call
return self._sudachi_tokenizer
```
- Notes: Follows lazy initialization pattern from `_get_client()` (lines 151-178). Sets `_sudachi_initialized = True` even on failure to prevent retry attempts on every call (library won't magically appear mid-process). If library is installed later, requires worker restart.
- [x] Task 7: Implement Japanese language detection method
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Add new method after `_get_sudachi_tokenizer`:
```python
def _is_japanese(self, text: str) -> bool:
"""Detect if text contains Japanese characters.
Checks for Hiragana, Katakana, or Kanji characters.
Args:
text: Text to analyze
Returns:
True if Japanese characters detected, False otherwise
"""
if not text:
return False
# Unicode ranges for Japanese scripts
# Hiragana: U+3040-U+309F
# Katakana: U+30A0-U+30FF
# Kanji: U+4E00-U+9FFF
japanese_pattern = r'[\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF]'
return bool(re.search(japanese_pattern, text))
```
- Notes: Conservative detection to avoid false positives. Requires `import re` (already present at line 10 of domain_term_extractor.py)
- [x] Task 8: Implement Japanese text segmentation method
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Add new method after `_is_japanese`:
```python
def _segment_japanese(self, text: str) -> str:
"""Segment Japanese text using SudachiPy.
Args:
text: Japanese text to segment
Returns:
Space-separated segmented text, or original text if segmentation fails
"""
tokenizer_obj = self._get_sudachi_tokenizer()
if tokenizer_obj is None:
return text # Fallback to original if tokenizer unavailable
try:
import time
# Use Mode C (long unit) to preserve compound words
# Note: Mode was already defined when tokenizer was created in _get_sudachi_tokenizer()
# We use the tokenizer's built-in mode enum: A=short, B=medium, C=long
from sudachipy.tokenizer import Tokenizer
mode = Tokenizer.SplitMode.C
# Measure segmentation performance
start_time = time.time()
tokens = [m.surface() for m in tokenizer_obj.tokenize(text, mode)]
elapsed_ms = (time.time() - start_time) * 1000
segmented = " ".join(tokens)
self.logger.info(
"japanese_segmentation_success",
char_count=len(text),
token_count=len(tokens),
elapsed_ms=round(elapsed_ms, 2)
)
return segmented
except Exception as e:
self.logger.warning(
"japanese_segmentation_failed",
error=str(e),
fallback="original_text"
)
return text # Graceful fallback
```
- Notes: Always return text (never raise exceptions)
- [x] Task 9: Modify _llm_extract to add segmentation preprocessing
- File: `src/extraction_v2/domain_term_extractor.py`
- Action: Modify `_llm_extract()` method - add segmentation BETWEEN client initialization and truncation:
```python
# Existing code - locate this in _llm_extract() method:
client = self._get_client()
# ADD THIS BLOCK HERE (AFTER client init, BEFORE truncation)
# PREPROCESSING: Segment Japanese text if enabled
if ENABLE_CJK_SEGMENTATION:
if self._is_japanese(content):
# Calculate ratio of Japanese characters to total
import re
japanese_chars = len(re.findall(r'[\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF]', content))
japanese_ratio = japanese_chars / len(content) if len(content) > 0 else 0
self.logger.info(
"japanese_text_detected",
char_count=len(content),
japanese_ratio=round(japanese_ratio, 3)
)
content = self._segment_japanese(content)
else:
self.logger.debug("cjk_segmentation_disabled", reason="env_var_false")
# Existing truncation code below (DO NOT MODIFY - segmentation must be ABOVE this)
if len(content) > 8000:
content = content[:8000] + "..."
# Rest of _llm_extract continues unchanged (prompt formatting, LLM call, etc.)...
```
- Notes: CRITICAL - Segmentation must happen AFTER client init (line 323) but BEFORE truncation (line 325-327). This ensures Japanese text is segmented with full context before being truncated.
**Phase 3: Testing**
- [x] Task 10: Create comprehensive test file
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Create new file with test classes:
- `TestJapaneseLanguageDetection` - test `_is_japanese()` method
- `TestJapaneseSegmentation` - test `_segment_japanese()` method
- `TestLazyTokenizerInitialization` - test `_get_sudachi_tokenizer()` method
- `TestEnvironmentVariableControl` - test env var enable/disable
- `TestLLMExtractIntegration` - test full integration with mocked LLM
- `TestJapaneseSegmentationIntegration` - test with real customer doc samples
- Notes: Follow patterns from `test_domain_term_extractor.py` (use @patch, patch.dict)
- [x] Task 11: Write unit tests for language detection
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Add tests covering:
- Japanese text only (Hiragana: "こんにちは", Katakana: "カタカナ", Kanji: "日本語") → returns True
- English text only ("Hello World") → returns False
- Mixed Japanese/English text ("Hello 日本") → returns True (ANY Japanese chars trigger detection)
- Empty string ("") → returns False
- Whitespace only (" ") → returns False
- Chinese text ("你好") → returns True (KNOWN LIMITATION: Kanji range U+4E00-U+9FFF overlaps with Chinese. This is expected behavior - segmentation will fail gracefully and fall back to original text)
- Notes: Use parametrized tests for efficiency. Mixed-script documents will be segmented in their entirety (both Japanese and English portions passed to tokenizer).
- [x] Task 12: Write unit tests for segmentation
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Add tests covering:
- Simple sentence: "売上高は増加した" → verify spaces added
- Compound preservation: verify Mode C keeps compounds together
- Empty input → returns empty/original
- Mock segmentation error → verify graceful fallback to original text
- Notes: May need to check actual segmentation output format
- [x] Task 13: Write unit tests for environment variable control
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Add tests using `patch.dict("os.environ", {...})`:
- `ENABLE_CJK_SEGMENTATION=true` → segmentation active
- `ENABLE_CJK_SEGMENTATION=false` → segmentation disabled
- Missing env var → defaults to true
- Invalid value ("invalid") → treated as false
- Notes: Follow pattern from line 158 in existing test file
- [x] Task 14: Write unit tests for lazy initialization
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Add tests covering:
- First call initializes tokenizer
- Second call returns cached instance (no re-init)
- ImportError handling (mock sudachipy as unavailable)
- Generic exception handling
- Notes: Use `patch.dict("sys.modules", {"sudachipy": None})` to simulate import failure
- [x] Task 15: Write integration tests with mocked LLM
- File: `tests/extraction_v2/test_japanese_segmentation.py`
- Action: Add tests using `@patch("...DomainTermExtractor._get_client")`:
- Japanese text + env=true → LLM receives segmented text
- Japanese text + env=false → LLM receives original text
- English text → LLM receives original text
- Segmentation failure → LLM receives original text (fallback works)
- Mock LLM response → verify term extraction still works
- Notes: Follow mock pattern from lines 144-163 in existing test file
- [x] Task 16: Run all tests and verify passing
- Command: `pytest tests/extraction_v2/test_japanese_segmentation.py -v`
- Action: Execute full test suite, verify all tests pass
- Notes: Fix any failures before proceeding
- [x] Task 17: Run existing domain extractor tests
- Command: `pytest tests/extraction_v2/test_domain_term_extractor.py -v`
- Action: Verify no regression in existing functionality
- Notes: All existing tests should still pass
**Phase 4: Validation & Deployment Prep**
- [ ] Task 18: Manual testing with real customer documents
- Script: `python test_domain_extraction.py` and `python experiment_japanese_segmentation.py`
- Action: Test with Japanese documents from CustomerDocument/ folder
- Expected: Term count improvement ≥3%, processing time <2.5s
- Notes: Compare results with baseline (env var disabled)
- [ ] Task 19: Performance validation
- Action: Measure segmentation overhead with different text sizes
- Expected: Small (<100 chars): <10ms, Medium (<500 chars): <25ms, Large (<1000 chars): <50ms
- Notes: Log elapsed_ms from segmentation events
- [ ] Task 20: Code review and cleanup
- Action: Review all changes, ensure code quality, add docstrings, check formatting
- Notes: Run `black` formatter if used in project
- [ ] Task 21: Pre-deployment checklist verification
- Action: Verify all items in "Pre-Deployment Checklist" section are complete
- Notes: Don't proceed to deployment without completing all items
- [ ] Task 22: Test rollback procedure and validate timing SLA
- Environment: Staging
- Action: Validate emergency rollback timing and effectiveness
- Steps:
1. Enable feature in staging: set `ENABLE_CJK_SEGMENTATION=true`, restart workers
2. Process several Japanese documents, verify segmentation logs appear
3. Record timestamp: Start rollback timer
4. Disable feature: set `ENABLE_CJK_SEGMENTATION=false` in .env
5. Restart Airflow workers
6. Process Japanese documents again, verify NO segmentation logs
7. Record timestamp: Stop rollback timer
8. Verify logs show `cjk_segmentation_disabled` events
- Expected: Complete rollback (no more segmentation) within 5 minutes of env var change
- Notes: Validates AC24. Measure actual timing for documentation. If >5 minutes, adjust SLA or worker restart procedure.
### Acceptance Criteria
**Functional Requirements:**
- [ ] AC1: Given Japanese text content, when `_is_japanese()` is called, then it returns True if Hiragana, Katakana, or Kanji characters are present
- [ ] AC2: Given English text content, when `_is_japanese()` is called, then it returns False
- [ ] AC3: Given Japanese text, when `_segment_japanese()` is called, then it returns space-separated tokens using SudachiPy Mode C
- [ ] AC4: Given `ENABLE_CJK_SEGMENTATION=true` and Japanese text, when `_llm_extract()` is called, then the content is segmented before being sent to the LLM
- [ ] AC5: Given `ENABLE_CJK_SEGMENTATION=false` and Japanese text, when `_llm_extract()` is called, then the content is NOT segmented (original text sent to LLM)
- [ ] AC6: Given Japanese text with segmentation enabled, when LLM extraction completes, then the average term count is ≥15.0 terms/doc (+3.4% improvement validated on 10 real customer documents)
- [ ] AC6a: Given validation testing with ≥20 Japanese documents, when comparing segmented vs baseline, then improvement is observed in ≥70% of individual documents (not just average)
- [ ] AC6b: Given production monitoring over 7 days, when analyzing Japanese document processing, then average term count remains ≥15.0 and segmentation success rate is >95%
**Error Handling & Fallback:**
- [ ] AC7: Given sudachipy library is not installed, when `_get_sudachi_tokenizer()` is called, then it logs "sudachi_init_failed" and returns None without crashing
- [ ] AC8: Given segmentation fails due to exception, when `_segment_japanese()` is called, then it logs "japanese_segmentation_failed" and returns the original text
- [ ] AC9: Given tokenizer initialization fails, when Japanese text is processed, then extraction continues with original (unsegmented) text
- [ ] AC10: Given any segmentation error, when `extract_from_chunk()` is called, then the pipeline completes successfully without raising exceptions
**Performance Requirements:**
- [ ] AC11: Given text <100 characters, when segmentation occurs, then elapsed time is <10ms
- [ ] AC12: Given text <500 characters, when segmentation occurs, then elapsed time is <25ms
- [ ] AC13: Given text <1000 characters, when segmentation occurs, then elapsed time is <50ms
- [ ] AC14: Given Japanese document processing with segmentation, when full extraction completes, then total time is <2.5 seconds (validated ~2.1s avg)
**Integration & Regression:**
- [ ] AC15: Given Airflow DAG processes a Japanese document, when domain extraction task runs, then segmentation automatically applies without DAG changes
- [ ] AC16: Given English document content, when `extract_from_chunk()` is called with segmentation enabled, then term extraction quality is unchanged (no regression)
- [ ] AC17: Given all existing unit tests, when test suite runs, then all tests pass (no regression in existing functionality)
**Logging & Monitoring:**
- [ ] AC18: Given Japanese text is detected, when segmentation succeeds, then "japanese_segmentation_success" event is logged with char_count and token_count
- [ ] AC19: Given Japanese text is detected, when segmentation fails, then "japanese_segmentation_failed" event is logged with error and fallback action
- [ ] AC20: Given segmentation is disabled via env var, when Japanese text is processed, then "cjk_segmentation_disabled" event is logged with reason
**Deployment Validation:**
- [ ] AC21: Given staging environment deployment, when Japanese documents are processed, then term extraction improvement is observed and no errors occur
- [ ] AC22: Given production deployment with env var disabled, when workers restart, then no segmentation events appear in logs
- [ ] AC23: Given production deployment with env var enabled, when Japanese documents are processed, then segmentation success rate is >99%
- [ ] AC24: Given emergency rollback needed, when `ENABLE_CJK_SEGMENTATION=false` is set and workers restart, then segmentation stops within 5 minutes
## Additional Context
### Dependencies
**New Python Packages:**
- `sudachipy>=0.6.0` - Japanese morphological analyzer
- `sudachidict_core>=20240716` - Dictionary for SudachiPy
**Installation:**
```bash
# Add to pyproject.toml dependencies list, then update lock file
uv add "sudachipy>=0.6.0" "sudachidict_core>=20240716"
# OR manually edit pyproject.toml and run:
uv lock
```
**Airflow Worker Setup:**
Dependencies are automatically installed via `uv sync` in Dockerfile.airflow (line 26). After adding to pyproject.toml and running `uv lock`, rebuild Docker images to pick up new dependencies.
### Testing Strategy
**Unit Tests** (File: `tests/extraction_v2/test_japanese_segmentation.py`)
Test Coverage Required:
1. **Language Detection Tests:**
- Japanese text only (Hiragana, Katakana, Kanji)
- English text only
- Mixed Japanese/English text
- Empty string
- Whitespace only
- Chinese/Korean text (should NOT detect as Japanese)
2. **Segmentation Tests:**
- Simple Japanese sentence → verify spaces added
- Compound words → verify Mode C preserves compounds
- Mixed script text → verify proper handling
- Empty input → verify returns empty/original
- Error simulation → verify graceful fallback
3. **Environment Variable Tests:**
- `ENABLE_CJK_SEGMENTATION=true` → segmentation active
- `ENABLE_CJK_SEGMENTATION=false` → segmentation disabled
- Missing env var → defaults to true
- Invalid value → treated as false
4. **Lazy Initialization Tests:**
- First call to `_get_sudachi_tokenizer()` → initializes
- Second call → returns cached instance
- Initialization failure → logged and returns None
5. **Integration with _llm_extract() Tests:**
- Japanese text with env=true → LLM receives segmented text
- Japanese text with env=false → LLM receives original text
- Non-Japanese text → LLM receives original text
- Segmentation failure → LLM receives original text (fallback)
- Mock LLM response → verify correct term extraction
**Mock Patterns to Use:**
```python
# Pattern 1: Mock LLM client
@patch("src.extraction_v2.domain_term_extractor.DomainTermExtractor._get_client")
def test_something(self, mock_get_client):
mock_client = MagicMock()
mock_response = MagicMock()
mock_response.choices[0].message.content = '["term1", "term2"]'
mock_client.chat.completions.create.return_value = mock_response
mock_get_client.return_value = mock_client
# Pattern 2: Mock environment variable
with patch.dict("os.environ", {"ENABLE_CJK_SEGMENTATION": "false"}):
result = extractor.extract_from_chunk(japanese_text)
# Pattern 3: Mock sudachi import failure
with patch.dict("sys.modules", {"sudachipy": None}):
tokenizer = extractor._get_sudachi_tokenizer()
assert tokenizer is None
```
**Integration Tests** (Use existing customer documents from validation):
Location: `tests/extraction_v2/test_japanese_segmentation.py` (separate test class)
1. Test with real Japanese customer document samples
2. Measure term count improvement (expect ≥3% improvement)
3. Validate category detection improvement
4. Verify no regression on English documents
**Performance Tests:**
1. Measure segmentation overhead:
- Small text (100 chars): <10ms
- Medium text (500 chars): <25ms
- Large text (1000+ chars): <50ms
2. Measure full extraction latency:
- Japanese text with segmentation: ~2.1s (validated)
- Baseline: ~2.1s (no degradation)
**Manual Validation** (Before production deployment):
1. Run `test_domain_extraction.py` with Japanese samples
2. Run `experiment_japanese_segmentation.py` to verify results
3. Test in staging with real customer documents
4. Validate Airflow DAG execution in staging environment
### Notes
**Validation Data:**
- Synthetic tests: +18% improvement (8.3 vs 7.0 avg terms)
- Real customer tests: +3.4% improvement (15.0 vs 14.5 avg terms)
- Processing time: ~2.1s avg (negligible overhead)
- Tested on 10 real customer documents (DOCX + XLSX)
**Deployment Considerations:**
- Recommend phased rollout: staging → 10% production → 50% → 100%
- Monitor metrics: term extraction count, processing time, error rates
- Rollback plan: Set `ENABLE_CJK_SEGMENTATION=false` if issues arise
**Future Enhancements:**
- Chinese segmentation (jieba library)
- Korean segmentation (KoNLPy library)
- Configurable segmentation modes per language
- Automated A/B testing infrastructure
## Deployment Procedures
### Pre-Deployment Checklist
- [ ] All unit tests pass (`pytest tests/extraction_v2/test_japanese_segmentation.py`)
- [ ] Integration tests pass with real customer documents
- [ ] Performance tests show <50ms overhead
- [ ] Code reviewed and approved
- [ ] Dependencies added to `pyproject.toml`
- [ ] `uv lock` executed to update lock file
- [ ] `.env.example` updated with new environment variable
- [ ] Documentation updated
### Phase 1: Staging Deployment (Week 1)
**Objective:** Validate functionality in staging environment with no production risk
**Steps:**
1. **Build and Deploy:**
```bash
# Add dependencies to pyproject.toml
# sudachipy>=0.6.0
# sudachidict_core>=20240716
# Update lock file (RUN ON HOST, NOT IN DOCKER)
uv lock
# CRITICAL: Verify lock file updated before building Docker
grep -q "sudachipy" uv.lock && echo "✓ sudachipy found in uv.lock" || echo "✗ ERROR: sudachipy NOT in uv.lock"
grep -q "sudachidict-core" uv.lock && echo "✓ sudachidict_core found in uv.lock" || echo "✗ ERROR: sudachidict_core NOT in uv.lock"
# NOW rebuild Docker images (will pick up new lock file via COPY command)
docker-compose -f docker-compose.staging.yml build airflow-worker
# Deploy to staging
docker-compose -f docker-compose.staging.yml up -d
```
2. **Configure Environment:**
```bash
# In staging .env file
ENABLE_CJK_SEGMENTATION=true
```
3. **Validation Tests:**
- Upload 5-10 Japanese customer documents
- Trigger Airflow DAG for document processing
- Monitor logs for segmentation events
- Verify term extraction improvement
- Check for any errors or exceptions
4. **Success Criteria:**
- ✓ No crashes or errors
- ✓ Japanese segmentation logs appear
- ✓ Term count improvement observed
- ✓ Processing time < 2.5s per document
- ✓ No degradation for English documents
**Timeline:** 2-3 days for validation
### Phase 2: Production Deployment - 10% Rollout (Week 2)
**Objective:** Deploy to production with minimal risk using feature flag control
**Steps:**
1. **Deploy Code:**
```bash
# Verify dependencies are in main branch
git checkout main
grep -q "sudachipy" pyproject.toml || echo "✗ ERROR: Dependencies missing from main branch!"
grep -q "sudachipy" uv.lock || echo "✗ ERROR: Lock file not updated!"
# Merge to main branch (if not already merged)
git merge feature/japanese-segmentation
git push origin main
# Rebuild production Docker images (picks up dependencies from uv.lock)
docker-compose -f docker-compose.prod.yml build airflow-worker
# Deploy (zero-downtime deployment)
docker-compose -f docker-compose.prod.yml up -d --no-deps airflow-worker
```
2. **Initial Configuration (Conservative):**
```bash
# In production .env - START WITH DISABLED
ENABLE_CJK_SEGMENTATION=false
```
3. **Enable for 10% of Processing:**
**Option A (Time-Based):**
- Enable segmentation for 2.4 hours per day (10% of 24 hours)
- Monitor during low-traffic periods initially
**Option B (Manual Sampling):**
- Keep disabled by default
- Enable temporarily for specific document batches
- Test with known Japanese documents
4. **Monitoring (First 48 Hours):**
- Check logs every 4 hours for:
- `japanese_segmentation_success` events
- `japanese_segmentation_failed` events (should be rare)
- Processing time metrics
- Error rates
- Validate no increase in LLM errors
- Validate term extraction improvement
5. **Success Criteria:**
- ✓ Zero crashes or pipeline failures
- ✓ < 1% segmentation failure rate
- ✓ Term extraction improvement matches testing (+3-4%)
- ✓ No performance degradation
- ✓ No increase in error logs
**Timeline:** 2-3 days monitoring before next phase
### Phase 3: Production Deployment - 50% Rollout (Week 2-3)
**Objective:** Expand to half of production traffic
**Steps:**
1. **Increase Exposure:**
```bash
# Enable segmentation full-time
ENABLE_CJK_SEGMENTATION=true
# Restart Airflow workers to pick up config change
docker-compose -f docker-compose.prod.yml restart airflow-worker
```
2. **Extended Monitoring (7 Days):**
- Daily log reviews
- Weekly metrics comparison
- User feedback collection (if any Japanese doc users)
- Performance metrics trending
3. **Success Criteria:**
- ✓ Stable for 7 consecutive days
- ✓ Consistent term extraction improvement
- ✓ No user complaints
- ✓ Metrics trend positive
**Timeline:** 1 week validation
### Phase 4: Full Production Rollout (Week 3-4)
**Objective:** Feature is fully deployed and monitored
**Steps:**
1. **Confirm Full Deployment:**
- Already running with `ENABLE_CJK_SEGMENTATION=true`
- No changes needed if Phase 3 successful
2. **Long-Term Monitoring Setup:**
- Add to regular monitoring dashboards
- Set up alerts for segmentation failure spikes (>5% failure rate)
- Document in runbook
3. **Documentation:**
- Update architecture docs
- Update deployment docs
- Document rollback procedure
**Timeline:** Ongoing monitoring
## Monitoring Strategy
### Log Events to Monitor
**Success Events:**
```
japanese_text_detected - Indicates Japanese text found
Fields: char_count, japanese_ratio
japanese_segmentation_success - Successful segmentation
Fields: char_count, token_count, elapsed_ms
llm_extraction_success - LLM extraction completed
Fields: terms_count (should be higher for Japanese)
```
**Warning/Error Events:**
```
japanese_segmentation_failed - Segmentation error (fallback activated)
Fields: error, fallback="original_text"
Alert: If >5% of Japanese docs fail
sudachi_init_failed - Tokenizer initialization failed
Fields: error
Alert: Immediate (indicates library issue)
cjk_segmentation_disabled - Feature disabled via env var
Fields: reason="env_var_false"
```
### Metrics to Track
**Primary Metrics:**
1. **Term Extraction Rate:**
- Baseline: 14.5 avg terms/doc (Japanese)
- Target: 15.0+ avg terms/doc (+3.4%)
- Alert: If drops below 14.0
2. **Segmentation Success Rate:**
- Target: >99%
- Alert: If <95%
3. **Processing Time:**
- Baseline: ~2.1s per document
- Target: <2.5s per document
- Alert: If >3.0s
4. **Error Rate:**
- Baseline: Current error rate
- Target: No increase
- Alert: If >10% increase
**Secondary Metrics:**
5. **Category Detection Accuracy:**
- Track percentage of Japanese docs correctly categorized
- Manual spot-checks monthly
6. **Segmentation Coverage:**
- % of documents that trigger Japanese detection
- % of detected documents successfully segmented
### Dashboards & Queries
**Suggested Log Queries (structlog format):**
```python
# Count segmentation successes per hour
SELECT
DATE_TRUNC('hour', timestamp) as hour,
COUNT(*) as segmentation_count
FROM logs
WHERE event = 'japanese_segmentation_success'
GROUP BY hour
ORDER BY hour DESC;
# Segmentation failure rate
SELECT
COUNT(CASE WHEN event = 'japanese_segmentation_failed' THEN 1 END) * 100.0 /
COUNT(CASE WHEN event = 'japanese_text_detected' THEN 1 END) as failure_rate_pct
FROM logs
WHERE timestamp > NOW() - INTERVAL '24 hours';
# Average term extraction count for Japanese docs
SELECT
AVG(terms_count) as avg_terms
FROM logs
WHERE event = 'llm_extraction_success'
AND EXISTS (
SELECT 1 FROM logs l2
WHERE l2.event = 'japanese_text_detected'
AND l2.timestamp BETWEEN logs.timestamp - INTERVAL '10 seconds'
AND logs.timestamp
);
```
## Rollback Procedures
### Immediate Rollback (If Critical Issues)
**Scenario:** Crashes, data corruption, >50% error rate
**Steps:**
1. **Disable Feature Immediately:**
```bash
# Update .env in production
ENABLE_CJK_SEGMENTATION=false
# Restart workers
docker-compose -f docker-compose.prod.yml restart airflow-worker
```
2. **Verify Rollback:**
- Check logs for `cjk_segmentation_disabled` event
- Confirm no more `japanese_segmentation_success` events
- Monitor error rates return to baseline
3. **Timeline:** <5 minutes
### Gradual Rollback (If Quality Issues)
**Scenario:** Lower term extraction quality, user complaints
**Steps:**
1. **Reduce Exposure:**
```bash
# Disable feature
ENABLE_CJK_SEGMENTATION=false
```
2. **Investigate:**
- Review logs for failure patterns
- Check specific documents causing issues
- Analyze term extraction quality
3. **Decision:**
- Fix and redeploy if fixable
- OR keep disabled and reschedule deployment
### Code Rollback (If Feature Flag Insufficient)
**Scenario:** Feature flag doesn't prevent issue, need code rollback
**Steps:**
1. **Revert Code:**
```bash
git revert
git push origin main
```
2. **Rebuild and Deploy:**
```bash
docker-compose -f docker-compose.prod.yml build airflow-worker
docker-compose -f docker-compose.prod.yml up -d --no-deps airflow-worker
```
3. **Timeline:** ~15-20 minutes
## Troubleshooting Guide
### Issue: Segmentation Not Happening
**Symptoms:**
- No `japanese_text_detected` events in logs
- Japanese documents still show poor term extraction
**Diagnosis:**
```bash
# Check environment variable
docker exec -it airflow-worker bash
echo $ENABLE_CJK_SEGMENTATION # Should be "true"
# Check logs for cjk_segmentation_disabled
grep "cjk_segmentation_disabled" /var/log/airflow/worker.log
```
**Resolution:**
1. Verify `.env` has `ENABLE_CJK_SEGMENTATION=true`
2. Restart workers to pick up env var change
3. Test with known Japanese document
### Issue: High Segmentation Failure Rate
**Symptoms:**
- Many `japanese_segmentation_failed` events
- Error: "No module named 'sudachipy'"
**Diagnosis:**
```bash
# Check if library installed
docker exec -it airflow-worker bash
python -c "import sudachipy; print('OK')"
# Check pyproject.toml
grep sudachipy pyproject.toml
```
**Resolution:**
1. Verify `sudachipy` in `pyproject.toml` dependencies
2. Run `uv lock` to update lock file
3. Rebuild Docker image
4. Redeploy
### Issue: Performance Degradation
**Symptoms:**
- Processing time >3s per document
- CPU usage spike
**Diagnosis:**
```bash
# Check segmentation timing in logs
grep "japanese_segmentation_success" /var/log/airflow/worker.log | grep "elapsed_ms"
# Expected: <50ms per document
```
**Resolution:**
1. If elapsed_ms >100ms consistently → investigate tokenizer initialization
2. Check if tokenizer being recreated (should be cached)
3. Review lazy initialization pattern
### Issue: Incorrect Language Detection
**Symptoms:**
- English documents detected as Japanese
- Japanese documents not detected
**Diagnosis:**
- Review `_is_japanese()` Unicode range checks
- Test specific document causing issue
**Resolution:**
1. Adjust Unicode range detection if false positives
2. For false negatives, verify document actually contains Hiragana/Katakana/Kanji
3. Consider mixed-script handling improvements
### Issue: Term Quality Degraded
**Symptoms:**
- Fewer terms extracted than baseline
- Category detection worse
**Diagnosis:**
```bash
# Compare term counts before/after
# Review segmented text in logs (if debug enabled)
```
**Resolution:**
1. Verify using Mode C (long unit) for SudachiPy
2. Check if truncation happening before segmentation (should be after)
3. Review specific examples causing issues
## Review Notes
**Adversarial Review Completed:** 2026-01-13
**Findings:** 15 total (4 HIGH, 7 MEDIUM, 4 LOW)
**Resolution Approach:** Auto-fix HIGH severity issues
**Fixes Applied:**
- F1 [HIGH]: Fixed misleading tokenizer mode comment - clarified mode is specified during tokenization
- F10 [HIGH]: Moved truncation before segmentation to prevent text expansion issues
- F11 [HIGH]: Added integration tests demonstrating Japanese domain term extraction improvement (2 tests)
- F13 [HIGH]: Added backward compatibility tests for non-Japanese languages (3 tests: English, Spanish, mixed)
**Test Results:**
- All 30 Japanese segmentation tests passing
- All 23 existing domain extractor tests passing (no regression)
- Total: 53 tests passing
**Remaining Findings (11):** Documented for future improvement
- 7 MEDIUM severity: Error handling, validation, test isolation, concurrency, logging, dependency versioning
- 4 LOW severity: Performance monitoring config, redundant calculations, flag semantics, test env var handling