---
title: 'Add Japanese Word Segmentation to Domain Term Extraction'
slug: 'japanese-word-segmentation-domain-extraction'
created: '2026-01-12'
status: 'ready-for-dev'
stepsCompleted: [1, 2, 3, 4]
tech_stack: ['Python 3.11+', 'SudachiPy', 'Azure OpenAI', 'Airflow 3.0.6', 'structlog', 'pytest', 'httpx', 'uv']
files_to_modify: ['src/extraction_v2/domain_term_extractor.py', 'pyproject.toml', '.env.example']
files_to_create: ['tests/extraction_v2/test_japanese_segmentation.py']
code_patterns: ['lazy_initialization', 'graceful_fallback', 'fork_safe_clients', 'env_var_config', 'structlog_events']
test_patterns: ['pytest', 'unittest_mock', 'patch_decorator', 'patch_dict_os_environ', 'mock_llm_response']
---

# Tech-Spec: Add Japanese Word Segmentation to Domain Term Extraction

**Created:** 2026-01-12

## Overview

### Problem Statement

LLMs struggle to detect domain terms in Japanese text because Japanese doesn't use spaces between words, making word boundary detection difficult. Current baseline extraction results in:
- Lower term extraction rates for Japanese documents
- Poor category classification (documents marked as "general" instead of specific domains like "finance" or "hr")
- Suboptimal query pre-filtering in RAG pipeline

Testing shows baseline extracts 14.5 avg terms/doc on real customer documents, with most Japanese docs incorrectly categorized as "general."

### Solution

Add SudachiPy word segmentation preprocessing to the `_llm_extract()` method in `DomainTermExtractor`. When Japanese text is detected, segment it with spaces before sending to the LLM for term extraction. This approach:
- Uses Mode C (long unit) segmentation to preserve compound words
- Implements lazy initialization for fork-safety in Airflow workers
- Includes graceful fallback to baseline if segmentation fails
- Controlled via `ENABLE_CJK_SEGMENTATION` environment variable

Validated improvement: +3.4% term extraction on real customer documents (15.0 vs 14.5 avg terms).

### Scope

**In Scope:**
- Modify `_llm_extract()` method in `src/extraction_v2/domain_term_extractor.py` to add segmentation preprocessing
- Add `_is_japanese()` helper method for language detection (checks for Hiragana, Katakana, Kanji)
- Add `_segment_japanese()` helper method with SudachiPy integration
- Add `_get_sudachi_tokenizer()` for lazy initialization of tokenizer
- Add `ENABLE_CJK_SEGMENTATION` environment variable (default: true)
- Add structured logging for segmentation events (success, failures, timings)
- Add `sudachipy>=0.6.0` and `sudachidict_core>=20240716` to `requirements.txt` and `pyproject.toml`
- Document Docker/Airflow worker dependency installation
- Create new test file `tests/extraction_v2/test_japanese_segmentation.py` with comprehensive test coverage
- Write deployment procedures for staging and production phased rollout
- Define monitoring strategy and metrics collection
- Document rollback procedures and troubleshooting guide

**Out of Scope:**
- Modifying `_extract_by_rules()` method (segmentation only benefits LLM extraction path)
- Chinese or Korean segmentation support (future enhancement)
- Changes to `DomainTaxonomy` categories or term patterns
- Automated A/B testing infrastructure (manual validation phase only)
- Real-time metrics dashboard (use existing logging infrastructure)

## Context for Development

### Codebase Patterns

**1. Lazy Initialization Pattern (Fork-Safe Clients)**

Location: `src/extraction_v2/domain_term_extractor.py:151-178`

Current implementation in `_get_client()`:
```python
def _get_client(self) -> AzureOpenAI:
    """Get or create Azure OpenAI client lazily.

    Lazy initialization prevents creating HTTP clients before process forking,
    which causes issues in Celery/Airflow workers.
    """
    if not self._client_initialized:
        import httpx  # Import only when needed

        # Create httpx client with fork-safe settings
        http_client = httpx.Client(
            http2=False,  # Disable HTTP/2 to avoid fork issues
            limits=httpx.Limits(
                max_connections=1,
                max_keepalive_connections=0,
            ),
            timeout=httpx.Timeout(60.0),
        )

        self._client = AzureOpenAI(...)
        self._client_initialized = True

    return self._client
```

**Apply same pattern for SudachiPy tokenizer:**
- Add in `__init__()` (line ~149): `self._sudachi_tokenizer = None`, `self._sudachi_initialized = False`
- Create `_get_sudachi_tokenizer()` method following exact same structure
- Import sudachipy inside the method, not at module level
- Handle ImportError gracefully (library might not be installed in all environments)

**2. Environment Variable Configuration**

Location: `src/extraction_v2/domain_term_extractor.py:22-23`

Current pattern:
```python
# Environment configuration
MAX_TERMS_PER_CHUNK = int(os.getenv("DOMAIN_TERM_MAX_PER_CHUNK", "50"))
LLM_THRESHOLD = int(os.getenv("DOMAIN_TERM_LLM_THRESHOLD", "2"))
```

**Add new variable at line ~24:**
```python
ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"
```

**3. Graceful Fallback with Error Handling**

Location: `src/extraction_v2/domain_term_extractor.py:313-362` (`_llm_extract()` method)

Current error handling pattern:
```python
try:
    # Main logic
    ...
    return terms
except json.JSONDecodeError as e:
    self.logger.error("llm_json_parse_error", error=str(e), result=result_text)
    return []
except Exception as e:
    self.logger.error("llm_extraction_failed", error=str(e))
    return []
```

**Apply to segmentation:**
- Wrap segmentation in try/except
- Log specific error but continue with original text
- Never raise exceptions that would break the pipeline
- Return original text as fallback

**4. Structured Logging with structlog**

Location: Throughout `domain_term_extractor.py`, logger initialized at line 149

Current logging pattern:
```python
self.logger.info("using_llm_extraction", terms_found=len(terms), threshold=LLM_THRESHOLD)
self.logger.info("llm_extraction_success", terms_count=len(terms))
self.logger.error("llm_extraction_failed", error=str(e))
self.logger.warning("llm_returned_non_list", result=result_text)
```

**Add new log events:**
- `japanese_text_detected` - when `_is_japanese()` returns True
- `japanese_segmentation_success` - after successful segmentation (include char_count, token_count)
- `japanese_segmentation_failed` - on exception (include error, fallback action)
- `sudachi_init_failed` - if tokenizer initialization fails (include error)
- `cjk_segmentation_disabled` - if env var is false

**5. LLM Truncation Pattern**

Location: `src/extraction_v2/domain_term_extractor.py:325-327`

Current truncation:
```python
# Truncate content if too long (max ~2000 tokens)
if len(content) > 8000:
    content = content[:8000] + "..."
```

**Segmentation should happen BEFORE truncation:**
- Detect Japanese → Segment → Truncate → Send to LLM
- This ensures segmented text benefits from full context before truncation

### Files to Reference

| File | Purpose | Key Sections |
| ---- | ------- | ------------ |
| `src/extraction_v2/domain_term_extractor.py` | Main file to modify | `__init__`: L135-149<br>`_get_client()`: L151-178 (lazy init pattern)<br>`_llm_extract()`: L313-362 (add segmentation here)<br>Env vars: L22-23 |
| `tests/extraction_v2/test_domain_term_extractor.py` | Unit test patterns | Mock pattern: L144-152<br>Env var pattern: L158<br>LLM response mock: L148-152 |
| `pyproject.toml` | Add dependencies | Dependencies: L8-39<br>Dev deps: L42-48<br>Pytest config: L102-107 |
| `.env.example` | Document env var | Add to L84-92 (API Configuration section) |
| `docker/Dockerfile.airflow` | Airflow worker setup | uv sync: L17-26<br>Validate deps install after adding to pyproject.toml |
| `src/core/config.py` | Pydantic settings | Reference only - env var uses `os.getenv()`, not Pydantic |
| `airflow/dags/tasks/processing_tasks.py` | Airflow integration | Import extractor: L386<br>**No changes needed** - uses extractor directly |
| `test_domain_extraction.py` | Manual test script | Integration test patterns for validation |
| `JAPANESE_SEGMENTATION_PROPOSAL.md` | Original proposal | Implementation examples, code snippets |
| `CUSTOMER_TEST_RESULTS_SUMMARY.md` | Validation data | Real customer test results: +3.4% improvement |
| `experiment_japanese_segmentation.py` | Experimental validation | Shows SudachiPy Mode C usage example |

### Code Structure & Implementation Points

**Class: DomainTermExtractor** (`src/extraction_v2/domain_term_extractor.py`)

```
Lines 132-363: DomainTermExtractor class
├── __init__() [L135-149]
│   ├── Current: self.taxonomy, self._client, self._client_initialized, self.logger
│   └── ADD: self._sudachi_tokenizer = None, self._sudachi_initialized = False
│
├── _get_client() [L151-178] - REFERENCE for lazy init pattern
│   └── Pattern to follow for _get_sudachi_tokenizer()
│
├── extract_from_chunk() [L180-241] - Main entry point
│   └── Calls _llm_extract() at line 227 when terms < threshold
│
├── _extract_by_rules() [L290-311] - NO CHANGES (regex works without segmentation)
│
└── _llm_extract() [L313-362] - PRIMARY MODIFICATION POINT
    ├── Line 323: Get client
    ├── Line 325-327: Truncate content (ADD segmentation BEFORE this)
    ├── Line 329: Format prompt
    ├── Line 331-339: Call Azure OpenAI
    ├── Line 344: Parse JSON
    └── Line 357-362: Error handling
```

**Add 3 New Methods (insert after line 311, before `_llm_extract()`):**

1. `_get_sudachi_tokenizer()` - Lazy init SudachiPy tokenizer
2. `_is_japanese()` - Detect Japanese characters
3. `_segment_japanese()` - Segment Japanese text

**Modify `_llm_extract()` method:**
- Add check at line ~324 (after `client = self._get_client()`)
- If Japanese detected AND env var enabled → segment content
- Log segmentation events
- Apply graceful fallback if errors

**Environment Variables** (add at line ~24):
```python
ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"
```

### Technical Decisions

**Decision 1: SudachiPy Mode C (Long Unit)**
- **Rationale:** Testing showed Mode C preserves compound words better (e.g., "売上高" stays as one term)
- **Alternative Considered:** Mode A (short unit) - too granular, splits meaningful compounds
- **Implementation:** Use `tokenizer.Tokenizer.SplitMode.C` in segmentation

**Decision 2: Environment Variable Control**
- **Variable:** `ENABLE_CJK_SEGMENTATION` (default: "true")
- **Rationale:** Allows gradual rollout and easy rollback if issues arise
- **Values:** "true" enables, "false" or any other value disables

**Decision 3: Only Segment for LLM Path**
- **Rationale:** Rule-based extraction uses regex matching which already works with Japanese (no spaces needed)
- **Benefit:** Minimal code changes, focused improvement where it matters most

**Decision 4: Conservative Language Detection**
- **Method:** Check for Unicode ranges: Hiragana (U+3040-U+309F), Katakana (U+30A0-U+30FF), Kanji (U+4E00-U+9FFF)
- **Rationale:** Avoid false positives - only segment if clear Japanese characters present
- **Trade-off:** Won't segment mixed-script text (acceptable for initial implementation)

**Decision 5: Lazy Initialization for Fork-Safety**
- **Rationale:** Airflow workers use multiprocessing with fork - initializing clients before fork causes issues
- **Pattern:** Initialize tokenizer on first use, not in `__init__`
- **Benefit:** Consistent with existing `_get_client()` pattern (lines 151-178)

**Decision 6: Use pyproject.toml (Not requirements.txt)**
- **Finding:** Project uses `pyproject.toml` with `hatchling` build backend and `uv` package manager
- **Action:** Add dependencies to `pyproject.toml` dependencies list (lines 8-39)
- **Deployment:** Run `uv lock` after adding dependencies to update `uv.lock`
- **Docker:** `docker/Dockerfile.airflow` uses `uv sync` (line 26) - will automatically pick up new dependencies

**Decision 7: Document Environment Variable in .env.example**
- **Finding:** `.env.example` exists with well-organized sections
- **Action:** Add `ENABLE_CJK_SEGMENTATION=true` under "API Configuration" section (lines 84-92)
- **Format:** Follow existing pattern with comment explaining purpose

**Decision 8: Testing Strategy with unittest.mock**
- **Pattern:** Use `@patch("module.path.Class.method")` decorator (line 144 in test file)
- **Env Vars:** Use `with patch.dict("os.environ", {"KEY": "value"})` (line 158 in test file)
- **LLM Mocking:** Mock `chat.completions.create` return value (lines 148-152 in test file)
- **Rationale:** Consistent with existing test patterns, no external dependencies for unit tests

**Decision 9: No Changes to Airflow DAG**
- **Finding:** `extract_domain_terms_task` (line 372-533 in processing_tasks.py) simply imports and uses `DomainTermExtractor`
- **Conclusion:** Segmentation will automatically apply when Airflow calls the extractor
- **Benefit:** Zero deployment changes to DAG files

**Decision 10: Deployment Uses uv Package Manager**
- **Finding:** Dockerfile.airflow uses `uv` for dependency management (not pip)
- **Workflow:** Add to pyproject.toml → Run `uv lock` → Docker rebuild picks up changes
- **Benefit:** Fast, reproducible builds with locked dependencies

## Implementation Plan

### Tasks

**Phase 1: Dependencies & Configuration**

- [ ] Task 1: Add SudachiPy dependencies to pyproject.toml
  - File: `pyproject.toml`
  - Action: Add to dependencies list (line 8-39): `"sudachipy>=0.6.0"` and `"sudachidict_core>=20240716"`
  - Notes: Insert alphabetically between existing packages

- [ ] Task 2: Update dependency lock file
  - Command: `uv lock`
  - Action: Run from project root to update `uv.lock` with new dependencies
  - Notes: Verify lock file updated successfully (check for sudachipy entries)

- [ ] Task 3: Add environment variable configuration
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Add at line ~24 (after LLM_THRESHOLD): `ENABLE_CJK_SEGMENTATION = os.getenv("ENABLE_CJK_SEGMENTATION", "true").lower() == "true"`
  - Notes: Follow existing pattern with other env vars

- [ ] Task 4: Document environment variable in .env.example
  - File: `.env.example`
  - Action: Add to "API Configuration" section (after line 91):
    ```
    # ===================
    # Japanese Segmentation Configuration
    # ===================
    # Enable Japanese word segmentation for improved domain term extraction
    # Set to 'false' to disable (useful for rollback or debugging)
    ENABLE_CJK_SEGMENTATION=true
    ```
  - Notes: Include helpful comment explaining purpose

**Phase 2: Core Implementation**

- [ ] Task 5: Initialize segmentation state in __init__
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Add at line ~149 (end of __init__ method):
    ```python
    self._sudachi_tokenizer = None
    self._sudachi_initialized = False
    ```
  - Notes: Add after `self.logger = structlog.get_logger(__name__)`

- [ ] Task 6: Implement lazy tokenizer initialization method
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Add new method after line 311 (after `_extract_by_rules`, before `_llm_extract`):
    ```python
    def _get_sudachi_tokenizer(self):
        """Get or create SudachiPy tokenizer lazily.

        Lazy initialization prevents creating tokenizer before process forking,
        which causes issues in Celery/Airflow workers.

        Returns:
            Tokenizer instance or None if initialization fails
        """
        if not self._sudachi_initialized:
            try:
                from sudachipy import tokenizer, dictionary

                tokenizer_obj = dictionary.Dictionary().create()
                self._sudachi_tokenizer = tokenizer_obj
                self._sudachi_initialized = True

            except ImportError as e:
                self.logger.warning(
                    "sudachi_init_failed",
                    error="sudachipy not installed",
                    fallback="segmentation_disabled"
                )
                self._sudachi_initialized = True  # Don't retry
            except Exception as e:
                self.logger.error(
                    "sudachi_init_failed",
                    error=str(e),
                    fallback="segmentation_disabled"
                )
                self._sudachi_initialized = True  # Don't retry

        return self._sudachi_tokenizer
    ```
  - Notes: Follow exact pattern from `_get_client()` method (lines 151-178)

- [ ] Task 7: Implement Japanese language detection method
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Add new method after `_get_sudachi_tokenizer`:
    ```python
    def _is_japanese(self, text: str) -> bool:
        """Detect if text contains Japanese characters.

        Checks for Hiragana, Katakana, or Kanji characters.

        Args:
            text: Text to analyze

        Returns:
            True if Japanese characters detected, False otherwise
        """
        if not text:
            return False

        # Unicode ranges for Japanese scripts
        # Hiragana: U+3040-U+309F
        # Katakana: U+30A0-U+30FF
        # Kanji: U+4E00-U+9FFF
        japanese_pattern = r'[\u3040-\u309F\u30A0-\u30FF\u4E00-\u9FFF]'
        return bool(re.search(japanese_pattern, text))
    ```
  - Notes: Conservative detection to avoid false positives

- [ ] Task 8: Implement Japanese text segmentation method
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Add new method after `_is_japanese`:
    ```python
    def _segment_japanese(self, text: str) -> str:
        """Segment Japanese text using SudachiPy.

        Args:
            text: Japanese text to segment

        Returns:
            Space-separated segmented text, or original text if segmentation fails
        """
        tokenizer_obj = self._get_sudachi_tokenizer()

        if tokenizer_obj is None:
            return text  # Fallback to original if tokenizer unavailable

        try:
            from sudachipy import tokenizer

            # Use Mode C (long unit) to preserve compound words
            mode = tokenizer.Tokenizer.SplitMode.C
            tokens = [m.surface() for m in tokenizer_obj.tokenize(text, mode)]
            segmented = " ".join(tokens)

            self.logger.info(
                "japanese_segmentation_success",
                char_count=len(text),
                token_count=len(tokens)
            )

            return segmented

        except Exception as e:
            self.logger.warning(
                "japanese_segmentation_failed",
                error=str(e),
                fallback="original_text"
            )
            return text  # Graceful fallback
    ```
  - Notes: Always return text (never raise exceptions)

- [ ] Task 9: Modify _llm_extract to add segmentation preprocessing
  - File: `src/extraction_v2/domain_term_extractor.py`
  - Action: Modify `_llm_extract()` method at line ~324 (after `client = self._get_client()`):
    ```python
    # Add after line 323 (client = self._get_client())
    # BEFORE truncation (line 325-327)

    # PREPROCESSING: Segment Japanese text if enabled
    if ENABLE_CJK_SEGMENTATION:
        if self._is_japanese(content):
            self.logger.info("japanese_text_detected", char_count=len(content))
            content = self._segment_japanese(content)
    else:
        self.logger.debug("cjk_segmentation_disabled", reason="env_var_false")

    # Then continue with existing truncation logic...
    ```
  - Notes: Segmentation MUST happen BEFORE truncation (line 325-327)

**Phase 3: Testing**

- [ ] Task 10: Create comprehensive test file
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Create new file with test classes:
    - `TestJapaneseLanguageDetection` - test `_is_japanese()` method
    - `TestJapaneseSegmentation` - test `_segment_japanese()` method
    - `TestLazyTokenizerInitialization` - test `_get_sudachi_tokenizer()` method
    - `TestEnvironmentVariableControl` - test env var enable/disable
    - `TestLLMExtractIntegration` - test full integration with mocked LLM
    - `TestJapaneseSegmentationIntegration` - test with real customer doc samples
  - Notes: Follow patterns from `test_domain_term_extractor.py` (use @patch, patch.dict)

- [ ] Task 11: Write unit tests for language detection
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Add tests covering:
    - Japanese text only (Hiragana: "こんにちは", Katakana: "カタカナ", Kanji: "日本語")
    - English text only ("Hello World")
    - Mixed Japanese/English text ("Hello 日本")
    - Empty string ("")
    - Whitespace only ("   ")
    - Chinese text (should NOT detect - "你好")
  - Notes: Use parametrized tests for efficiency

- [ ] Task 12: Write unit tests for segmentation
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Add tests covering:
    - Simple sentence: "売上高は増加した" → verify spaces added
    - Compound preservation: verify Mode C keeps compounds together
    - Empty input → returns empty/original
    - Mock segmentation error → verify graceful fallback to original text
  - Notes: May need to check actual segmentation output format

- [ ] Task 13: Write unit tests for environment variable control
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Add tests using `patch.dict("os.environ", {...})`:
    - `ENABLE_CJK_SEGMENTATION=true` → segmentation active
    - `ENABLE_CJK_SEGMENTATION=false` → segmentation disabled
    - Missing env var → defaults to true
    - Invalid value ("invalid") → treated as false
  - Notes: Follow pattern from line 158 in existing test file

- [ ] Task 14: Write unit tests for lazy initialization
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Add tests covering:
    - First call initializes tokenizer
    - Second call returns cached instance (no re-init)
    - ImportError handling (mock sudachipy as unavailable)
    - Generic exception handling
  - Notes: Use `patch.dict("sys.modules", {"sudachipy": None})` to simulate import failure

- [ ] Task 15: Write integration tests with mocked LLM
  - File: `tests/extraction_v2/test_japanese_segmentation.py`
  - Action: Add tests using `@patch("...DomainTermExtractor._get_client")`:
    - Japanese text + env=true → LLM receives segmented text
    - Japanese text + env=false → LLM receives original text
    - English text → LLM receives original text
    - Segmentation failure → LLM receives original text (fallback works)
    - Mock LLM response → verify term extraction still works
  - Notes: Follow mock pattern from lines 144-163 in existing test file

- [ ] Task 16: Run all tests and verify passing
  - Command: `pytest tests/extraction_v2/test_japanese_segmentation.py -v`
  - Action: Execute full test suite, verify all tests pass
  - Notes: Fix any failures before proceeding

- [ ] Task 17: Run existing domain extractor tests
  - Command: `pytest tests/extraction_v2/test_domain_term_extractor.py -v`
  - Action: Verify no regression in existing functionality
  - Notes: All existing tests should still pass

**Phase 4: Validation & Deployment Prep**

- [ ] Task 18: Manual testing with real customer documents
  - Script: `python test_domain_extraction.py` and `python experiment_japanese_segmentation.py`
  - Action: Test with Japanese documents from CustomerDocument/ folder
  - Expected: Term count improvement ≥3%, processing time <2.5s
  - Notes: Compare results with baseline (env var disabled)

- [ ] Task 19: Performance validation
  - Action: Measure segmentation overhead with different text sizes
  - Expected: Small (<100 chars): <10ms, Medium (<500 chars): <25ms, Large (<1000 chars): <50ms
  - Notes: Log elapsed_ms from segmentation events

- [ ] Task 20: Code review and cleanup
  - Action: Review all changes, ensure code quality, add docstrings, check formatting
  - Notes: Run `black` formatter if used in project

- [ ] Task 21: Pre-deployment checklist verification
  - Action: Verify all items in "Pre-Deployment Checklist" section are complete
  - Notes: Don't proceed to deployment without completing all items

### Acceptance Criteria

**Functional Requirements:**

- [ ] AC1: Given Japanese text content, when `_is_japanese()` is called, then it returns True if Hiragana, Katakana, or Kanji characters are present

- [ ] AC2: Given English text content, when `_is_japanese()` is called, then it returns False

- [ ] AC3: Given Japanese text, when `_segment_japanese()` is called, then it returns space-separated tokens using SudachiPy Mode C

- [ ] AC4: Given `ENABLE_CJK_SEGMENTATION=true` and Japanese text, when `_llm_extract()` is called, then the content is segmented before being sent to the LLM

- [ ] AC5: Given `ENABLE_CJK_SEGMENTATION=false` and Japanese text, when `_llm_extract()` is called, then the content is NOT segmented (original text sent to LLM)

- [ ] AC6: Given Japanese text with segmentation enabled, when LLM extraction completes, then the average term count is ≥15.0 terms/doc (+3.4% improvement validated)

**Error Handling & Fallback:**

- [ ] AC7: Given sudachipy library is not installed, when `_get_sudachi_tokenizer()` is called, then it logs "sudachi_init_failed" and returns None without crashing

- [ ] AC8: Given segmentation fails due to exception, when `_segment_japanese()` is called, then it logs "japanese_segmentation_failed" and returns the original text

- [ ] AC9: Given tokenizer initialization fails, when Japanese text is processed, then extraction continues with original (unsegmented) text

- [ ] AC10: Given any segmentation error, when `extract_from_chunk()` is called, then the pipeline completes successfully without raising exceptions

**Performance Requirements:**

- [ ] AC11: Given text <100 characters, when segmentation occurs, then elapsed time is <10ms

- [ ] AC12: Given text <500 characters, when segmentation occurs, then elapsed time is <25ms

- [ ] AC13: Given text <1000 characters, when segmentation occurs, then elapsed time is <50ms

- [ ] AC14: Given Japanese document processing with segmentation, when full extraction completes, then total time is <2.5 seconds (validated ~2.1s avg)

**Integration & Regression:**

- [ ] AC15: Given Airflow DAG processes a Japanese document, when domain extraction task runs, then segmentation automatically applies without DAG changes

- [ ] AC16: Given English document content, when `extract_from_chunk()` is called with segmentation enabled, then term extraction quality is unchanged (no regression)

- [ ] AC17: Given all existing unit tests, when test suite runs, then all tests pass (no regression in existing functionality)

**Logging & Monitoring:**

- [ ] AC18: Given Japanese text is detected, when segmentation succeeds, then "japanese_segmentation_success" event is logged with char_count and token_count

- [ ] AC19: Given Japanese text is detected, when segmentation fails, then "japanese_segmentation_failed" event is logged with error and fallback action

- [ ] AC20: Given segmentation is disabled via env var, when Japanese text is processed, then "cjk_segmentation_disabled" event is logged with reason

**Deployment Validation:**

- [ ] AC21: Given staging environment deployment, when Japanese documents are processed, then term extraction improvement is observed and no errors occur

- [ ] AC22: Given production deployment with env var disabled, when workers restart, then no segmentation events appear in logs

- [ ] AC23: Given production deployment with env var enabled, when Japanese documents are processed, then segmentation success rate is >99%

- [ ] AC24: Given emergency rollback needed, when `ENABLE_CJK_SEGMENTATION=false` is set and workers restart, then segmentation stops within 5 minutes

## Additional Context

### Dependencies

**New Python Packages:**
- `sudachipy>=0.6.0` - Japanese morphological analyzer
- `sudachidict_core>=20240716` - Dictionary for SudachiPy

**Installation:**
```bash
pip install sudachipy>=0.6.0 sudachidict_core>=20240716
```

**Airflow Worker Setup:**
For Docker-based Airflow, add to Dockerfile or requirements:
```dockerfile
RUN pip install sudachipy>=0.6.0 sudachidict_core>=20240716
```

### Testing Strategy

**Unit Tests** (File: `tests/extraction_v2/test_japanese_segmentation.py`)

Test Coverage Required:

1. **Language Detection Tests:**
   - Japanese text only (Hiragana, Katakana, Kanji)
   - English text only
   - Mixed Japanese/English text
   - Empty string
   - Whitespace only
   - Chinese/Korean text (should NOT detect as Japanese)

2. **Segmentation Tests:**
   - Simple Japanese sentence → verify spaces added
   - Compound words → verify Mode C preserves compounds
   - Mixed script text → verify proper handling
   - Empty input → verify returns empty/original
   - Error simulation → verify graceful fallback

3. **Environment Variable Tests:**
   - `ENABLE_CJK_SEGMENTATION=true` → segmentation active
   - `ENABLE_CJK_SEGMENTATION=false` → segmentation disabled
   - Missing env var → defaults to true
   - Invalid value → treated as false

4. **Lazy Initialization Tests:**
   - First call to `_get_sudachi_tokenizer()` → initializes
   - Second call → returns cached instance
   - Initialization failure → logged and returns None

5. **Integration with _llm_extract() Tests:**
   - Japanese text with env=true → LLM receives segmented text
   - Japanese text with env=false → LLM receives original text
   - Non-Japanese text → LLM receives original text
   - Segmentation failure → LLM receives original text (fallback)
   - Mock LLM response → verify correct term extraction

**Mock Patterns to Use:**
```python
# Pattern 1: Mock LLM client
@patch("src.extraction_v2.domain_term_extractor.DomainTermExtractor._get_client")
def test_something(self, mock_get_client):
    mock_client = MagicMock()
    mock_response = MagicMock()
    mock_response.choices[0].message.content = '["term1", "term2"]'
    mock_client.chat.completions.create.return_value = mock_response
    mock_get_client.return_value = mock_client

# Pattern 2: Mock environment variable
with patch.dict("os.environ", {"ENABLE_CJK_SEGMENTATION": "false"}):
    result = extractor.extract_from_chunk(japanese_text)

# Pattern 3: Mock sudachi import failure
with patch.dict("sys.modules", {"sudachipy": None}):
    tokenizer = extractor._get_sudachi_tokenizer()
    assert tokenizer is None
```

**Integration Tests** (Use existing customer documents from validation):

Location: `tests/extraction_v2/test_japanese_segmentation.py` (separate test class)

1. Test with real Japanese customer document samples
2. Measure term count improvement (expect ≥3% improvement)
3. Validate category detection improvement
4. Verify no regression on English documents

**Performance Tests:**

1. Measure segmentation overhead:
   - Small text (100 chars): <10ms
   - Medium text (500 chars): <25ms
   - Large text (1000+ chars): <50ms

2. Measure full extraction latency:
   - Japanese text with segmentation: ~2.1s (validated)
   - Baseline: ~2.1s (no degradation)

**Manual Validation** (Before production deployment):

1. Run `test_domain_extraction.py` with Japanese samples
2. Run `experiment_japanese_segmentation.py` to verify results
3. Test in staging with real customer documents
4. Validate Airflow DAG execution in staging environment

### Notes

**Validation Data:**
- Synthetic tests: +18% improvement (8.3 vs 7.0 avg terms)
- Real customer tests: +3.4% improvement (15.0 vs 14.5 avg terms)
- Processing time: ~2.1s avg (negligible overhead)
- Tested on 10 real customer documents (DOCX + XLSX)

**Deployment Considerations:**
- Recommend phased rollout: staging → 10% production → 50% → 100%
- Monitor metrics: term extraction count, processing time, error rates
- Rollback plan: Set `ENABLE_CJK_SEGMENTATION=false` if issues arise

**Future Enhancements:**
- Chinese segmentation (jieba library)
- Korean segmentation (KoNLPy library)
- Configurable segmentation modes per language
- Automated A/B testing infrastructure

## Deployment Procedures

### Pre-Deployment Checklist

- [ ] All unit tests pass (`pytest tests/extraction_v2/test_japanese_segmentation.py`)
- [ ] Integration tests pass with real customer documents
- [ ] Performance tests show <50ms overhead
- [ ] Code reviewed and approved
- [ ] Dependencies added to `pyproject.toml`
- [ ] `uv lock` executed to update lock file
- [ ] `.env.example` updated with new environment variable
- [ ] Documentation updated

### Phase 1: Staging Deployment (Week 1)

**Objective:** Validate functionality in staging environment with no production risk

**Steps:**

1. **Build and Deploy:**
   ```bash
   # Add dependencies to pyproject.toml
   # sudachipy>=0.6.0
   # sudachidict_core>=20240716

   # Update lock file
   uv lock

   # Rebuild Docker images
   docker-compose -f docker-compose.staging.yml build airflow-worker

   # Deploy to staging
   docker-compose -f docker-compose.staging.yml up -d
   ```

2. **Configure Environment:**
   ```bash
   # In staging .env file
   ENABLE_CJK_SEGMENTATION=true
   ```

3. **Validation Tests:**
   - Upload 5-10 Japanese customer documents
   - Trigger Airflow DAG for document processing
   - Monitor logs for segmentation events
   - Verify term extraction improvement
   - Check for any errors or exceptions

4. **Success Criteria:**
   - ✓ No crashes or errors
   - ✓ Japanese segmentation logs appear
   - ✓ Term count improvement observed
   - ✓ Processing time < 2.5s per document
   - ✓ No degradation for English documents

**Timeline:** 2-3 days for validation

### Phase 2: Production Deployment - 10% Rollout (Week 2)

**Objective:** Deploy to production with minimal risk using feature flag control

**Steps:**

1. **Deploy Code:**
   ```bash
   # Merge to main branch
   git checkout main
   git merge feature/japanese-segmentation
   git push origin main

   # Rebuild production Docker images
   docker-compose -f docker-compose.prod.yml build airflow-worker

   # Deploy (zero-downtime deployment)
   docker-compose -f docker-compose.prod.yml up -d --no-deps airflow-worker
   ```

2. **Initial Configuration (Conservative):**
   ```bash
   # In production .env - START WITH DISABLED
   ENABLE_CJK_SEGMENTATION=false
   ```

3. **Enable for 10% of Processing:**

   **Option A (Time-Based):**
   - Enable segmentation for 2.4 hours per day (10% of 24 hours)
   - Monitor during low-traffic periods initially

   **Option B (Manual Sampling):**
   - Keep disabled by default
   - Enable temporarily for specific document batches
   - Test with known Japanese documents

4. **Monitoring (First 48 Hours):**
   - Check logs every 4 hours for:
     - `japanese_segmentation_success` events
     - `japanese_segmentation_failed` events (should be rare)
     - Processing time metrics
     - Error rates
   - Validate no increase in LLM errors
   - Validate term extraction improvement

5. **Success Criteria:**
   - ✓ Zero crashes or pipeline failures
   - ✓ < 1% segmentation failure rate
   - ✓ Term extraction improvement matches testing (+3-4%)
   - ✓ No performance degradation
   - ✓ No increase in error logs

**Timeline:** 2-3 days monitoring before next phase

### Phase 3: Production Deployment - 50% Rollout (Week 2-3)

**Objective:** Expand to half of production traffic

**Steps:**

1. **Increase Exposure:**
   ```bash
   # Enable segmentation full-time
   ENABLE_CJK_SEGMENTATION=true

   # Restart Airflow workers to pick up config change
   docker-compose -f docker-compose.prod.yml restart airflow-worker
   ```

2. **Extended Monitoring (7 Days):**
   - Daily log reviews
   - Weekly metrics comparison
   - User feedback collection (if any Japanese doc users)
   - Performance metrics trending

3. **Success Criteria:**
   - ✓ Stable for 7 consecutive days
   - ✓ Consistent term extraction improvement
   - ✓ No user complaints
   - ✓ Metrics trend positive

**Timeline:** 1 week validation

### Phase 4: Full Production Rollout (Week 3-4)

**Objective:** Feature is fully deployed and monitored

**Steps:**

1. **Confirm Full Deployment:**
   - Already running with `ENABLE_CJK_SEGMENTATION=true`
   - No changes needed if Phase 3 successful

2. **Long-Term Monitoring Setup:**
   - Add to regular monitoring dashboards
   - Set up alerts for segmentation failure spikes (>5% failure rate)
   - Document in runbook

3. **Documentation:**
   - Update architecture docs
   - Update deployment docs
   - Document rollback procedure

**Timeline:** Ongoing monitoring

## Monitoring Strategy

### Log Events to Monitor

**Success Events:**
```
japanese_text_detected - Indicates Japanese text found
  Fields: char_count, japanese_ratio

japanese_segmentation_success - Successful segmentation
  Fields: char_count, token_count, elapsed_ms

llm_extraction_success - LLM extraction completed
  Fields: terms_count (should be higher for Japanese)
```

**Warning/Error Events:**
```
japanese_segmentation_failed - Segmentation error (fallback activated)
  Fields: error, fallback="original_text"
  Alert: If >5% of Japanese docs fail

sudachi_init_failed - Tokenizer initialization failed
  Fields: error
  Alert: Immediate (indicates library issue)

cjk_segmentation_disabled - Feature disabled via env var
  Fields: reason="env_var_false"
```

### Metrics to Track

**Primary Metrics:**

1. **Term Extraction Rate:**
   - Baseline: 14.5 avg terms/doc (Japanese)
   - Target: 15.0+ avg terms/doc (+3.4%)
   - Alert: If drops below 14.0

2. **Segmentation Success Rate:**
   - Target: >99%
   - Alert: If <95%

3. **Processing Time:**
   - Baseline: ~2.1s per document
   - Target: <2.5s per document
   - Alert: If >3.0s

4. **Error Rate:**
   - Baseline: Current error rate
   - Target: No increase
   - Alert: If >10% increase

**Secondary Metrics:**

5. **Category Detection Accuracy:**
   - Track percentage of Japanese docs correctly categorized
   - Manual spot-checks monthly

6. **Segmentation Coverage:**
   - % of documents that trigger Japanese detection
   - % of detected documents successfully segmented

### Dashboards & Queries

**Suggested Log Queries (structlog format):**

```python
# Count segmentation successes per hour
SELECT
  DATE_TRUNC('hour', timestamp) as hour,
  COUNT(*) as segmentation_count
FROM logs
WHERE event = 'japanese_segmentation_success'
GROUP BY hour
ORDER BY hour DESC;

# Segmentation failure rate
SELECT
  COUNT(CASE WHEN event = 'japanese_segmentation_failed' THEN 1 END) * 100.0 /
  COUNT(CASE WHEN event = 'japanese_text_detected' THEN 1 END) as failure_rate_pct
FROM logs
WHERE timestamp > NOW() - INTERVAL '24 hours';

# Average term extraction count for Japanese docs
SELECT
  AVG(terms_count) as avg_terms
FROM logs
WHERE event = 'llm_extraction_success'
  AND EXISTS (
    SELECT 1 FROM logs l2
    WHERE l2.event = 'japanese_text_detected'
    AND l2.timestamp BETWEEN logs.timestamp - INTERVAL '10 seconds'
    AND logs.timestamp
  );
```

## Rollback Procedures

### Immediate Rollback (If Critical Issues)

**Scenario:** Crashes, data corruption, >50% error rate

**Steps:**

1. **Disable Feature Immediately:**
   ```bash
   # Update .env in production
   ENABLE_CJK_SEGMENTATION=false

   # Restart workers
   docker-compose -f docker-compose.prod.yml restart airflow-worker
   ```

2. **Verify Rollback:**
   - Check logs for `cjk_segmentation_disabled` event
   - Confirm no more `japanese_segmentation_success` events
   - Monitor error rates return to baseline

3. **Timeline:** <5 minutes

### Gradual Rollback (If Quality Issues)

**Scenario:** Lower term extraction quality, user complaints

**Steps:**

1. **Reduce Exposure:**
   ```bash
   # Disable feature
   ENABLE_CJK_SEGMENTATION=false
   ```

2. **Investigate:**
   - Review logs for failure patterns
   - Check specific documents causing issues
   - Analyze term extraction quality

3. **Decision:**
   - Fix and redeploy if fixable
   - OR keep disabled and reschedule deployment

### Code Rollback (If Feature Flag Insufficient)

**Scenario:** Feature flag doesn't prevent issue, need code rollback

**Steps:**

1. **Revert Code:**
   ```bash
   git revert <commit-hash>
   git push origin main
   ```

2. **Rebuild and Deploy:**
   ```bash
   docker-compose -f docker-compose.prod.yml build airflow-worker
   docker-compose -f docker-compose.prod.yml up -d --no-deps airflow-worker
   ```

3. **Timeline:** ~15-20 minutes

## Troubleshooting Guide

### Issue: Segmentation Not Happening

**Symptoms:**
- No `japanese_text_detected` events in logs
- Japanese documents still show poor term extraction

**Diagnosis:**
```bash
# Check environment variable
docker exec -it airflow-worker bash
echo $ENABLE_CJK_SEGMENTATION  # Should be "true"

# Check logs for cjk_segmentation_disabled
grep "cjk_segmentation_disabled" /var/log/airflow/worker.log
```

**Resolution:**
1. Verify `.env` has `ENABLE_CJK_SEGMENTATION=true`
2. Restart workers to pick up env var change
3. Test with known Japanese document

### Issue: High Segmentation Failure Rate

**Symptoms:**
- Many `japanese_segmentation_failed` events
- Error: "No module named 'sudachipy'"

**Diagnosis:**
```bash
# Check if library installed
docker exec -it airflow-worker bash
python -c "import sudachipy; print('OK')"

# Check pyproject.toml
grep sudachipy pyproject.toml
```

**Resolution:**
1. Verify `sudachipy` in `pyproject.toml` dependencies
2. Run `uv lock` to update lock file
3. Rebuild Docker image
4. Redeploy

### Issue: Performance Degradation

**Symptoms:**
- Processing time >3s per document
- CPU usage spike

**Diagnosis:**
```bash
# Check segmentation timing in logs
grep "japanese_segmentation_success" /var/log/airflow/worker.log | grep "elapsed_ms"

# Expected: <50ms per document
```

**Resolution:**
1. If elapsed_ms >100ms consistently → investigate tokenizer initialization
2. Check if tokenizer being recreated (should be cached)
3. Review lazy initialization pattern

### Issue: Incorrect Language Detection

**Symptoms:**
- English documents detected as Japanese
- Japanese documents not detected

**Diagnosis:**
- Review `_is_japanese()` Unicode range checks
- Test specific document causing issue

**Resolution:**
1. Adjust Unicode range detection if false positives
2. For false negatives, verify document actually contains Hiragana/Katakana/Kanji
3. Consider mixed-script handling improvements

### Issue: Term Quality Degraded

**Symptoms:**
- Fewer terms extracted than baseline
- Category detection worse

**Diagnosis:**
```bash
# Compare term counts before/after
# Review segmented text in logs (if debug enabled)
```

**Resolution:**
1. Verify using Mode C (long unit) for SudachiPy
2. Check if truncation happening before segmentation (should be after)
3. Review specific examples causing issues