# Analysis Plan for CustomerDocument Folder

## 1. Objective
To analyze the `CustomerDocument` folder to understand the domain knowledge, information structure, and categorization of the contained documents.

## 2. Initial Findings
- **Location**: `/home/hieutt50/projects/textiq-doc-extraction/CustomerDocument`
- **Content**: The folder contains numerous Excel files (`.xls`, `.xlsx`) organized into subdirectories.
- **Language**: The content is primarily in **Japanese**.
- **Domain**: The documents appear to be **System Design Documents**, specifically covering:
    - **JP1 Job Scheduling**: Job nets, execution agents, schedules.
    - **Screen Transitions**: UI flow diagrams.
    - **Interface Definitions**: Data exchange formats (e.g., SOAP Faults).
- **File System Issues**: Directory and filenames exhibit encoding issues (mojibake), which may require careful handling during automated processing.

## 3. Proposed Analysis Plan

### Phase 1: Inventory and Metadata Extraction
**Goal**: Create a comprehensive catalog of all assets.
- **Action**: Develop a Python script to traverse the directory structure.
- **Data to Extract**:
    - Full file paths (handling encoding).
    - File metadata (size, modification date).
    - Internal Excel metadata:
        - Sheet names.
        - Column headers (row 1-5).
        - Title cells (often in cell A1).
- **Output**: A `document_inventory.csv` listing all files with their metadata.

### Phase 2: Categorization and Tagging
**Goal**: Group documents by their functional role.
- **Action**: Analyze the inventory to classify files into categories such as:
    - **Job Management**: Files containing "JP1", "Job", "Schedule".
    - **User Interface**: Files containing "Screen", "Transition", "UI".
    - **Interfaces/API**: Files containing "IF", "Interface", "SOAP", "Data".
    - **Business Logic**: Files describing specific business rules or "Construction Information".
- **Output**: A categorized list/report of documents.

### Phase 3: Domain Knowledge Extraction
**Goal**: Understand the business entities and system logic.
- **Action**:
    - **Glossary Generation**: Extract frequent Japanese terms from headers and sheet names. Translate them to English to build a "Domain Glossary".
    - **Entity Recognition**: Identify key business entities (e.g., "Construction", "Order", "Customer") and System entities (e.g., "Unit", "Service").
    - **Structure Analysis**: For JP1 files, map the hierarchy of Job Groups -> Job Nets -> Jobs.

### Phase 4: Synthesis and Reporting
**Goal**: Summarize the findings for the client.
- **Action**: Compile the findings into a "Domain Knowledge Report".
- **Report Sections**:
    - **System Overview**: High-level architecture inferred from documents.
    - **Document Map**: How the folders and files relate to system components.
    - **Key Terminology**: The constructed glossary.
    - **Observations**: Notes on the completeness or complexity of the documentation.

## 4. Next Steps
1.  Approve this plan.
2.  Execute Phase 1 (Inventory Script).