# Brainstorming Session Results

**Session Date:** 2025-11-26
**Facilitator:** BMad Brainstorming Facilitator
**Participant:** TextIQ

## Session Start

**Approach:** Focused ideation
**Domain:** Complex Excel file processing

## Executive Summary

**Topic:** DSOL - Processing complex Excel files with varied formats, complex tables, merged cells, and diagrams

**Session Goals:** Focused ideation on solving the challenges of extracting and processing data from messy, inconsistent Excel files with complex structures

**Techniques Used:** Mind Mapping, Five Whys, First Principles Thinking, SCAMPER

**Total Ideas Generated:** 25+

### Key Themes Identified:

1. Structure Preservation - Don't destroy what makes Excel meaningful
2. Multi-Pass Processing - Resolve dependencies before final extraction
3. Hybrid Approaches - Combine structured + semantic, rules + ML
4. Domain Specificity - NTT conventions baked into the system
5. Traceability - Every answer traceable to exact source location

## Technique Sessions

### Technique 1: Mind Mapping

**Central Concept:** Complex Excel Files → DSOL → AI-Powered Q&A

**Branches Identified:**

1. **Document Types (Confirmed):**
   - Excel files (complex tables, merged cells, embedded diagrams)
   - PDFs and misc non-standard docs
   - Cross-sheet Symbol dictionaries (within Excel)

   *Note: 119 files in scope for this phase*

2. **Input Challenges:**
   - Macro-enabled workbooks with hidden logic
   - Vector charts / diagrams
   - Multi-level header tables
   - Long cross-references
   - Embedded images (flowcharts, screen flows)
   - Hidden sequence/trigger semantics
   - Symbol shorthand requiring cross-sheet resolution
   - Non-Excel files (PDF, misc docs)

3. **Question Types (from Q&A dataset):**
   - LOOKUP: "What does code X mean?"
   - CROSS-REF: "Item Y on Sheet Z"
   - MAPPING: "Code A → Result B"
   - SUMMARIZE: "Contents of page 4-5"
   - DEDUCTION: "Which screen shows X?"

4. **Current Failures:**
   - AI can't find right info
   - Symbolic relationships missed
   - Cross-sheet references broken
   - Page/item number lookups fail

5. **Scale (This Phase):** 119 files

**Key Insight:** The core problem is SEMANTIC EXTRACTION that preserves meaning, relationships, and context so AI can reason over it.

---

### Technique 2: Five Whys

**Starting Problem:** AI cannot correctly answer client questions from complex documents

**WHY 1:** Why does AI give wrong answers?
→ **Chunking destroys context** - table headers separate from data, page structure lost, spatial relationships broken

**WHY 2:** Why does chunking destroy context?
→ **Standard chunking treats docs as linear text**, but Excel meaning is encoded in 2D spatial positions and cell relationships that chunking flattens

**WHY 3:** Why is smart chunking hard?
→ Multiple compounding factors:
- Tables aren't labeled (unknown structure types)
- Formats vary wildly (no consistent patterns)
- Semantics are domain-specific (symbols, shorthand)
- Cross-references everywhere (multi-step resolution needed)
- Mixed content types (each needs different strategy)

**WHY 4:** Root cause underneath all problems?
→ **Documents designed for HUMAN visual reading, not machine parsing** - no semantic layer between raw Excel structure and meaning

**WHY 5:** Why don't existing semantic extraction tools work?
→ All existing tools fail because they are:
- Generic, not domain-specific
- Single-pass (can't do multi-pass symbol resolution)
- No cross-document reasoning
- Weak layout detection
- No human-in-the-loop for edge cases

**ROOT CAUSE IDENTIFIED:** Need domain-aware, multi-pass, cross-document, adaptive, teachable semantic extraction system

### Technique 3: First Principles Thinking

**Fundamental Truths:**
1. Input: 119 Excel files with complex structures (tables, merged cells, diagrams, symbols)
2. Output needed: Accurate answers to client questions with source references
3. Core problem: Preserve semantic meaning during extraction so AI can reason correctly

**Assumptions Challenged:**

| Common Assumption | Challenge |
|-------------------|-----------|
| Use vector embeddings | Knowledge graph may be better for tables |
| Chunk documents | Keep tables WHOLE as atomic units |
| One-pass extraction | Multi-pass: symbols → resolve → extract |
| Generic parser | Domain-specific rules for NTT conventions |
| Fully automated | Human-in-loop for teaching new patterns |

**First Principles Solution Components:**
1. Table-aware extraction - Treat tables as 2D structures, not linear text
2. Symbol resolution layer - Pre-process symbol dictionaries, resolve before indexing
3. Relationship preservation - Keep cross-references as explicit links
4. Hybrid retrieval - Structured query (exact lookups) + semantic search (concepts)
5. Source traceability - Every answer includes "Sheet X, Row Y, Column Z"

---

### Technique 4: SCAMPER

**S - Substitute:**
- Substitute chunking with table-unit indexing (whole tables as atomic units)
- Substitute embeddings-only with knowledge graph + embeddings hybrid

**C - Combine:**
- Combine OCR + layout detection for embedded diagrams
- Combine rule-based extraction + LLM understanding
- Combine symbol dictionary + main content in unified knowledge base

**A - Adapt:**
- Adapt database schema design to Excel (cells, ranges, relationships)
- Adapt code documentation tools (AST parsing) to table structure parsing

**M - Modify/Magnify:**
- Magnify cross-reference resolution (make it first-class feature)
- Modify retrieval to return CONTEXT (surrounding cells, headers) not just matches

**P - Put to other uses:**
- Could become general "Complex Excel → Knowledge Base" platform
- Could serve as training data generator for domain-specific LLMs

**E - Eliminate:**
- Eliminate assumption that all docs need same processing
- Eliminate single-format output (support graph + vector + structured)

**R - Reverse:**
- Instead of "document → chunks → index" try "question → what structure needed → extract that"
- Build retrieval BACKWARDS from question types

---

## Idea Categorization

### Immediate Opportunities

_Ideas ready to implement now_

1. **Table boundary detection** - Identify where tables start/end in each sheet
2. **Header extraction** - Capture multi-level headers with hierarchy
3. **Symbol dictionary builder** - Auto-detect sheets that define symbols (○, ×, etc.)
4. **Reference pattern matcher** - Regex for "Sheet X, Item Y" patterns
5. **Merged cell resolver** - Propagate merged cell values to child cells

### Future Innovations

_Ideas requiring development/research_

1. **Hybrid knowledge graph + vector store** - Tables as nodes, relationships as edges
2. **Multi-pass extraction pipeline** - Configurable passes for different content types
3. **Layout-aware chunking** - Chunks that respect table boundaries
4. **Cross-document linker** - Resolve references across 119 files
5. **Domain ontology** - NTT-specific concept hierarchy

### Moonshots

_Ambitious, transformative concepts_

1. **Self-improving extraction** - Learn from Q&A feedback to improve
2. **Visual reasoning** - LLM that "sees" table structure like humans
3. **Auto-schema discovery** - Automatically detect table types and their semantics
4. **Natural language to structured query** - "What does code 07 mean?" → precise cell lookup

### Insights and Learnings

_Key realizations from the session_

1. **The problem isn't "parsing Excel"** - it's semantic extraction that preserves reasoning ability
2. **Tables are 2D databases** - treat them as structured data, not documents
3. **Symbols are a hidden language** - must be resolved BEFORE indexing
4. **Cross-references are relationships** - model them explicitly in knowledge graph
5. **Question types dictate extraction needs** - work backwards from what users ask

## Key Themes Identified

1. **Structure Preservation** - Don't destroy what makes Excel meaningful
2. **Multi-Pass Processing** - Resolve dependencies before final extraction
3. **Hybrid Approaches** - Combine structured + semantic, rules + ML
4. **Domain Specificity** - NTT conventions baked into the system
5. **Traceability** - Every answer traceable to exact source location

## Action Planning

### Top 3 Priority Ideas

#### #1 Priority: Table-Aware Extraction Engine

- **Rationale:** Foundation for everything else - without preserving table structure, nothing works
- **Next steps:** Build table boundary detector → Create header hierarchy extractor → Handle merged cells properly
- **Resources needed:** Python + openpyxl/xlrd, layout detection model

#### #2 Priority: Symbol Resolution Pipeline

- **Rationale:** Many answers require understanding ○/×/symbols - must resolve before indexing
- **Next steps:** Auto-detect symbol dictionary sheets → Build symbol → meaning mapping → Pre-process all files
- **Resources needed:** Pattern matching, sheet classification logic

#### #3 Priority: Hybrid Retrieval System

- **Rationale:** Some questions need exact lookup ("code 07"), others need semantic search ("tell me about...")
- **Next steps:** Knowledge graph for structured relationships → Vector store for semantic → Query router
- **Resources needed:** Graph database (Neo4j or similar), embedding model, query classifier

## Reflection and Follow-up

### What Worked Well

- Mind Mapping revealed the full problem space quickly
- Five Whys drilled to the root cause (semantic extraction, not just parsing)
- Examining the actual Q&A dataset provided concrete examples of failures
- First Principles helped challenge assumptions about chunking/embeddings

### Areas for Further Exploration

- How exactly do multi-level headers work in the NTT files?
- What are all the symbol conventions used?
- How are cross-references formatted (patterns)?
- What percentage of questions are lookup vs semantic?

### Recommended Follow-up Techniques

- **Morphological Analysis** - Systematically explore all parameter combinations for extraction
- **User Journey Mapping** - Map the full flow from question → answer → verification
- **Competitive Analysis** - Deep dive into Unstructured, LlamaParse limitations

### Questions That Emerged

1. Can we use the Q&A dataset as ground truth for testing extraction quality?
2. Should we build a visual tool for humans to correct/teach extraction rules?
3. Is there a "minimum viable extraction" that handles 80% of questions?
4. How do we handle embedded images/diagrams - OCR or skip for now?

### Next Session Planning

- **Suggested topics:** Deep dive into one Excel file to prototype extraction rules
- **Recommended timeframe:** After initial technical spike on table detection
- **Preparation needed:** Sample of 5-10 diverse Excel files representing different patterns

---

_Session facilitated using the BMAD CIS brainstorming framework_