# Infrastructure Estimate: Wiki Generation for 30M Lines of Code

**Date:** 2026-03-31
**Target:** `/home/code/customer_source` (Kopt enterprise Java monorepo)
**Pipeline:** RepoPilot Scanner + CodeWiki Generator

---

## 1. Codebase Profile

| Metric | Value |
|--------|-------|
| Total files | 85,090 |
| Java source files | 59,002 |
| Java lines of code | 26.6M |
| Total LoC (all file types) | 30.6M |
| Disk size (excl. .git) | 1.4 GB |
| Unique Java packages | 3,284 |
| Sub-projects | 16 |
| Files > 500KB | 111 |
| Files > 1MB | 42 (largest 2.6MB) |

### Sub-project Breakdown

| Sub-project | Java files | Java LoC |
|-------------|-----------|----------|
| koptWebF | 1,133 | 1,147,183 |
| koptWebA | 895 | 1,103,351 |
| koptWebB | 6,952 | 973,322 |
| koptModel | 18,132 | 537,802 |
| koptWebFrontCommon | 707 | 473,900 |
| koptWebR | 245 | 329,878 |
| koptBp | 24,915 | 302,501 |
| koptCommon | 142 | 128,883 |
| koptBatch | 5,410 | 98,052 |
| koptKVBatch | 63 | 40,579 |
| koptWebS | 296 | 31,346 |
| koptWebRest | 108 | 14,217 |
| koptWebE | 4 | 4,203 |

### Top Packages by File Count

| Package | Files |
|---------|-------|
| `com.fujitsu.futurity.bp.custom.constant` | 8,191 |
| `com.fujitsu.futurity.bp.custom.mapping` | 6,441 |
| `eo.ejb.cbs.cbsmsg` | 5,208 |
| `eo.ejb.check.item` | 3,135 |
| `eo.ejb.cbm.entity` | 2,887 |
| `eo.business.service` | 2,459 |

---

## 2. Pipeline Architecture

The wiki generation pipeline has four sequential macro-phases:

```
Phase 1: Scan & Index (CPU + I/O bound)
  1a. File discovery         — recursive directory walk
  1b. Parse & extract        — tree-sitter AST + regex fallback per file
  1c. Relationship resolution — cross-file edge linking
  1d. Index serialization    — msgpack binary to disk

Phase 2: Module Tree Building (CPU bound)
  2a. Package collection     — graph queries
  2b. Tree construction      — hierarchical grouping
  2c. Large module splitting — CLASS_GROUP_THRESHOLD=8

Phase 3: Wiki Generation (LLM API bound) ← BOTTLENECK
  3a. Leaf module docs       — 1 LLM agent call per leaf
  3b. Parent module docs     — 1 LLM agent call per parent (depth-ordered)
  3c. Root overview          — 1 LLM agent call
  3d. Guide pages            — 1 LLM agent call per guide

Phase 4: Finalization (CPU + I/O)
  4a. Mermaid validation
  4b. Navigation tree build
  4c. Structured artifact write
```

---

## 3. Phase-by-Phase Time Complexity

### 3.1 Phase 1a: File Discovery

**Algorithm:** `discoverJavaFiles()` — synchronous recursive `readdirSync()` + `statSync()`

```
Complexity: O(F + D)
  F = 59,002 files
  D = ~5,000 directories
```

| Operation | Count | Unit cost | Total |
|-----------|-------|-----------|-------|
| `readdirSync()` | ~5,000 | ~0.1ms | 0.5s |
| `statSync()` per file | 59,002 | ~0.05ms | 3.0s |
| `.sort()` on path strings | 59,002 | O(F log F) | 0.5s |
| **Subtotal** | | | **~4s** |

**Memory:** ~10MB (string array of file paths)
**Parallelizable:** No (single synchronous recursive walk)
**I/O pattern:** ~64K filesystem syscalls, entirely metadata (no file content read)

### 3.2 Phase 1b: Parse & Extract

**Algorithm:** Sequential loop over all Java files. Per file:

```
1. readFileSync(path)                    O(file_size)
2. parseJavaFile() via tree-sitter       O(file_size)
3. extractFromFile() — AST walk          O(AST_nodes)
4. regexExtractFromFile() — fallback     O(file_size × regex_count)
5. graph.addNode() / graph.addEdge()     O(1) amortized
```

Tree-sitter parsing is the dominant cost. Parse time scales linearly with file size but with significant constant factors for Java grammar complexity.

| File size category | Count | Avg parse time | Subtotal |
|-------------------|-------|----------------|----------|
| Normal (< 50KB) | 57,845 | 3ms | 174s |
| Medium (50KB - 500KB) | 1,046 | 30ms | 31s |
| Large (500KB - 1MB) | 69 | 150ms | 10s |
| Huge (> 1MB) | 42 | 500ms | 21s |
| **Total** | **59,002** | | **~236s (3.9 min)** |

**Memory during scanning:**

The knowledge graph grows monotonically as files are processed:

| Structure | Entries (final) | Avg bytes each | Total |
|-----------|----------------|---------------|-------|
| `_nodes` Map | ~360K | ~500B (with snippets) | ~180MB |
| `_outEdges` Map | ~700K edges | ~200B | ~140MB |
| `_inEdges` Map | ~700K edges | ~200B | ~140MB |
| **Total graph** | | | **~460MB** |

Transient per-file memory (released after each file): source buffer (~22KB avg, up to 2.6MB) + tree-sitter AST (~3-5x source size).

**Parallelizable:** YES — files are independent units. A `worker_threads` pool could split the file list into N batches, each building a partial graph, then merge into the main graph. Expected speedup: near-linear up to I/O saturation (~4 workers on SSD).

**Impact of `includeSnippets: false`:**

With snippets disabled, class nodes drop ~2KB each and method nodes drop ~1KB each:
- ~59K classes × 2KB = ~118MB saved
- ~300K methods × 1KB = ~300MB saved
- **Graph drops from ~460MB to ~100MB**

Snippets are redundant for wiki generation (agents read source files via tools).

### 3.3 Phase 1c: Relationship Resolution

**Algorithm:** `resolveRelationships()` — builds lookup maps, resolves placeholder edges

```
1. Build classLookup Map<name, nodeId>:     O(N) scan, N=360K
2. Build methodLookup Map<name, nodeId[]>:  O(N) scan
3. Resolve extends/implements:              O(C), C=59K classes
4. Resolve import edges:                    O(F × I_avg), F=59K, I_avg≈3
5. Resolve method call edges:               O(M × calls_avg), M=300K, calls_avg≈2
   (capped at 5 targets per method name)
6. Build package dependency edges:          O(F × I_avg)
```

| Step | Iterations | Time |
|------|-----------|------|
| Build lookups (2 full scans) | 2 × 360K | ~100ms |
| Resolve extends | 59K × O(1) | ~20ms |
| Resolve imports | ~180K lookups | ~30ms |
| Resolve method calls | ~600K lookups, capped | ~200ms |
| Package dependencies | ~180K | ~30ms |
| **Total** | | **~0.5-1s** |

**Memory:** classLookup (~8MB) + methodLookup (~30MB) = ~40MB additional (temporary)

### 3.4 Phase 1d: Index Serialization

**Algorithm:** Graph → plain JS objects → msgpack → disk

```
1. graph.toSerializable():  O(N + E) — copies all node/edge data into arrays
2. msgpack.encode():        O(output_bytes)
3. writeFileSync():         O(output_bytes)
```

| Step | Data size | Time |
|------|-----------|------|
| toSerializable() — copy 360K nodes + 700K edges | ~200MB objects | ~2s |
| msgpack encode | 200MB → ~120MB binary | ~3s |
| Write to SSD | 120MB | ~1s |
| **Total** | | **~6s** |

**Peak memory during serialization:** This is the **highest memory moment** in the entire pipeline:
- Original graph in Maps: ~460MB
- Serialized copy (plain objects): ~200MB
- msgpack output buffer: ~120MB
- **Peak: ~780MB** (with snippets) / **~250MB** (without snippets)

### 3.5 Phase 2: Module Tree Building

**Algorithm:** `buildModuleTree()` — collect packages, group, build hierarchy, split large leaves

```
1. graph.getNodesByType(PACKAGE):   O(N) — full scan of 360K nodes
2. graph.getNodesByType(FILE):      O(N)
3. graph.getNodesByType(CLASS):     O(N)
4. graph.getNodesByType(INTERFACE): O(N)
5. graph.getNodesByType(ENUM):      O(N)
6. Tree construction:               O(P × depth), P=3,284 packages
7. splitLargeModules (DFS):         O(C_total × patterns), C_total=59K
```

`getNodesByType()` performs a linear scan of the entire `_nodes` Map each time (no secondary index). Five calls = 5 × 360K = **1.8M iterations** just for type filtering.

| Step | Complexity | Time |
|------|-----------|------|
| 5× getNodesByType full scans | 5 × O(360K) | ~3s |
| Tree construction (3,284 packages) | O(P × depth_avg) | <0.1s |
| splitLargeModules (regex on 59K classes) | O(C × 22 patterns) | ~0.5s |
| **Total** | | **~4s** |

**Expected module tree dimensions:**

| Module level | Count |
|-------------|-------|
| Root | 1 |
| Top-level packages | ~16 |
| Subpackages (intermediate) | ~500 |
| **Leaf modules (after splitting)** | **~5,400** |
| **Parent modules (non-leaf, non-root)** | **~500** |

Splitting arithmetic: packages with >8 classes subdivide by naming pattern (Handler, Service, Bean, Dto, Dao, Controller, etc. — 22 patterns). The largest package (8,191 files) could split into 15-25 class groups.

### 3.6 Phase 3: Wiki Generation (BOTTLENECK)

#### Per-Module Call Anatomy

Each wiki module generation spawns a Rust agent subprocess that makes LLM API calls:

```
generateModuleDoc(module):
  1. buildModuleEvidence()          — 50-200ms CPU
     ├─ getClassEntries()           — O(N) full graph scan
     ├─ getMethodEntries()          — O(N) full graph scan
     ├─ buildStructureDiagram()     — string building from evidence
     └─ buildDependencyDiagram()    — graph traversal
  2. buildLeafPrompt()              — <10ms (string concatenation)
  3. runSubagent(prompt)            — 15-45s (LLM API, DOMINANT COST)
     ├─ spawn Rust binary           — ~200ms process startup
     ├─ LLM round 1 (prompt)        — 5-15s (TTFT + generation)
     ├─ Tool call: read_file         — ~500ms (disk I/O)
     ├─ LLM round 2 (tool result)   — 3-8s
     ├─ Tool call: write_file        — ~100ms
     └─ LLM round 3 (finalize)      — 3-8s
  4. writeFinalDocument()           — <50ms (disk write)
  5. buildModuleStructuredPages()   — <100ms (JSON construction)
```

**Average time per module: ~25 seconds** (dominated by 3 LLM round-trips)

#### Evidence Building CPU Overhead

`buildModuleEvidence()` calls `getClassEntries()` and `getMethodEntries()`, each invoking `graph.getNodesByType()` — a full O(N) scan of 360K nodes.

For 5,400 leaf modules executed sequentially:

```
5,400 modules × 2 getNodesByType scans × 360K nodes = 3.9 billion iterations
Estimated CPU time: ~50 minutes (pure overhead)
```

This is an O(N) operation that should be O(1) with a type index.

#### Parallelism Constraints

The wiki generation has three parallelism domains with different constraints:

**Domain A — Leaf modules (5,400 modules):**
No dependencies between leaves. Fully parallelizable up to concurrency limit C.

```
T_leaves = ceil(5,400 / C) × 25s
```

**Domain B — Parent modules (500 modules, depth-ordered):**
Parent documentation depends on child documentation being complete. Parents at the **same depth level** can run in parallel, but depth levels must execute sequentially (deepest first).

Estimated depth distribution:
```
Depth 5+: ~54 parents   → ceil(54/C) batches
Depth 4:  ~150 parents  → ceil(150/C) batches
Depth 3:  ~200 parents  → ceil(200/C) batches
Depth 2:  ~80 parents   → ceil(80/C) batches
Depth 1:  ~16 parents   → ceil(16/C) batches
```

```
T_parents = sum over depths d: ceil(count_d / C) × 25s
```

**Domain C — Guides (14 pages):**
Getting Started + 5 cross-cutting + 8 deep-dive pages. No dependencies. Fully parallelizable.

```
T_guides = ceil(14 / C) × 25s
```

**Domain D — Root overview (1 page):**
Depends on all modules being complete. Always sequential.

```
T_root = 25s
```

**Total generation time:**

```
T_gen = T_leaves + T_parents + T_root + T_guides
```

---

## 4. Concurrency Scaling Analysis

### 4.1 Wall Time by Concurrency Level

| C | T_leaves | T_parents | T_root | T_guides | **T_gen** | T_scan | **T_total** |
|---|---------|-----------|--------|----------|-----------|--------|-------------|
| 1 | 37.5h | 3.5h | 25s | 5.8min | **41.0h** | 4.2min | **41.1h** |
| 4 | 9.4h | 52min | 25s | 1.5min | **10.3h** | 4.2min | **10.4h** |
| 8 | 4.7h | 26min | 25s | 45s | **5.2h** | 4.2min | **5.2h** |
| 16 | 2.3h | 13min | 25s | 25s | **2.6h** | 4.2min | **2.7h** |
| 32 | 1.2h | 7min | 25s | 12s | **1.3h** | 4.2min | **1.4h** |
| 64 | 35min | 4min | 25s | 6s | **40min** | 4.2min | **44min** |

### 4.2 Memory by Concurrency Level

```
Base memory = Node.js process (150MB) + KnowledgeGraph (460MB) + Module tree (5MB) = 615MB
Per agent subprocess = ~50MB (Rust binary + stdio buffers)
Peak during index save = 615MB + serialization overhead (320MB) = 935MB
Peak during generation = 615MB + (C × 50MB)
```

| C | Generation RAM | Index save peak | **Overall peak** |
|---|---------------|----------------|-----------------|
| 1 | 665MB | 935MB | **935MB** |
| 4 | 815MB | 935MB | **935MB** |
| 8 | 1,015MB | 935MB | **1,015MB** |
| 16 | 1,415MB | 935MB | **1,415MB** |
| 32 | 2,215MB | 935MB | **2,215MB** |
| 64 | 3,815MB | 935MB | **3,815MB** |

With `includeSnippets: false`: subtract ~360MB from all values.

### 4.3 CPU Utilization by Concurrency Level

```
Scanning phase:   1 core saturated (tree-sitter CPU-bound)
Evidence building: 1 core (Node.js event loop, blocked on O(N) graph scans)
Agent subprocesses: ~0.1 core each (mostly idle, waiting on LLM API)
```

| C | Active cores | Utilization on 20-thread machine |
|---|-------------|--------------------------------|
| 1 | 2 | 10% |
| 4 | 2-3 | 12-15% |
| 8 | 2-4 | 15-20% |
| 16 | 3-5 | 15-25% |
| 32 | 4-8 | 20-40% |
| 64 | 6-12 | 30-60% |

CPU is never the bottleneck. Network latency to the LLM API dominates.

### 4.4 Network / API Rate Limits

```
Per module: ~3 LLM round-trips
Per round-trip: ~10K input tokens + ~2K output tokens
Per module total: ~36K tokens
Total across all modules: 5,900 × 36K = ~212M tokens
```

| C | Requests/min | Tokens/min (input) | Risk |
|---|--------------|--------------------|------|
| 1 | ~7 | ~250K | None |
| 4 | ~29 | ~1M | None |
| 8 | ~58 | ~2M | Low |
| 16 | ~115 | ~4M | Check tier limits |
| 32 | ~230 | ~8M | May hit rate limits |
| 64 | ~460 | ~16M | Likely throttled |

### 4.5 Disk I/O

| Phase | Read pattern | Write pattern |
|-------|-------------|---------------|
| Discovery | 64K stat() metadata calls | — |
| Scanning | 1.4GB sequential file reads | — |
| Index save | — | ~120MB single write |
| Generation (per agent) | 2-5 file reads (~200KB) per iteration | 1 markdown file (~5KB) |
| Generation (C=16) | ~50 concurrent reads | ~3 concurrent writes |

**SSD:** Not a bottleneck at any concurrency level.
**HDD:** Could bottleneck above C=8 during agent file reads.

---

## 5. LLM Token Cost Estimate

Total token consumption across all 5,900 module/guide/overview calls:

| Metric | Estimate |
|--------|----------|
| Total input tokens | ~170M |
| Total output tokens | ~42M |
| **Total tokens** | **~212M** |

| Model | Input cost | Output cost | **Total cost** |
|-------|-----------|-------------|---------------|
| GPT-4o-mini | $0.15/1M × 170M = $25.50 | $0.60/1M × 42M = $25.20 | **~$51** |
| GPT-5-mini (est.) | ~$0.15/1M × 170M = $25.50 | ~$0.60/1M × 42M = $25.20 | **~$51** |
| Claude Haiku 4.5 | $0.80/1M × 170M = $136 | $4/1M × 42M = $168 | **~$304** |
| Claude Sonnet 4.6 | $3/1M × 170M = $510 | $15/1M × 42M = $630 | **~$1,140** |
| GPT-4o | $2.50/1M × 170M = $425 | $10/1M × 42M = $420 | **~$845** |

---

## 6. Critical Bug: Parallelism Regression

### Problem

During the refactoring of `src/server/wiki-generator.ts` into `src/server/wiki/` sub-modules, all three parallel execution calls were replaced with sequential `for` loops.

**Old code** (`wiki-generator.ts`, pre-refactor):
```typescript
// Leaf docs — parallel
await runParallelSubagents(
  leaves.map((leaf) => async () => { ... }),
  concurrency,     // from serverConfig.wikiAgent.concurrency
);

// Parent docs — parallel within each depth level
for (const depth of depthLevels) {
  const group = parentsByDepth.get(depth)!;
  await runParallelSubagents(
    group.map((parent) => async () => { ... }),
    concurrency,
  );
}

// Guides — parallel
await runParallelSubagents(guideTasks, concurrency);
```

**New code** (`src/server/wiki/generator.ts`, post-refactor):
```typescript
// Leaf docs — SEQUENTIAL
for (const leaf of leaves) {
  await generateModuleDoc(leaf, ...);
}

// Parent docs — SEQUENTIAL
for (const parent of parents) {
  await synthesizeParentDoc(parent, ...);
}

// Guides — SEQUENTIAL
for (const pattern of crossCuttingPatterns) {
  await generateCrossCuttingPage(pattern, ...);
}
```

### Impact

The `WIKI_AGENT_CONCURRENCY` environment variable is now **dead code**. All LLM calls execute one at a time.

For this 30M LoC codebase:
- **With parallelism (C=16):** ~2.7 hours
- **Without parallelism (C=1):** ~41 hours
- **Performance regression: 15x slower**

### Fix Required

Restore `runParallelSubagents()` calls in `src/server/wiki/generator.ts` for all three phases. Additionally, parent module parallelism requires grouping by depth level (deepest-first) so children are documented before their parents.

---

## 7. Performance Optimization Opportunities

### 7.1 Restore Parallelism (Critical, 0 infra cost)

**Impact:** Reduces generation time from 41h → 2.7h at C=16.
**Effort:** Modify `src/server/wiki/generator.ts` to use `runParallelSubagents()`.
**Risk:** Low — the worker pool function already exists and was battle-tested.

### 7.2 Add Type Index to KnowledgeGraph (High impact)

**Problem:** `getNodesByType()` in `src/lib/repopilot/graph/knowledge-graph.ts:90` performs a full O(N) scan of the entire `_nodes` Map (360K entries) on every call. This function is called:
- 5 times during module tree building
- 2 times per module during evidence building (5,400 × 2 = 10,800 calls)

Total: **10,805 full scans × 360K nodes = 3.9 billion iterations (~50 min CPU)**

**Fix:** Add a secondary index `_nodesByType: Map<NodeType, Set<NodeId>>` maintained during `addNode()`. Reduces each call from O(N) to O(result_size).

**Impact:** Saves ~50 minutes of CPU time; reduces evidence building from 200ms to <5ms per module.

### 7.3 Disable Snippets for Large Repos

**Problem:** `includeSnippets: true` stores 2KB per class + 1KB per method in graph nodes. With 59K classes + 300K methods = ~418MB of snippet data. These snippets are redundant because agents read source files directly via tools.

**Fix:** Pass `{ includeSnippets: false }` to `RepoScanner` when source exceeds a threshold (e.g., >10K files).

**Impact:** Reduces graph memory from ~460MB to ~100MB. Reduces index file from ~120MB to ~30MB. Reduces serialization peak from ~780MB to ~200MB.

### 7.4 Parallelize File Scanning

**Problem:** The file parse loop is sequential — one file at a time on a single thread.

**Fix:** Use `worker_threads` to split the 59K files into N batches. Each worker builds a partial `KnowledgeGraph`, main thread merges them.

**Impact:** Reduces scan time from ~4 min to ~1 min with 4 workers. Modest benefit relative to the 2.7h generation phase.

### 7.5 Batch Small Modules

**Problem:** Many leaf modules have ≤3 classes. Each still requires a full Rust subprocess spawn + 3 LLM round-trips (~25s).

**Fix:** Batch adjacent small modules (≤5 classes total) into a single agent call with combined evidence. One prompt generates docs for 2-3 tiny modules.

**Impact:** Could reduce leaf module calls from ~5,400 to ~3,000 (saving ~1h at C=16).

---

## 8. Trade-off Summary

| Configuration | Wall time | Peak RAM | API cost | Risk level |
|--------------|-----------|----------|----------|------------|
| C=1 (current bug) | ~41h | 935MB | $51 | None, but impractical |
| C=4 | ~10.4h | 935MB | $51 | Low |
| C=8 | ~5.2h | 1.0GB | $51 | Low |
| **C=16 (recommended)** | **~2.7h** | **1.4GB** | **$51** | **Low** |
| C=32 | ~1.4h | 2.2GB | $51 | Medium (rate limits) |
| C=64 | ~44min | 3.8GB | $51 | High (rate limits) |

Cost is model-dependent — $51 estimate is for GPT-4o-mini / GPT-5-mini class models.

---

## 9. Recommended Infrastructure

### For This Specific Run

| Resource | Minimum | Recommended |
|----------|---------|-------------|
| CPU | 4 cores | 8-16 cores |
| RAM | 4GB | 8GB (with snippets disabled) |
| RAM (snippets enabled) | 4GB | 16GB |
| Disk (SSD) | 10GB free | 20GB free |
| Concurrency (C) | 4 | **16** |
| LLM API tier | Standard | Tier with ≥200 RPM |

### Current Machine Assessment

| Spec | Value | Verdict |
|------|-------|---------|
| CPU | Intel i5-14600KF, 20 threads | Sufficient for C=32 |
| RAM | 32GB (22GB available) | Sufficient even with snippets at C=32 |
| Disk | Needs verification | Need 10+ GB free on SSD |

**This machine can run the full pipeline at C=16 in ~2.7 hours with no infrastructure changes** — once the parallelism bug is fixed.

### Environment Variables

```env
WIKI_AGENT_CONCURRENCY=16
WIKI_AGENT_MODEL=gpt-5-mini
WIKI_AGENT_MAX_TURNS=24
```

---

## 10. Pre-Run Checklist

1. **Fix parallelism regression** in `src/server/wiki/generator.ts`
2. Verify LLM API key has sufficient rate limits for 115 RPM sustained
3. Ensure 10+ GB free disk space on the target volume
4. Consider setting `includeSnippets: false` to halve memory usage
5. Set `WIKI_AGENT_CONCURRENCY=16` in `.env`
6. Monitor with `htop` during the run — watch for RSS exceeding available RAM
7. Expect ~5,400 `.md` files + index artifacts in `.codewiki/` output directory