---
title: 'Accept .md uploads and store source to S3'
type: 'feature'
created: '2026-05-15'
status: 'done'
baseline_commit: '64acabc8d2d472988421a0ffe3cc2be43fa0d654'
context:
  - '_planning/docs/services/textiq-doc-extraction/architecture.md'
---

<frozen-after-approval reason="human-owned intent — do not modify unless human renegotiates">

## Intent

**Problem:** The Airflow-plugin upload endpoint (`POST /api/v1/documents/upload`) rejects markdown files. We need to ingest `.md` source documents into S3 so downstream features can consume them, but markdown does not flow through the extraction DAG.

**Approach:** Extend the existing Airflow-plugin upload endpoint at `airflow/plugins/api/upload.py` to accept `.md`. The S3 put, DB insert, and HMAC/tenant scoping are reused as-is. For `.md` only: skip magic-byte and password-protection checks (markdown has no signature and is plaintext), and on insert mark the row terminal (`status="completed"`, `approval_status="approved"`) so the approval-gated DAG is never triggered for markdown.

## Boundaries & Constraints

**Always:**
- Modify the Airflow plugin module (`airflow/plugins/api/`). The plugin runs in the Airflow api-server process and **must not** import from `src.*`.
- Reuse the existing `s3.put_object`, `ensure_bucket`, `validate_s3_path` helpers in `airflow/plugins/api/s3.py`.
- Reuse the existing S3 key pattern used by other uploads: `{tenant_id}/{node_id}/{document_id}/{safe_filename}`. Do not invent a new path layout.
- Reuse the existing `IMAGE`-side bucket via `get_upload_bucket()` — same bucket as today's uploads.
- For `.md` rows: `status="completed"` and `approval_status="approved"` are set at insert time (no DAG trigger, no review needed). For all other extensions: behavior is byte-identical to today (`status="pending"`, `approval_status="pending"`, approval gate triggers DAG).
- Validation chain for `.md`: extension → size → UTF-8 decodable. Skip `validate_magic_bytes` and `check_password_protected`.
- `sanitize_filename` and `FILENAME_TOO_LONG` checks still apply to `.md`.
- Existing metrics, structured logging, response shape, and 202 status code are reused — no new endpoint, no new response field.

**Ask First:**
- Adding a new endpoint instead of extending the existing upload route.
- Changing the response shape of the upload endpoint.
- Routing `.md` through the approval workflow (i.e. `approval_status="pending"`) instead of auto-approving.
- Triggering a DAG for `.md` uploads.

**Never:**
- Do not add a markdown extraction DAG, chunking, or Milvus indexing in this spec — out of scope.
- Do not touch `src/api/...` (that surface is unused by this ingestion path).
- Do not bypass tenant scoping (`validate_s3_path` must still be called).
- Do not change behavior for `.xlsx/.xls/.xlsm/.docx/.doc/.docm/.pdf/.pptx/.ppt/.pptm`.

## I/O & Edge-Case Matrix

| Scenario | Input / State | Expected Output / Behavior | Error Handling |
|---|---|---|---|
| Single .md upload happy path | `POST /api/v1/documents/upload` with valid `foo.md`, valid HMAC ctx, size ≤ max | 202 `{document_id, filename, status:"completed"}`; S3 object exists at `{tenant}/{node}/{doc_id}/foo.md`; DB row has `status="completed"`, `approval_status="approved"`; no DAG trigger | N/A |
| Unsupported extension | `foo.txt` | 4xx `INVALID_EXTENSION` (existing handler — error message lists `.md`) | Existing handler |
| .md over size limit | `foo.md`, size > max | 4xx `FILE_TOO_LARGE` | Existing handler |
| .md with invalid UTF-8 | `foo.md` bytes not decodable as UTF-8 | 4xx `INVALID_FORMAT` `"File is not valid UTF-8 markdown"` | Raised by `validate_markdown_utf8` |
| Approve a previously uploaded .md | `POST /{doc_id}/approve` for a `.md` row | Existing approve handler runs; because `status != "pending"`, no DAG is triggered (already true today) | None — relies on existing branch |
| Non-md formats | Any of the existing extensions | Unchanged behavior: magic-bytes + password checks run, `status="pending"`, `approval_status="pending"`, DAG triggers on approve | Unchanged |

</frozen-after-approval>

## Code Map

- `airflow/plugins/api/validation.py` — add `.md` to `VALID_EXTENSIONS`; add `validate_markdown_utf8(content: bytes)`.
- `airflow/plugins/api/upload.py` — in `upload()`: after `validate_extension`, branch on `ext == ".md"`: for markdown, run `validate_markdown_utf8` and skip `validate_magic_bytes` + `check_password_protected` + temp-file write; at DB insert, set `status` and `approval_status` based on the markdown branch.

## Tasks & Acceptance

**Execution:**
- [ ] `airflow/plugins/api/validation.py` -- Add `".md"` to `VALID_EXTENSIONS`; add `validate_markdown_utf8(content: bytes) -> None` that raises `ValueError("File is not valid UTF-8 markdown")` on `UnicodeDecodeError`. Do NOT change `validate_magic_bytes` -- it stays binary-only; the caller decides whether to invoke it.
- [ ] `airflow/plugins/api/upload.py` -- In `upload()` after `validate_extension(filename)` and the size check, capture `ext = Path(filename).suffix.lower()` and set `is_markdown = (ext == ".md")`. If markdown: call `validate_markdown_utf8(content)` instead of `validate_magic_bytes(content)`, skip the temp-file write and `check_password_protected` step. Continue through `sanitize_filename`, S3 key build, `validate_s3_path`, `ensure_bucket`, `put_object`. At the DB insert, when `is_markdown` set `status="completed"` and `approval_status="approved"`; otherwise keep the current `"pending"`/`"pending"` values. Final response payload: when markdown, return `"status": "completed"` (instead of `"pending"`) so the client sees the terminal state.

**Acceptance Criteria:**
- Given a valid `.md` ≤ max size and a valid HMAC context, when `POST /api/v1/documents/upload` is called, then the response is 202 with `status="completed"`, an S3 object exists at `{tenant}/{node}/{document_id}/{safe_filename}`, and the DB row has `status="completed"` and `approval_status="approved"`.
- Given a `.md` upload with bytes that are not valid UTF-8, when validation runs, then the response is the existing `INVALID_FORMAT` error shape with a message mentioning UTF-8 markdown, and no DB row or S3 object is created.
- Given a `.md` upload, when processing completes, then no Airflow DAG is triggered (the existing approve-side branch only triggers DAGs when `status == "pending"`, and `.md` rows are already `completed`).
- Given an upload of any pre-existing format (`.xlsx/.docx/.pdf/...`), when processing completes, then behavior is byte-identical to today: magic-byte check runs, password-protection check runs, `status="pending"`, `approval_status="pending"`, DAG triggers only on approval.

## Verification

**Commands:**
- `python -c "from airflow.plugins.api.validation import validate_extension, validate_markdown_utf8, VALID_EXTENSIONS; assert '.md' in VALID_EXTENSIONS; validate_extension('a.md'); validate_markdown_utf8(b'# hi\n'); print('ok')"` -- expected: prints `ok`.
- `python -c "from airflow.plugins.api.validation import validate_markdown_utf8; import pytest; \
  raised=False
  try: validate_markdown_utf8(b'\xff\xfe not utf8')
  except ValueError: raised=True
  assert raised, 'expected ValueError'; print('ok')"` -- expected: prints `ok`.

**Manual checks (no CLI):**
- Send a multipart `POST /api/v1/documents/upload` with a small valid `.md` (test client or `curl`). Confirm 202 response with `status="completed"`. Verify the SeaweedFS object exists under `{tenant}/{node}/{document_id}/{filename}` and the `documents` row has `status='completed'` and `approval_status='approved'`. (No automated test harness exists for the Airflow plugin module.)
- Send the same `.md` upload again but corrupt the bytes (e.g. `head -c 4 /dev/urandom`). Expect 4xx `INVALID_FORMAT` and no DB row.
- Send a `.xlsx` upload and confirm behavior is unchanged: `status="pending"`, `approval_status="pending"`, no DAG trigger until approved.
