--- title: 'Accept .md uploads and store source to S3' type: 'feature' created: '2026-05-15' status: 'done' baseline_commit: '64acabc8d2d472988421a0ffe3cc2be43fa0d654' context: - '_planning/docs/services/textiq-doc-extraction/architecture.md' --- ## Intent **Problem:** The Airflow-plugin upload endpoint (`POST /api/v1/documents/upload`) rejects markdown files. We need to ingest `.md` source documents into S3 so downstream features can consume them, but markdown does not flow through the extraction DAG. **Approach:** Extend the existing Airflow-plugin upload endpoint at `airflow/plugins/api/upload.py` to accept `.md`. The S3 put, DB insert, and HMAC/tenant scoping are reused as-is. For `.md` only: skip magic-byte and password-protection checks (markdown has no signature and is plaintext), and on insert mark the row terminal (`status="completed"`, `approval_status="approved"`) so the approval-gated DAG is never triggered for markdown. ## Boundaries & Constraints **Always:** - Modify the Airflow plugin module (`airflow/plugins/api/`). The plugin runs in the Airflow api-server process and **must not** import from `src.*`. - Reuse the existing `s3.put_object`, `ensure_bucket`, `validate_s3_path` helpers in `airflow/plugins/api/s3.py`. - Reuse the existing S3 key pattern used by other uploads: `{tenant_id}/{node_id}/{document_id}/{safe_filename}`. Do not invent a new path layout. - Reuse the existing `IMAGE`-side bucket via `get_upload_bucket()` — same bucket as today's uploads. - For `.md` rows: `status="completed"` and `approval_status="approved"` are set at insert time (no DAG trigger, no review needed). For all other extensions: behavior is byte-identical to today (`status="pending"`, `approval_status="pending"`, approval gate triggers DAG). - Validation chain for `.md`: extension → size → UTF-8 decodable. Skip `validate_magic_bytes` and `check_password_protected`. - `sanitize_filename` and `FILENAME_TOO_LONG` checks still apply to `.md`. - Existing metrics, structured logging, response shape, and 202 status code are reused — no new endpoint, no new response field. **Ask First:** - Adding a new endpoint instead of extending the existing upload route. - Changing the response shape of the upload endpoint. - Routing `.md` through the approval workflow (i.e. `approval_status="pending"`) instead of auto-approving. - Triggering a DAG for `.md` uploads. **Never:** - Do not add a markdown extraction DAG, chunking, or Milvus indexing in this spec — out of scope. - Do not touch `src/api/...` (that surface is unused by this ingestion path). - Do not bypass tenant scoping (`validate_s3_path` must still be called). - Do not change behavior for `.xlsx/.xls/.xlsm/.docx/.doc/.docm/.pdf/.pptx/.ppt/.pptm`. ## I/O & Edge-Case Matrix | Scenario | Input / State | Expected Output / Behavior | Error Handling | |---|---|---|---| | Single .md upload happy path | `POST /api/v1/documents/upload` with valid `foo.md`, valid HMAC ctx, size ≤ max | 202 `{document_id, filename, status:"completed"}`; S3 object exists at `{tenant}/{node}/{doc_id}/foo.md`; DB row has `status="completed"`, `approval_status="approved"`; no DAG trigger | N/A | | Unsupported extension | `foo.txt` | 4xx `INVALID_EXTENSION` (existing handler — error message lists `.md`) | Existing handler | | .md over size limit | `foo.md`, size > max | 4xx `FILE_TOO_LARGE` | Existing handler | | .md with invalid UTF-8 | `foo.md` bytes not decodable as UTF-8 | 4xx `INVALID_FORMAT` `"File is not valid UTF-8 markdown"` | Raised by `validate_markdown_utf8` | | Approve a previously uploaded .md | `POST /{doc_id}/approve` for a `.md` row | Existing approve handler runs; because `status != "pending"`, no DAG is triggered (already true today) | None — relies on existing branch | | Non-md formats | Any of the existing extensions | Unchanged behavior: magic-bytes + password checks run, `status="pending"`, `approval_status="pending"`, DAG triggers on approve | Unchanged | ## Code Map - `airflow/plugins/api/validation.py` — add `.md` to `VALID_EXTENSIONS`; add `validate_markdown_utf8(content: bytes)`. - `airflow/plugins/api/upload.py` — in `upload()`: after `validate_extension`, branch on `ext == ".md"`: for markdown, run `validate_markdown_utf8` and skip `validate_magic_bytes` + `check_password_protected` + temp-file write; at DB insert, set `status` and `approval_status` based on the markdown branch. ## Tasks & Acceptance **Execution:** - [ ] `airflow/plugins/api/validation.py` -- Add `".md"` to `VALID_EXTENSIONS`; add `validate_markdown_utf8(content: bytes) -> None` that raises `ValueError("File is not valid UTF-8 markdown")` on `UnicodeDecodeError`. Do NOT change `validate_magic_bytes` -- it stays binary-only; the caller decides whether to invoke it. - [ ] `airflow/plugins/api/upload.py` -- In `upload()` after `validate_extension(filename)` and the size check, capture `ext = Path(filename).suffix.lower()` and set `is_markdown = (ext == ".md")`. If markdown: call `validate_markdown_utf8(content)` instead of `validate_magic_bytes(content)`, skip the temp-file write and `check_password_protected` step. Continue through `sanitize_filename`, S3 key build, `validate_s3_path`, `ensure_bucket`, `put_object`. At the DB insert, when `is_markdown` set `status="completed"` and `approval_status="approved"`; otherwise keep the current `"pending"`/`"pending"` values. Final response payload: when markdown, return `"status": "completed"` (instead of `"pending"`) so the client sees the terminal state. **Acceptance Criteria:** - Given a valid `.md` ≤ max size and a valid HMAC context, when `POST /api/v1/documents/upload` is called, then the response is 202 with `status="completed"`, an S3 object exists at `{tenant}/{node}/{document_id}/{safe_filename}`, and the DB row has `status="completed"` and `approval_status="approved"`. - Given a `.md` upload with bytes that are not valid UTF-8, when validation runs, then the response is the existing `INVALID_FORMAT` error shape with a message mentioning UTF-8 markdown, and no DB row or S3 object is created. - Given a `.md` upload, when processing completes, then no Airflow DAG is triggered (the existing approve-side branch only triggers DAGs when `status == "pending"`, and `.md` rows are already `completed`). - Given an upload of any pre-existing format (`.xlsx/.docx/.pdf/...`), when processing completes, then behavior is byte-identical to today: magic-byte check runs, password-protection check runs, `status="pending"`, `approval_status="pending"`, DAG triggers only on approval. ## Verification **Commands:** - `python -c "from airflow.plugins.api.validation import validate_extension, validate_markdown_utf8, VALID_EXTENSIONS; assert '.md' in VALID_EXTENSIONS; validate_extension('a.md'); validate_markdown_utf8(b'# hi\n'); print('ok')"` -- expected: prints `ok`. - `python -c "from airflow.plugins.api.validation import validate_markdown_utf8; import pytest; \ raised=False try: validate_markdown_utf8(b'\xff\xfe not utf8') except ValueError: raised=True assert raised, 'expected ValueError'; print('ok')"` -- expected: prints `ok`. **Manual checks (no CLI):** - Send a multipart `POST /api/v1/documents/upload` with a small valid `.md` (test client or `curl`). Confirm 202 response with `status="completed"`. Verify the SeaweedFS object exists under `{tenant}/{node}/{document_id}/{filename}` and the `documents` row has `status='completed'` and `approval_status='approved'`. (No automated test harness exists for the Airflow plugin module.) - Send the same `.md` upload again but corrupt the bytes (e.g. `head -c 4 /dev/urandom`). Expect 4xx `INVALID_FORMAT` and no DB row. - Send a `.xlsx` upload and confirm behavior is unchanged: `status="pending"`, `approval_status="pending"`, no DAG trigger until approved.