From 1bf717a54d4413e0e5cfe5e2b7ff008489b97c04 Mon Sep 17 00:00:00 2001 From: lyingbug Date: Fri, 25 Sep 2026 13:26:16 +0800 Subject: [PATCH] feat(chat): open cited documents at the cited passage (#3700) * feat(chat): open cited documents at the cited passage Clicking a citation used to show only the parsed chunk. It now opens the original file in the references panel, jumps to the cited place and highlights it, narrowed to the sentence the citation supports. Parsing records where each chunk sits in the original file (chunks.source_locators): PDF page and paragraph boxes from the docreader, MinerU and PaddleOCR-VL layouts; Word paragraphs, slides, sheet rows and EPUB chapters aligned in Go against any parser engine's Markdown; text offsets for Markdown/TXT; segment times for audio. Pipeline text rewrites are followed with a line-diff remapper. Scanned PDF pages get their OCR text as quotes at search time so a sentence can pick its page. The frontend renders PDFs with pdf.js and every other previewable type with the existing preview, then highlights with the CSS Custom Highlight API in a highlighter yellow that stays visible against the brand green. * fix(docreader): serialize source blocks outside main so tests import it test_docx_tables stubs docreader.parser in sys.modules, so importing docreader.main from a later test failed in the full unittest discover run. The wire conversion now lives in docreader.source_wire, which needs only the proto. * fix(chat): tighten source locators from review - Remap rewritten-line-for-line stretches by line so scanned PDF pages no longer drift onto the wrong page. - Keep the source preview mounted for citations in the file already open. - Wait for in-flight PDF page renders before looking for the text layer. - Count image rune offsets in one pass; stream xlsx rows up to maxUnits. - Start text locators after a UTF-8 BOM, as viewers decode without it. - Only re-query image children for scanned-page results. - Seek audio citations without autoplay and drop stale seek listeners. - Fix the SQLite migration's cross-reference to versioned/000113. * fix(migrations): renumber chunk source locators after main's chunk_images main took versioned 000113 and sqlite 000032 for chunk_images, so the source_locators column moves to versioned 000114 and sqlite 000033. --- docreader/main.py | 3 + docreader/models/document.py | 10 + docreader/parser/excel_parser.py | 28 +- docreader/parser/pdf_parser.py | 162 ++++- docreader/parser/source_locator.py | 293 ++++++++ docreader/proto/docreader.pb.go | 200 ++++-- docreader/proto/docreader.proto | 13 + docreader/proto/docreader_grpc.pb.go | 2 +- docreader/proto/docreader_pb2.py | 44 +- docreader/proto/docreader_pb2.pyi | 22 +- docreader/source_wire.py | 34 + docreader/tests/test_excel_parser.py | 23 + docreader/tests/test_source_locator.py | 143 ++++ frontend/package-lock.json | 263 +++++++ frontend/package.json | 1 + frontend/src/assets/theme/theme.css | 8 + .../components/ChatReferenceSourceView.vue | 349 +++++++++ .../src/components/ChatReferencesDrawer.vue | 123 +++- .../src/components/chat/SandboxSidePanel.vue | 7 +- frontend/src/components/document-preview.vue | 492 ++++++++++++- .../source-preview/PdfSourceViewer.vue | 664 ++++++++++++++++++ .../components/source-preview/pdfjsLoader.ts | 24 + .../composables/useChatReferencesDrawer.ts | 38 +- .../composables/useCitationPopover.test.ts | 1 + .../src/composables/useCitationPopover.ts | 8 +- frontend/src/i18n/locales/en-US.ts | 9 + frontend/src/i18n/locales/ja-JP.ts | 9 + frontend/src/i18n/locales/ko-KR.ts | 9 + frontend/src/i18n/locales/ru-RU.ts | 9 + frontend/src/i18n/locales/zh-CN.ts | 9 + frontend/src/utils/citationAnchor.ts | 84 +++ frontend/src/utils/epubPreview.ts | 113 +++ frontend/src/utils/filePreview.ts | 4 + frontend/src/utils/referenceSources.test.mjs | 9 + frontend/src/utils/referenceSources.ts | 68 ++ frontend/src/utils/sourceLocator.test.ts | 132 ++++ frontend/src/utils/sourceLocator.ts | 281 ++++++++ frontend/src/utils/sourceLocatorDom.ts | 200 ++++++ .../chat/components/AgentStreamDisplay.vue | 5 + .../chat/components/chatLinksNewTab.test.mjs | 3 +- frontend/src/views/chat/index.vue | 12 +- .../service/chat_pipeline/merge_overlap.go | 3 + .../service/chat_pipeline/search_entity.go | 1 + .../application/service/image_multimodal.go | 2 + .../service/knowledge_clone_move.go | 1 + .../application/service/knowledge_process.go | 56 +- .../service/knowledge_source_locators.go | 133 ++++ .../service/knowledge_source_locators_test.go | 80 +++ .../service/knowledgebase_search_results.go | 1 + .../migration_sqlite_versioned_schema_test.go | 3 +- internal/handler/session/helpers.go | 9 + internal/handler/session/helpers_test.go | 4 + .../infrastructure/docparser/grpc_parser.go | 2 + .../infrastructure/docparser/http_parser.go | 2 + .../docparser/image_resolver.go | 3 + .../docparser/mineru_cloud_converter.go | 44 +- .../docparser/mineru_converter.go | 42 +- .../infrastructure/docparser/mineru_layout.go | 154 ++++ .../docparser/mineru_v1_client.go | 26 +- .../docparser/paddleocr_vl_cloud_converter.go | 27 +- .../docparser/paddleocr_vl_converter.go | 35 +- .../docparser/paddleocr_vl_layout.go | 82 +++ .../infrastructure/docparser/source_blocks.go | 57 ++ .../docparser/source_blocks_test.go | 92 +++ internal/searchutil/imageinfo.go | 124 +++- internal/searchutil/imageinfo_pages_test.go | 66 ++ internal/sourceloc/align.go | 229 ++++++ internal/sourceloc/assign.go | 215 ++++++ internal/sourceloc/office.go | 557 +++++++++++++++ internal/sourceloc/remap.go | 246 +++++++ internal/sourceloc/sourceloc_test.go | 268 +++++++ internal/sourceloc/text.go | 48 ++ internal/types/chunk.go | 4 + internal/types/docparser.go | 16 +- internal/types/search.go | 4 + internal/types/source_locator.go | 109 +++ internal/types/task.go | 3 + .../000033_chunk_source_locators.down.sql | 1 + .../000033_chunk_source_locators.up.sql | 2 + .../000114_chunk_source_locators.down.sql | 1 + .../000114_chunk_source_locators.up.sql | 3 + .../03-features/03-document-parsing.md | 19 +- .../03-features/18-chat-experience.md | 15 + website-docs/04-api/01-api-overview.md | 16 + website-docs/04-api/02-api-chunks.md | 2 +- .../06-development/02-database-schema.md | 2 +- 86 files changed, 6496 insertions(+), 224 deletions(-) create mode 100644 docreader/parser/source_locator.py create mode 100644 docreader/source_wire.py create mode 100644 docreader/tests/test_source_locator.py create mode 100644 frontend/src/components/ChatReferenceSourceView.vue create mode 100644 frontend/src/components/source-preview/PdfSourceViewer.vue create mode 100644 frontend/src/components/source-preview/pdfjsLoader.ts create mode 100644 frontend/src/utils/citationAnchor.ts create mode 100644 frontend/src/utils/epubPreview.ts create mode 100644 frontend/src/utils/sourceLocator.test.ts create mode 100644 frontend/src/utils/sourceLocator.ts create mode 100644 frontend/src/utils/sourceLocatorDom.ts create mode 100644 internal/application/service/knowledge_source_locators.go create mode 100644 internal/application/service/knowledge_source_locators_test.go create mode 100644 internal/infrastructure/docparser/mineru_layout.go create mode 100644 internal/infrastructure/docparser/paddleocr_vl_layout.go create mode 100644 internal/infrastructure/docparser/source_blocks.go create mode 100644 internal/infrastructure/docparser/source_blocks_test.go create mode 100644 internal/searchutil/imageinfo_pages_test.go create mode 100644 internal/sourceloc/align.go create mode 100644 internal/sourceloc/assign.go create mode 100644 internal/sourceloc/office.go create mode 100644 internal/sourceloc/remap.go create mode 100644 internal/sourceloc/sourceloc_test.go create mode 100644 internal/sourceloc/text.go create mode 100644 internal/types/source_locator.go create mode 100644 migrations/sqlite/000033_chunk_source_locators.down.sql create mode 100644 migrations/sqlite/000033_chunk_source_locators.up.sql create mode 100644 migrations/versioned/000114_chunk_source_locators.down.sql create mode 100644 migrations/versioned/000114_chunk_source_locators.up.sql diff --git a/docreader/main.py b/docreader/main.py index 51f2cd0c..5098c4f2 100644 --- a/docreader/main.py +++ b/docreader/main.py @@ -26,6 +26,7 @@ from docreader.proto.docreader_pb2 import ( ListEnginesResponse, ParserEngineInfo, ) +from docreader.source_wire import source_blocks_to_proto from docreader.utils.request import init_logging_request_id, request_id_context _SURROGATE_RE = re.compile(r"[\ud800-\udfff]") @@ -209,6 +210,7 @@ class DocReaderServicer(docreader_pb2_grpc.DocReaderServicer): metadata={k: _c(str(v)) for k, v in result.metadata.items()} if result.metadata else {}, + source_blocks=source_blocks_to_proto(result), ) logger.info( "Read response: content_len=%d, images=%d", @@ -258,6 +260,7 @@ class DocReaderServicer(docreader_pb2_grpc.DocReaderServicer): if result.metadata else {}, image_count=image_count, + source_blocks=source_blocks_to_proto(result), ) ) diff --git a/docreader/models/document.py b/docreader/models/document.py index 1ab8c46e..a165e1c7 100644 --- a/docreader/models/document.py +++ b/docreader/models/document.py @@ -70,6 +70,16 @@ class Document(BaseModel): ) chunks: List[Chunk] = Field(default_factory=list, description="document chunks") + source_blocks: List[Dict[str, Any]] = Field( + default_factory=list, + description=( + "Ranges of content mapped to positions in the original file: " + '{"start": int, "end": int, "locator": {"type": "pdf", "page": 1, ' + '"bbox": [x0, y0, x1, y1]}}. Offsets are code point indices into ' + "content, end exclusive; ordinals are 1-based; bbox is fractions of " + "the page with the origin at the top-left corner." + ), + ) metadata: Dict[str, Any] = Field( default_factory=dict, description="metadata fields", diff --git a/docreader/parser/excel_parser.py b/docreader/parser/excel_parser.py index e302a7fc..48e37819 100644 --- a/docreader/parser/excel_parser.py +++ b/docreader/parser/excel_parser.py @@ -93,6 +93,7 @@ class ExcelParser(BaseParser): """ chunks: List[Chunk] = [] text: List[str] = [] + source_blocks: List[dict] = [] start, end = 0, 0 excel_file = _open_excel_file(content, file_type=self.file_type) @@ -107,8 +108,10 @@ class ExcelParser(BaseParser): # Remove rows where all values are NaN (completely empty rows) df.dropna(how="all", inplace=True) - # Process each row in the DataFrame - for _, row in df.iterrows(): + # Process each row in the DataFrame. The index is the 0-based + # sheet row (rows are read from row 1 and dropped rows keep + # their labels), so index + 1 is the row number users see. + for row_index, row in df.iterrows(): page_content = [] # Build key-value pairs for non-null values for k, v in row.items(): @@ -128,10 +131,29 @@ class ExcelParser(BaseParser): chunks.append( Chunk(content=content_row, seq=len(chunks), start=start, end=end) ) + try: + row_number = int(row_index) + 1 + except (TypeError, ValueError): + row_number = 0 + if row_number > 0: + source_blocks.append( + { + "start": start, + "end": end, + "locator": { + "type": "sheet", + "sheet": str(excel_sheet_name), + "row_start": row_number, + "row_end": row_number, + }, + } + ) start = end # Combine all text and return as Document - return Document(content="".join(text), chunks=chunks) + return Document( + content="".join(text), chunks=chunks, source_blocks=source_blocks + ) def _read_sheet_dataframe( diff --git a/docreader/parser/pdf_parser.py b/docreader/parser/pdf_parser.py index 4a541965..1b31db72 100644 --- a/docreader/parser/pdf_parser.py +++ b/docreader/parser/pdf_parser.py @@ -28,6 +28,7 @@ from docreader.config import CONFIG from docreader.models.document import Document from docreader.parser.base_parser import BaseParser from docreader.parser.concurrency import parser_worker_limit +from docreader.parser.source_locator import PageGeometry, page_blocks logger = logging.getLogger(__name__) @@ -134,6 +135,12 @@ MAX_FIGURE_HEIGHT_RATIO = _env_float("DOCREADER_PDF_MAX_FIGURE_HEIGHT_RATIO", 0. # Can be overridden per-upload via parser_engine_overrides.pdf_force_scanned. FORCE_SCANNED_PDF = _env_bool("DOCREADER_PDF_FORCE_SCANNED", False) +# --- Source locators --------------------------------------------------------- +# Record where each paragraph of a native text page sits on the page, so a +# citation can open the PDF there and outline the paragraph. Costs one glyph +# pass per text page; disable to only record page numbers. +SOURCE_BOXES = _env_bool("DOCREADER_PDF_SOURCE_BOXES", True) + # pdfium / Adobe text layers often emit U+FFFE for missing hyphenation or ligatures. _PDF_ARTIFACT_RE = re.compile(r"[\u00ad\u200b-\u200f\ufeff\ufffe\uffff]") _PDF_ARTIFACT_JOIN_RE = re.compile(r"(\w)[\u00ad\ufffe](\w)") @@ -629,12 +636,13 @@ def _point_in_boxes(x: float, y: float, boxes: list) -> bool: return False -def _page_chars(textpage, page, raw) -> tuple: +def _page_chars(textpage, page, raw, loose: bool = False) -> tuple: """Return ``(chars, page_width)`` with hidden/off-page glyphs filtered. Working at the glyph level (instead of pdfium rect segments) keeps mixed CJK + Latin/number lines in their true left-to-right order, which the - rect-level ``get_text_bounded`` API scrambles. + rect-level ``get_text_bounded`` API scrambles. ``loose`` boxes span the + font's ascent to descent, so every glyph of a line shares one height. """ n = textpage.count_chars() if n <= 0: @@ -645,7 +653,7 @@ def _page_chars(textpage, page, raw) -> tuple: chars: list = [] for i in range(n): try: - left, bottom, right, top = textpage.get_charbox(i) + left, bottom, right, top = textpage.get_charbox(i, loose=loose) except Exception: continue ch = textpage.get_text_range(i, 1) @@ -658,7 +666,7 @@ def _page_chars(textpage, page, raw) -> tuple: continue # off-page glyph if invisible and _point_in_boxes((x0 + x1) / 2, (y0 + y1) / 2, invisible): continue # covered by an invisible text object - chars.append({"x0": x0, "y0": y0, "x1": x1, "y1": y1, "ch": ch}) + chars.append({"x0": x0, "y0": y0, "x1": x1, "y1": y1, "ch": ch, "i": i}) return chars, width @@ -988,6 +996,81 @@ def _extract_layout_text(page, raw) -> str: _close_pdfium_resource(textpage) +# A visual line is cut where a gap wider than one glyph height coincides with +# a jump in the text stream: the gutter between two columns, not a word space. +LINE_SEGMENT_GAP_RATIO = 1.0 +LINE_SEGMENT_STREAM_JUMP = 8 +LINE_SEGMENT_MIN_CHARS = 3 + + +def _split_line_at_gaps(line: dict) -> list: + """Cut a visual line where it crosses from one column into another. + + Narrow two-column gutters (ACL/IEEE papers leave ~16pt) and anything that + spans them (a title, a wide table) defeat the full-height gutter search, + so glyphs of both columns land in one line; each side must be placed and + boxed on its own. Columns are far apart in the text stream while glyphs of + one line are consecutive, which keeps wide justified word spaces intact. + """ + whole = [{"text": line["text"], "bbox": line["bbox"]}] + chars = [c for c in line["chars"] if not c["ch"].isspace()] + if len(chars) < 2 * LINE_SEGMENT_MIN_CHARS: + return whole + min_gap = max(max(c["y1"] - c["y0"] for c in chars), 1.0) * LINE_SEGMENT_GAP_RATIO + segments: list = [[chars[0]]] + for prev, cur in zip(chars, chars[1:]): + jump = abs(cur.get("i", 0) - prev.get("i", 0)) + if cur["x0"] - prev["x1"] > min_gap and jump > LINE_SEGMENT_STREAM_JUMP: + segments.append([cur]) + else: + segments[-1].append(cur) + if len(segments) == 1 or any(len(seg) < LINE_SEGMENT_MIN_CHARS for seg in segments): + return whole + out = [] + for seg in segments: + text = _join_line_glyphs(seg) + if text.strip(): + out.append({"text": text, "bbox": _chars_bbox(seg)}) + return out or whole + + +def _page_line_boxes(page, raw) -> list: + """Visual lines of a text page as ``{"text", "bbox"}`` in PDF points. + + Lines are grouped per column so a two-column page does not fuse the + columns' glyphs into one line. Any failure yields no lines, which only + downgrades the page's locators to page numbers. + """ + textpage = None + try: + textpage = page.get_textpage() + # Tight glyph boxes put "a" and "d" at different heights, which splits + # one line of text into fragments by letter shape. + chars, width = _page_chars(textpage, page, raw, loose=True) + if not chars: + return [] + heights = [c["y1"] - c["y0"] for c in chars if c["y1"] - c["y0"] > 0] + scale = (statistics.median(heights) if heights else 1.0) or 1.0 + lines = [] + for col in _split_columns(chars, scale, width): + for ln in _group_lines_with_chars(col): + lines.extend(_split_line_at_gaps(ln)) + return lines + except Exception: + logger.debug("line box extraction failed", exc_info=True) + return [] + finally: + _close_pdfium_resource(textpage) + + +def _page_geometry(page): + try: + return PageGeometry(page.get_cropbox(), page.get_rotation()) + except Exception: + logger.debug("page geometry unavailable", exc_info=True) + return None + + def _effective_scale(page, scale: float, max_edge: int) -> float: """Reduce ``scale`` so the rendered long edge never exceeds ``max_edge`` px. @@ -1393,10 +1476,23 @@ class PDFScannedParser(BaseParser): finally: _close_pdfium_resource(pdf) + source_blocks = [] + offset = 0 for i in range(page_count): page_filename = f"{base_name}_page_{i+1}.jpg" ref_path = f"images/{page_filename}" - markdown_lines.append(f"![{page_filename}]({ref_path})") + line = f"![{page_filename}]({ref_path})" + if markdown_lines: + offset += 2 + markdown_lines.append(line) + source_blocks.append( + { + "start": offset, + "end": offset + len(line), + "locator": {"type": "pdf", "page": i + 1}, + } + ) + offset += len(line) images[ref_path] = base64.b64encode(rendered[i]).decode("utf-8") text = "\n\n".join(markdown_lines) @@ -1407,6 +1503,7 @@ class PDFScannedParser(BaseParser): "image_source_type": "scanned_pdf", "page_count": page_count, }, + source_blocks=source_blocks, ) except Exception as e: logger.exception("PDFScannedParser failed to parse PDF: %s", e) @@ -1499,9 +1596,12 @@ class PDFParser(BaseParser): texts: list = [] classes: list = [] vector_clips: dict = {} + line_boxes: dict = {} + geometries: dict = {} for i in range(page_count): page = pdf[i] try: + geometries[i] = _page_geometry(page) plain = _extract_page_text(page) ratio = _page_image_area_ratio(page, pdfium_r) cls = _classify_page(ratio, len(plain.strip())) @@ -1534,6 +1634,8 @@ class PDFParser(BaseParser): for ref_path, b64, _y, _cap in clips: images[ref_path] = b64 text = _postprocess_pdf_text(text) + if cls == "text" and SOURCE_BOXES: + line_boxes[i] = _page_line_boxes(page, pdfium_r) if cls == "text" and vector_clips.get(i): text = _inject_figure_markdown_before_captions( text, vector_clips[i] @@ -1574,27 +1676,60 @@ class PDFParser(BaseParser): finally: _close_pdfium_resource(pdf) - # Assemble markdown in reading order. + # Assemble markdown in reading order, recording where each page's + # text lands so citations can point back at the page (and paragraph). embedded_count = 0 vector_figure_count = 0 blocks = [] + source_blocks = [] + offset = 0 + + def place(part: str) -> int: + nonlocal offset + if blocks: + offset += 2 # the "\n\n" separator + start = offset + blocks.append(part) + offset += len(part) + return start + for i in range(page_count): + page_only = {"type": "pdf", "page": i + 1} if classes[i] == "scanned": page_filename = f"{base_name}_page_{i+1}.jpg" - blocks.append(f"![{page_filename}](images/{page_filename})") + part = f"![{page_filename}](images/{page_filename})" + start = place(part) + source_blocks.append( + {"start": start, "end": start + len(part), "locator": page_only} + ) else: stripped = texts[i].strip() if stripped: - blocks.append(stripped) + start = place(stripped) + source_blocks.extend( + page_blocks( + stripped, + start, + i + 1, + line_boxes.get(i) or [], + geometries.get(i), + ) + ) vector_figure_count += len(vector_clips.get(i, [])) page_images = list(embedded.get(i, [])) page_images.sort(key=lambda item: item[2], reverse=True) for ref_path, _b64, _y in page_images: fname = os.path.basename(ref_path) - blocks.append(f"![{fname}]({ref_path})") + part = f"![{fname}]({ref_path})" + start = place(part) + source_blocks.append( + {"start": start, "end": start + len(part), "locator": dict(page_only)} + ) embedded_count += 1 - content_text = "\n\n".join(blocks).strip() + # Every part is already stripped, so the join needs no trimming and + # the recorded offsets stay exact. + content_text = "\n\n".join(blocks) metadata = { "page_count": page_count, @@ -1615,4 +1750,9 @@ class PDFParser(BaseParser): embedded_count, len(content_text), ) - return Document(content=content_text, images=images, metadata=metadata) + return Document( + content=content_text, + images=images, + metadata=metadata, + source_blocks=source_blocks, + ) diff --git a/docreader/parser/source_locator.py b/docreader/parser/source_locator.py new file mode 100644 index 00000000..29ef3e9b --- /dev/null +++ b/docreader/parser/source_locator.py @@ -0,0 +1,293 @@ +"""Source locators: map ranges of parsed markdown back to the original file. + +A source block is ``{"start": int, "end": int, "locator": {...}}`` where +``start``/``end`` are code point offsets into the document content (end +exclusive) and the locator says where that text sits in the original file. +For PDFs the locator is ``{"type": "pdf", "page": n}`` plus an optional +``"bbox": [x0, y0, x1, y1]`` given as fractions of the displayed page with +the origin at its top-left corner, so viewers can draw it at any zoom. +""" + +from __future__ import annotations + +import re +import statistics +import unicodedata +from typing import Iterable, List, Optional, Sequence, Tuple + +# How far past the cursor a line is searched for, in normalized characters. +_SEARCH_WINDOW = 6000 +# How much of a line's normalized text is searched for. +_KEY_CHARS = 24 +# Lines shorter than this (normalized) are too ambiguous to place. +_MIN_KEY_CHARS = 2 +# Lines shorter than this are placed only when their text is unique on the +# page: labels inside figures ("LLM", "EBAE") otherwise steal body offsets. +# Wide (CJK) characters count as four, since each carries about a word. +_SHORT_KEY_WEIGHT = 16 + +_LIST_START_RE = re.compile( + r"^\s*(?:" + r"\d{1,3}(?:\.\d{1,3}){0,4}[\.、.))]?\s" # 1 1. 8.14.11 2) + r"|[((]\d{1,3}[))]" # (1) (1) + r"|[一二三四五六七八九十百]+[、..]" # 一、 + r"|[•·●○■□◆◇▪\-–*]\s" # bullets + r"|第[一二三四五六七八九十百零\d]+[章节条款部分篇]" # 第三章 + r")" +) +_SENTENCE_END = tuple("。..!?!?::;;」』”))") + + +def _fold(ch: str) -> str: + """Letters and digits only, lower-cased, full-width folded.""" + if not ch.isalnum(): + return "" + return unicodedata.normalize("NFKC", ch).lower() + + +def normalize_with_positions(text: str) -> Tuple[str, List[int]]: + """Project text onto letters and digits. + + Returns the projection and, for each of its characters, the offset in + ``text`` of the character it came from. + """ + out: List[str] = [] + pos: List[int] = [] + for i, ch in enumerate(text): + folded = _fold(ch) + for f in folded: + out.append(f) + pos.append(i) + return "".join(out), pos + + +def normalize(text: str) -> str: + return "".join(_fold(ch) for ch in text) + + +def _key_weight(key: str) -> int: + return sum(4 if unicodedata.east_asian_width(ch) in ("W", "F") else 1 for ch in key) + + +def _column_cursor(boxes, idx: int, ends: List[Optional[int]]) -> Optional[int]: + """End of the nearest placed line above ``boxes[idx]`` in the same column.""" + x0, y0, x1, _ = boxes[idx] + for j in range(idx - 1, max(-1, idx - 80), -1): + if ends[j] is None: + continue + px0, py0, px1, _ = boxes[j] + if px0 < x1 and x0 < px1 and py0 > y0: + return ends[j] + return None + + +def locate_lines( + page_text: str, + line_texts: Sequence[str], + boxes: Optional[Sequence[Tuple[float, float, float, float]]] = None, +) -> List[Optional[int]]: + """Find where each visual line starts in the page text. + + Lines are searched in order after the previous hit so repeated phrases + resolve to the right occurrence. With ``boxes`` (PDF points, bottom-left + origin) the search first continues from the line above in the same + column, since glyph lines of side-by-side columns arrive interleaved. A + line not found after either cursor is searched from the start of the page, + which tolerates text extractors that order columns differently from the + glyph grouping. Returns one offset (or None) per line. + """ + norm, pos = normalize_with_positions(page_text) + hits: List[Optional[int]] = [] + ends: List[Optional[int]] = [] + cursor = 0 + for idx, line in enumerate(line_texts): + key = normalize(line)[:_KEY_CHARS] + if len(key) < _MIN_KEY_CHARS or ( + _key_weight(key) < _SHORT_KEY_WEIGHT and norm.count(key) > 1 + ): + hits.append(None) + ends.append(None) + continue + cursors = [cursor] + if boxes is not None: + column = _column_cursor(boxes, idx, ends) + if column is not None: + cursors.insert(0, column) + at = -1 + for start_at in cursors: + at = norm.find(key, start_at, start_at + _SEARCH_WINDOW + len(key)) + if at >= 0: + break + if at < 0: + at = norm.find(key) + if at < 0: + hits.append(None) + ends.append(None) + continue + else: + cursor = at + len(key) + ends.append(at + len(key)) + start = pos[at] + # Pull in leading punctuation or markup on the same text line + # (brackets, "## ", list bullets) that normalization skipped. + while start > 0 and page_text[start - 1] != "\n" and not page_text[start - 1].isalnum(): + start -= 1 + hits.append(start) + return hits + + +class PageGeometry: + """Converts PDF user-space points to fractions of the displayed page.""" + + def __init__(self, cropbox: Tuple[float, float, float, float], rotation: int = 0): + left, bottom, right, top = cropbox + self.left, self.bottom = min(left, right), min(bottom, top) + self.width = abs(right - left) or 1.0 + self.height = abs(top - bottom) or 1.0 + self.rotation = int(rotation or 0) % 360 + + def bbox(self, box: Tuple[float, float, float, float]) -> List[float]: + x0, y0, x1, y1 = box + # Unrotated, top-left origin. + ux0 = (x0 - self.left) / self.width + ux1 = (x1 - self.left) / self.width + uy0 = 1.0 - (y1 - self.bottom) / self.height + uy1 = 1.0 - (y0 - self.bottom) / self.height + corners = [self._rotate(ux0, uy0), self._rotate(ux1, uy1)] + xs = [c[0] for c in corners] + ys = [c[1] for c in corners] + return [ + round(_clamp(min(xs)), 4), + round(_clamp(min(ys)), 4), + round(_clamp(max(xs)), 4), + round(_clamp(max(ys)), 4), + ] + + def _rotate(self, x: float, y: float) -> Tuple[float, float]: + # Viewers display /Rotate clockwise. + if self.rotation == 90: + return 1.0 - y, x + if self.rotation == 180: + return 1.0 - x, 1.0 - y + if self.rotation == 270: + return y, 1.0 - x + return x, y + + +def _clamp(v: float) -> float: + return 0.0 if v < 0 else 1.0 if v > 1 else v + + +def _union(boxes: Iterable[Tuple[float, float, float, float]]): + boxes = list(boxes) + return ( + min(b[0] for b in boxes), + min(b[1] for b in boxes), + max(b[2] for b in boxes), + max(b[3] for b in boxes), + ) + + +def _line_gap(prev: dict, cur: dict) -> Optional[float]: + """Vertical gap from ``prev`` down to ``cur`` when they share a column.""" + px0, py0, px1, _ = prev["bbox"] + cx0, _, cx1, cy1 = cur["bbox"] + if cx1 < px0 or cx0 > px1 or cy1 > py0: + return None + return py0 - cy1 + + +def _starts_paragraph( + prev: dict, cur: dict, line_h: float, usual_gap: float, col_right: float +) -> bool: + """Whether ``cur`` opens a new visual paragraph after ``prev``. + + Boxes are PDF points with a bottom-left origin, so reading downwards means + decreasing y. ``usual_gap`` is the page's typical gap between consecutive + lines of a paragraph, which absorbs double-spaced layouts. + """ + px0, py0, px1, _ = prev["bbox"] + cx0, _, cx1, cy1 = cur["bbox"] + if cx1 < px0 or cx0 > px1: + return True # moved to another column + if cy1 > py0 + line_h * 0.5: + return True # went back up the page: a new column or region + if py0 - cy1 > usual_gap * 1.5 + line_h * 0.3: + return True # vertical gap wider than the usual line spacing + if _LIST_START_RE.match(cur["text"]): + return True + ends_sentence = prev["text"].rstrip().endswith(_SENTENCE_END) + short = col_right - px1 > line_h * 2 + return ends_sentence and short + + +def page_blocks( + page_text: str, + offset: int, + page_number: int, + lines: Sequence[dict], + geometry: Optional[PageGeometry], +) -> List[dict]: + """Source blocks for one text page placed at ``offset`` in the document. + + ``lines`` are visual lines ``{"text", "bbox"}`` (bbox in PDF points). Lines + are placed in ``page_text``, grouped into paragraphs by geometry, and each + paragraph becomes a block spanning from its first line to the next + paragraph, boxed by the union of its lines. Text before the first placed + line, or a page with no placeable lines, gets a page-only block, so the + blocks always tile the page's text. + """ + end = offset + len(page_text) + page_only = {"type": "pdf", "page": page_number} + if not page_text: + return [] + placed = [] + if lines and geometry is not None: + starts = locate_lines( + page_text, [ln["text"] for ln in lines], [ln["bbox"] for ln in lines] + ) + placed = sorted( + ( + (at, ln) + for at, ln in zip(starts, lines) + if at is not None + ), + key=lambda item: item[0], + ) + if not placed: + return [{"start": offset, "end": end, "locator": dict(page_only)}] + + heights = [ln["bbox"][3] - ln["bbox"][1] for _, ln in placed if ln["bbox"][3] > ln["bbox"][1]] + line_h = statistics.median(heights) if heights else 10.0 + gaps = [ + g + for g in (_line_gap(a[1], b[1]) for a, b in zip(placed, placed[1:])) + if g is not None and g >= 0 + ] + usual_gap = statistics.median(gaps) if gaps else line_h * 0.5 + + paragraphs: List[Tuple[int, List[dict]]] = [] + for at, ln in placed: + if paragraphs: + prev = paragraphs[-1][1][-1] + col_right = max(p["bbox"][2] for p in paragraphs[-1][1]) + col_right = max(col_right, ln["bbox"][2]) + if not _starts_paragraph(prev, ln, line_h, usual_gap, col_right): + paragraphs[-1][1].append(ln) + continue + paragraphs.append((at, [ln])) + + blocks: List[dict] = [] + first_at = paragraphs[0][0] + if first_at > 0 and page_text[:first_at].strip(): + blocks.append({"start": offset, "end": offset + first_at, "locator": dict(page_only)}) + for i, (at, members) in enumerate(paragraphs): + stop = paragraphs[i + 1][0] if i + 1 < len(paragraphs) else len(page_text) + if i == 0: + at = 0 if not blocks else at + if stop <= at: + continue + locator = dict(page_only) + locator["bbox"] = geometry.bbox(_union(m["bbox"] for m in members)) + blocks.append({"start": offset + at, "end": offset + stop, "locator": locator}) + return blocks diff --git a/docreader/proto/docreader.pb.go b/docreader/proto/docreader.pb.go index f071eecb..507c1bd1 100644 --- a/docreader/proto/docreader.pb.go +++ b/docreader/proto/docreader.pb.go @@ -1,7 +1,7 @@ // Code generated by protoc-gen-go. DO NOT EDIT. // versions: -// protoc-gen-go v1.36.7 -// protoc v5.29.3 +// protoc-gen-go v1.36.6 +// protoc v6.33.4 // source: docreader.proto package proto @@ -242,6 +242,71 @@ func (x *ImageRef) GetImageData() []byte { return nil } +// SourceBlock ties a range of markdown_content to the place in the original +// file it was parsed from, so a citation can open the file there. +type SourceBlock struct { + state protoimpl.MessageState `protogen:"open.v1"` + // Unicode code point offsets into markdown_content, end exclusive. + Start uint32 `protobuf:"varint,1,opt,name=start,proto3" json:"start,omitempty"` + End uint32 `protobuf:"varint,2,opt,name=end,proto3" json:"end,omitempty"` + // JSON-encoded locator: {"type":"pdf","page":3,"bbox":[x0,y0,x1,y1]} + // with ordinals 1-based and bbox as fractions of the page, origin top-left. + LocatorJson string `protobuf:"bytes,3,opt,name=locator_json,json=locatorJson,proto3" json:"locator_json,omitempty"` + unknownFields protoimpl.UnknownFields + sizeCache protoimpl.SizeCache +} + +func (x *SourceBlock) Reset() { + *x = SourceBlock{} + mi := &file_docreader_proto_msgTypes[3] + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + ms.StoreMessageInfo(mi) +} + +func (x *SourceBlock) String() string { + return protoimpl.X.MessageStringOf(x) +} + +func (*SourceBlock) ProtoMessage() {} + +func (x *SourceBlock) ProtoReflect() protoreflect.Message { + mi := &file_docreader_proto_msgTypes[3] + if x != nil { + ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) + if ms.LoadMessageInfo() == nil { + ms.StoreMessageInfo(mi) + } + return ms + } + return mi.MessageOf(x) +} + +// Deprecated: Use SourceBlock.ProtoReflect.Descriptor instead. +func (*SourceBlock) Descriptor() ([]byte, []int) { + return file_docreader_proto_rawDescGZIP(), []int{3} +} + +func (x *SourceBlock) GetStart() uint32 { + if x != nil { + return x.Start + } + return 0 +} + +func (x *SourceBlock) GetEnd() uint32 { + if x != nil { + return x.End + } + return 0 +} + +func (x *SourceBlock) GetLocatorJson() string { + if x != nil { + return x.LocatorJson + } + return "" +} + type ReadResponse struct { state protoimpl.MessageState `protogen:"open.v1"` MarkdownContent string `protobuf:"bytes,1,opt,name=markdown_content,json=markdownContent,proto3" json:"markdown_content,omitempty"` @@ -249,13 +314,14 @@ type ReadResponse struct { ImageDirPath string `protobuf:"bytes,3,opt,name=image_dir_path,json=imageDirPath,proto3" json:"image_dir_path,omitempty"` Metadata map[string]string `protobuf:"bytes,4,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` Error string `protobuf:"bytes,5,opt,name=error,proto3" json:"error,omitempty"` + SourceBlocks []*SourceBlock `protobuf:"bytes,6,rep,name=source_blocks,json=sourceBlocks,proto3" json:"source_blocks,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *ReadResponse) Reset() { *x = ReadResponse{} - mi := &file_docreader_proto_msgTypes[3] + mi := &file_docreader_proto_msgTypes[4] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -267,7 +333,7 @@ func (x *ReadResponse) String() string { func (*ReadResponse) ProtoMessage() {} func (x *ReadResponse) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[3] + mi := &file_docreader_proto_msgTypes[4] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -280,7 +346,7 @@ func (x *ReadResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ReadResponse.ProtoReflect.Descriptor instead. func (*ReadResponse) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{3} + return file_docreader_proto_rawDescGZIP(), []int{4} } func (x *ReadResponse) GetMarkdownContent() string { @@ -318,6 +384,13 @@ func (x *ReadResponse) GetError() string { return "" } +func (x *ReadResponse) GetSourceBlocks() []*SourceBlock { + if x != nil { + return x.SourceBlocks + } + return nil +} + // Metadata header for a streamed read. Sent exactly once, before any image. type ReadStreamMeta struct { state protoimpl.MessageState `protogen:"open.v1"` @@ -326,13 +399,14 @@ type ReadStreamMeta struct { Metadata map[string]string `protobuf:"bytes,3,rep,name=metadata,proto3" json:"metadata,omitempty" protobuf_key:"bytes,1,opt,name=key" protobuf_val:"bytes,2,opt,name=value"` Error string `protobuf:"bytes,4,opt,name=error,proto3" json:"error,omitempty"` ImageCount uint32 `protobuf:"varint,5,opt,name=image_count,json=imageCount,proto3" json:"image_count,omitempty"` // best-effort total image count (0 if unknown) + SourceBlocks []*SourceBlock `protobuf:"bytes,6,rep,name=source_blocks,json=sourceBlocks,proto3" json:"source_blocks,omitempty"` unknownFields protoimpl.UnknownFields sizeCache protoimpl.SizeCache } func (x *ReadStreamMeta) Reset() { *x = ReadStreamMeta{} - mi := &file_docreader_proto_msgTypes[4] + mi := &file_docreader_proto_msgTypes[5] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -344,7 +418,7 @@ func (x *ReadStreamMeta) String() string { func (*ReadStreamMeta) ProtoMessage() {} func (x *ReadStreamMeta) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[4] + mi := &file_docreader_proto_msgTypes[5] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -357,7 +431,7 @@ func (x *ReadStreamMeta) ProtoReflect() protoreflect.Message { // Deprecated: Use ReadStreamMeta.ProtoReflect.Descriptor instead. func (*ReadStreamMeta) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{4} + return file_docreader_proto_rawDescGZIP(), []int{5} } func (x *ReadStreamMeta) GetMarkdownContent() string { @@ -395,6 +469,13 @@ func (x *ReadStreamMeta) GetImageCount() uint32 { return 0 } +func (x *ReadStreamMeta) GetSourceBlocks() []*SourceBlock { + if x != nil { + return x.SourceBlocks + } + return nil +} + // One frame of a ReadStream. The first frame carries `meta`; every subsequent // frame carries a single `image`. type ReadStreamResponse struct { @@ -410,7 +491,7 @@ type ReadStreamResponse struct { func (x *ReadStreamResponse) Reset() { *x = ReadStreamResponse{} - mi := &file_docreader_proto_msgTypes[5] + mi := &file_docreader_proto_msgTypes[6] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -422,7 +503,7 @@ func (x *ReadStreamResponse) String() string { func (*ReadStreamResponse) ProtoMessage() {} func (x *ReadStreamResponse) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[5] + mi := &file_docreader_proto_msgTypes[6] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -435,7 +516,7 @@ func (x *ReadStreamResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ReadStreamResponse.ProtoReflect.Descriptor instead. func (*ReadStreamResponse) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{5} + return file_docreader_proto_rawDescGZIP(), []int{6} } func (x *ReadStreamResponse) GetPayload() isReadStreamResponse_Payload { @@ -488,7 +569,7 @@ type ListEnginesRequest struct { func (x *ListEnginesRequest) Reset() { *x = ListEnginesRequest{} - mi := &file_docreader_proto_msgTypes[6] + mi := &file_docreader_proto_msgTypes[7] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -500,7 +581,7 @@ func (x *ListEnginesRequest) String() string { func (*ListEnginesRequest) ProtoMessage() {} func (x *ListEnginesRequest) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[6] + mi := &file_docreader_proto_msgTypes[7] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -513,7 +594,7 @@ func (x *ListEnginesRequest) ProtoReflect() protoreflect.Message { // Deprecated: Use ListEnginesRequest.ProtoReflect.Descriptor instead. func (*ListEnginesRequest) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{6} + return file_docreader_proto_rawDescGZIP(), []int{7} } func (x *ListEnginesRequest) GetConfigOverrides() map[string]string { @@ -536,7 +617,7 @@ type ParserEngineInfo struct { func (x *ParserEngineInfo) Reset() { *x = ParserEngineInfo{} - mi := &file_docreader_proto_msgTypes[7] + mi := &file_docreader_proto_msgTypes[8] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -548,7 +629,7 @@ func (x *ParserEngineInfo) String() string { func (*ParserEngineInfo) ProtoMessage() {} func (x *ParserEngineInfo) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[7] + mi := &file_docreader_proto_msgTypes[8] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -561,7 +642,7 @@ func (x *ParserEngineInfo) ProtoReflect() protoreflect.Message { // Deprecated: Use ParserEngineInfo.ProtoReflect.Descriptor instead. func (*ParserEngineInfo) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{7} + return file_docreader_proto_rawDescGZIP(), []int{8} } func (x *ParserEngineInfo) GetName() string { @@ -608,7 +689,7 @@ type ListEnginesResponse struct { func (x *ListEnginesResponse) Reset() { *x = ListEnginesResponse{} - mi := &file_docreader_proto_msgTypes[8] + mi := &file_docreader_proto_msgTypes[9] ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) ms.StoreMessageInfo(mi) } @@ -620,7 +701,7 @@ func (x *ListEnginesResponse) String() string { func (*ListEnginesResponse) ProtoMessage() {} func (x *ListEnginesResponse) ProtoReflect() protoreflect.Message { - mi := &file_docreader_proto_msgTypes[8] + mi := &file_docreader_proto_msgTypes[9] if x != nil { ms := protoimpl.X.MessageStateOf(protoimpl.Pointer(x)) if ms.LoadMessageInfo() == nil { @@ -633,7 +714,7 @@ func (x *ListEnginesResponse) ProtoReflect() protoreflect.Message { // Deprecated: Use ListEnginesResponse.ProtoReflect.Descriptor instead. func (*ListEnginesResponse) Descriptor() ([]byte, []int) { - return file_docreader_proto_rawDescGZIP(), []int{8} + return file_docreader_proto_rawDescGZIP(), []int{9} } func (x *ListEnginesResponse) GetEngines() []*ParserEngineInfo { @@ -671,24 +752,30 @@ const file_docreader_proto_rawDesc = "" + "\vstorage_key\x18\x04 \x01(\tR\n" + "storageKey\x12\x1d\n" + "\n" + - "image_data\x18\x05 \x01(\fR\timageData\"\xa9\x02\n" + + "image_data\x18\x05 \x01(\fR\timageData\"X\n" + + "\vSourceBlock\x12\x14\n" + + "\x05start\x18\x01 \x01(\rR\x05start\x12\x10\n" + + "\x03end\x18\x02 \x01(\rR\x03end\x12!\n" + + "\flocator_json\x18\x03 \x01(\tR\vlocatorJson\"\xe6\x02\n" + "\fReadResponse\x12)\n" + "\x10markdown_content\x18\x01 \x01(\tR\x0fmarkdownContent\x122\n" + "\n" + "image_refs\x18\x02 \x03(\v2\x13.docreader.ImageRefR\timageRefs\x12$\n" + "\x0eimage_dir_path\x18\x03 \x01(\tR\fimageDirPath\x12A\n" + "\bmetadata\x18\x04 \x03(\v2%.docreader.ReadResponse.MetadataEntryR\bmetadata\x12\x14\n" + - "\x05error\x18\x05 \x01(\tR\x05error\x1a;\n" + + "\x05error\x18\x05 \x01(\tR\x05error\x12;\n" + + "\rsource_blocks\x18\x06 \x03(\v2\x16.docreader.SourceBlockR\fsourceBlocks\x1a;\n" + "\rMetadataEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + - "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\x9a\x02\n" + + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"\xd7\x02\n" + "\x0eReadStreamMeta\x12)\n" + "\x10markdown_content\x18\x01 \x01(\tR\x0fmarkdownContent\x12$\n" + "\x0eimage_dir_path\x18\x02 \x01(\tR\fimageDirPath\x12C\n" + "\bmetadata\x18\x03 \x03(\v2'.docreader.ReadStreamMeta.MetadataEntryR\bmetadata\x12\x14\n" + "\x05error\x18\x04 \x01(\tR\x05error\x12\x1f\n" + "\vimage_count\x18\x05 \x01(\rR\n" + - "imageCount\x1a;\n" + + "imageCount\x12;\n" + + "\rsource_blocks\x18\x06 \x03(\v2\x16.docreader.SourceBlockR\fsourceBlocks\x1a;\n" + "\rMetadataEntry\x12\x10\n" + "\x03key\x18\x01 \x01(\tR\x03key\x12\x14\n" + "\x05value\x18\x02 \x01(\tR\x05value:\x028\x01\"}\n" + @@ -728,43 +815,46 @@ func file_docreader_proto_rawDescGZIP() []byte { return file_docreader_proto_rawDescData } -var file_docreader_proto_msgTypes = make([]protoimpl.MessageInfo, 13) +var file_docreader_proto_msgTypes = make([]protoimpl.MessageInfo, 14) var file_docreader_proto_goTypes = []any{ (*ReadConfig)(nil), // 0: docreader.ReadConfig (*ReadRequest)(nil), // 1: docreader.ReadRequest (*ImageRef)(nil), // 2: docreader.ImageRef - (*ReadResponse)(nil), // 3: docreader.ReadResponse - (*ReadStreamMeta)(nil), // 4: docreader.ReadStreamMeta - (*ReadStreamResponse)(nil), // 5: docreader.ReadStreamResponse - (*ListEnginesRequest)(nil), // 6: docreader.ListEnginesRequest - (*ParserEngineInfo)(nil), // 7: docreader.ParserEngineInfo - (*ListEnginesResponse)(nil), // 8: docreader.ListEnginesResponse - nil, // 9: docreader.ReadConfig.ParserEngineOverridesEntry - nil, // 10: docreader.ReadResponse.MetadataEntry - nil, // 11: docreader.ReadStreamMeta.MetadataEntry - nil, // 12: docreader.ListEnginesRequest.ConfigOverridesEntry + (*SourceBlock)(nil), // 3: docreader.SourceBlock + (*ReadResponse)(nil), // 4: docreader.ReadResponse + (*ReadStreamMeta)(nil), // 5: docreader.ReadStreamMeta + (*ReadStreamResponse)(nil), // 6: docreader.ReadStreamResponse + (*ListEnginesRequest)(nil), // 7: docreader.ListEnginesRequest + (*ParserEngineInfo)(nil), // 8: docreader.ParserEngineInfo + (*ListEnginesResponse)(nil), // 9: docreader.ListEnginesResponse + nil, // 10: docreader.ReadConfig.ParserEngineOverridesEntry + nil, // 11: docreader.ReadResponse.MetadataEntry + nil, // 12: docreader.ReadStreamMeta.MetadataEntry + nil, // 13: docreader.ListEnginesRequest.ConfigOverridesEntry } var file_docreader_proto_depIdxs = []int32{ - 9, // 0: docreader.ReadConfig.parser_engine_overrides:type_name -> docreader.ReadConfig.ParserEngineOverridesEntry + 10, // 0: docreader.ReadConfig.parser_engine_overrides:type_name -> docreader.ReadConfig.ParserEngineOverridesEntry 0, // 1: docreader.ReadRequest.config:type_name -> docreader.ReadConfig 2, // 2: docreader.ReadResponse.image_refs:type_name -> docreader.ImageRef - 10, // 3: docreader.ReadResponse.metadata:type_name -> docreader.ReadResponse.MetadataEntry - 11, // 4: docreader.ReadStreamMeta.metadata:type_name -> docreader.ReadStreamMeta.MetadataEntry - 4, // 5: docreader.ReadStreamResponse.meta:type_name -> docreader.ReadStreamMeta - 2, // 6: docreader.ReadStreamResponse.image:type_name -> docreader.ImageRef - 12, // 7: docreader.ListEnginesRequest.config_overrides:type_name -> docreader.ListEnginesRequest.ConfigOverridesEntry - 7, // 8: docreader.ListEnginesResponse.engines:type_name -> docreader.ParserEngineInfo - 1, // 9: docreader.DocReader.Read:input_type -> docreader.ReadRequest - 1, // 10: docreader.DocReader.ReadStream:input_type -> docreader.ReadRequest - 6, // 11: docreader.DocReader.ListEngines:input_type -> docreader.ListEnginesRequest - 3, // 12: docreader.DocReader.Read:output_type -> docreader.ReadResponse - 5, // 13: docreader.DocReader.ReadStream:output_type -> docreader.ReadStreamResponse - 8, // 14: docreader.DocReader.ListEngines:output_type -> docreader.ListEnginesResponse - 12, // [12:15] is the sub-list for method output_type - 9, // [9:12] is the sub-list for method input_type - 9, // [9:9] is the sub-list for extension type_name - 9, // [9:9] is the sub-list for extension extendee - 0, // [0:9] is the sub-list for field type_name + 11, // 3: docreader.ReadResponse.metadata:type_name -> docreader.ReadResponse.MetadataEntry + 3, // 4: docreader.ReadResponse.source_blocks:type_name -> docreader.SourceBlock + 12, // 5: docreader.ReadStreamMeta.metadata:type_name -> docreader.ReadStreamMeta.MetadataEntry + 3, // 6: docreader.ReadStreamMeta.source_blocks:type_name -> docreader.SourceBlock + 5, // 7: docreader.ReadStreamResponse.meta:type_name -> docreader.ReadStreamMeta + 2, // 8: docreader.ReadStreamResponse.image:type_name -> docreader.ImageRef + 13, // 9: docreader.ListEnginesRequest.config_overrides:type_name -> docreader.ListEnginesRequest.ConfigOverridesEntry + 8, // 10: docreader.ListEnginesResponse.engines:type_name -> docreader.ParserEngineInfo + 1, // 11: docreader.DocReader.Read:input_type -> docreader.ReadRequest + 1, // 12: docreader.DocReader.ReadStream:input_type -> docreader.ReadRequest + 7, // 13: docreader.DocReader.ListEngines:input_type -> docreader.ListEnginesRequest + 4, // 14: docreader.DocReader.Read:output_type -> docreader.ReadResponse + 6, // 15: docreader.DocReader.ReadStream:output_type -> docreader.ReadStreamResponse + 9, // 16: docreader.DocReader.ListEngines:output_type -> docreader.ListEnginesResponse + 14, // [14:17] is the sub-list for method output_type + 11, // [11:14] is the sub-list for method input_type + 11, // [11:11] is the sub-list for extension type_name + 11, // [11:11] is the sub-list for extension extendee + 0, // [0:11] is the sub-list for field type_name } func init() { file_docreader_proto_init() } @@ -772,7 +862,7 @@ func file_docreader_proto_init() { if File_docreader_proto != nil { return } - file_docreader_proto_msgTypes[5].OneofWrappers = []any{ + file_docreader_proto_msgTypes[6].OneofWrappers = []any{ (*ReadStreamResponse_Meta)(nil), (*ReadStreamResponse_Image)(nil), } @@ -782,7 +872,7 @@ func file_docreader_proto_init() { GoPackagePath: reflect.TypeOf(x{}).PkgPath(), RawDescriptor: unsafe.Slice(unsafe.StringData(file_docreader_proto_rawDesc), len(file_docreader_proto_rawDesc)), NumEnums: 0, - NumMessages: 13, + NumMessages: 14, NumExtensions: 0, NumServices: 1, }, diff --git a/docreader/proto/docreader.proto b/docreader/proto/docreader.proto index f1f8a441..be6ab544 100644 --- a/docreader/proto/docreader.proto +++ b/docreader/proto/docreader.proto @@ -43,12 +43,24 @@ message ImageRef { bytes image_data = 5; // inline bytes fallback } +// SourceBlock ties a range of markdown_content to the place in the original +// file it was parsed from, so a citation can open the file there. +message SourceBlock { + // Unicode code point offsets into markdown_content, end exclusive. + uint32 start = 1; + uint32 end = 2; + // JSON-encoded locator: {"type":"pdf","page":3,"bbox":[x0,y0,x1,y1]} + // with ordinals 1-based and bbox as fractions of the page, origin top-left. + string locator_json = 3; +} + message ReadResponse { string markdown_content = 1; repeated ImageRef image_refs = 2; string image_dir_path = 3; map metadata = 4; string error = 5; + repeated SourceBlock source_blocks = 6; } // Metadata header for a streamed read. Sent exactly once, before any image. @@ -58,6 +70,7 @@ message ReadStreamMeta { map metadata = 3; string error = 4; uint32 image_count = 5; // best-effort total image count (0 if unknown) + repeated SourceBlock source_blocks = 6; } // One frame of a ReadStream. The first frame carries `meta`; every subsequent diff --git a/docreader/proto/docreader_grpc.pb.go b/docreader/proto/docreader_grpc.pb.go index abb5bbc8..6e20ba9d 100644 --- a/docreader/proto/docreader_grpc.pb.go +++ b/docreader/proto/docreader_grpc.pb.go @@ -1,7 +1,7 @@ // Code generated by protoc-gen-go-grpc. DO NOT EDIT. // versions: // - protoc-gen-go-grpc v1.5.1 -// - protoc v5.29.3 +// - protoc v6.33.4 // source: docreader.proto package proto diff --git a/docreader/proto/docreader_pb2.py b/docreader/proto/docreader_pb2.py index 8e3d16e0..7a92fad0 100644 --- a/docreader/proto/docreader_pb2.py +++ b/docreader/proto/docreader_pb2.py @@ -24,7 +24,7 @@ _sym_db = _symbol_database.Default() -DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0f\x64ocreader.proto\x12\tdocreader\"\xba\x01\n\nReadConfig\x12\x15\n\rparser_engine\x18\x01 \x01(\t\x12Q\n\x17parser_engine_overrides\x18\x02 \x03(\x0b\x32\x30.docreader.ReadConfig.ParserEngineOverridesEntry\x1a<\n\x1aParserEngineOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01J\x04\x08\x03\x10\x04\"\xa0\x01\n\x0bReadRequest\x12\x14\n\x0c\x66ile_content\x18\x01 \x01(\x0c\x12\x11\n\tfile_name\x18\x02 \x01(\t\x12\x11\n\tfile_type\x18\x03 \x01(\t\x12\x0b\n\x03url\x18\x04 \x01(\t\x12\r\n\x05title\x18\x05 \x01(\t\x12%\n\x06\x63onfig\x18\x06 \x01(\x0b\x32\x15.docreader.ReadConfig\x12\x12\n\nrequest_id\x18\x07 \x01(\t\"n\n\x08ImageRef\x12\x10\n\x08\x66ilename\x18\x01 \x01(\t\x12\x14\n\x0coriginal_ref\x18\x02 \x01(\t\x12\x11\n\tmime_type\x18\x03 \x01(\t\x12\x13\n\x0bstorage_key\x18\x04 \x01(\t\x12\x12\n\nimage_data\x18\x05 \x01(\x0c\"\xe2\x01\n\x0cReadResponse\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\'\n\nimage_refs\x18\x02 \x03(\x0b\x32\x13.docreader.ImageRef\x12\x16\n\x0eimage_dir_path\x18\x03 \x01(\t\x12\x37\n\x08metadata\x18\x04 \x03(\x0b\x32%.docreader.ReadResponse.MetadataEntry\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\xd2\x01\n\x0eReadStreamMeta\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\x16\n\x0eimage_dir_path\x18\x02 \x01(\t\x12\x39\n\x08metadata\x18\x03 \x03(\x0b\x32\'.docreader.ReadStreamMeta.MetadataEntry\x12\r\n\x05\x65rror\x18\x04 \x01(\t\x12\x13\n\x0bimage_count\x18\x05 \x01(\r\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"p\n\x12ReadStreamResponse\x12)\n\x04meta\x18\x01 \x01(\x0b\x32\x19.docreader.ReadStreamMetaH\x00\x12$\n\x05image\x18\x02 \x01(\x0b\x32\x13.docreader.ImageRefH\x00\x42\t\n\x07payload\"\x9a\x01\n\x12ListEnginesRequest\x12L\n\x10\x63onfig_overrides\x18\x01 \x03(\x0b\x32\x32.docreader.ListEnginesRequest.ConfigOverridesEntry\x1a\x36\n\x14\x43onfigOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"x\n\x10ParserEngineInfo\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x12\n\nfile_types\x18\x03 \x03(\t\x12\x11\n\tavailable\x18\x04 \x01(\x08\x12\x1a\n\x12unavailable_reason\x18\x05 \x01(\t\"C\n\x13ListEnginesResponse\x12,\n\x07\x65ngines\x18\x01 \x03(\x0b\x32\x1b.docreader.ParserEngineInfo2\xdf\x01\n\tDocReader\x12\x39\n\x04Read\x12\x16.docreader.ReadRequest\x1a\x17.docreader.ReadResponse\"\x00\x12G\n\nReadStream\x12\x16.docreader.ReadRequest\x1a\x1d.docreader.ReadStreamResponse\"\x00\x30\x01\x12N\n\x0bListEngines\x12\x1d.docreader.ListEnginesRequest\x1a\x1e.docreader.ListEnginesResponse\"\x00\x42\x35Z3github.com/Tencent/WeKnora/internal/docreader/protob\x06proto3') +DESCRIPTOR = _descriptor_pool.Default().AddSerializedFile(b'\n\x0f\x64ocreader.proto\x12\tdocreader\"\xba\x01\n\nReadConfig\x12\x15\n\rparser_engine\x18\x01 \x01(\t\x12Q\n\x17parser_engine_overrides\x18\x02 \x03(\x0b\x32\x30.docreader.ReadConfig.ParserEngineOverridesEntry\x1a<\n\x1aParserEngineOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01J\x04\x08\x03\x10\x04\"\xa0\x01\n\x0bReadRequest\x12\x14\n\x0c\x66ile_content\x18\x01 \x01(\x0c\x12\x11\n\tfile_name\x18\x02 \x01(\t\x12\x11\n\tfile_type\x18\x03 \x01(\t\x12\x0b\n\x03url\x18\x04 \x01(\t\x12\r\n\x05title\x18\x05 \x01(\t\x12%\n\x06\x63onfig\x18\x06 \x01(\x0b\x32\x15.docreader.ReadConfig\x12\x12\n\nrequest_id\x18\x07 \x01(\t\"n\n\x08ImageRef\x12\x10\n\x08\x66ilename\x18\x01 \x01(\t\x12\x14\n\x0coriginal_ref\x18\x02 \x01(\t\x12\x11\n\tmime_type\x18\x03 \x01(\t\x12\x13\n\x0bstorage_key\x18\x04 \x01(\t\x12\x12\n\nimage_data\x18\x05 \x01(\x0c\"?\n\x0bSourceBlock\x12\r\n\x05start\x18\x01 \x01(\r\x12\x0b\n\x03\x65nd\x18\x02 \x01(\r\x12\x14\n\x0clocator_json\x18\x03 \x01(\t\"\x91\x02\n\x0cReadResponse\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\'\n\nimage_refs\x18\x02 \x03(\x0b\x32\x13.docreader.ImageRef\x12\x16\n\x0eimage_dir_path\x18\x03 \x01(\t\x12\x37\n\x08metadata\x18\x04 \x03(\x0b\x32%.docreader.ReadResponse.MetadataEntry\x12\r\n\x05\x65rror\x18\x05 \x01(\t\x12-\n\rsource_blocks\x18\x06 \x03(\x0b\x32\x16.docreader.SourceBlock\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"\x81\x02\n\x0eReadStreamMeta\x12\x18\n\x10markdown_content\x18\x01 \x01(\t\x12\x16\n\x0eimage_dir_path\x18\x02 \x01(\t\x12\x39\n\x08metadata\x18\x03 \x03(\x0b\x32\'.docreader.ReadStreamMeta.MetadataEntry\x12\r\n\x05\x65rror\x18\x04 \x01(\t\x12\x13\n\x0bimage_count\x18\x05 \x01(\r\x12-\n\rsource_blocks\x18\x06 \x03(\x0b\x32\x16.docreader.SourceBlock\x1a/\n\rMetadataEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"p\n\x12ReadStreamResponse\x12)\n\x04meta\x18\x01 \x01(\x0b\x32\x19.docreader.ReadStreamMetaH\x00\x12$\n\x05image\x18\x02 \x01(\x0b\x32\x13.docreader.ImageRefH\x00\x42\t\n\x07payload\"\x9a\x01\n\x12ListEnginesRequest\x12L\n\x10\x63onfig_overrides\x18\x01 \x03(\x0b\x32\x32.docreader.ListEnginesRequest.ConfigOverridesEntry\x1a\x36\n\x14\x43onfigOverridesEntry\x12\x0b\n\x03key\x18\x01 \x01(\t\x12\r\n\x05value\x18\x02 \x01(\t:\x02\x38\x01\"x\n\x10ParserEngineInfo\x12\x0c\n\x04name\x18\x01 \x01(\t\x12\x13\n\x0b\x64\x65scription\x18\x02 \x01(\t\x12\x12\n\nfile_types\x18\x03 \x03(\t\x12\x11\n\tavailable\x18\x04 \x01(\x08\x12\x1a\n\x12unavailable_reason\x18\x05 \x01(\t\"C\n\x13ListEnginesResponse\x12,\n\x07\x65ngines\x18\x01 \x03(\x0b\x32\x1b.docreader.ParserEngineInfo2\xdf\x01\n\tDocReader\x12\x39\n\x04Read\x12\x16.docreader.ReadRequest\x1a\x17.docreader.ReadResponse\"\x00\x12G\n\nReadStream\x12\x16.docreader.ReadRequest\x1a\x1d.docreader.ReadStreamResponse\"\x00\x30\x01\x12N\n\x0bListEngines\x12\x1d.docreader.ListEnginesRequest\x1a\x1e.docreader.ListEnginesResponse\"\x00\x42\x35Z3github.com/Tencent/WeKnora/internal/docreader/protob\x06proto3') _globals = globals() _builder.BuildMessageAndEnumDescriptors(DESCRIPTOR, _globals) @@ -48,24 +48,26 @@ if not _descriptor._USE_C_DESCRIPTORS: _globals['_READREQUEST']._serialized_end=380 _globals['_IMAGEREF']._serialized_start=382 _globals['_IMAGEREF']._serialized_end=492 - _globals['_READRESPONSE']._serialized_start=495 - _globals['_READRESPONSE']._serialized_end=721 - _globals['_READRESPONSE_METADATAENTRY']._serialized_start=674 - _globals['_READRESPONSE_METADATAENTRY']._serialized_end=721 - _globals['_READSTREAMMETA']._serialized_start=724 - _globals['_READSTREAMMETA']._serialized_end=934 - _globals['_READSTREAMMETA_METADATAENTRY']._serialized_start=674 - _globals['_READSTREAMMETA_METADATAENTRY']._serialized_end=721 - _globals['_READSTREAMRESPONSE']._serialized_start=936 - _globals['_READSTREAMRESPONSE']._serialized_end=1048 - _globals['_LISTENGINESREQUEST']._serialized_start=1051 - _globals['_LISTENGINESREQUEST']._serialized_end=1205 - _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_start=1151 - _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_end=1205 - _globals['_PARSERENGINEINFO']._serialized_start=1207 - _globals['_PARSERENGINEINFO']._serialized_end=1327 - _globals['_LISTENGINESRESPONSE']._serialized_start=1329 - _globals['_LISTENGINESRESPONSE']._serialized_end=1396 - _globals['_DOCREADER']._serialized_start=1399 - _globals['_DOCREADER']._serialized_end=1622 + _globals['_SOURCEBLOCK']._serialized_start=494 + _globals['_SOURCEBLOCK']._serialized_end=557 + _globals['_READRESPONSE']._serialized_start=560 + _globals['_READRESPONSE']._serialized_end=833 + _globals['_READRESPONSE_METADATAENTRY']._serialized_start=786 + _globals['_READRESPONSE_METADATAENTRY']._serialized_end=833 + _globals['_READSTREAMMETA']._serialized_start=836 + _globals['_READSTREAMMETA']._serialized_end=1093 + _globals['_READSTREAMMETA_METADATAENTRY']._serialized_start=786 + _globals['_READSTREAMMETA_METADATAENTRY']._serialized_end=833 + _globals['_READSTREAMRESPONSE']._serialized_start=1095 + _globals['_READSTREAMRESPONSE']._serialized_end=1207 + _globals['_LISTENGINESREQUEST']._serialized_start=1210 + _globals['_LISTENGINESREQUEST']._serialized_end=1364 + _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_start=1310 + _globals['_LISTENGINESREQUEST_CONFIGOVERRIDESENTRY']._serialized_end=1364 + _globals['_PARSERENGINEINFO']._serialized_start=1366 + _globals['_PARSERENGINEINFO']._serialized_end=1486 + _globals['_LISTENGINESRESPONSE']._serialized_start=1488 + _globals['_LISTENGINESRESPONSE']._serialized_end=1555 + _globals['_DOCREADER']._serialized_start=1558 + _globals['_DOCREADER']._serialized_end=1781 # @@protoc_insertion_point(module_scope) diff --git a/docreader/proto/docreader_pb2.pyi b/docreader/proto/docreader_pb2.pyi index e10b4e53..0f16dc0d 100644 --- a/docreader/proto/docreader_pb2.pyi +++ b/docreader/proto/docreader_pb2.pyi @@ -53,8 +53,18 @@ class ImageRef(_message.Message): image_data: bytes def __init__(self, filename: _Optional[str] = ..., original_ref: _Optional[str] = ..., mime_type: _Optional[str] = ..., storage_key: _Optional[str] = ..., image_data: _Optional[bytes] = ...) -> None: ... +class SourceBlock(_message.Message): + __slots__ = ("start", "end", "locator_json") + START_FIELD_NUMBER: _ClassVar[int] + END_FIELD_NUMBER: _ClassVar[int] + LOCATOR_JSON_FIELD_NUMBER: _ClassVar[int] + start: int + end: int + locator_json: str + def __init__(self, start: _Optional[int] = ..., end: _Optional[int] = ..., locator_json: _Optional[str] = ...) -> None: ... + class ReadResponse(_message.Message): - __slots__ = ("markdown_content", "image_refs", "image_dir_path", "metadata", "error") + __slots__ = ("markdown_content", "image_refs", "image_dir_path", "metadata", "error", "source_blocks") class MetadataEntry(_message.Message): __slots__ = ("key", "value") KEY_FIELD_NUMBER: _ClassVar[int] @@ -67,15 +77,17 @@ class ReadResponse(_message.Message): IMAGE_DIR_PATH_FIELD_NUMBER: _ClassVar[int] METADATA_FIELD_NUMBER: _ClassVar[int] ERROR_FIELD_NUMBER: _ClassVar[int] + SOURCE_BLOCKS_FIELD_NUMBER: _ClassVar[int] markdown_content: str image_refs: _containers.RepeatedCompositeFieldContainer[ImageRef] image_dir_path: str metadata: _containers.ScalarMap[str, str] error: str - def __init__(self, markdown_content: _Optional[str] = ..., image_refs: _Optional[_Iterable[_Union[ImageRef, _Mapping]]] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ...) -> None: ... + source_blocks: _containers.RepeatedCompositeFieldContainer[SourceBlock] + def __init__(self, markdown_content: _Optional[str] = ..., image_refs: _Optional[_Iterable[_Union[ImageRef, _Mapping]]] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ..., source_blocks: _Optional[_Iterable[_Union[SourceBlock, _Mapping]]] = ...) -> None: ... class ReadStreamMeta(_message.Message): - __slots__ = ("markdown_content", "image_dir_path", "metadata", "error", "image_count") + __slots__ = ("markdown_content", "image_dir_path", "metadata", "error", "image_count", "source_blocks") class MetadataEntry(_message.Message): __slots__ = ("key", "value") KEY_FIELD_NUMBER: _ClassVar[int] @@ -88,12 +100,14 @@ class ReadStreamMeta(_message.Message): METADATA_FIELD_NUMBER: _ClassVar[int] ERROR_FIELD_NUMBER: _ClassVar[int] IMAGE_COUNT_FIELD_NUMBER: _ClassVar[int] + SOURCE_BLOCKS_FIELD_NUMBER: _ClassVar[int] markdown_content: str image_dir_path: str metadata: _containers.ScalarMap[str, str] error: str image_count: int - def __init__(self, markdown_content: _Optional[str] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ..., image_count: _Optional[int] = ...) -> None: ... + source_blocks: _containers.RepeatedCompositeFieldContainer[SourceBlock] + def __init__(self, markdown_content: _Optional[str] = ..., image_dir_path: _Optional[str] = ..., metadata: _Optional[_Mapping[str, str]] = ..., error: _Optional[str] = ..., image_count: _Optional[int] = ..., source_blocks: _Optional[_Iterable[_Union[SourceBlock, _Mapping]]] = ...) -> None: ... class ReadStreamResponse(_message.Message): __slots__ = ("meta", "image") diff --git a/docreader/source_wire.py b/docreader/source_wire.py new file mode 100644 index 00000000..8166b1b2 --- /dev/null +++ b/docreader/source_wire.py @@ -0,0 +1,34 @@ +"""Wire form of source blocks (see docreader.parser.source_locator). + +Kept apart from docreader.main so it can be imported without the parser +registry. +""" + +import json + +from docreader.proto.docreader_pb2 import SourceBlock + + +def source_blocks_to_proto(document) -> list: + """Convert a document's source blocks to their wire form. + + Malformed entries are skipped: positions are an optional extra and must + never fail a parse. + """ + out = [] + for block in getattr(document, "source_blocks", None) or []: + try: + start, end = int(block["start"]), int(block["end"]) + locator = block["locator"] + if end <= start or start < 0 or not locator.get("type"): + continue + out.append( + SourceBlock( + start=start, + end=end, + locator_json=json.dumps(locator, ensure_ascii=False), + ) + ) + except (KeyError, TypeError, ValueError, AttributeError): + continue + return out diff --git a/docreader/tests/test_excel_parser.py b/docreader/tests/test_excel_parser.py index 682613ff..abc0fcd7 100644 --- a/docreader/tests/test_excel_parser.py +++ b/docreader/tests/test_excel_parser.py @@ -511,3 +511,26 @@ class ExcelParserTest(unittest.TestCase): if __name__ == "__main__": unittest.main() + + +class ExcelSourceBlocksTest(unittest.TestCase): + def test_rows_carry_sheet_and_row_number(self): + import io + + import openpyxl + + from docreader.parser.excel_parser import ExcelParser + + wb = openpyxl.Workbook() + ws = wb.active + ws.title = "明细" + ws["B3"] = "螺栓" + ws["A5"] = "合计" + buf = io.BytesIO() + wb.save(buf) + + doc = ExcelParser(file_name="a.xlsx", file_type="xlsx").parse_into_text(buf.getvalue()) + rows = [(b["locator"]["sheet"], b["locator"]["row_start"]) for b in doc.source_blocks] + self.assertEqual(rows, [("明细", 3), ("明细", 5)]) + first = doc.source_blocks[0] + self.assertEqual(doc.content[first["start"]:first["end"]], "B: 螺栓\n") diff --git a/docreader/tests/test_source_locator.py b/docreader/tests/test_source_locator.py new file mode 100644 index 00000000..f84cf2f4 --- /dev/null +++ b/docreader/tests/test_source_locator.py @@ -0,0 +1,143 @@ +import json +import unittest + +from docreader.models.document import Document +from docreader.parser.source_locator import ( + PageGeometry, + locate_lines, + page_blocks, +) + + +def _ln(text, x0, y0, x1, y1): + return {"text": text, "bbox": (x0, y0, x1, y1)} + + +class LocateLinesTest(unittest.TestCase): + def test_finds_lines_in_order_and_pulls_in_leading_markup(self): + text = "## 标题\n【版权声明】\n正文 第一行\n正文 第一行" + hits = locate_lines(text, ["标题", "版权声明】", "正文第一行", "正文 第一行", "无此行"]) + self.assertEqual(hits[0], 0) + self.assertEqual(text[hits[1]:hits[1] + 2], "【版") + self.assertEqual(hits[2], text.index("正文")) + self.assertEqual(hits[3], text.rindex("正文")) + self.assertIsNone(hits[4]) + + +class PageGeometryTest(unittest.TestCase): + def test_flips_to_top_left_origin(self): + g = PageGeometry((0, 0, 200, 100)) + self.assertEqual(g.bbox((20, 80, 100, 90)), [0.1, 0.1, 0.5, 0.2]) + + def test_applies_cropbox_offset_and_rotation(self): + g = PageGeometry((100, 100, 300, 200), rotation=90) + # Unrotated [0.1, 0.1, 0.5, 0.2] turns clockwise. + self.assertEqual(g.bbox((120, 180, 200, 190)), [0.8, 0.1, 0.9, 0.5]) + + +class PageBlocksTest(unittest.TestCase): + def test_groups_lines_into_boxed_paragraphs(self): + page = ( + "8.14.11 索夹与吊索施工应符合下列规定:\n" + "1 在满足施工需要的前提下,应减小猫道面层开孔面积,并应在开孔位置四周绑\n" + "扎防滑木条,设立警示标志。\n" + "2 索夹在主缆上定位后,应紧固螺栓。" + ) + lines = [ + _ln("8.14.11 索夹与吊索施工应符合下列规定:", 50, 700, 300, 712), + _ln("1 在满足施工需要的前提下,应减小猫道面层开孔面积,并应在开孔位置四周绑", 60, 684, 540, 696), + _ln("扎防滑木条,设立警示标志。", 50, 668, 200, 680), + _ln("2 索夹在主缆上定位后,应紧固螺栓。", 60, 652, 400, 664), + ] + blocks = page_blocks(page, 10, 57, lines, PageGeometry((0, 0, 600, 800))) + self.assertEqual(len(blocks), 3) + self.assertEqual(blocks[0]["start"], 10) + item1 = blocks[1] + self.assertEqual(page[item1["start"] - 10:item1["end"] - 10].split("\n")[0][:1], "1") + self.assertIn("设立警示标志。", page[item1["start"] - 10:item1["end"] - 10]) + self.assertEqual(item1["locator"]["page"], 57) + x0, y0, x1, y1 = item1["locator"]["bbox"] + self.assertAlmostEqual(y0, 1 - 696 / 800, places=3) + self.assertAlmostEqual(y1, 1 - 668 / 800, places=3) + self.assertEqual(blocks[-1]["end"], 10 + len(page)) + + def test_page_without_lines_gets_page_block(self): + blocks = page_blocks("文字", 5, 2, [], None) + self.assertEqual(blocks, [{"start": 5, "end": 7, "locator": {"type": "pdf", "page": 2}}]) + + +class WireTest(unittest.TestCase): + def test_source_blocks_serialize_and_skip_malformed(self): + from docreader.source_wire import source_blocks_to_proto + + doc = Document( + content="abc", + source_blocks=[ + {"start": 0, "end": 3, "locator": {"type": "pdf", "page": 1}}, + {"start": 2, "end": 2, "locator": {"type": "pdf", "page": 1}}, + {"start": 0, "end": 1}, + ], + ) + wire = source_blocks_to_proto(doc) + self.assertEqual(len(wire), 1) + self.assertEqual(json.loads(wire[0].locator_json), {"type": "pdf", "page": 1}) + + +class LocateColumnsTest(unittest.TestCase): + def test_repeated_line_follows_its_own_column(self): + text = ( + "alpha left first line\nsame tail phrase here\n" + "bravo left third line\nsame tail phrase here\n" + "omega right first line\nsigma right second line" + ) + left, right = (0, 100), (120, 220) + lines = [ + ("alpha left first line", left, 700), + ("omega right first line", right, 700), + ("same tail phrase here", left, 680), + ("sigma right second line", right, 680), + ("bravo left third line", left, 660), + ("same tail phrase here", left, 640), + ] + hits = locate_lines( + text, + [t for t, _, _ in lines], + [(x[0], y, x[1], y + 10) for _, x, y in lines], + ) + self.assertEqual(hits[2], text.index("same tail")) + self.assertEqual(hits[5], text.rindex("same tail")) + + def test_short_repeated_label_is_not_placed(self): + text = "Figure shows the LLM pipeline\nThe LLM is prompted twice" + hits = locate_lines(text, ["LLM", "The LLM is prompted twice"]) + self.assertIsNone(hits[0]) + self.assertEqual(hits[1], text.index("The LLM")) + + +class SplitLineAtGapsTest(unittest.TestCase): + @staticmethod + def _line(chars): + glyphs = [ + {"ch": ch, "x0": x, "x1": x + 5, "y0": 0.0, "y1": 10.0, "i": i} + for ch, x, i in chars + ] + return {"text": "".join(c for c, _, _ in chars), "bbox": (0, 0, 0, 0), "chars": glyphs} + + def test_cuts_fused_columns_at_a_stream_jump(self): + from docreader.parser.pdf_parser import _split_line_at_gaps + + left = [(ch, 10 + 5 * k, 100 + k) for k, ch in enumerate("left side")] + right = [(ch, 80 + 5 * k, 900 + k) for k, ch in enumerate("right side")] + parts = _split_line_at_gaps(self._line(left + right)) + self.assertEqual([p["bbox"][0] for p in parts], [10, 80]) + + def test_keeps_a_wide_word_space_within_one_column(self): + from docreader.parser.pdf_parser import _split_line_at_gaps + + chars = [(ch, 10 + 5 * k, k) for k, ch in enumerate("wide")] + chars += [(ch, 60 + 5 * k, 5 + k) for k, ch in enumerate("space")] + self.assertEqual(len(_split_line_at_gaps(self._line(chars))), 1) + + +if __name__ == "__main__": + unittest.main() diff --git a/frontend/package-lock.json b/frontend/package-lock.json index bdbe9e17..0f70a6ed 100644 --- a/frontend/package-lock.json +++ b/frontend/package-lock.json @@ -26,6 +26,7 @@ "mermaid": "^11.15.0", "pagefind": "^1.5.2", "papaparse": "^5.5.3", + "pdfjs-dist": "^6.3.289", "pinia": "^3.0.4", "swiper": "^12.1.4", "tdesign-icons-vue-next": "0.4.4", @@ -1061,6 +1062,256 @@ "resolved": "https://mirrors.tencent.com/npm/@microsoft/fetch-event-source/-/fetch-event-source-2.0.1.tgz", "integrity": "sha512-W6CLUJ2eBMw3Rec70qrsEW0jOm/3twwJv21mrmj2yORiaVmVYGS4sSS5yUwvQc1ZlDLYGPnClVWmUUMagKNsfA==" }, + "node_modules/@napi-rs/canvas": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas/-/canvas-1.0.9.tgz", + "integrity": "sha512-QviPdJImDi/jMAvBfqaw+19BndMd/sizXVW3NnpMd3VJGz++QXkOHcP9kWR/smHG0hNjHeyuHFyrx/5lD0oNcQ==", + "license": "MIT", + "optional": true, + "workspaces": [ + "e2e/*" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + }, + "optionalDependencies": { + "@napi-rs/canvas-android-arm64": "1.0.9", + "@napi-rs/canvas-darwin-arm64": "1.0.9", + "@napi-rs/canvas-darwin-x64": "1.0.9", + "@napi-rs/canvas-linux-arm-gnueabihf": "1.0.9", + "@napi-rs/canvas-linux-arm64-gnu": "1.0.9", + "@napi-rs/canvas-linux-arm64-musl": "1.0.9", + "@napi-rs/canvas-linux-riscv64-gnu": "1.0.9", + "@napi-rs/canvas-linux-x64-gnu": "1.0.9", + "@napi-rs/canvas-linux-x64-musl": "1.0.9", + "@napi-rs/canvas-win32-arm64-msvc": "1.0.9", + "@napi-rs/canvas-win32-x64-msvc": "1.0.9" + } + }, + "node_modules/@napi-rs/canvas-android-arm64": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-android-arm64/-/canvas-android-arm64-1.0.9.tgz", + "integrity": "sha512-4LGXk2/0HVzE29K8SzML5WubgCp++B1FH3qgl35XmSZE+lLdr6P9VRQEnZ0MCLZMTSuJP41yyhtoiVNEuvrTIA==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "android" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-darwin-arm64": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-darwin-arm64/-/canvas-darwin-arm64-1.0.9.tgz", + "integrity": "sha512-YNdfLBzY0W/Pep9fo2L6RmoNlNksnn05LRnX66W63R3ij58S25QOTcjdtEt2v8+PnCESzqZsYzUo+QPeIR44NA==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-darwin-x64": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-darwin-x64/-/canvas-darwin-x64-1.0.9.tgz", + "integrity": "sha512-ceZQSknTEcy3dOXoekv59LTCkXjvnLsq+VW5PeNNDHEPQbRS5Ervkm1EaDa7WLAjiYWMLuSQTRTHao1dEX4prg==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "darwin" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm-gnueabihf": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-arm-gnueabihf/-/canvas-linux-arm-gnueabihf-1.0.9.tgz", + "integrity": "sha512-XhfI0Wwv4llhd6nnWDtY3kQKjq0r+y1i91PlJlJI24ag2U9WrnwbG1qS3+fDLEyouwEVFchiKkTEHozK+5iUNA==", + "cpu": [ + "arm" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm64-gnu": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-arm64-gnu/-/canvas-linux-arm64-gnu-1.0.9.tgz", + "integrity": "sha512-012oiYtKaE7i9oxc8q7nraT7kDOpLcaCmFLzVe9Ty34RHDdoDzbWLrVh827CNxYh/EADX1eSikA3ymLjo/nNuw==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-arm64-musl": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-arm64-musl/-/canvas-linux-arm64-musl-1.0.9.tgz", + "integrity": "sha512-Ls5UWYFFn63casTZEczbeyEg3vRDRkv9lscuGwfchtY5yLLQhgOB8SN4YGxmfJ5vTaBwZ2YUxBS3NtmjJmFXdA==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-riscv64-gnu": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-riscv64-gnu/-/canvas-linux-riscv64-gnu-1.0.9.tgz", + "integrity": "sha512-hLKEGxV7ZiRHqndePTokgDMdBlo/rDfzg7P4p4QIv9pUhuYobnu3R2NIFLCRghG0nwfo+s2sw+c1xZFeCmEAsw==", + "cpu": [ + "riscv64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-gnu": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-x64-gnu/-/canvas-linux-x64-gnu-1.0.9.tgz", + "integrity": "sha512-6kaz3w0QMy77PDWk6rJ1ksIihdad3qzEyX2o2oGT8GwCaypfT5mhjr8buOO5hstyLxcWXDScuz56RsINLtBPIQ==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-linux-x64-musl": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-linux-x64-musl/-/canvas-linux-x64-musl-1.0.9.tgz", + "integrity": "sha512-xrGvmS3v55hmZ86ls/kBLVNMUTYio3f6Ik0DireemG994VfPAwiA3ZXA0Uf1bByctkB3NQ1Sfb+H5bkdUnnzfQ==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "linux" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-arm64-msvc": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-win32-arm64-msvc/-/canvas-win32-arm64-msvc-1.0.9.tgz", + "integrity": "sha512-yjmVS3ArZeRVCP7jqbPq4rpZa/BhTeI7ELE2XqJg3snICQBDevLZyArxswHkiTnT34KRic33/4fLirrHI+SY8A==", + "cpu": [ + "arm64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, + "node_modules/@napi-rs/canvas-win32-x64-msvc": { + "version": "1.0.9", + "resolved": "https://mirrors.tencent.com/npm/@napi-rs/canvas-win32-x64-msvc/-/canvas-win32-x64-msvc-1.0.9.tgz", + "integrity": "sha512-QlSYQdMQslB81nlABo9wNfQ6npFhE7/O+saCZdqVGueGanyRk4jCogD5EwQenfP3kIq9e+mm6GreQBjX5MrA8g==", + "cpu": [ + "x64" + ], + "license": "MIT", + "optional": true, + "os": [ + "win32" + ], + "engines": { + "node": ">= 10" + }, + "funding": { + "type": "github", + "url": "https://github.com/sponsors/Brooooooklyn" + } + }, "node_modules/@novnc/novnc": { "version": "1.7.0", "resolved": "https://registry.npmjs.org/@novnc/novnc/-/novnc-1.7.0.tgz", @@ -4302,6 +4553,18 @@ "integrity": "sha512-WUjGcAqP1gQacoQe+OBJsFA7Ld4DyXuUIjZ5cc75cLHvJ7dtNsTugphxIADwspS+AraAUePCKrSVtPLFj/F88w==", "license": "MIT" }, + "node_modules/pdfjs-dist": { + "version": "6.3.289", + "resolved": "https://mirrors.tencent.com/npm/pdfjs-dist/-/pdfjs-dist-6.3.289.tgz", + "integrity": "sha512-ZHjSVpDa3D6izMq8/04lvkhkATUmL9px6ChPaXc1k6nU2Mrhlg1/7F0bdUqCwUjw3NsPTfPZsMDUU6ZIcRaeQw==", + "license": "Apache-2.0", + "engines": { + "node": ">=22.13.0 || >=24" + }, + "optionalDependencies": { + "@napi-rs/canvas": "^1.0.0" + } + }, "node_modules/perfect-debounce": { "version": "1.0.0", "resolved": "https://mirrors.tencent.com/npm/perfect-debounce/-/perfect-debounce-1.0.0.tgz", diff --git a/frontend/package.json b/frontend/package.json index 1b1b5354..ee38c482 100644 --- a/frontend/package.json +++ b/frontend/package.json @@ -34,6 +34,7 @@ "mermaid": "^11.15.0", "pagefind": "^1.5.2", "papaparse": "^5.5.3", + "pdfjs-dist": "^6.3.289", "pinia": "^3.0.4", "swiper": "^12.1.4", "tdesign-icons-vue-next": "0.4.4", diff --git a/frontend/src/assets/theme/theme.css b/frontend/src/assets/theme/theme.css index 51aa6541..aef4d07c 100644 --- a/frontend/src/assets/theme/theme.css +++ b/frontend/src/assets/theme/theme.css @@ -217,6 +217,14 @@ select:-webkit-autofill { --app-accent-purple: #9d7cff; } +/* 引用原文定位的高亮:荧光笔黄,和品牌绿的界面元素(表头、选中态)区分开 */ +:root { + --app-source-highlight: #ffd84d; +} +:root[theme-mode="dark"] { + --app-source-highlight: #b58f1c; +} + /* 全局旋转动画:spinner / 刷新图标统一用 `animation: wk-spin 0.8s linear infinite` */ @keyframes wk-spin { to { diff --git a/frontend/src/components/ChatReferenceSourceView.vue b/frontend/src/components/ChatReferenceSourceView.vue new file mode 100644 index 00000000..38c2931b --- /dev/null +++ b/frontend/src/components/ChatReferenceSourceView.vue @@ -0,0 +1,349 @@ + + + + + diff --git a/frontend/src/components/ChatReferencesDrawer.vue b/frontend/src/components/ChatReferencesDrawer.vue index 39b98279..b886d5ec 100644 --- a/frontend/src/components/ChatReferencesDrawer.vue +++ b/frontend/src/components/ChatReferencesDrawer.vue @@ -4,13 +4,25 @@