Omnipub

Overview

Omnipub is a universal document representation system that transforms diverse file formats into a unified, searchable graph structure. It accepts EPUB, PDF, Markdown, HTML, XML, and office documents, normalizing them into a common schema that enables consistent content retrieval regardless of source format.

The system addresses four fundamental challenges in document management. First, different formats store content and structure in incompatible ways, making cross-format search and navigation difficult. Second, traditional document storage lacks the semantic richness needed for modern AI-powered retrieval. Third, professional publishing workflows require tracking content through complex lifecycles from upload through processing, validation, and publication. Fourth, content licensing often varies at granular levels within documents, requiring block-level entitlement management and consumption tracking.

Omnipub solves these challenges by extracting content into a knowledge graph called the omnigraph—a hierarchical structure of typed blocks that preserves both content and its structural relationships. Navigation indexes link the graph to human-readable tables of contents. Vector embeddings enable semantic search alongside traditional keyword matching. Entity extraction identifies people, organizations, concepts, and citations within content, linking them to external knowledge graphs to enable relationship-aware discovery and ground AI retrieval in published sources. The result is a content system that serves both human readers navigating documents and AI systems retrieving relevant passages.

Schema

The data model centers on two primary entities that together form the omnigraph: the publication record and content blocks. A publication record represents a complete document with its metadata and processing state. Blocks form a hierarchical tree of content blocks within each publication.

Publication

A publication record stores document-level information including title, authors, publisher, and cover image in a flexible metadata structure. Each Omnibook tracks its processing status through eighteen distinct states from initial upload through parsing, validation, and publication. Parse version and chunking strategy fields record which algorithm versions produced the current representation, enabling selective reprocessing when algorithms improve.

The navigation structure links table-of-contents entries to their corresponding sections within the content tree. Each navigation item captures its hierarchical level, display label, anchor reference, and the identifier of its associated section block. This separation allows efficient navigation queries without loading the full content graph.

Block

Blocks form a tree using the materialized path pattern, where each node's path encodes its position in the hierarchy. This approach supports efficient subtree queries and ordering without recursive database operations. The block taxonomy covers the range of document content: root omnipub nodes, sections for navigation targets, text paragraphs with size variants, lists with items, tables with rows and cells, images, figures, mathematical expressions, and marker blocks for metadata anchors.

Each block carries properties appropriate to its type. Text blocks store their content string. Images store source references. Sections store their heading text. A separate format structure captures presentation information including text size, inline styles, and CSS classes. Word and token counts are computed and cached for sections and the entire document.

Entity

Entities represent identified concepts, people, organizations, and citations extracted from publication content. Each entity record captures the entity type, canonical name, and links to external knowledge graphs such as Wikidata, ORCID, or DOI registries. Mention spans connect entities to specific text locations within blocks, preserving the context in which each entity appears.

Entity types include persons (authors, historical figures, researchers), organizations (institutions, publishers, companies), concepts (technical terms, topics, methodologies), and citations (references to other publications). Citation entities carry structured metadata including title, authors, publication venue, and persistent identifiers that enable cross-publication linking.

Relationships between entities are stored in the graph database, capturing both explicit relationships stated in the text and inferred relationships derived from co-occurrence patterns and external knowledge graph data. These relationships enable traversal queries that discover connections across publications—finding all works by a researcher, tracing citation networks, or identifying publications discussing related concepts.

Entitlements

Block-level UUIDs enable granular rights management across document content. Different block types can carry different licensing terms within the same publication. For example, a publication may include text content under one license while embedded images or tables require separate rights agreements with their original creators or rights holders.

Entitlement rules can be defined at multiple granularities: per block type across all publications, per block type within a specific publication, or on individual blocks when specific content requires unique handling. This flexibility accommodates complex licensing scenarios common in professional publishing, where a single document may aggregate content from multiple sources with varying rights.

Access checks occur at retrieval time against the requesting user's entitlements. When content is requested, the system evaluates the user's permissions against the entitlement requirements of each block in the response. Blocks the user lacks rights to access can be filtered, replaced with placeholder content, or flagged for separate licensing. This per-block evaluation ensures that entitlement enforcement is precise rather than all-or-nothing at the document level.

Tracking and Analytics

Block-level UUIDs enable precise consumption tracking throughout the content lifecycle. Every content retrieval is logged with block identifiers, creating an auditable record of exactly what content was accessed, when, and by whom. This granularity supports both compliance requirements and business intelligence.

Compliance reporting draws on these consumption logs to demonstrate adherence to licensing terms. Publishers can verify that content was only accessed by authorized users, that access volumes stayed within licensed limits, and that time-restricted content was not accessed outside its availability window. Audit trails provide evidence for rights holder reporting and contractual compliance.

Usage-based billing models rely on accurate consumption data at the block level. Different pricing can apply to different content types—premium images or tables might carry higher per-access costs than standard text. Metered access models can count retrievals against quotas, with the block-level tracking ensuring accurate accounting even when users access partial documents.

Analytics aggregation supports business insights across multiple dimensions: by publication to understand document popularity, by section to identify which parts of documents engage users, by block type to understand consumption patterns across content categories, and by time period to track trends and seasonality. These aggregations inform content acquisition decisions, pricing strategies, and platform development priorities.

Databases

The system employs three complementary database technologies, each optimized for specific access patterns.

Relational Storage

The primary data store uses a relational database for persistent storage of all entities. Document metadata, content trees, user accounts, collections, and consumption records all reside here. The hierarchical content structure uses a materialized path encoding that supports efficient tree operations without recursive queries. JSON fields provide flexibility for variable metadata that differs across document formats.

Indexes optimize the most common query patterns: filtering documents by processing status, listing documents by publisher, and traversing content trees by path prefix. Read replicas distribute query load for high-traffic operations.

Graph Storage

A graph database mirrors the content hierarchy and stores derived relationships. Block vertices replicate the relational content tree, enabling graph traversal queries that would require multiple round-trips in relational storage. AI-generated summaries attach to content nodes as separate vertices, linked by contextual relationships. Entity vertices represent extracted people, organizations, concepts, and citations, with edges connecting entities to the blocks where they appear and to related entities in external knowledge graphs.

Graph queries excel at relationship-centric operations: finding all summaries for a subtree, traversing parent-child chains, discovering content connected through shared attributes, or navigating citation networks and entity relationships across publications.

Search Index

A dedicated search index provides full-text and semantic search capabilities. Document content is chunked into searchable segments, each preserving its position within the source hierarchy through section identifiers. Vector embeddings enable semantic similarity search alongside traditional keyword matching.

The index supports hybrid queries that combine keyword relevance with semantic similarity, weighted according to the use case. Collection filters restrict searches to authorized content. Recency functions can boost newer documents when freshness matters. Multiple embedding models accommodate different precision and performance trade-offs.

Interfaces

Content is accessible through three interfaces serving different use cases: web administration for publishers and operators, a REST API for programmatic integration, and the Model Context Protocol for AI assistants.

Web Administration

The administrative interface provides publishers with content management capabilities through a web browser. Users can view document metadata, monitor processing status, and inspect parsing results. Batch operations enable queuing documents for reprocessing, embedding generation, or validation testing. Error management tools display parsing issues and allow marking them as resolved or ignored.

Administrative actions are restricted based on user roles. Publishers manage their own content while operators can access cross-publisher views and system-wide operations.

REST API

The HTTP API enables programmatic access to all content operations. Publication endpoints support creating documents from uploaded files or URLs, listing documents with filtering and pagination, and retrieving document metadata. Section endpoints return content in multiple formats: rendered HTML for display, structured JSON for processing, or plain text for analysis.

Search endpoints provide both keyword and semantic search across document content, with filters for collections, specific documents, or external identifiers. Collection endpoints enable organizing documents into groups for access control and discovery.

Authentication uses API keys with configurable rate limits. Consumption is tracked per request for usage reporting and billing.

Model Context Protocol

The MCP interface exposes content operations as tools that AI assistants can invoke. Search, list, and retrieval tools mirror the REST API capabilities in a format optimized for language model interaction. Tool descriptions guide the AI in selecting appropriate operations.

Authentication supports both OAuth flows for interactive sessions and API keys for automated integrations. Usage reporting tools provide consumption summaries filtered by date range, license type, or external user identifiers. The interface is read-only by design, ensuring AI assistants can discover and retrieve content without modifying it.

How the Omnigraph is Constructed

Overview

The omnigraph construction follows a multi-stage pipeline that transforms raw documents into a structured graph representation. The process moves from source file to HTML, then to sectioned HTML with custom section tags, and finally to the omnigraph data structure that gets persisted to the database.

File Reading and HTML Generation

Each parser subclass reads its specific source format and converts it to HTML. EPUB parsers combine all document items into a single HTML body. PDF parsers extract content as HTML using text extraction tools. Markdown parsers convert markdown syntax to HTML. Webpage parsers read HTML directly from the source file.

The result is a unified HTML representation stored as both raw text and a parsed object for manipulation.

Each parser creates navigation items from the document's table of contents or heading structure. EPUB files extract navigation from their built-in table of contents. PDFs derive navigation from PDF outlines or by inferring structure from heading font sizes. Markdown and webpage files generate navigation directly from heading tags.

Navigation items capture the label, anchor identifier, nesting level, sequential order, and reference links for each section.

HTML Sectioning

The HTML is restructured by wrapping content into custom section tags using the Closest Common Ancestor algorithm. This algorithm handles irregular HTML structures where headings are not direct siblings by finding the common ancestor between consecutive headings to determine correct content boundaries.

For each navigation item, the algorithm locates the heading element, identifies the next same-or-higher-level heading, computes the closest common ancestor between them, creates a section wrapper with navigation attributes, and pulls all content from the current heading to the boundary into that section.

The result is a hierarchically nested HTML structure where each section contains its heading and all associated content.

Omnigraph Construction

The sectioned HTML is converted to the omnigraph data structure. Styles are extracted and preserved while style and script tags are removed from the content. The HTML is then transformed into graph nodes called blocks, with relationships established between blocks to represent the parent-child hierarchy. Element properties, text content, and styling information are preserved on each block.

Finally, navigation items are linked to their corresponding section blocks by matching identifiers.

Entity Extraction

After the omnigraph structure is established, entity extraction identifies significant concepts, people, organizations, and citations within the content. Named entity recognition models process text blocks to locate entity mentions, while citation parsers identify references to other publications.

Each extracted entity is resolved against external knowledge graphs. Person entities link to ORCID identifiers where available. Organization entities link to institutional identifiers. Concept entities link to domain ontologies and Wikidata entries. Citation entities resolve to DOI records, enabling cross-publication linking.

Mention spans record the exact text locations where each entity appears, preserving context for downstream applications. Relationship extraction identifies explicit connections stated in the text—authorship, affiliation, citation—as well as implicit relationships inferred from co-occurrence patterns.

Database Persistence

The omnigraph and extracted entities are persisted to the database. Section counts and node paths are computed as derived properties. All blocks are bulk-created as records in the relational database, while entity vertices and their relationships are written to the graph database for traversal queries.

Subclass Variations

EPUB parsing includes an additional flattening step before sectioning to handle the complex structure of EPUB documents. It also tracks EPUB item identifiers on sections for content validation during testing.

PDF parsing includes fallback logic to switch between extraction methods if the primary extraction fails. Error tracking captures when this fallback occurs.

Markdown and webpage parsing follow a simpler flow since their HTML structure is already well-formed. These parsers skip the validation step as it is not applicable to their format.

Quality, Coverage, and Updates

Quality assurance operates through a multi-layer validation approach. Section content validation enforces minimum content thresholds to ensure meaningful sections. Navigation structure validation requires at least one top-level item to confirm the document has usable navigation. Partition validation checks backmatter ratios to detect documents where supplementary material dominates the actual content. Errors encountered during parsing are tracked with resolution statuses that distinguish between new issues, ignored issues, and resolved issues. Navigation quality varies by source, with document-defined navigation being highest quality, heading-based navigation being derived quality, and fallback navigation being generated when no structure exists.

Format coverage spans EPUB, PDF, Markdown, Webpage, XML, and a universal handler that processes office documents through conversion. All formats normalize to HTML through a universal conversion pipeline, which enables consistent downstream processing regardless of source format. Navigation extraction varies by format, drawing from embedded tables of contents, document outlines, or heading hierarchies depending on what each format provides. Metadata extraction similarly adapts to format-specific sources, pulling title, author, and other properties from wherever each format stores them.

Update frequency is managed through semantic versioning of the parsing algorithm. Changes to parsing logic increment the version number, allowing identification of which algorithm version produced any given omnigraph. The status lifecycle tracks content through eighteen distinct states from initial upload through parsing, validation, and publication. Reparse capability allows updated parsing logic to be applied to existing content, regenerating the omnigraph with improvements while preserving the source file. Version history is maintained for audit purposes, recording when content was parsed and with which algorithm version.

Performance and Integration

Hierarchical navigation structures for document retrieval have demonstrated strong performance on industry benchmarks. PageIndex, an open-source project at github.com/VectifyAI/PageIndex, implements a similar architecture using tree-structured document indexes for retrieval. Systems using this navigation-based approach have achieved 98.7% accuracy on FinanceBench, a benchmark for question-answering over financial documents. The key insight from these benchmarks is that navigation-based retrieval outperforms pure vector similarity for complex documents where semantic similarity alone fails to capture contextual relevance.

The BEIR benchmark at github.com/beir-cellar/beir evaluates retrieval models across diverse domains and demonstrates the complementary strengths of different search approaches. Modern embedding models now surpass traditional keyword search in aggregate performance, with particular advantages on semantically challenging domains like financial and scientific question-answering. However, keyword search remains superior for exact phrase matching and certain specialized domains. Hybrid approaches combining both methods show the strongest results, with semantic reranking improving keyword search performance by approximately 39% across benchmark suites. In practice, dual-pathway retrieval has reduced hallucinations by 18% in biomedical applications and improved multi-hop question-answering recall by over six points compared to single-method baselines.

Knowledge graph architectures provide additional performance advantages for complex retrieval tasks. Research comparing graph-based RAG to traditional vector-only approaches demonstrates substantial improvements on multi-hop reasoning: GraphRAG methods achieve higher F1 scores on benchmarks like HotPotQA that require connecting information across multiple passages. Industry evaluations report accuracy improvements ranging from 35% to 3x on enterprise question-answering tasks, with knowledge graphs sustaining stable performance even as query complexity increases beyond five entities—a threshold where pure vector approaches show significant degradation. Hybrid strategies that route fact-based queries to vector retrieval and reasoning-based queries to graph retrieval show further gains, suggesting the omnigraph's hierarchical structure complements rather than replaces semantic search.

The system supports multiple integration approaches to accommodate different use cases. Text search enables keyword-based discovery across titles, authors, section labels, and content. Semantic search uses vector embeddings to find conceptually related passages regardless of exact terminology. Hybrid search combines both approaches, weighting text matches against semantic similarity to balance precision and recall. For agentic integrations, the hierarchical section structure enables navigation-based retrieval where systems traverse the document tree to locate contextually relevant sections through reasoning rather than similarity matching. Each chunk preserves its position within the document hierarchy through section identifiers, enabling retrieval systems to expand or refine results by traversing to parent or sibling sections.

Entity extraction provides particular advantages for academic and research content. Citation linking connects publications through their references, enabling AI systems to trace claims back to primary sources and verify assertions against the published literature. When an agentic system retrieves a passage, it can follow citation links to assess source credibility, find supporting evidence, or discover contradictory findings. This grounding in published research reduces hallucination by anchoring responses to verifiable sources. Cross-publication entity linking surfaces related content that shares authors, institutions, or concepts—connections that pure semantic similarity might miss because the same ideas are expressed in domain-specific terminology.

Summary

Omnipub provides a complete document management system that transforms diverse file formats into a unified, searchable representation. The publication and block schema captures both content and structure, enabling consistent operations regardless of source format. Block-level UUIDs enable granular entitlement management and consumption tracking, supporting complex licensing scenarios and usage-based billing. Three database technologies work together: relational storage for persistent data and entitlements, graph storage for relationship traversal, and search indexes for text and semantic retrieval.

Content flows through a multi-stage pipeline that reads source files, generates navigation structures, sections the content using hierarchical algorithms, constructs the graph representation, extracts and links entities, and persists the results. Quality assurance validates content thresholds and navigation structure. Multiple integration approaches support text search, semantic search, hybrid retrieval, and navigation-based access for agentic systems.

Three interfaces expose the system to different users: web administration for publishers and operators, a REST API for programmatic integration, and the Model Context Protocol for AI assistants. Together, these components enable professional publishing workflows that serve both human readers and AI-powered applications, with the entitlements and analytics capabilities ensuring proper rights management and business intelligence throughout.

References

  1. PageIndex. Tree-structured document indexes for retrieval. https://github.com/VectifyAI/PageIndex
  2. BEIR Benchmark. Heterogeneous benchmark for information retrieval. https://github.com/beir-cellar/beir
  3. FinanceBench. Benchmark for question-answering over financial documents. https://github.com/patronus-ai/financebench
  4. Hu, Z., et al. (2025). RAG vs. GraphRAG: A Systematic Evaluation and Key Insights. arXiv:2502.11371. https://arxiv.org/abs/2502.11371
  5. AWS Machine Learning Blog. (2024). Improving Retrieval Augmented Generation accuracy with GraphRAG. https://aws.amazon.com/blogs/machine-learning/improving-retrieval-augmented-generation-accuracy-with-graphrag/
  6. FalkorDB. (2025). GraphRAG vs Vector RAG: Accuracy Benchmark Insights. https://www.falkordb.com/blog/graphrag-accuracy-diffbot-falkordb/