Preparing Structured Data for Ingestion

Organizations with structured or semi-structured datasets—company directories, investment databases, deal trackers, fund analytics, and the like—should prepare their content for Cashmere by converting each entity into a single materialized article view and submitting it to the POST /omnipub endpoint with the correct metadata.

This guide uses a fictional Private Markets Intelligence Platform as the running example. In that platform, the core entities are Companies, Investors, Funds, Service Providers, and Deals. These entities live in their own relational tables but are presented to users as rich, joined profile pages. Cashmere needs those unified profile pages—converted to HTML or Markdown—as the ingestible content units.

1. Materialize each entity into a single article view

How relational data becomes a profile page

In a relational system, an entity exists as a single row, but the profile page users see is built from many joins. A Company profile might include company details, the deals it was involved in, investors in those deals, funds associated with those investors, service providers on related deals, and relevant news. An Investor profile might include investor details, all deals the investor participated in, all companies tied to those deals, and any funds the investor manages.

These views naturally contain cross-linked content and repeated rows. This repetition is correct and required.

Repeat data everywhere it appears

When you materialize a profile into an article view:

• Do not de-duplicate entities.
• Do not reference other entities indirectly.
• Do not assume "the LLM can fetch that elsewhere."

For example, if an investor made three separate investments into a company across three rounds, the Company profile should list that investor three separate times, each with the correct round and amount. On the Investor profile, the same company may appear multiple times. On the Deal profile, show all participants, even if they appear elsewhere. This gives Cashmere a complete, self-contained, text-based representation of the relationships in your dataset.

Remove internal / self-referential links

Your application UI probably links between profiles (clicking a company name opens the company profile). For Cashmere ingestion:

• Remove all internal hyperlinks and replace them with plain text.
• Keep only URLs that point to external sources (news articles, filings, and so on).

Internal links confuse LLM interpretation of the content and undermine the goal of routing users to your canonical source_url.

Output the profile as HTML or Markdown with clear sections

Use headings to separate logical segments:

title: Apex Robotics
external_id: company_145726
source_url: https://exampledata.com/companies/apex-robotics
creation_date: 01/15/2025

# Apex Robotics
Apex Robotics is a Seattle, WA-based company founded in 2014. Apex Robotics develops autonomous robotic systems for industrial logistics. The company operates in the Information Technology sector within the Robotics group.

Apex Robotics is currently Generating Revenue and is Venture Capital-Backed. Apex Robotics employs 320 people and reported $48M in revenue for FY2024.

## Apex Robotics Deal Activity
### Apex Robotics Series A — Jan 2020
In 01/15/2020, Apex Robotics raised $20M in a Series A round. Investors in this round included Highland Ridge Capital ($12M) and Westport Ventures ($8M).

### Apex Robotics Seed — Jun 2018
In 06/01/2018, Apex Robotics raised $3M in a Seed round. Investors in this round included Highland Ridge Capital ($3M).

## Apex Robotics Current Investors
Highland Ridge Capital has been an investor in Apex Robotics since 06/01/2018. Status: Active. Holding: Minority.
Westport Ventures has been an investor in Apex Robotics since 01/15/2020. Status: Active. Holding: Minority.

Note how the entity name "Apex Robotics" appears on nearly every line—this is a critical pattern (see Best Practice 6 below). Sectioning improves chunking, relationship inference, and grounding.

2. Submit materialized views via the Cashmere API

Submit each document to the POST /omnipub endpoint. Each document should include a unique external_id, one or more collection_id values, optional tags, metadata (title, publisher, source_url), and the materialized HTML/Markdown as content.

Suggested fields

collection_id — a comma-delimited list of collection IDs this content belongs to. Collections must exist before upload; retrieve their IDs via the list_collections endpoint. Example: 14,29,56.

external_id — your internal ID for this record. If IDs overlap across tables (e.g., all are integers), prefix them for global uniqueness: company_145726, investor_22291, deal_9081, fund_112.

tags — an optional comma-delimited list of descriptive labels. Tags don't need to exist before upload (they can be created on the fly). Examples: robotics,series_a,west_coast or geo:north_america,topic:health,topic:education.

Required metadata

• title — the human-readable title of the entity (e.g., "Apex Robotics", "Series B — Apex Robotics (2022)", "Highland Ridge Capital").
• publisher — your organization's display name, as you want it shown in downstream surfaces. Don't use your domain or URL.
• source_url — where users should be taken when they click your content in an AI surface. Ideally the canonical version of this profile on your website; if omitted, Cashmere displays the content directly.

Example payload

{
  "collection_id": "18,26",
  "external_id": "company_145726",
  "tags": "robotics,public_company",
  "metadata": {
    "title": "Apex Robotics",
    "publisher": "Private Markets Intelligence",
    "source_url": "https://exampledata.com/companies/apex-robotics"
  },
  "html_content": "<h1>Apex Robotics</h1><h2>Overview</h2>..."
}

Final checklist

• Materialize each entity into one HTML/Markdown article.
• Repeat all joined data; do not de-duplicate.
• Structure the document with headings.
• Remove internal / self-referential links.
• Ensure globally unique external_ids.
• Include all relevant collection_ids.
• Add optional tags for discovery.
• Provide metadata: title, publisher, source_url.

Best practices

1. Don't include LLM instructions inside your content

The article view should contain only the core content—not LLM processing guides, retrieval instructions, behavioral hints, or prompts. Including this type of information dilutes the relevance and value of your content and can keep it from appearing in results where it otherwise would. Cashmere attaches all necessary instructions at the system, collection, and tool-description levels. Keep your content clean.

2. Remove self-referential internal URLs

Internal links to your own entities confuse LLM interpretation, imply a navigable structure that doesn't exist, undermine routing users to your canonical site via source_url, and can distort chunk boundaries. Replace them with plain text. External URLs (press releases, filings, regulatory docs) are fine.

3. Include all metadata you want preserved

Metadata is a first-class citizen in Cashmere. Beyond the required fields, consider sector/industry, region/geography, entity type, dates (founded, active, last updated), valuation bands, risk categories, and flags (e.g., has_fund, has_recent_news). Well-structured metadata enables powerful filtering, dynamic collections, better ranking and grounding, cleaner versioning, and source attribution.

4. Repeat joined data exactly as it appears in your UI

The structured view should reflect the real relationships in your dataset. Don't simplify or "normalize" the content away.

5. Use consistent headings and formatting

This improves both chunk quality and retrieval performance.

6. Repeat the primary entity name on every line

This is the single most impactful structural practice for LLM retrieval. When content is chunked, individual lines are often served to the LLM without the document header. If a line reads only "raised $20M in a Series A round", the LLM has no way to know which company raised that money. Instead:

• ❌ "raised $20M in a Series A round"
• ✅ "Apex Robotics raised $20M in a Series A round"
• ❌ "Highland Ridge Capital has been an investor since 2018. Status: Active."
• ✅ "Highland Ridge Capital has been an investor in Apex Robotics since 2018. Status: Active."

This redundancy is intentional: it makes every chunk self-identifying. Apply it to funding rounds, investor lists, deal records, competitor lists, and any tabular or list-format data.

7. Include the entity name in section headings

Section headings anchor chunking and retrieval. Generic headings like "Funding History" lose all context when separated from the document title:

• ❌ ## Funding History → ✅ ## Apex Robotics Funding History
• ❌ ## Current Investors → ✅ ## Apex Robotics Current Investors

Even when a chunk begins at a section boundary, the LLM immediately knows which entity it belongs to.

8. Lead with a natural-language overview paragraph

Before diving into structured fields and lists, include a prose paragraph summarizing the entity. It gives the LLM rich semantic context, improves relevance matching for open-ended queries, and provides a human-readable summary usable directly in responses. Weave together what the entity does, where it operates, its stage, and defining characteristics—and include the entity name in the paragraph itself.

9. Use consistent, predictable templates for repeated records

When an entity contains lists of similar sub-records (funding rounds, investments, deals), every record should follow the same sentence structure—for example, "In [DATE], [ENTITY] raised [AMOUNT] in a [ROUND TYPE] round. Investors in this round included [INVESTORS]." Consistency lets LLMs pattern-match across records more reliably and reduces ambiguity during extraction.

10. Include contextual data inline with each record

When temporal or contextual data exists at the time of a record, include it inline rather than in a separate section—for example, "In 01/2020, Apex Robotics raised $20M in a Series A round. Revenue at time of deal was $12M." This lets the LLM answer questions like "what was the revenue when they raised their Series A?" from a single chunk, and makes each record a self-contained fact.

11. Add structured metadata headers at the top of each document

Before the Markdown body, include a concise block of key-value fields:

title: Apex Robotics
external_id: company_145726
source_url: https://exampledata.com/companies/apex-robotics
creation_date: 01/15/2025

This machine-readable preamble helps retrieval systems identify and deduplicate content, gives the LLM immediate context, and supports pipelines that parse metadata separately from content. Keep it to identification and routing information—not the full content.