OpenAbstracts

Log In

RAG over academic literature: using a paper search API as your retriever

Published September 26, 2026

Retrieval-augmented generation (RAG) over scientific literature usually means one of two things: build and maintain your own index of papers, or stitch together several free APIs and a PDF scraper. Both are more work than most teams want. A hosted search API can act as the retriever instead, so you only build the part that is unique to your product.

The architecture

user question → query rewrite → paper search API → select passages → LLM answer with citations

The search API replaces your own index and crawler. Your code handles query rewriting, passage selection and generation.

1. Rewrite the question into search queries

Search engines match terms, not questions. Have your model turn "does remote work reduce productivity?" into two or three short queries such as remote work productivity and telework performance employees. Run them and merge the results by DOI.

2. Retrieve

OpenAbstracts gives you two ways to retrieve:

  • mode=papers returns records ranked by match on titles, abstracts, keywords and authors. Use the abstract as your passage.
  • mode=content searches inside the PDFs and returns highlights, short snippets around the matching terms. Use these when you need the exact wording.
import requests

def retrieve(query, mode="papers", limit=8, **filters):
    r = requests.get(
        "https://openabstracts.com/api/references-search",
        params={"q": query, "mode": mode, "limit": limit, **filters},
        headers={"x-api-key": "isk_live_YOUR_KEY"}, timeout=30,
    )
    return r.json().get("references", [])

papers = retrieve("remote work productivity", yearFrom=2019, quartile="Q1,Q2")
snippets = retrieve("remote work productivity", mode="content", limit=5)

Filters for category, yearFrom/yearTo and journal quartile let you bias retrieval toward recent, higher-quality sources without extra logic.

3. Build the context with provenance

Keep the DOI and title attached to every passage you send to the model, and require the model to cite by source number. Provenance makes answers auditable and lets your UI link each claim back to a real paper. The chatbot citation guide shows a prompt that enforces this.

4. Decide what to cache

Papers don't change, so cache search results by normalised query. A cache also caps your cost: you only pay for uncached searches that return results.

Using LangChain or LlamaIndex

The API is a plain HTTP GET with a header, so a custom retriever is a few lines: call the endpoint, map each result to a document with page_content set to the abstract (or snippet) and the DOI, title, authors and year in the metadata. There is no SDK to install.

Or let the assistant retrieve for itself

If your "application" is an MCP-capable assistant, connect the MCP server and let it call search_papers as it needs.

What this approach does and doesn't do

It gives you real, citable sources from 100,000+ journals without operating infrastructure. It is not a citation-graph service: results come from keyword and full-text matching, so plan for good query rewriting. If you need citation graphs, pair it with a graph API — we cover when in our comparison with OpenAlex.