2 Day 2: Literature Review and Synthesis with AI Tools
An index card pulled halfway from a stack of old library cards: a literature review, most sources still unchecked.
Trust, but verify.
Russian proverb (doveryai, no proveryai), used repeatedly by U.S. President Ronald Reagan in nuclear arms-reduction talks with Mikhail Gorbachev (Russian proverb, popularized in English by Ronald Reagan, 1987)
2.1 Learning objectives
By the end of this day you should be able to:
- Compile a working bibliography efficiently, from a database of record rather than one-by-one chat prompts, reserving AI tools for discovery and synthesis rather than as the source of the citation itself.
- Triage a large compiled reference set down to a readable evidence base, checking the discarded papers rather than the kept ones, and extract structured, source-anchored notes from what survives.
- State the additional constraints that apply when AI-assisted screening is used in a formal systematic review rather than a narrative one.
- Use an AI tool to accelerate literature search and summarization without accepting its output uncorrected.
- Verify every AI-produced citation against a primary source before it enters a draft.
- Synthesize across multiple papers with an AI assistant while preserving genuine disagreement between sources rather than flattening it.
- Recognize the specific signatures of a fabricated or misattributed citation.
- Decide when AI-assisted synthesis is and is not appropriate for a given review task.
2.2 Lecture
Literature review is the task in this workshop with the clearest efficiency payoff and, done carelessly, the highest cost of an unverified error: a fabricated citation that reaches a manuscript damages the researcher’s credibility directly. The two are related rather than opposed, and this chapter opens with the technique that delivers both at once.
2.2.1 Compiling a bibliography efficiently: best practice versus brute force
We begin with the approach most researchers try first. The brute-force method of building a bibliography is conversational: ask a general-purpose AI assistant, in a chat window, “give me some references on topic X,” read the result, ask for more, and repeat. It feels efficient because each individual exchange is fast. Unfortunately, it is not efficient at all, for two reasons. It does not scale (each reference costs a chat turn, and a literature review needs dozens), and every citation it produces is unverified in exactly the sense Day 1 introduced, generated from the model’s trained parameters rather than retrieved from any actual database, which is precisely the mechanism behind the fabrication risk this chapter spends the rest of its time addressing.
The efficient alternative is to treat bibliography compilation as a database-query problem, not a conversation. Two free, keyless services cover most of academic literature. Crossref’s REST API resolves a title or author query directly against the metadata for every DOI it has registered, and returns real bibliographic records rather than a model’s reconstruction of one; content negotiation returns the result directly in BibTeX format (Crossref, 2024):
curl -s "https://api.crossref.org/works?query.bibliographic=attention+is+all+you+need&rows=1" \
| python3 -c "import json,sys; print(json.load(sys.stdin)['message']['items'][0]['DOI'])"For biomedical literature, the NCBI E-utilities provide the same capability against PubMed: esearch finds matching record IDs and efetch retrieves their full metadata, both scriptable against dozens or hundreds of records in a single batch rather than one chat turn each (National Center for Biotechnology Information, 2024). A reference manager such as Zotero sits on top of the same idea at the interactive level: its browser connector and “Add Item by Identifier” feature pull a citation directly from the publisher or a DOI, not from a chatbot’s memory, and its own citation-graph search is often faster than typing a natural-language request into a chat window at all (Corporation for Digital Scholarship, 2024).
This does not mean AI tools have no role in literature discovery; it means their role is narrower and better-placed than originating the citation. A search-augmented AI tool (Day 1) is genuinely useful for breadth, surfacing candidate papers or search terms the researcher would not have thought to query directly. The efficient workflow uses the AI tool for that discovery step and then resolves every candidate through Crossref, PubMed, or a reference manager before it becomes a bibliography entry, rather than accepting the AI tool’s own citation as the final record. Done this way, efficiency and verification are the same action: querying a database of record is both faster at scale than chatting one reference at a time, and structurally immune to the citation-fabrication risk the rest of this chapter addresses, because the bibliographic record never passed through the part of the AI tool that can hallucinate one.
2.2.2 From a comprehensive bibliography to a usable evidence base
A sieve over a tray, most of the gravel fallen through and a few stones retained: a comprehensive reference set triaged down to the papers worth reading.
Compiling comprehensively, as the previous section describes, creates a problem that used to be a good reason not to do it. For most of the history of literature review, breadth was self-limiting: a researcher kept the search narrow because every additional reference had to be read, and reading was the bottleneck. The database-query approach removes the compilation constraint entirely (a Crossref or PubMed query returns four hundred records as easily as forty) without touching the reading constraint at all. The bottleneck simply moved downstream, and that is where the next three techniques apply.
The relationship is worth stating plainly because it inverts an old tradeoff. Comprehensive compilation is now cheap, and AI-assisted triage makes a large compiled set tractable to work through, so the rational strategy shifts from “search narrowly to keep the reading manageable” toward “search broadly and triage aggressively.” That is a genuine methodological improvement, not merely a speed gain: a narrow search silently risks missing the paper that would have changed the conclusion, and no amount of careful reading of the papers actually found will surface it. But how trustworthy is the triage step? The shift only holds if it can be relied upon, and that is the subject of the rest of this section.
Triaging a reading backlog. Given a compiled set of candidate references (or a stack of PDFs already sitting unread), an AI tool can perform a first-pass relevance screen against criteria stated explicitly, returning a keep/skip decision with a one-line reason for each. The reason matters more than the decision: a decision alone is unauditable, while a stated reason lets the researcher spot a systematic misreading of the criteria that would otherwise silently discard a whole category of relevant work.
Task: Screen each of the following abstracts for relevance to my
review question and return keep or skip for each, with a
one-sentence reason.
Context: My question is whether remote work affects promotion
rates for employees in their first five years of employment.
[paste abstracts, numbered]
Constraints: Judge relevance only against my stated question. Do
not judge study quality. If an abstract is ambiguous about
whether it addresses my question, mark it keep, not skip.
Output format: A numbered list: number, keep/skip, reason.We note the asymmetry built into that prompt’s constraints, and it should be carried into how the output is checked. A false keep costs a few minutes of reading and is self-correcting, because the irrelevance becomes apparent as soon as the paper is opened. A false skip is invisible: the paper silently never enters the review, and nothing downstream will ever signal its absence. So spot-check the skips, not the keeps, and instruct the tool to resolve ambiguity toward keeping.
Screening for a systematic review. The same technique under formal systematic-review conditions is a stricter proposition, because a systematic review makes a methodological claim about its own completeness and reproducibility that a narrative review does not. Three constraints follow. The screen must run against the protocol’s pre-specified inclusion and exclusion criteria rather than an informal description; AI screening belongs as one pass within a documented dual-screening workflow (a second independent screen, human or otherwise, with disagreements adjudicated by a human), not as a replacement for it; and the use must be reported. PRISMA 2020 already requires that authors describe how automation tools were used in the review, including in the search, study-selection, and data-collection processes (Page et al., 2021), which means AI-assisted screening in a systematic review is not merely permitted-if-disclosed in the general sense developed on Day 1, but falls under an existing, specific reporting standard that readers and reviewers will expect to have been followed.
Extracting structured notes from papers. Once a set is triaged down to what will actually be read, the third technique converts each paper into consistent structured fields (design, population, sample size, outcome measure, effect estimate, stated limitations) that can be compared across studies in a table rather than recalled from prose. This is where most of the downstream synthesis time is actually saved, because a comparison table makes the disagreements between studies visible in a way that a folder of PDFs never does.
It is also where this chapter’s verification discipline applies most sharply, because extraction failures are exactly the “real paper cited for a claim it does not make” signature described below, produced at scale and in a tidy table that looks authoritative. The mitigation is to require provenance in the extraction itself: instruct the tool to include, for every extracted value, the verbatim sentence and section it came from. An extracted effect size accompanied by the sentence stating it can be checked in seconds; the same number sitting alone in a table cell cannot be checked at all without re-reading the paper, which defeats the purpose of having extracted it.
2.2.3 AI-assisted search and summarization
Two different kinds of tool both call themselves useful for literature work, and Day 1 already introduced the distinction that matters here. A plain general-purpose assistant, asked to summarize research on a topic, draws on whatever it absorbed during training; it has no live connection to a paper database and cannot signal that it is drawing a blank rather than recalling accurately. A search-augmented or retrieval-augmented tool instead issues a real query against a real corpus (a search engine, a paper database, an index the researcher controls) and grounds its summary in the documents that query returns (Lewis et al., 2020). The retrieval step does not make the tool infallible, but it changes the nature of the risk: an error in a retrieval-augmented summary is more often a misreading of a real document than an invention of a nonexistent one, and a misreading is far easier to catch by opening the source.
For literature work specifically, prefer a tool that shows which documents it drew from over one that does not, even if the one that does not is more fluent, for exactly the reason Day 1’s “Check your understanding: tool choice” made explicit: traceability, not polish, is what lets a researcher convert an unverified claim into a verified one.
2.2.4 Verifying citations against primary sources
Every citation an AI tool produces is unverified, in the sense defined on the Conventions page, until the researcher has checked it against a primary source. The check itself is mechanical and fast: confirm the title, author list, year, and venue against a database of record such as PubMed, Crossref (which resolves DOIs), or a publisher’s own site, rather than trusting a search-engine snippet or the AI tool’s own restatement of the citation. A citation that resolves to a real DOI and a real author list is not yet fully verified: the last step is opening the source and confirming it actually supports the specific claim attached to it, not merely that the citation exists.
This is not a pedantic extra step. In Mata v. Avianca, introduced on Day 1, the fabricated citations had the surface form of real case law: plausible party names, plausible citation formatting, invented but internally consistent quotations (United States District Court for the Southern District of New York, 2023). Nothing about how the citation looked distinguished it from a real one; only checking it against the actual case reporter did. The same is true of a fabricated academic citation: format is not evidence of existence.
Verify citations as you receive them, not in a batch at the end of a writing session. A citation is cheapest to check the moment you have it in front of you and cannot yet remember why you believed it; it is most expensive to check after it has already shaped three paragraphs of argument built on top of it.
2.2.5 Synthesizing across papers without flattening disagreement
Asked to synthesize several papers, an AI tool has a structural tendency to produce a smooth, well-organized narrative, because that is what its training rewards, and a smooth narrative is often achieved by averaging over disagreement rather than stating it. Two papers that reach opposite conclusions on the same question can come back from a naive synthesis prompt as a single hedged sentence that represents neither paper’s actual finding.
We counteract this tendency by prompting for disagreement explicitly rather than hoping the tool preserves it on its own:
Task: Compare the findings of these two papers on <topic>.
Context: [paste abstracts or key sections of both papers]
Constraints: Do not average or split the difference between
conflicting findings. If the papers disagree, state the
disagreement explicitly: what each paper found, and one
plausible reason (different population, different method,
different time period) the findings might differ.
Output format: A short table, one row per paper, columns for
finding, method, and population; followed by one paragraph
naming the disagreement if one exists.A synthesis that says “the evidence is mixed” without saying what the actual conflicting findings were has quietly discarded the information a literature review exists to preserve.
2.2.6 Recognizing a fabricated citation
A magnifying glass held over a library card, its printed lines dissolving into a smudge under close inspection: a citation that looks right until it is examined.
A handful of concrete signatures recur across documented cases of fabricated or misattributed AI citations (Academ-AI project, 2024; United States District Court for the Southern District of New York, 2023):
- A plausible-sounding author, journal, or case name that does not resolve to any real record in a database of record, even though it sounds exactly like a real one for the subfield.
- Mismatched or shifted bibliographic details: a real paper’s title paired with the wrong year, wrong volume, or an author list that has been reordered or altered.
- A real paper cited for a claim it does not make. The citation exists and is correctly formatted; the specific factual claim attached to it is not actually in the source. This is the hardest signature to catch by database lookup alone, since the citation itself checks out; catching it requires reading the source.
- Suspiciously perfect support. A citation that appears to confirm the hypothesis exactly, with no caveats, on a topic where the real literature is actually contested, merits extra scrutiny; real findings are rarely that clean.
None of these signatures is a substitute for the verification workflow above; they are what to look for when deciding where to spend limited verification effort first.
2.3 Further reading
- Crossref REST API documentation (Crossref, 2024) and NCBI E-utilities documentation (National Center for Biotechnology Information, 2024). The two database APIs behind this chapter’s bibliography-compilation technique; worth reading once to see the full range of queryable fields beyond title and author.
- Page, M. J. et al. (2021). The PRISMA 2020 Statement. BMJ 372:n71 (Page et al., 2021). Read item-level guidance on reporting automation tools before using AI-assisted screening in any review that claims to be systematic.
- Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) (United States District Court for the Southern District of New York, 2023). The primary legal record of a real, professionally costly fabricated-citation incident.
- Academ-AI project (2024). Documenting the Undisclosed Use of Generative Artificial Intelligence in Academic Publishing (Academ-AI project, 2024). An ongoing effort to catalog real instances of undisclosed AI use and fabricated content reaching published academic work.
- Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020). Revisit this from Day 1 for the technical basis of why search-augmented tools are the safer default for literature work.
- ICMJE, Recommendations: Artificial Intelligence (International Committee of Medical Journal Editors, 2025). Read the passage on authors’ responsibility to assert the absence of plagiarism in AI-produced text and images, previewed here and developed fully in Day 5’s ethics section.
2.4 Worked example: reviewing AI-assisted summaries of five papers
Five folders fanned on a table, four clipped in green and one in red: a five-paper review, four verified and one set aside.
To illustrate, consider an AI research-search tool that has produced one-paragraph summaries of five papers relevant to a review, each with a citation. Before any of the five summaries enters the draft, we work through them with the discipline this chapter has built:
- Resolve each citation independently. For each of the five, look up the title and author list in a database of record. Suppose four resolve cleanly and the fifth’s DOI points to a different paper by the same last author, one of the signatures from above.
- Mark the fifth unverified and set it aside, rather than silently correcting the DOI and moving on. Either find the paper the summary actually describes by searching the title directly, or drop the claim if you cannot locate a source that matches it.
- Spot-check the four verified summaries against their sources, not just their existence. Suppose paper 3’s summary states a stronger causal claim than the paper’s own abstract does; that is the “real paper cited for a claim it does not make” signature, and the fix is to revise the summary to match what the paper actually says, not to drop the citation.
- Ask for a synthesis across the four now-verified papers, using the disagreement-preserving prompt structure above, rather than the original five.
What enters the draft is now four claims personally verified against their sources, one flagged and excluded citation, and a synthesis built only from material the researcher can stand behind. That is a slower process than accepting the tool’s first output, and it is the entire point of this chapter: the AI tool’s speed is real, and so is the requirement that a human close the loop before anything reaches a manuscript.
2.5 Homework
Attempt each problem in your own environment before checking the solution.
Request a literature summary. Ask an AI tool to summarize the state of research on a narrow topic in your field, with citations. List the citations it produced.
Verify every citation. For each citation from Problem 1, confirm it exists (title, authors, year, venue) using a primary database. Report how many verified and how many did not.
Find a flattened disagreement. Ask an AI tool to synthesize two papers you know take different positions. Did the synthesis preserve the disagreement or blur it?
Draft a verification checklist. Write a short checklist you would apply to any AI-produced bibliography before using it in a manuscript.
Compare AI synthesis to your own reading. For one paper you have read carefully, compare an AI-generated summary of it against your own understanding. Note any distortion.
Decide a boundary case. Describe a literature-review task you would not delegate to an AI tool even for a first pass, and explain why.
2.6 Solutions
Problem 1. The output to record is the citation list itself, in whatever form the tool produced it. Two features of that list matter for the problems that follow.
Note first whether the tool named its sources or generated them from trained parameters alone. The chapter’s distinction between search-augmented and general-purpose tools predicts different failure rates here (Lewis et al., 2020). Note second whether each citation carries a resolvable identifier such as a DOI. A citation given without one is not yet checkable, and supplying that identifier is the tool’s cheapest contribution to the work that follows.
Problem 2. The verification itself is mechanical. Search the title and author list in a database of record, Crossref or PubMed rather than a search-engine snippet, and confirm that the year and venue match (Crossref, 2024; National Center for Biotechnology Information, 2024).
Report three categories rather than two. A citation may verify cleanly, fail to resolve at all, or resolve to a real record whose details differ from what the tool supplied. The third category is the interesting one, and it corresponds to the mismatched-details signature described in this chapter.
Rates vary widely by tool and by how obscure the topic is, so there is no single expected number. What matters is that you now have one, measured in your own field rather than taken from a general claim about AI reliability. A retrieval-grounded tool will usually do better than a general-purpose assistant, and neither result licenses skipping the check next time.
Problem 3. A blurred synthesis is the common outcome, and it has a recognizable shape. The disagreement is replaced by a hedge (“findings are mixed,” “further research is needed”) that names neither position. Alternatively, one paper’s conclusion is presented as the consensus and the other is reduced to a caveat.
A good answer quotes the flattening sentence and states what each paper actually found. That comparison is the evidence. A general impression that the synthesis seemed vague is not.
If the synthesis did preserve the disagreement, record what the prompt asked for. An explicit instruction not to average conflicting findings, as in the lecture’s prompt, materially changes the result. That is the practical point of the exercise.
Problem 4. A workable checklist has five items, and it applies to every citation rather than to a sample:
- The record resolves in a database of record by title and author, not merely in a web search.
- Year, venue, and volume match the resolved record exactly.
- The DOI, if one was supplied, resolves to that same record rather than to another paper by the same author.
- The source actually contains the claim attached to it, which requires opening it.
- The citation is one you would defend if an editor asked how you found it.
Item (d) is the one most often skipped, and it is the only one that catches a real paper cited for a claim it does not make. Item (e) checks provenance rather than the record. It separates a bibliography you assembled from one you inherited without inspection.
Problem 5. Distortions in AI summaries are usually not fabrications. They are compressions that drop the qualifier the authors were careful to include.
Look specifically for four shifts: a correlational finding described in causal language, a subgroup result stated as a whole-sample result, a stated limitation omitted, and a hedged conclusion firmed up. A good answer names which shift occurred and quotes both the paper and the summary.
This exercise is the reason the extraction discipline in this chapter asks for the verbatim source sentence alongside each extracted value. A summary you cannot trace back to a sentence cannot be checked at the speed at which it was produced.
Problem 6. Any review whose validity depends on completeness or on contested judgment is a poor candidate for unsupervised delegation. The clearest case is a systematic review’s inclusion decision. There the methodological claim is precisely that screening was reproducible against pre-specified criteria, and PRISMA already requires you to report how any automation was used (Page et al., 2021).
A second defensible answer is a review of a literature you are entering for the first time. The chapter’s warning applies with full force, since you cannot audit a synthesis of work you have not read. The delegation removes exactly the judgment that would have made it safe.
A weaker answer names a task that is merely tedious. Tedium is an argument for delegation rather than against it. The criterion is whether you can check the result.
2.7 What’s next
Day 3 turns from prose to code. We shall apply the same verify-before-trust discipline to AI-assisted coding and data analysis, where the failure mode shifts from fabricated citations to code that runs and looks correct while silently mishandling the data.