1  Day 1: Foundations, How LLMs Work and Where They Fit in Research

A single glowing filament woven through a dark wire lattice sculpture on a desk

A single glowing filament woven through a dark wire lattice on a desk: the associative connections inside a language model, mostly hidden, briefly visible.

Text generated by an LM is not grounded in communicative intent, any model of the world, or any model of the reader’s state of mind.

Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, On the Dangers of Stochastic Parrots (Bender et al., 2021)

1.1 Learning objectives

By the end of this day you should be able to:

  • State what the current professional consensus actually permits in AI-assisted academic work, and answer the question “is it legitimate for me to use these tools at all?” with reference to specific published policy rather than intuition.
  • Describe, at a level useful for a researcher (not a model developer), what a large language model does when it generates text.
  • Explain why hallucination is a structural property of these tools rather than an occasional bug, and predict when it is most likely.
  • Describe the role of context window and tokens in what a model can and cannot “see” in a session.
  • Select among tools on the five criteria that stay stable as products change, and state when a locally run open-weight model is the right choice for confidential material.
  • Choose an appropriate AI tool for a given research task (search and synthesis, coding, writing, general reasoning).
  • Write a prompt that states task, context, constraints, and the form of the desired output.

1.2 Lecture

We begin with two questions, in the order most researchers actually ask them. First, the one that keeps people from starting at all: is using these tools on academic work legitimate? Second, the mechanical one: what exactly are we relying on when we do, and where does that reliance become risky?

1.2.1 Is this legitimate? What the consensus actually permits

A heavy library reading-room door standing open, warm light spilling through onto a dark stone floor

A library door standing open, warm light spilling through: the professional consensus on AI-assisted research is permission with conditions, not a closed door.

We begin with the worry itself. Many researchers approach AI tools with a low-grade unease that has nothing to do with hallucination or any other technical failure. It is a professional worry rather than a technical one: that using an AI assistant is a kind of cheating; that a colleague or reviewer would think less of the work if they knew; that relying on a tool erodes the skill and judgment that make someone a scholar in the first place. That worry merits a direct answer before any technique in this book is worth learning, because a researcher who believes the whole enterprise is faintly illegitimate will either avoid the tools entirely or, worse, use them and not disclose it.

The direct answer is that the major bodies governing academic publishing have already settled the central question, and they settled it permissively. The International Committee of Medical Journal Editors, the Committee on Publication Ethics, and the large publishers all reached the same two-part position: AI tools cannot be authors, because authorship requires accountability a tool cannot bear, and AI tools may be used by authors who disclose that use and take responsibility for the result (Committee on Publication Ethics, 2023; International Committee of Medical Journal Editors, 2025). We should note what the second half of that position is not. It is not grudging tolerance of an unfortunate practice; it is an explicit accommodation, written into the policies of the bodies with the most at stake in research integrity, of a category of assistance they expect authors to use.

Springer Nature’s policy goes further in a way worth knowing precisely, because it marks how routine some AI use has already become: AI-assisted copy editing, meaning improvements to human-written text for readability, style, grammar, spelling, punctuation, and tone, does not need to be declared in the manuscript at all, provided the authors remain accountable for the final text and it reflects their own work; only generative editorial work and autonomous content creation require declaration (Springer Nature, 2024). A researcher running their own prose through an AI editor for clarity is, under that policy, doing something closer to using a spell-checker than to committing an ethical violation. Funders have drawn a line in a similar place rather than banning the tools: NSF requires disclosure of the extent and manner of generative AI use in proposal preparation (National Science Foundation, 2024), and NIH permits AI assistance for limited aspects of application preparation while refusing to treat an application substantially developed by AI as the applicant’s own original work (National Institutes of Health, Office of Extramural Research, 2025).

The legitimate/illegitimate boundary these policies draw is therefore not “AI or no AI.” It runs between assistance, where a human directs the work, verifies the output, and takes responsibility for it, and substitution, where the tool’s output stands in for the researcher’s own judgment, unverified and unacknowledged. Every technique in this book sits on the assistance side of that line by construction: compiling a bibliography faster from a database of record (Day 2), scaffolding a project skeleton (Day 3), or structuring an outline from one’s own notes (Day 4) accelerates work the researcher would otherwise do unaided, and none of it hands the scholarly judgment to the tool. The residual anxiety about skill erosion merits being taken seriously on its own terms rather than dismissed. The honest answer is that it depends entirely on which side of that line a researcher’s habits fall. One who verifies every citation and reviews every generated function is exercising exactly the judgment they are worried about losing, while one who accepts unverified output uncritically really is outsourcing it.

TipTip

If you find yourself reluctant to describe a particular use of an AI tool to a co-author or reviewer, that reluctance is useful diagnostic information. Under the disclosure norms developed in Day 4 and Day 5, anything you would be comfortable stating plainly in a methods or acknowledgments section is almost certainly on the legitimate side of the line; anything you would rather not mention is worth re-examining before you do it, not after.

Q. A colleague says using an AI tool to improve the clarity of your own drafted paragraphs is “basically academic dishonesty.” On what specific published grounds can you disagree?

A. ICMJE and COPE both permit AI-assisted work by accountable human authors and restrict only authorship itself; Springer Nature’s policy goes further and does not require declaration of AI-assisted copy editing (readability, style, grammar) in the manuscript at all, provided the authors remain accountable and the text reflects their own work. The prohibited category is substitution for scholarly judgment without verification or disclosure, not assistance as such.

1.2.2 What a large language model does

We begin with the mechanism. A large language model (LLM) is, mechanically, a next-token predictor. Given the text so far, it produces a probability distribution over the vocabulary and samples the next unit of text (a token, roughly a word or word-fragment) from that distribution, then repeats. Nothing in this process consults a database of facts at generation time. What the model has instead is a set of statistical regularities learned from a very large training corpus of text, encoded as billions of numerical parameters, and refined afterward by additional training (often called instruction tuning and reinforcement learning from human feedback) that shapes the model toward producing helpful, confident-sounding, well-formatted answers (Ji et al., 2023).

We note an immediate consequence for research use. Fluency is not the same property as correctness, and nothing about the generation process guarantees the second even when the first is perfect. A response can be grammatically flawless, well-organized, and confidently stated while being partly or entirely fabricated. The model is not lying in the sense of knowing the truth and choosing to conceal it; it is producing the most statistically plausible continuation of the prompt, and plausible is not the same target as true.

Q. A colleague says “the AI looked up the answer and told me.” What is technically wrong with this description of what a general-purpose LLM did, and why does the distinction matter for how you use its output?

A. A base LLM does not look anything up; it generates the statistically most plausible next tokens given its training and the conversation so far. It matters because “looked up” implies a retrieval-and-check step guaranteeing the answer exists somewhere real, which is precisely the guarantee a plain LLM does not provide. (Some tools do add a genuine retrieval step, discussed under “Choosing a tool for the task” below; the distinction is exactly which kind of tool you are using.)

1.2.3 Hallucination as a structural property

Because generation is driven by plausibility rather than by a lookup against ground truth, a large language model can produce a confident, well-formed statement that is simply false: a citation that does not exist, a function that does not exist in the library it is named for, a date or a quotation invented to complete the pattern the surrounding text set up. This behavior is usually called hallucination, and the research literature treats it as a structural property of how these models generate text, not as a bug that a future patch removes (Ji et al., 2023). Unfortunately, instruction tuning tends to make the problem worse in one specific way: it optimizes the model to always produce a confident, complete-sounding answer, which means a model asked something it does not know is more likely to invent a plausible-sounding answer than to say so.

To illustrate the cost of trusting this uncritically, we consider a case from outside academia. In 2023, lawyers representing a plaintiff in Mata v. Avianca submitted a legal brief citing six court decisions that ChatGPT had fabricated whole cloth, complete with invented quotations and invented case numbers. When the fabrication was discovered, the presiding judge sanctioned the attorneys under Federal Rule of Civil Procedure 11 for failing to verify the citations they had submitted under their own names (United States District Court for the Southern District of New York, 2023). The lesson generalizes directly to research writing: an AI tool inventing a citation is not a remote, hypothetical risk, and the professional consequence falls on the human who submitted the work, not on the tool. Day 2 develops the verification habit that this case makes necessary.

WarningWarning

Hallucination is worst exactly where you are least equipped to catch it: narrow, specialized, or obscure topics, where a fabricated answer sounds just as authoritative as a correct one and you have no independent basis for suspicion. The topics where an AI tool feels most impressively fluent are often the topics where you should verify hardest, not least.

Q. Why might an AI assistant be more likely to fabricate an answer to an obscure question than to a common one, rather than simply saying that it does not know?

A. Instruction tuning rewards confident, complete-sounding answers over hedged or refused ones, and the model’s training data thins out on obscure topics, so it has less signal to draw a correct answer from but the same pressure to produce something fluent. The result is that confidence in the output and correctness of the output can come apart precisely where the underlying evidence is thin.

1.2.4 Context windows and what the model can see

An LLM does not have persistent memory of a conversation the way a person does. Everything it “knows” about the current exchange is whatever text sits inside its context window, the fixed-size span of tokens (system instructions, conversation history, and any documents supplied by the user) that the model actually processes on each turn. Once a conversation grows past that window, or once a tool summarizes or truncates older turns to save space, instructions and facts supplied earlier can simply fall out of what the model is conditioning on, and it will proceed without them, often without any visible warning that it has done so.

For research work we draw two practical consequences. First, restate load-bearing constraints (a citation style, a variable naming convention, a scope boundary) periodically in long sessions rather than trusting the model to recall an instruction from fifty turns earlier. Second, when a task depends on a specific document (a paper, a dataset codebook, a style guide), paste or attach the relevant excerpt directly into the prompt rather than assuming the model already knows its contents, even if you discussed that same document earlier in the session.

Q. You instructed an assistant at the start of a long session to always answer in a formal register. Fifty exchanges later it produces a casual answer. What is the most likely explanation, and what should you do about it?

A. The instruction most likely fell outside the model’s context window, or was dropped or compressed by the tool’s history-management logic; the model is not disobeying, it simply no longer has the instruction in view. Restate the instruction rather than repeating the request and hoping.

A long strip of densely printed paper unrolled flat across a desk, legible only where a band of lamplight falls, both ends lost to shadow

A long strip of text unrolled across a desk, legible only where the lamp falls and lost to shadow at both ends: what sits inside, and outside, a model’s context window.

1.2.5 Choosing a tool for the task

Not every AI tool does the same thing, and the differences matter more than most marketing suggests. But how is one to choose among them? We distinguish three broad categories, which recur across this book:

  • General-purpose assistants (for example Claude or ChatGPT (Anthropic, 2024; OpenAI, 2024)) are strong at reasoning, drafting, and explaining, and by default generate purely from their trained parameters, with the hallucination risk described above.
  • Search-augmented and retrieval-augmented tools attach a live retrieval step: the tool looks up real documents (a web search, a paper database, the researcher’s own files) and conditions its answer on the retrieved text rather than on trained parameters alone. This measurably reduces, but does not eliminate, hallucination, because the model can still misread, over-generalize from, or selectively quote the retrieved material (Lewis et al., 2020). Day 2’s literature-review tools are mostly of this kind.
  • Coding assistants (for example Claude Code or GitHub Copilot (Anthropic, 2025; GitHub, 2024)) are tuned and tooled specifically for reading and writing code inside a project, often with the ability to run commands and tests. Day 3 is devoted to them.

The practical rule is to match the tool’s design to the task’s verifiability, not to its polish. A retrieval-augmented tool that shows its sources is worth more for literature work than a more articulate general-purpose assistant that shows none.

1.2.6 Selection criteria, and the case for a local model

Within those three categories the specific products change constantly, and any ranking of named models in a printed book would be wrong within months. The criteria for choosing between them are more stable, and we recommend selecting on these five rather than on benchmark scores or reputation.

Retrieval grounding. Does the tool show which documents it drew from, and can you open them? This is the single most useful property for literature work, for the reason Day 2 develops at length.

Context window. How much material can the tool hold in view at once? A tool that accepts a whole manuscript reasons about it differently from one that must be fed a section at a time. We note the practical test rather than the advertised number: paste in the document you actually work with and ask a question whose answer sits near the end.

Execution. Can the tool run code and inspect the result, or only produce text? Day 3’s assistants can execute; a chat window cannot. Execution is what makes the difference between a suggested analysis and a checked one, and it is also what raises the stakes discussed under failure modes.

Where the data goes. Does the prompt leave your machine, and if so, to whom? This is the criterion most often ignored, and it is the one that determines whether a tool is usable at all for confidential material.

Cost and latency per task. Reasoning-heavy models cost more and answer more slowly. They earn that cost on genuinely hard problems and waste it on formatting a bibliography.

The local option. The fourth criterion has an answer that many researchers do not know they have. Open-weight models can be run entirely on your own hardware, through a runner such as Ollama or the llama.cpp engine beneath it (Gerganov & contributors, 2026; Ollama, 2026). A local model exposes an interface much like a hosted one, but no prompt, document, or dataset leaves the machine, because there is no remote service to send it to.

This matters directly for the confidentiality problem developed in Day 5 and in the faculty appendix. A researcher holding identifiable interview transcripts, unpublished data under a data-use agreement, or a colleague’s manuscript under review cannot paste any of it into a hosted assistant. Until now this book’s advice in that situation has been simply to refrain. A locally run model changes the question from whether to use AI on such material to which model is good enough to be useful on it, and that is a far better position to be in.

The trade-off is real and should be stated plainly. On the plus side, the confidentiality problem disappears at its root, the tool works offline, and there is no per-token cost. On the minus side, a model small enough to run on a laptop is meaningfully less capable than the frontier hosted models, hardware sets a ceiling on the model size you can load, and setup takes an hour you would not otherwise spend. Unfortunately, the capability gap is largest on exactly the reasoning-heavy tasks where assistance helps most. For summarizing, extracting structured fields, and first-pass triage, a local model is often entirely adequate, and those happen to be the tasks most likely to involve confidential material in the first place.

TipTip

Local models do not remove the verification obligation. They run the same kind of next-token prediction described above and hallucinate for the same structural reasons. What changes is who else sees your data, and nothing else.

Q. A colleague dismisses local models on the grounds that they perform worse than the best hosted model on published benchmarks. On what grounds might a researcher still prefer one?

A. Benchmark rank answers only the capability question. For material that cannot leave the institution, such as identifiable subject data or a manuscript under review, a hosted model’s capability is unavailable at any score, because using it would breach confidentiality. A less capable model that can lawfully see the data is more useful than a better one that cannot.

Q. Two literature-search tools return equally polished summaries of a topic. One shows the specific passages it drew from; the other does not. Which should you prefer for research work, and why?

A. The one that shows its sources, because it lets you verify the claim against the underlying document directly, converting an unverified claim into a verified one at low cost. Polish is not evidence of accuracy; traceability is.

1.2.7 Anatomy of a good prompt

A prompt that gets a useful, checkable answer typically states four things explicitly:

  1. Task. What you actually want done (summarize, draft, critique, generate code), stated as an instruction, not a question about whether the model can do it.
  2. Context. The material the model needs and does not already have: a pasted excerpt, a dataset’s variable names, the audience for a piece of writing.
  3. Constraints. Length, format, style, citation requirements, what to exclude, and any known-wrong answers to avoid repeating.
  4. Output format. The shape you want back: a bulleted list, a table, a function signature, plain prose of a stated length.

Compare a weak prompt with a stronger one built from the same four parts:

Tell me about attention mechanisms.
Task: Explain the attention mechanism in transformer models
to a second-year statistics graduate student who knows linear
algebra but has not read the original paper.
Context: This is for a one-paragraph background section in a
methods write-up, not a full tutorial.
Constraints: No code. Do not claim a specific equation is "the"
formulation used by every model; note that variants exist.
Output format: One paragraph, 5-8 sentences.

The second prompt is longer to write, but it is far more likely to produce an answer usable with only light editing, and it is easier to judge as right or wrong, because it states exactly what “right” would look like.

Q. A one-line prompt gets back a response that is correct but unusably long for your purpose. Which of the four prompt components did you most likely omit?

A. Constraints (and usually output format alongside it). The model was not wrong; it had no signal about the length or form you needed and defaulted to a thorough general answer.

1.3 Further reading

  • ICMJE, Recommendations: Artificial Intelligence (International Committee of Medical Journal Editors, 2025); COPE, Authorship and AI Tools (Committee on Publication Ethics, 2023); Springer Nature, Editorial Policies: Artificial Intelligence (Springer Nature, 2024). Read these three directly if the legitimacy question is what is holding you back; they are short, and they are the actual basis for the permission this chapter describes.
  • Ji, Z. et al. (2023). Survey of Hallucination in Natural Language Generation. ACM Computing Surveys, 55(12) (Ji et al., 2023). The standard technical survey behind this chapter’s hallucination claims.
  • Wei, J. et al. (2022). Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. NeurIPS (Wei et al., 2022). A foundational prompting technique; worth reading once you are comfortable with the four-part prompt structure above and want to go further on multi-step reasoning tasks.
  • Lewis, P. et al. (2020). Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS (Lewis et al., 2020). The paper behind the retrieval-augmented tool category described above.
  • Ollama (Ollama, 2026) and llama.cpp (Gerganov & contributors, 2026). The two pieces of software behind the local-model option. Read the Ollama quickstart before deciding whether a local model is practical on the hardware you have; it is a shorter setup than most researchers expect.
  • Mata v. Avianca, Inc., 678 F. Supp. 3d 443 (S.D.N.Y. 2023) (United States District Court for the Southern District of New York, 2023). Primary legal record of the sanctions opinion; read it once, in any field, as a concrete picture of what unverified AI output can cost.

1.4 Worked example: framing a research question for an AI assistant

Scattered points of light converging through a prism to a single sharp point

Scattered points of light converging through a prism to a single sharp point: a vague question narrowed into a well-specified prompt.

To illustrate, consider a researcher seeking an AI assistant’s help getting oriented on a question: does remote work affect junior employees’ long-term career progression? Asked exactly that way, a general-purpose assistant will readily produce two or three confident, well-organized paragraphs, citing named studies and stating a qualified but definite-sounding conclusion. That response is a plausible answer to a vague prompt, which is a different thing from being a reliable answer to the actual question, and at this stage every citation and statistic in it is unverified in the sense defined on the Conventions page.

Applying the four-part structure changes what comes back:

Task: Summarize what empirical research says about the effect
of remote work on junior employees' long-term career
progression (promotion rate, mentorship access, visibility).
Context: I am scoping a literature review for a paper in
organizational behavior; I have not yet read the primary
literature.
Constraints: Distinguish correlational findings from causal
claims. Do not present a single study's finding as consensus.
For every claim, name the specific study, its year, and its
method, so I can look each one up myself.
Output format: A bulleted list, one bullet per distinct
finding, each bullet naming the source.

The response to the revised prompt is still not verified; it is, however, now structured for verification, because each claim carries a named source that can be checked rather than an unattributed assertion that cannot. That shift from unattributed confidence to attributed, checkable claims is the entire point of the prompt, and it is also exactly the discipline Day 2 extends to full literature review: never let a claim’s fluency stand in for its verification.

1.5 Homework

Attempt each problem in your own environment before checking the solution.

  1. Describe token prediction in your own words. In two or three sentences, explain what a large language model is doing when it generates the next word of a response.

  2. Provoke a hallucination. Ask an AI assistant a factual question about a narrow topic you know well. Identify one claim in the response that is subtly wrong or unsupported.

  3. Test the context window. In a long conversation, refer back to an instruction you gave much earlier. Does the tool still follow it? What does this suggest about how you should structure long sessions?

  4. Compare two tools on the same task. Give the same research question to two different AI tools. Note where the responses agree, disagree, and differ in usefulness.

  5. Rewrite a weak prompt. Take a one-line prompt lacking context or constraints and rewrite it using the task, context, constraints, output-format structure from the lecture. Compare the two responses.

  6. Name a task this generation of tools should not be trusted with unsupervised. Justify your answer with reference to the failure modes discussed in this chapter.

1.6 Solutions

Problem 1. A satisfactory answer makes three points. The model produces a probability distribution over its vocabulary and samples the next token from it. It conditions only on the text in front of it, consulting no database of facts at generation time. It then repeats that step until the response is complete.

As an illustration, an answer of this shape would qualify: “The model computes which token is most likely to follow the prompt and everything it has generated so far, samples one, and repeats. Those probabilities come from statistical regularities learned during training, not from a lookup against a source.”

An answer that says the model “searches for” or “looks up” the answer has missed the central point of the day. That is exactly the confusion the check-your-understanding callout above corrects.

Problem 2. The instruction to pick a topic you know well is the whole design of the exercise. On an unfamiliar topic you cannot distinguish a fabricated claim from a correct one, which is the warning stated earlier in this chapter.

Most readers will find a claim that is wrong in a small and specific way rather than one that is grossly false. Common shapes include a real author credited with someone else’s finding, a plausible but incorrect date, and a contested question described as settled. A good answer names the claim, states what is actually the case, and says how you knew.

Two other outcomes are worth recording rather than treating as failures. The tool may hedge or decline, which is the behavior we want and is more common on well-known topics. It may also produce a claim you cannot check without opening a source. That is itself the finding, since confidence in the response carries no information about whether verification is needed.

Problem 3. What you observe depends on the length of the session and on how the tool manages its history, so both outcomes are legitimate results. If the instruction was still followed, the session most likely remained inside the context window. If it was dropped, the instruction fell outside that window or was compressed away by the tool’s own summarization.

The conclusion is the same in both cases. Do not treat an early instruction as durable. Restate load-bearing constraints periodically, and paste the material a task depends on into the prompt that needs it rather than relying on an earlier turn.

A stronger answer notes that this failure is silent. The tool does not announce that it has lost an instruction, so the only reliable detection is your own check of the output against the constraint you set.

Problem 4. Three observations distinguish a good answer.

Agreement between two tools is weak evidence of correctness. Models trained on overlapping corpora can be wrong in the same way, so convergence tells you less than it appears to.

Disagreement is genuinely diagnostic. Where the two responses conflict, you have located a claim that at least one tool got wrong, and that is where verification effort belongs first.

Usefulness often differs more than accuracy does. A response that names its sources can be checked in minutes. An equally fluent response without them cannot be checked at all without redoing the search yourself. For research work that difference matters more than the polish of the prose.

Problem 5. Consider a weak prompt of the kind that produces a technically correct but unusable answer:

Summarize this paper.

The four-part rewrite states the task, supplies the context the model lacks, sets the constraints, and names the output format:

Task: Summarize the attached paper for a reader deciding
whether it belongs in a literature review.
Context: [paste the abstract and methods section] The review
concerns whether remote work affects promotion rates in an
employee's first five years.
Constraints: Distinguish what the paper measured from what it
concluded. Do not describe the findings as more definitive than
the paper itself does. Name the study design and sample.
Output format: Four bullets: design, sample, primary finding,
relevance to the stated question.

The comparison to record is not that the second response is longer or better written. It is that the second response can be judged right or wrong, because the prompt stated what a correct answer would contain. The first cannot be judged at all.

Problem 6. Any task whose output is expensive to verify and costly to get wrong belongs on this list. Producing citations for a manuscript without checking each one is the clearest example in this chapter, for the reasons Mata v. Avianca makes concrete (United States District Court for the Southern District of New York, 2023). The fabrication is invisible in the output, and the professional consequence falls on the person who signed the work.

A good justification names a specific failure mode rather than a general unease. Three from this chapter support the answer. Hallucination is structural rather than incidental (Ji et al., 2023). Confidence in a response is uncorrelated with its correctness. Both are worst on narrow or specialized topics, which is where research questions usually sit.

Other defensible answers include the final interpretation of a statistical result, any task involving confidential or unpublished material, and any output submitted under your name without a human review step. The common feature is not difficulty but unverifiability at a glance.

1.7 What’s next

Day 2 applies these foundations to the task most exposed to hallucination: literature review. We shall develop a verification habit for AI-assisted search and synthesis, so that speed does not come at the cost of a single fabricated citation reaching a manuscript.