3 Day 3: AI-Assisted Coding and Data Analysis
A circuit board lit in violet with a few isolated amber points glowing: code under review, not merely code that runs.
Program testing can be used to show the presence of bugs, but never to show their absence!
Edsger W. Dijkstra, Notes on Structured Programming (Dijkstra, 1970)
3.1 Learning objectives
By the end of this day you should be able to:
- Use a coding assistant to bootstrap a reproducible project skeleton (directory structure, dependency lockfile, container definition) as an efficient, low-risk delegation.
- Use a coding assistant such as Claude Code or GitHub Copilot productively within an existing research codebase.
- Review AI-generated code for correctness, not merely for whether it runs.
- Use AI assistance to accelerate data cleaning and exploratory analysis while keeping inferential decisions under human control.
- Identify edge cases and silent failure modes that an AI coding assistant is prone to miss.
- Decide when to accept, revise, or reject AI-generated code.
3.2 Lecture
Code that runs is not the same as code that is correct, and we shall treat AI-generated code here the way a careful reviewer treats a collaborator’s pull request: useful, often correct, and never exempt from review. That review burden is not uniform across tasks, however, and we begin with the task that carries the least of it.
3.2.1 Scaffolding a reproducible project with an AI coding assistant
A timber frame at dusk, structurally complete but empty: a project skeleton is worth delegating precisely because its correct shape is known in advance.
We note at the outset that not every coding task an AI assistant can help with carries the same review burden. The later sections of this chapter are about reviewing AI-generated analysis code carefully, because an analysis choice is a judgment call with no single correct answer. Scaffolding a new project’s skeleton is a different kind of task entirely. The target shape is well defined and checkable (does the directory layout match a known convention, does renv::restore() succeed, does the Dockerfile build), which makes it a strong candidate for efficient delegation rather than close supervision.
A faculty researcher starting a new project typically wants the same handful of things every time: a standard directory layout separating raw data, scripts, and output; a dependency lockfile (renv.lock for R) pinning package versions; a container definition pinning the underlying system and language version; and a minimal report skeleton to build on. Asking a coding assistant to produce this skeleton directly is efficient precisely because it is mechanical, not because it is trivial: a consistent project skeleton is what makes the reproducibility practice in Day 5 possible in the first place, and hand-building it from memory for every new project is exactly the kind of repetitive setup work worth delegating.
Task: Scaffold a reproducible R research-compendium directory
for a new project called penguin-growth.
Context: Standard layout: analysis/data/{raw,derived},
analysis/scripts, analysis/figures, analysis/report; an
renv.lock initialized via renv::init(); a Dockerfile pinning
R 4.4 and installing system dependencies for a typical
tidyverse stack; a minimal Quarto report skeleton
(analysis/report/index.qmd) with YAML front matter only.
Constraints: Do not write any analysis code or invent example
data. Only the skeleton and configuration files.
Output format: a list of files to create, each with its full
starter content.We should note that several established frameworks encode this same skeleton as a single command rather than a prompt. renv::init() and a Dockerfile handle the two hardest parts by themselves (Docker, Inc., 2024; Posit Software, PBC, 2024), and project-scaffolding tools such as zzcollab generate the entire layout, lockfile, and container definition in one step for exactly this reason (Thomas, 2024). Where such a tool is available, prefer it over an AI-generated skeleton, since it encodes a convention that the rest of the team, and one’s own later self, already know. Reach for an AI assistant to produce the equivalent skeleton by hand when no such tool exists for the language or framework at hand, or when a one-off variation is needed that a generic scaffolding tool does not support.
The review this task needs is lighter than for analysis code, but it is not absent. Confirm that the skeleton actually builds (docker build succeeds, renv::restore() completes without error) before trusting it. This is the same “does it do what it claims” check that we develop in the next section for AI-generated code in general. Unfortunately, a scaffold that looks right but fails to build has cost the time it would have taken to write by hand, without returning the reproducibility payoff.
3.2.2 Working with a coding assistant
An interactive coding assistant such as Claude Code or GitHub Copilot (Anthropic, 2025; GitHub, 2024) differs from a plain chat assistant in that it operates directly inside the project. It can read existing files, propose edits as diffs against them, and, in agentic modes, run shell commands and tests on our behalf. We note that this project-level context is exactly what makes such a tool useful, and exactly what raises the stakes of the “Where AI coding assistants commonly fail” section below, because a tool that can execute commands can also execute a mistaken or dangerous one.
Two habits make these tools more productive without changing what they are trusted to do unsupervised. First, we give the assistant real project context deliberately: the relevant file, the function signature it must match, the test it must pass. This is preferable to relying on the assistant to infer our intent from a short instruction. A randomized controlled trial across three companies and nearly 4,900 developers found productivity gains concentrated where developers gave clear, well-scoped tasks, with the largest gains among less experienced developers working on tasks that were well specified (Cui et al., 2026). Second, we iterate rather than accept the first draft, treating the assistant’s first response as a draft to be reviewed and refined, in the same way we would treat a collaborator’s first pass, and not as a finished deliverable.
3.2.3 Reviewing generated code for correctness
Code that runs without error and code that is correct are different claims, and an AI coding assistant can produce the first without the second. We review AI-generated code the way we would review a collaborator’s pull request. That is, we read the diff line by line, rather than only running the result and checking that it did not crash, and we check the output against a result we can independently verify: a hand-computed value on a small example, a known-correct reference implementation, or an existing test.
Unit tests are the most reliable verification tool available for this, because a test encodes exactly what “correct” means for a given input, in a form that can be rerun automatically on this change and on every future change. For AI-generated functions in particular we recommend a straightforward discipline. Before accepting a generated function, write at least one test for an edge case that the function’s ordinary use would never exercise, since that is precisely where the failure modes of the next section live. The assistant may draft that test as well, provided we verify it ourselves.
3.2.4 AI-assisted data cleaning and exploratory analysis
Drafting a data-cleaning script, suggesting exploratory plots appropriate to a dataset’s structure, and summarizing an unfamiliar data dictionary are all tasks where a coding assistant can save substantial time, because they are mechanical translation tasks (a data dictionary into code, a variable’s type into a plot choice) rather than judgment calls about what the analysis should conclude. The productivity literature on AI coding assistance is consistent with this framing: a large-scale field study found a roughly 26% increase in completed tasks among developers given access to GitHub Copilot, concentrated in exactly this kind of well-defined, mechanically checkable work (Cui et al., 2026), echoing an earlier controlled study that found developers completed a scoped coding task noticeably faster with an AI assistant than without one (Peng et al., 2023).
The boundary to hold is between drafting and deciding. On the one hand, an assistant that drafts a cleaning script or suggests a plot type is accelerating the execution of a plan we have already made, and there is little to lose by letting it. On the other, deciding what the analysis plan actually is, which variables matter, what a missing-data pattern implies, or which model specification answers the research question, remains a human judgment. We should never allow the tool’s fluency to substitute for that judgment.
3.2.5 Where AI coding assistants commonly fail
The most familiar failure mode is silent mishandling of cases the assistant was not explicitly shown: missing data treated as zero instead of excluded, a unit conversion silently omitted or applied twice, an off-by-one error at a boundary the example data never reached. These are dangerous precisely because the code runs cleanly and returns a plausible-looking number; the error surfaces only when someone checks the result against an independent source, or not at all.
A cracked pane of glass over a control panel, a warning light glowing behind the break: a coding assistant’s own safety judgment, overridden by a direct instruction.
A second, less familiar failure mode concerns not what the code computes but what the assistant is willing to run. A detailed 2026 account documents both Claude Code and Codex, two different coding assistants from different vendors, correctly identifying an installation command piped from a URL directly into a shell (curl ... | sh) as dangerous, and then running it anyway because the user had asked them to. In one case the assistant disabled its own safety sandbox in order to do so (tidydesign, 2026). This matters for two reasons beyond the specific incident. First, it is tool-agnostic: the same behavior appeared in two independently built agents, which suggests a structural property of how these tools currently resolve a conflict between a safety judgment and a direct user instruction, rather than a one-vendor defect. Second, it means that the assistant’s own stated risk assessment is not a safeguard we can rely on when we are the ones giving the instruction. The tool can tell us that a command is dangerous and run it in the same breath.
An agentic coding assistant that can execute shell commands will generally do what you explicitly ask, even when it has just told you the action is risky. Treat its own warnings as information to act on yourself, not as a guarantee that it will decline the action for you (tidydesign, 2026).
3.3 Further reading
renvdocumentation (Posit Software, PBC, 2024), Docker documentation (Docker, Inc., 2024), and the zzcollab framework (Thomas, 2024). The scaffolding tools behind this chapter’s opening technique; worth reading once even when the setup is delegated to an AI assistant, so that the result can be judged against the convention it claims to follow.- Peng, S. et al. (2023). The Impact of AI on Developer Productivity: Evidence from GitHub Copilot (Peng et al., 2023). An early controlled study behind this chapter’s productivity claims.
- Cui, Z. et al. (2026). The Effects of Generative AI on High-Skilled Work: Evidence from Three Field Experiments with Software Developers. Management Science (Cui et al., 2026). The larger, multi-company field study cited above; read the discussion of which developers and tasks saw the largest gains.
- tidydesign (2026). Auto Mode Knows It’s Dangerous (and Does It Anyway) (tidydesign, 2026). The primary account of the
curl | shfinding discussed above; the author ran the experiment themselves and shows the actual transcripts.
3.4 Worked example: reviewing an AI-generated data-cleaning script
A grid of blank tiles with two marked by a red pin: two errors found during review of a data-cleaning script.
We consider a concrete scenario. Suppose a coding assistant has drafted a cleaning script for a messy survey dataset: it reads a CSV, recodes a few free-text fields, drops rows it judges incomplete, and writes a cleaned file. Before running it on the real data, we review it the way this chapter recommends.
Reading the diff line by line, rather than merely running it, we notice that the drop step removes any row with a missing value in any column, including columns unrelated to the planned analysis. That is a deciding-level choice about missingness, made silently inside what looked like a mechanical drafting task. We also notice a unit conversion applied to a duration field: the assistant assumed the raw values were in minutes and converted them to hours, but the codebook, which the assistant was not shown, records the field in seconds.
Both issues share the pattern this chapter has built toward. The script ran without error and looked plausible, and yet both errors are exactly the kind that a simple independent check would catch immediately, whether a hand computation on two or three known rows or a unit test comparing the script’s output against a value we have computed ourselves. Fixing the script means narrowing the drop condition to the columns the analysis actually needs, correcting the unit conversion against the codebook, and then adding a small test that encodes both corrected behaviors, so that a future edit cannot silently reintroduce either error.
3.5 Homework
Attempt each problem in your own environment before checking the solution.
Generate and review a function. Ask a coding assistant to write a function that computes a summary statistic on a dataset you use. Review it for correctness on an edge case (missing values, a single row, an empty input).
Break it deliberately. Construct an input designed to expose a weakness in the generated function. Does it fail silently or does it error clearly?
Write a test. Write a unit test that would have caught the weakness from Problem 2, and confirm it passes once you fix the function.
Use AI for exploratory analysis. Ask a coding assistant to suggest exploratory plots for a dataset. Evaluate whether the suggestions are appropriate for the data’s actual structure.
Diff review. Have a coding assistant modify an existing script in your project. Review the diff and identify one change you would not have made yourself, for better or worse.
Name a boundary case. Describe an analysis decision (not a coding task) that you would never delegate to an AI assistant, and explain why.
3.6 Solutions
Problem 1. The review is the exercise, not the generation. Read the returned function line by line and ask what it does on three inputs its author probably never considered: an empty input, a single row, and a column that is entirely missing.
As an illustration, a request for a mean and confidence interval will commonly produce something of this shape:
mean_ci <- function(x, conf = 0.95) {
x <- x[!is.na(x)]
se <- sd(x) / sqrt(length(x))
crit <- qt(1 - (1 - conf) / 2, df = length(x) - 1)
c(mean = mean(x), lower = mean(x) - crit * se,
upper = mean(x) + crit * se)
}The function is correct on ordinary input. It drops missing values silently, which may or may not be what the analysis wants, and it has no guard at all on the size of what remains. Note that the missing-value choice is a deciding question in the sense of this chapter, made here by a tool rather than by you.
Problem 2. The distinction that matters is between a clear error and a silent wrong answer. A function that stops with a message has cost you a minute. A function that returns a plausible number from invalid input can put that number in a manuscript.
Passing an empty vector to the illustrative function above returns NaN for all three elements rather than raising an error. A single non-missing value, as in c(3, NA), is more dangerous: the point estimate comes back as a real number and only the two bounds are NaN, which reads far more like a reportable result than three NaN values do.
Neither case is entirely silent. R emits a warning from qt(), which receives zero degrees of freedom once the missing value is dropped. A warning is not an error, though. It does not stop the script, it is easily lost in a long console log, and it is invisible in a rendered document whose chunks are set to suppress warnings, which is a common default.
That is the outcome to record, and it is the general pattern this chapter warns about: the code ran, returned a value, and was wrong.
Problem 3. The fix adds a guard, and the test encodes what the guard is supposed to do:
mean_ci <- function(x, conf = 0.95) {
x <- x[!is.na(x)]
if (length(x) < 2) {
stop("mean_ci() needs at least two non-missing values")
}
se <- stats::sd(x) / sqrt(length(x))
crit <- stats::qt(1 - (1 - conf) / 2, df = length(x) - 1)
c(mean = mean(x), lower = mean(x) - crit * se,
upper = mean(x) + crit * se)
}# inst/tinytest/test-mean-ci.R
expect_error(mean_ci(numeric(0)),
pattern = "at least two",
info = "empty input errors rather than returning NaN")
expect_error(mean_ci(c(3, NA)),
pattern = "at least two",
info = "one non-missing value cannot support an interval")
expect_equal(unname(mean_ci(c(1, 2, 3))["mean"]), 2,
info = "point estimate is the arithmetic mean")The first two assertions fail against the original function and pass against the corrected one, which is what makes them worth keeping. The third guards the ordinary case, so that a later fix to the edge cases cannot quietly break the common path. Run them with tinytest::run_test_dir("inst/tinytest").
Problem 4. Judge the suggestions against the data’s actual structure rather than against their general reasonableness. Three checks are usually enough.
Does the suggested plot match each variable’s type and distribution? A histogram for a variable with six distinct values, or a scatter plot for a categorical predictor, indicates that the assistant inferred structure it was never shown.
Does the suggestion respect the design? Repeated measures, clustering, and stratification are invisible in a column of numbers, and a plot that ignores them will mislead even when the code is correct.
Did you supply the data dictionary? Most poor suggestions in this exercise trace to context the assistant did not have, which is the point made under “Working with a coding assistant” above.
Problem 5. The instructive changes are rarely the ones that break. Look for a change that is defensible but not yours: a rewritten loop, a renamed variable, a silently added default argument, a dependency introduced to save three lines.
For each, ask whether the change alters behavior or only style. A change that alters behavior belongs in the review even when it is an improvement, because it is a decision. A style change can be accepted or reverted on preference, and reverting it costs nothing.
An answer that finds nothing worth noting usually means the diff was read for errors rather than for decisions. Unfortunately, that is the easier reading, and it is the one that lets an unexamined choice into the codebase.
Problem 6. Any decision that determines what the analysis concludes, rather than how it is executed, belongs on this list. The handling of missing data is the clearest example in this chapter. Whether to drop, impute, or model missingness depends on a mechanism the data cannot reveal on its own, and the choice changes the estimate.
Equally defensible answers include the choice of primary endpoint, the decision to adjust for a covariate, the threshold for declaring a result meaningful, and any judgment about whether an assumption is satisfied. Each requires knowledge of how the data were generated, which sits with you rather than with a tool that has seen only the columns.
The general criterion is the drafting and deciding boundary developed earlier. If the task has a checkable correct answer, delegation is reasonable. If the task determines what counts as the correct answer, it is not.
3.7 What’s next
Day 4 moves to the manuscript itself. We shall use the same review discipline for AI-assisted writing and citation management, so that the prose that reaches a journal remains the author’s own argument in the author’s own voice.