Skip to content

1.2 Importing data

Most of step one is setting up how you and the agent will work and loading in data. The example in the recording is deforestation against GDP per person, but the sequence is the same whatever data you bring.

Setting the folder up

Download the human-agentic-data-analysis vault and open it in Obsidian with Open folder as vault. Three things need installing once, and Setup.md covers all of them: the Claudian plugin, which puts the agent inside Obsidian so the conversation happens beside the documents; Claude Code, which Claudian drives but does not include; and uv, which builds the Python environment from a lockfile so your results reproduce on someone else's machine. The environment is one command, cd Code && ./setup_env.sh.

The instructions.md are read by the agent automatically at the start of every conversation. Open a blank Claude window and you negotiate how to work while you are working; here the terms are already agreed — one step per request, say which step you are on, propose a method before applying it, do not run ahead. If you don't know how to start just ask in the Claude window!

The report and the appendix

Report.md starts as an empty template: abstract, introduction, methods, results, discussion. Under each heading is a note saying which step fills it. Research questions go in at step 2, figures at step 3, methods and results together at step 4, the discussion at step 5. The background comes near the end, and the abstract last, because it is the only section that has to be true of every other one. By step 7 there is no writing left to do, only a read-through.

Appendix.md fills up alongside it and takes everything the report does not: the decisions and their reasoning, the workings, the checks that changed nothing.

Getting the data

Say what you want to investigate. That is not yet the research question — settling that is step 2 — just enough for the agent to work out what data you need. It then either uses data you already have or goes looking. You will often get a multiple-choice question here. That is the agent surfacing a decision rather than making it, so read it rather than clicking the recommended option.

Then open the provider's own web page and check it. Who collected the data, over what period, measuring what. This is the step people skip, and it is the only way to catch a confident description of a dataset that is not quite the dataset you are getting. Many sites block agents from scraping them. Some need an API key, some simply refuse, and sometimes you will have to fetch a file yourself.

Two things exist afterwards that are worth opening. The download code goes in Code/ — in the recording it is the SQL that was actually run — so you can read what was requested rather than taking the agent's word for it. The data goes in Attachments/ and is never modified.

The agent then describes the file: shape, columns, coverage, completeness, and anything odd, including the gaps. It does this in two passes, first what is in the file and then whether it can answer your question. Merged into one you get a summary that quietly argues for a decision you have not been asked to make yet.

All of it goes to Data Summary in Appendix.md, with where the file came from, when, and a checksum. Report.md is not touched. When you pick a subset, that decision and its cost are recorded and the description of the data you are not using is deleted.

The agent will try to move on. In the recording it finishes the import and proposes framing the research question, and has to be told to wait. The hold is worth making. Pushed on whether the data could answer the question, it proposed a restriction on which countries to keep and said what that cost. A second question — what to divide through by — it declined, saying that belonged to step 2. That is the right answer. Some questions get noted and carried forward rather than settled where they are raised.

Also see