How to get the most out of your tokens

[01] · ADVANCED

Get more value from each research session

This guide builds on the beginner exercise. It is for researchers who can inspect a file change and run a command, and who want to stop re-explaining the same analysis procedures. You will follow a small experiment comparison from a vague request to a reusable tool and skill.

Read the guide in order for the worked example, or use Contents to return to a particular step. The downloadable data is deliberately tiny: you can check the arithmetic yourself before trusting a tool with real results.

Tokens carry your instructions, context and the model’s response. A short request can still lead to a long, unproductive session if the goal is unclear or essential context is missing.

For repeated research work, the opportunity is larger: avoid re-explaining your methods and rebuilding the same checks. Use Ask → Plan → Act to turn a successful task into a process you can reuse.

[02] · ADVANCED

Ask → Plan → Act is the backbone

The stages do two jobs. They help you spend effort on the right problem now, and they reveal what should be saved for later. Questions that recur may belong in a checklist; stable calculations may belong in a script; the sequence around that script may belong in a skill.

Keep the research goal visible throughout. “Write a script” is an implementation step. “Produce a comparable, traceable summary of these two methods” explains why the script is needed and gives you a way to judge the result.

Ask → Plan → Act is a way of working, independent of the tool’s mode names. A small, clear task can move through these stages quickly; uncertain work needs more time in Ask and Plan.

[03] · ADVANCED

Ask: what are we trying to establish?

Imagine that you have repeated runs of two methods and want a table for a paper. Asking for “a comparison table” leaves important choices open: which runs count, which split was used, what the metric means and whether a missing run should block the comparison.

For the follow-along exercise, create a disposable research folder. Download the two files below into results/ and configs/ respectively. The CSV contains six invented measurements. The protocol fixes the methods, seeds, metric units and aggregation rule.

Download runs.csv · Download evaluation.json

Start with the research goal, not a request to “analyse everything”. Identify what evidence is needed and which decisions are still open.

Goal: compare Baseline and Candidate on the same evaluation.
Read the protocol and the results schema before making changes.
Identify the required runs, comparability checks and missing data.
Explain which questions these results can and cannot answer.

[04] · ADVANCED

Plan: define the result and how to check it

The supplied protocol expects one row per method and seed: two methods and three seeds, giving six rows. It fixes one dataset version, split and metric. An acceptable tool must check those conditions before computing a mean.

Ask the agent for a plan using these files. Review the proposed inputs, outputs and failure cases. Agree that the comparison should stop on an incomplete or incompatible input rather than silently produce a misleading table. For a real study, the researcher must decide these rules.

Convert the clarified goal into a bounded procedure. Decide the aggregation rules before looking for the most favourable result.

Inputs: results CSV, protocol and expected run list.
Steps: validate → aggregate → compare → render → review.
Outputs: validation report, comparison table and source manifest.
Done when: agreed checks pass and every number is traceable.
Stop when: required runs are missing or protocols differ.
Boundaries: preserve raw results; do not invent or drop runs.

[05] · ADVANCED

Act: run the plan and show the evidence

At this stage you have a procedure, but you may not yet have a tool to execute it. The next sections show how to build one with the same Ask → Plan → Act pattern. Return to this step once that tool passes its small checks.

Inspect both the result and the route used to produce it. If validation fails, that can be the correct outcome: discovering a missing seed is more useful than receiving an apparently complete table. Keep the failure report so the next action is clear.

Carry out the agreed comparison on the small fixture first.
Then process the supplied results if validation passes.
Return the table, its source manifest and a concise interpretation.
If a check fails, report the offending records and stop that step.

[06] · ADVANCED

Choose the repetition worth capturing

Do not automate every task immediately. Start with one you have already performed and can recognise when it goes wrong. Repetition gives you examples of the common path and the exceptions; those examples help you define a useful contract.

The table below separates routine operations from research judgment. An agent can help draft an interpretation, but a schema check alone cannot establish that a dataset is representative or that an experiment supports a scientific claim.

[07] · ADVANCED

Put repeated work in the right place

A tool performs an operation; a skill tells the agent when and how to use it. The distinction matters because a model should not have to reconstruct the same calculation from prose each time you request a table.

For the example, the script checks rows and computes means. The skill establishes the intended comparison, invokes the script, handles a validation failure and explains the result. Project instructions hold conventions that apply across many tasks, such as preserving raw observations.

[08] · ADVANCED

Build the tool once, check it each time

Use the downloaded CSV and protocol as concrete inputs when asking the agent to build the script. Request a command-line interface with the three arguments shown below, using the standard library if that is sufficient. Specify summary.json, table.csv and manifest.json as the output files so that both humans and later tools can inspect the result.

Ask it to explain the proposed schema and checks before editing. After implementation, inspect the code and run the known-answer checks in the next section. The interface is an exercise specification: this website supplies the data and protocol, while you and your agent build the tool in your practice repository.

Apply Ask → Plan → Act to the tool itself: inspect a real example, define its contract, then implement and test one operation. The following is a proposed interface to build in your research repository, not a command shipped with this presentation.

python scripts/compare_runs.py \
  --input results/runs.csv \
  --protocol configs/evaluation.json \
  --output results/comparison-v1/

Build a comparison tool for this CSV and protocol.
First inspect the inputs and propose a small interface.
Keep the aggregation rules in code and document them.
Add tiny fixtures with hand-calculated expected results.
Report missing runs; never silently drop or invent them.
Generate a table and provenance manifest from the same inputs.

[09] · ADVANCED

Make correctness observable

A fixture is a small, controlled input whose expected behaviour you know. The provided CSV is the valid fixture. The values below let you verify the arithmetic without relying on the assistant’s own assessment of its output.

Then create an invalid copy for each failure case. Keep these checks separate from the valid input, and confirm that each produces a clear error. This establishes that the tool rejects misleading inputs as well as handling the happy path.

Use a small fixture you can calculate by hand before processing a real experiment. These numbers are illustrative, not research results.

Expected difference: +0.03 on the 0–1 scale, or +3 percentage points. This toy example establishes the arithmetic, not statistical significance or generalisation.

• Check the valid fixture against its hand-calculated answer.
• Remove one expected seed: the tool should report an incomplete comparison.
• Duplicate a method–seed row: it should fail rather than count the run twice.
• Change the dataset version, split or metric units: it should reject the comparison.
• Check that the rendered table matches the machine-readable output and retains the run count.

[10] · ADVANCED

Save the procedure as a reusable skill

Only wrap a process you have actually checked. Ask the agent to draft a skill from the successful run, compare that draft with the commands and decisions you used, and remove steps that were speculative or unnecessary.

The example references a protocol document in the skill’s references/ folder. Create that file with the comparison rules and the location of your project’s protocol JSON, or change the reference to an existing document. Install the skill in a location your tool supports, then invoke it explicitly on a fresh copy of the fixture.

Once the tool works, capture the repeated procedure in a skill. Keep detailed schemas and examples in linked files. This illustrative skill assumes the comparison script has been implemented and tested.

# compare-experiments/SKILL.md
---
name: compare-experiments
description: Produce checked comparisons of experiment runs. Use when updating benchmark tables or summarising repeated runs.
---
1. ASK: Read [the protocol](references/evaluation-protocol.md).
2. PLAN: Confirm methods, dataset version, split, metric and seeds.
   Ask if a required choice is missing; do not infer a new protocol.
3. ACT: Run scripts/compare_runs.py with explicit input, protocol
   and a new output directory. Follow the tool's --help.
4. If validation fails, report offending records and stop.
   Do not omit failed runs or fabricate missing measurements.
5. Check the table against the summary and provenance manifest.
6. Draft an interpretation citing the output files and limits.
7. Report commands, checks, outputs and unresolved questions.
   Propose workflow improvements for review before saving them.

Use a supported skill location and required frontmatter. Invoke the skill explicitly for its first trial, then check that the agent loaded it and followed the procedure. File locations and activation rules vary by tool.

Source: VS Code skill structure and loading

[11] · ADVANCED

Close the loop: fewer repeated corrections

The value of a saved workflow appears on the next run. Give the agent an input with one missing seed and invoke the skill. It should identify the missing record and explain why the comparison cannot yet proceed; it should not fill in a plausible number.

If it misses the problem, locate the gap. Was the check absent from the script, the skill not loaded, or the failure ignored? Improve the relevant part and rerun the same case. This turns a correction into an inspectable improvement instead of another instruction buried in chat.

Use the compare-experiments skill for Baseline and Candidate.
Inputs: results/runs.csv and configs/evaluation.json.
Write a new results/comparison-v2/ directory.
Report the validation outcome and source files for every number.
If data is incomplete, explain what is missing before summarising.

[12] · ADVANCED

Keep durable context close to the research

The directory layout below is a map of what you want to retain. It separates rules, executable operations, small tests and observed outputs. You can adapt the names to your project; the useful property is that another person can find the method and its evidence.

A file is durable storage, but it is not automatically active context. Link to the relevant protocol from the skill and point the agent to the right files for each task. Keep large logs available for investigation rather than pasting them into every conversation.

README.md                         # first working command
AGENTS.md                         # project conventions
configs/evaluation.json           # declared comparison rules
scripts/compare_runs.py            # tested calculations
fixtures/                         # valid and invalid tiny examples
results/comparison-v1/             # outputs + provenance
skills/compare-experiments/          # use your tool’s supported location
  SKILL.md
  references/evaluation-protocol.md

Keep raw observations separate from derived outputs. Record the dataset version, code revision, environment, seeds and exact command for each result. For literature work, retain the source location and supporting passage for each extracted claim.

[13] · ADVANCED

Spend context deliberately

Once the process is reusable, focus on what changes between runs. You should usually need to provide the new inputs and the goal, while the saved procedure supplies the established checks. Review the result each time; reuse reduces repeated explanation, not the need for evidence.

If a session becomes confusing, write a short handoff before restarting. Include what has been done and what remains uncertain. A new conversation without that state may repeat the same investigation, so a restart should be a deliberate reset rather than a universal token-saving rule.

[14] · ADVANCED

Check whether the process is improving

To see whether the workflow helps, compare similar tasks and keep a small run log. Record corrections you had to make, checks that failed, time spent reviewing and any usage information the tool exposes. An apparently fast run that needs substantial repair may be less useful than a slower, correct one.

Look for improvement across several runs rather than one convenient success. If maintaining a skill costs more effort than the repeated task saves, simplify it or return to a plain documented command.

Trial the workflow on comparable tasks before and after a change. Keep the inputs and acceptance checks fixed where possible. Recheck after changing a tool, skill or model.

[15] · ADVANCED

Same goal, a better starting point next time

You have completed the exercise when the valid fixture produces the known answer, invalid fixtures produce useful failures, and the skill can guide a repeat run without you re-explaining every step. Keep the script, checks and reviewed procedure together.

Now choose one real research process with the same shape: a clear input, a repeated method and an output you can check. Replace the teaching rules with your actual protocol, then work through Ask → Plan → Act again.

• ASK to clarify the goal and gather relevant context.
• PLAN the procedure, boundaries and success checks.
• ACT, verify the result and report the evidence.
• Save tested commands in tools and reusable procedures in skills.
• Use the next run to check whether those changes reduced rework.

← Beginner presentation