Finish a brief someone can check

You are organizing a workshop and need to know whether a room is suitable. Two notes describe its seating, entrance and hire price. A polished assistant response is less useful than a short brief that shows exactly where each answer came from and admits what the notes do not say.

Your finished artifact: brief.md, a planning note with source labels and an explicit unanswered question. This is the research-assistant track of our practical agent learning path. You can start here without reading the trading experiments.

Input: Can Harbor Room seat our group, provide step-free access, and tell us the cancellation notice? Finished result: 24 seats, an east entrance with step-free access, and cancellation notice marked NOT FOUND. The capacity and access answers retain complete quotations. Harbor Room, its notes and its prices are fictional teaching data.

For two tiny documents, reading them yourself is probably simpler. A model becomes worth testing when the material is longer or differently worded. Before delegating that work, build the small evidence check below. A citation connects an answer to a source; it does not establish that the source is correct or that its sentence answers your question.

Why this problem deserves a lesson

A public LangChain feature request about inline citations, opened July 5, 2023, describes models citing document numbers outside the supplied set. It is an individual developer report, not a current failure rate. Separately, Stack Overflow's 2026 trust survey item, checked October 15, records 6,872 of 14,304 respondents choosing trust when output is easy to verify. That self-selected developer sample does not measure this newsletter's audience or search traffic.

The ALCE citation research, version dated October 31, 2023, separates answer correctness from citation quality. We use that distinction, not its benchmark scores, as our design starting point. This lesson reproduces no published model experiment and claims no new citation architecture.

Research question: Can a small, inspectable program turn proposed evidence into a useful brief while rejecting invented references and making the remaining review work visible?

1. Prepare one clean folder

You need Python 3.10 or newer, a terminal and a text editor. Check with python --version; use python3 throughout if that is your installation's command. Create an empty folder, save the complete file below as lab.py, then run python lab.py from that folder. No packages, API keys or private repository access are required. Allow about 25 minutes.

All sample data lives in SOURCES. Its short identifiers, room and hire, are labels assigned by the program, not URLs a model must invent. A proposal is a list of candidate evidence rows. Each row names a question slot, one approved source, and one exact sentence. The program checks the proposal before writing the brief.

The dates describe the fictional notes. They do not prove that a real venue's conditions are current. For your own project, record when you obtained the documents and who approved using them. Do not place private customer records in this exercise.

2. Build and run the evidence check

The default proposal is deliberately scripted. Running this file makes no model calls, performs no search and sends nothing. It tests one component that a research assistant could use, rather than pretending to be an autonomous agent.

import argparse
import json
from pathlib import Path

# Fictional, approved notes. No web requests or model calls occur here.
SOURCES = {
    "room": {"title": "Harbor Room facts", "date": "2026-10-01",
             "sentences": ["The room seats 24 people.",
                           "Step-free access is available through the east entrance."]},
    "hire": {"title": "Harbor Room hire note", "date": "2026-10-02",
             "sentences": ["A half-day booking costs 180 demo units."]},
}
QUESTIONS = {"capacity": "Capacity", "access": "Access",
             "cancellation": "Cancellation notice"}
SAMPLE = [
    {"slot": "capacity", "source": "room", "quote": SOURCES["room"]["sentences"][0]},
    {"slot": "access", "source": "room", "quote": SOURCES["room"]["sentences"][1]},
]


def unique_object(pairs):
    result = {}
    for key, value in pairs:
        if key in result:
            raise ValueError("duplicate JSON key")
        result[key] = value
    return result


def validate(rows):
    if type(rows) is not list or len(rows) > len(QUESTIONS):
        raise ValueError("expected at most three evidence rows")
    accepted = {}
    for row in rows:
        if type(row) is not dict or set(row) != {"slot", "source", "quote"}:
            raise ValueError("unexpected or missing fields")
        if any(type(value) is not str for value in row.values()):
            raise ValueError("fields must be strings")
        slot, source, quote = row["slot"], row["source"], row["quote"]
        if slot not in QUESTIONS:
            raise ValueError("unknown question")
        if slot in accepted:
            raise ValueError("duplicate question")
        if source not in SOURCES:
            raise ValueError("unknown source")
        if quote not in SOURCES[source]["sentences"]:
            raise ValueError("quote is not an approved complete sentence")
        accepted[slot] = row
    return accepted


def render(rows):
    accepted = validate(rows)
    lines = ["# Workshop planning brief", "", "Status: needs human relevance review", ""]
    for slot, label in QUESTIONS.items():
        row = accepted.get(slot)
        if row is None:
            lines.append(f"- {label}: NOT FOUND in the supplied notes.")
        else:
            source = SOURCES[row["source"]]
            lines.append(f'- {label}: "{row["quote"]}" '
                         f'[{row["source"]}: {source["title"]}; {source["date"]}]')
    return "\n".join(lines) + "\n"


def checks():
    bad = [
        [{**SAMPLE[0], "source": "invented"}],
        [{**SAMPLE[0], "quote": "The room seats 240 people."}],
        [SAMPLE[0], SAMPLE[0]],
        [{**SAMPLE[0], "send_email": True}],
        [{**SAMPLE[0], "source": ["room"]}],
        [{**SAMPLE[0], "slot": "book_now"}],
        SAMPLE * 2,
        {},
    ]
    for rows in bad:
        try:
            validate(rows)
        except ValueError:
            pass
        else:
            raise AssertionError("invalid proposal accepted")
    assert render([]).count("NOT FOUND") == 3
    irrelevant = [{**SAMPLE[0], "quote": SOURCES["hire"]["sentences"][0], "source": "hire"}]
    assert validate(irrelevant)  # Deliberately shows the relevance gap.
    return len(bad)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--proposal", type=Path)
    args = parser.parse_args()
    rows = SAMPLE
    if args.proposal:
        raw = args.proposal.read_bytes()
        if len(raw) > 4096:
            parser.error("proposal exceeds 4096 bytes")
        try:
            rows = json.loads(raw, object_pairs_hook=unique_object)
            validate(rows)
        except (ValueError, UnicodeError) as error:
            parser.error(str(error))
    rejected = checks()
    brief = render(rows)
    Path("brief.md").write_text(brief, encoding="utf-8")
    print("OFFLINE EXERCISE: no model or external service called")
    print(brief, end="")
    print(f"NEGATIVE CHECKS: {rejected} invalid proposals rejected")
    print("RELEVANCE CHECK: a real quote can still answer the wrong question")
    print("SAVED: brief.md; review before sharing")


if __name__ == "__main__":
    main()

Expected complete output:

OFFLINE EXERCISE: no model or external service called
# Workshop planning brief

Status: needs human relevance review

- Capacity: "The room seats 24 people." [room: Harbor Room facts; 2026-10-01]
- Access: "Step-free access is available through the east entrance." [room: Harbor Room facts; 2026-10-01]
- Cancellation notice: NOT FOUND in the supplied notes.
NEGATIVE CHECKS: 8 invalid proposals rejected
RELEVANCE CHECK: a real quote can still answer the wrong question
SAVED: brief.md; review before sharing

Open brief.md in your editor. It contains the heading, review status and three question lines above. The final test messages belong to the terminal, not the saved brief. Running again replaces that local draft; copy it elsewhere first if you want to retain an edited version.

3. Understand what the checks actually enforce

The allowed questions keep the report tied to the task. Exact field names prevent an extra send_email field from becoming an instruction. The approved-source lookup rejects invented references. Complete-sentence membership rejects a change from 24 seats to 240, even when the source label is genuine. Duplicate question rows are rejected so one answer cannot quietly overwrite another.

An empty proposal remains valid: it produces three unanswered slots. That is a useful fallback when evidence is unavailable. NOT FOUND means the supplied proposal contains no accepted sentence for that slot. It is not proof that the answer is absent from the wider world, or even that a model searched every supplied note properly.

The deliberately irrelevant proposal is the most important negative case. It places a genuine hire-price sentence under the capacity question. The mechanical validator accepts it because the citation exists and the quotation matches. A person must reject it during relevance review. The program advertises this limitation rather than calling every accepted row a verified answer.

4. Try model assistance without hiding the boundary

Once the replay works, give a language model only the fictional SOURCES, the three slot names and this instruction: return a JSON list with slot, source and quote; use complete supplied sentences; omit a slot if no sentence answers it; include no other fields or surrounding commentary. JSON is a text format with double-quoted keys and strings. The sample list in the file shows the required shape.

Save the model's list as proposal.json in the same folder. Run python lab.py --proposal proposal.json. The command reads the candidate file, rejects invalid rows and writes a draft only after validation. Duplicate JSON keys are rejected during decoding. Invalid input should exit with an error, not silently repair a quotation. A previous brief remains on disk after a rejected run, so never treat file existence as evidence that the latest proposal succeeded.

This optional step is a single model-assisted workflow, not a model choosing tools repeatedly. No API integration or model accuracy was tested for this edition. You manually carry the fictional data and response between the model and the checker. Record which model and date you used if you compare results; do not expect its wording or choices to match the scripted replay.

5. Apply it to your own research task

Replace the fictional notes with a small approved set and change QUESTIONS to the decisions you actually need to support. Keep source labels stable within a run. Preserve complete sentences and important qualifications rather than clipping a phrase that reverses the meaning. If two documents disagree, retain both in a separate review note; this simple one-row-per-question format is intentionally insufficient for resolving conflicting evidence.

Before sharing a brief, check every quotation against its source, then ask whether it answers the named question with all conditions intact. Mark missing answers for follow-up. Compare the effort with reading the documents manually, including preparation and review time. Additional automation earns its place only if that comparison supports it.

Troubleshooting and completion

File not found usually means the terminal is in a different folder. A JSON error often comes from Markdown fences or single quotes; save only the list. An unknown-source error means a label is outside SOURCES. A rejected quotation must be fixed by consulting the note, not by weakening validation. An irrelevant but accepted sentence needs human review or a better-designed relevance test.

Completion check: reproduce every output line, open the saved brief, show that an invented quotation is rejected, and explain why the hire-price sentence cannot establish seating capacity. Then remove both sample rows and confirm all three slots remain unanswered.

The experiment establishes deterministic source and quotation checks on a tiny synthetic set. It does not establish document authenticity, retrieval completeness, semantic relevance, production security, time savings or an accurate AI model. Next experiment: let a model choose which approved note to read, limit its reading budget, and compare its completed brief with this fixed workflow on the same questions.

References you can use

Sources checked October 15, 2026. The code and fictional exercise are original educational material.