15. Final project
Put together everything you have seen in Section 1 in a program you could actually use:
- download data in streaming mode (ch. 11);
- validate every record with a typed model (ch. 7–8), discarding the broken ones;
- aggregate (counts, averages, top-N) with the language’s data structures;
- write a readable JSON report (ch. 4–6);
- verify the result against an independent source (
/api/v1/stats).
The task
Section titled “The task”From the Aurora Bookshop catalog, produce report.json with this shape:
{ "books": 80, "avg_price": 19.9, "by_category": [ { "category": "saggistica", "books": 14, "avg_price": 17.04 }, … ], "top_authors": [ { "name": "Matteo Rinaldi", "books": 8 }, … ], "formats": { "tascabile": 30, "brossura": 26, "rilegato": 24 }}Source: GET /api/v1/export/books.ndjson (one book per line, with author_name and category already resolved by the server). Check: GET /api/v1/stats, which computes the same things in SQL.
- Download the export in streaming mode and, line by line, validate it with a model (
Libro). Count the discarded lines. - Accumulate the aggregates while you read (do not keep all the books in memory, even if here there are only 80: it is the habit that counts).
- Build the report and write it to a file with indentation and UTF-8.
- Download
/statsand compare the average price and books per category: they must match.
curl, explained
Section titled “curl, explained”The two calls the program will make, seen “bare”:
curl -s https://corso-api-json.giuseppebosi.com/api/v1/export/books.ndjson | wc -lcurl -s https://corso-api-json.giuseppebosi.com/api/v1/stats | jq '{prices, cat: .books_by_category[0:2], top: .top_authors[0:2]}'- | wc -l
- counts the lines of the export: 80, that is the records, without parsing
- jq '{prices, cat: .books_by_category[0:2], top: .top_authors[0:2]}'
- extracts the three pieces the report must reproduce;
[0:2]is a slice: the first two elements
80{ "prices": { "min": 7.13, "avg": 19.9, "max": 34.49 }, "cat": [ { "slug": "saggistica", "name": "Saggistica", "books": 14, "avg_price": 17.04 }, { "slug": "ragazzi", "name": "Ragazzi", "books": 12, "avg_price": 16.72 } ], "top": [ { "id": 18, "name": "Matteo Rinaldi", "books": 8, "avg_rating": 4.22 }, { "id": 1, "name": "Elena Marchetti", "books": 6, "avg_rating": 3.12 } ]}Python and TypeScript
Section titled “Python and TypeScript”"""Progetto finale: da NDJSON a report. pip install httpx pydantic · python progetto.py [report.json]"""import json, sysfrom collections import Counter, defaultdictimport httpxfrom pydantic import BaseModel, Field, ValidationError
BASE = "https://corso-api-json.giuseppebosi.com"
class Libro(BaseModel): # il contratto: ciò che DEVE esserci per contare il libro id: int title: str = Field(min_length=1) price: float = Field(ge=0) author_name: str category: str in_stock: bool attributes: dict
def scarica() -> list[Libro]: libri, scartati = [], 0 with httpx.stream("GET", f"{BASE}/api/v1/export/books.ndjson", timeout=30) as r: r.raise_for_status() for riga in r.iter_lines(): if not riga: continue try: libri.append(Libro.model_validate_json(riga)) # parsing + validazione in un colpo except ValidationError as e: scartati += 1 print("riga scartata:", e.errors()[0]["msg"], file=sys.stderr) print(f"letti {len(libri)} libri, scartati {scartati}", file=sys.stderr) return libri
def report(libri: list[Libro]) -> dict: per_cat = defaultdict(list) for l in libri: per_cat[l.category].append(l.price) autori = Counter(l.author_name for l in libri) return { "books": len(libri), "avg_price": round(sum(l.price for l in libri) / len(libri), 2), "by_category": sorted( [{"category": c, "books": len(p), "avg_price": round(sum(p) / len(p), 2)} for c, p in per_cat.items()], key=lambda x: -x["books"]), "top_authors": [{"name": n, "books": k} for n, k in autori.most_common(3)], "formats": dict(Counter(l.attributes.get("format", "n/d") for l in libri)), }
def confronta(mio: dict) -> None: ufficiale = httpx.get(f"{BASE}/api/v1/stats").json() print("prezzo medio: mio", mio["avg_price"], "| server", ufficiale["prices"]["avg"], file=sys.stderr) miei = {c["category"]: c["books"] for c in mio["by_category"]} loro = {c["slug"]: c["books"] for c in ufficiale["books_by_category"]} print("categorie uguali:", miei == loro, file=sys.stderr)
if __name__ == "__main__": libri = scarica() r = report(libri) confronta(r) destinazione = sys.argv[1] if len(sys.argv) > 1 else "report.json" with open(destinazione, "w", encoding="utf-8") as f: json.dump(r, f, indent=2, ensure_ascii=False) print("scritto", destinazione, file=sys.stderr)Output on stderr (the report goes to a file, so python progetto.py > altro does not mix them):
letti 80 libri, scartati 0prezzo medio: mio 19.9 | server 19.9categorie uguali: Truescritto report.jsonChoices worth noticing: model_validate_json parses and validates in one go (faster than json.loads + model_validate); the libri list is kept only for clarity — for a huge export you would pass the data to report() as you read; ensure_ascii=False keeps accented letters readable in the file.
// Progetto finale: da NDJSON a report. npm install zod · node --experimental-strip-types progetto.ts [report.json]import * as readline from "node:readline";import { Readable } from "node:stream";import { writeFile } from "node:fs/promises";import { z } from "zod";
const BASE = "https://corso-api-json.giuseppebosi.com";
const Libro = z.object({ // il contratto id: z.number().int(), title: z.string().min(1), price: z.number().nonnegative(), author_name: z.string(), category: z.string(), in_stock: z.boolean(), attributes: z.record(z.string(), z.unknown()),});type Libro = z.infer<typeof Libro>; // il tipo TS deriva dallo schema: una sola fonte di verità
async function scarica(): Promise<Libro[]> { const r = await fetch(`${BASE}/api/v1/export/books.ndjson`); if (!r.ok) throw new Error(`HTTP ${r.status}`); const libri: Libro[] = []; let scartati = 0; for await (const riga of readline.createInterface({ input: Readable.fromWeb(r.body as any) })) { if (!riga) continue; const esito = Libro.safeParse(JSON.parse(riga)); // safeParse: niente eccezioni, un esito if (esito.success) libri.push(esito.data); else { scartati++; console.error("riga scartata:", esito.error.issues[0].message); } } console.error(`letti ${libri.length} libri, scartati ${scartati}`); return libri;}
function report(libri: Libro[]) { const perCat = new Map<string, number[]>(); const autori = new Map<string, number>(); const formati: Record<string, number> = {}; for (const l of libri) { perCat.set(l.category, [...(perCat.get(l.category) ?? []), l.price]); autori.set(l.author_name, (autori.get(l.author_name) ?? 0) + 1); const f = String(l.attributes.format ?? "n/d"); formati[f] = (formati[f] ?? 0) + 1; } const media = (p: number[]) => Math.round((p.reduce((a, b) => a + b, 0) / p.length) * 100) / 100; return { books: libri.length, avg_price: media(libri.map((l) => l.price)), by_category: [...perCat].map(([category, p]) => ({ category, books: p.length, avg_price: media(p) })).sort((a, b) => b.books - a.books), top_authors: [...autori].sort((a, b) => b[1] - a[1]).slice(0, 3).map(([name, books]) => ({ name, books })), formats: formati, };}
async function confronta(mio: ReturnType<typeof report>) { const ufficiale = await (await fetch(`${BASE}/api/v1/stats`)).json(); console.error("prezzo medio: mio", mio.avg_price, "| server", ufficiale.prices.avg); const loro: Record<string, number> = Object.fromEntries(ufficiale.books_by_category.map((c: any) => [c.slug, c.books])); const miei: Record<string, number> = Object.fromEntries(mio.by_category.map((c) => [c.category, c.books])); console.error("categorie uguali:", Object.keys(loro).every((k) => loro[k] === miei[k]));}
const libri = await scarica();const r = report(libri);await confronta(r);const destinazione = process.argv[2] ?? "report.json";await writeFile(destinazione, JSON.stringify(r, null, 2) + "\n", "utf-8");console.error("scritto", destinazione);Output:
letti 80 libri, scartati 0prezzo medio: mio 19.9 | server 19.9categorie uguali: truescritto report.jsonChoices worth noticing: z.infer saves you from writing the shape of the book twice; a Map instead of an object for the accumulators (arbitrary keys, ordered iteration); JSON.stringify(r, null, 2) for a readable file.
Try it now
Section titled “Try it now”Response headers
Extension exercises
Section titled “Extension exercises”Try it online, nothing to install:
The GET request straight in your browser: https://corso-api-json.giuseppebosi.com/api/v1/stats
Python on Google Colab (a free Google account is needed to run the cells); TypeScript on StackBlitz (it opens and starts by itself in the browser). If the practice API is switched off, turn it on from /en/api-attiva/.
- Robustness: simulate a corrupted line (edit a local copy of the export: remove a comma, put
"price": "n/d") and check that the program discards it without stopping. - Top authors with ties: the server sorts authors with the same number of books by name. Do the same in your report and compare
top_authorstoo. - A second report:
ordini.jsonfromGET /api/v1/orders?limit=100(paginated: followlinks.nextuntil it isnull) with revenue by status and by city (shipping_address.city), to compare with/api/v1/stats(orders_by_status) and/api/v1/stats/sales?group_by=city.
Hints
- Save the export with
curl -o libri.ndjson, edit it, and have the program read the file (open("libri.ndjson")/createReadStream): the rest of the code does not change — that is the benefit of having separated reading from validation. - Python:
sorted(autori.items(), key=lambda x: (-x[1], x[0]))[:3]. TS:.sort((a, b) => b[1] - a[1] || a[0].localeCompare(b[0])). - A loop
while url: r = get(url); …; url = r["links"]["next"]. Revenue by status: the sum oftotalgrouped bystatus, excludingcancelledif you want to match/stats/sales.
Common mistakes
Section titled “Common mistakes”- Validating after aggregating: a broken line ends up in the totals before you notice. Validate at the boundary, right away.
- Rounding halfway through: round only in the final report; use full numbers in the intermediate sums.
- Printing the report on stdout together with the logs: use
stderrfor messages and a file/stdout for data. sum(p) / len(p)with an emptyp: division by zero; it does not happen here, but in a real dataset it does.
Summary
Section titled “Summary”Download in streaming mode, validate at the boundary, aggregate while you read, write readable UTF-8, verify against an independent source. Next step: Section 2, chapter 1 — HTTP and REST.