← Wiki

Lecture 1: enough Python to finish the exercise

2026-10-14 · 15 min read · python, tooling, lecture

Everything the hermes-protocol exercises need. Examples are from this repo. OCaml comparisons are there to locate the concept, not to justify it.

1. Project and environment

The layers

nix develop        gives you the python 3.14 binary, uv, ruff, pyright
  └─ uv            resolves and installs python libraries into .venv
       └─ .venv    the actual interpreter + libraries your code runs against

Nix owns tools. uv owns libraries. There is no global install anywhere; a Python project is a directory with a manifest and a private interpreter.

Files

filerole
pyproject.tomlthe manifest: name, dependencies, tool config
uv.lockexact resolved versions of everything, transitively. Committed.
.venv/installed packages. A build artifact, gitignored.

pyproject.toml is both the package manifest and the config file for ruff, pyright-adjacent settings, pytest, and so on. Multiple tools, one file, separated by [tool.<name>] tables.

Commands

uv sync --all-packages      make .venv match uv.lock
uv run --all-packages X      run X with .venv activated
uv add msgpack               add a dependency (edits pyproject.toml + uv.lock)
uv lock                      re-resolve without installing

uv run is the important one: it activates the venv for one command, so you never source .venv/bin/activate. Everything in the exercise is uv run --all-packages pytest.

The workspace

This repo is a uv workspace: one lockfile, one .venv, several packages.

pyproject.toml                  [tool.uv.workspace] members = ["libs/*", "apps/*"]
libs/hermes-protocol/
    pyproject.toml              its own dependencies: pydantic, msgpack
    src/hermes_protocol/        the importable code

Members declare their own dependencies. That is the whole point: driver-whatsapp will depend on neonize, and chiron must not, so the dependency lists are separate even though the lockfile is shared.

--all-packages is needed because a bare uv sync installs only the root's dev group, which omits hypothesis.

How import hermes_protocol finds the code

Workspace members are installed editable: the venv contains a pointer to libs/hermes-protocol/src/, not a copy. So:

  1. import hermes_protocol searches sys.path
  2. sys.path includes libs/hermes-protocol/src/ via the editable install
  3. it finds the directory hermes_protocol/ containing __init__.py
  4. edits to your source are live — no rebuild, no reinstall

The distribution is named hermes-protocol (hyphen) and the import name is hermes_protocol (underscore). Unrelated strings that happen to correspond by convention; the link is [tool.hatch.build.targets.wheel] packages = [...].

vs OCaml — closest to a dune workspace with several libraries and one lock file. The difference is that there is no build step and no separate artifact: the interpreter reads your .py files directly, so "installed editable" means a path entry, not a compiled .cmxa.

2. Modules and packages

The mapping

There is no interface file and no separate signature. A module's contents are whatever it defines at the top level, and all of it is reachable.

Import forms

import msgpack  # bind the module name
from pydantic import BaseModel  # bind one name out of it
from ._ids import DriverId  # relative: a sibling module
from ._ids import DriverId, StreamId  # several
import msgpack.exceptions as mexc  # rename

from . import x means "this package". from .. import x means the parent. Inside hermes_protocol, always use relative imports for siblings — it makes the package renameable and the dependency direction obvious.

Execution model

A module's top level runs on first import, once per process. The result is cached in sys.modules. So this line in _codec.py:

_FRAMES: TypeAdapter[Frame] = TypeAdapter(Frame)

executes when _codec is first imported, and the TypeAdapter is built exactly once. That is why it lives at module level rather than inside decode — it is expensive to construct and immutable afterwards.

Class and function definitions are also statements that run at import time. There is no declaration/definition split.

Circular imports

If a.py imports b.py and b.py imports a.py, one of them observes the other half-initialised and you get an ImportError or an AttributeError at import time. The fix is always to break the cycle, not to work around it.

This is why hermes_protocol has a strict layering:

_ids, _errors          (no internal imports)
_handle, _message      → _ids
_status                (no internal imports)
_inbound               → _handle, _ids, _message, _status
_handshake             → _errors, _ids
_outbound              → _handle, _ids
_frames                → _errors, _handshake, _ids, _inbound, _outbound
_codec                 → _errors, _frames
_stream                → _codec, _errors, _frames
__init__               → everything

Strictly acyclic, and that is not an accident — it is the constraint that decides which module a type goes in.

__init__.py and the public surface

__init__.py runs when the package is imported. Here it does two things:

from ._codec import decode, encode  # pull names up

__all__ = ["decode", "encode", ...]  # declare the public surface

__all__ is a list of strings. It controls from hermes_protocol import * and, more usefully, is read by type checkers and linters as the declared API.

The underscore prefix on module names is convention, not enforcement — import hermes_protocol._frames works. What makes it bite is the py.typed marker file: with it present, pyright's reportPrivateImportUsage flags consumers importing anything this package does not explicitly re-export.

Same convention one level down: _to_utc, _FRAMES, _next_id are private to their module or class, and pyright's reportPrivateUsage enforces that in strict mode.

vs OCaml — there is no .mli. __all__ plus the underscore convention is the whole mechanism, and it is advisory until a type checker is involved. Also note: no functors, no module-level parameterisation. A module is a namespace with mutable global state, not a value.

3. Classes, instantiation, inheritance

Definition and instantiation

class Correlator:
    def __init__(self, limit: int) -> None:
        self.limit = limit  # attributes are created by assignment

    def issue(self) -> str:
        return str(self.limit)


c = Correlator(10)  # no `new`; calling the class constructs
c.issue()  # c is passed as `self`

__init__ is the initialiser, not a constructor — the object already exists when it runs. self is explicit and is just the first parameter, by convention named self; the method call c.issue() is sugar for Correlator.issue(c).

Attributes are not declared. self.limit = limit creates it. An annotation at class level with no value is a declaration for the type checker only:

class Correlator:
    limit: int  # declares the type; creates nothing

Class attributes vs instance attributes

class Handle(BaseModel):
    model_config = ConfigDict(frozen=True)  # class attribute: one, shared
    driver: DriverId  # annotation: per-instance field

A plain assignment at class level belongs to the class. An annotation without a value does not create anything at runtime. Pydantic reads those annotations to build the model, which is why model_config (assigned) and driver (annotated) mean completely different things in the same block.

@property

@property
def outstanding(self) -> frozenset[RequestId]:
    return frozenset(self._outstanding)

Makes a method accessible as an attribute: c.outstanding, no parentheses. Use it when the value is conceptually a field but needs computing — here, to hand out a copy that cannot be mutated.

Inheritance

class FramingError(HermesProtocolError):  # single inheritance
    """..."""

Subclasses inherit attributes and methods; super().__init__(...) calls the parent. Multiple inheritance exists and resolves by MRO (a linearisation of the class graph). You can ignore all of it: in this codebase inheritance appears in exactly two roles, and neither involves overriding anything.

  1. Exception hierarchies — FramingError(HermesProtocolError) exists so a caller can except HermesProtocolError and catch all of them.
  2. BaseModel — subclassing is how you tell pydantic "build a validated model out of my annotations".

The idiomatic Python answer to "share behaviour between types" is not inheritance. It is composition, or typing.Protocol for structural typing. You will not need either for this exercise.

class VersionMismatch(HermesProtocolError):
    def __init__(self, *, local: int, remote: int) -> None:
        super().__init__(f"...speaks {local}...{remote}")  # set the message
        self.local: int = local  # keep the fields
        self.remote: int = remote

Dataclasses

@dataclass(slots=True)
class Correlator:
    _next_id: int = field(default=1, init=False)
    _outstanding: set[RequestId] = field(default_factory=set[RequestId], init=False)

@dataclass is a decorator that reads the annotations and generates __init__, __repr__, __eq__. Four things in play:

Decorators (@dataclass, @property) are functions that take the thing below them and return a replacement. @dataclass on a class means Correlator = dataclass(Correlator).

When dataclass, when pydantic

use
@dataclassinternal state, never crosses a boundary, no validation needed
BaseModelanything parsed from or serialised to the outside world

Correlator is a dataclass because its contents never leave the process. Everything that goes on the wire is a BaseModel, because it must validate bytes that another process produced.

vs OCaml — a class is closer to a mutable record with methods attached than to an object system you would build with. No method keyword, no explicit self type, no immutability by default, and the type checker is optional and external.

4. Typing

Annotations are optional at runtime and mandatory in this repo — pyright runs strict and ruff's ANN rules require them.

Basics

def feed(self, data: bytes) -> None: ...


x: int = 3
names: tuple[str, ...] = ()
maybe: str | None = None

X | Y is a union (PEP 604). "Optional" is spelled X | None; there is no separate option type, and None is a real value of its own type.

For a field, nullable and omittable are different, and you usually want both:

reason: str | None = None  # may be null, and may be left out entirely

Literal

ConnectionState = Literal["connected", "degraded", "logged_out"]

A type whose only members are those exact values. This is as close as Python gets to an enum of constants in a type position, and pydantic validates it at the boundary for free.

Annotated

UtcTimestamp = Annotated[AwareDatetime, AfterValidator(_to_utc)]

Annotated[T, meta...] is T with metadata attached. The type checker sees only T; libraries read the metadata. Pydantic uses it for validators, discriminators, and constraints. It is the mechanism for "a type that is also a constraint".

NewType

RequestId = NewType("RequestId", str)
r = RequestId("1")  # erased at runtime: this is just the str "1"

A distinct type to the checker, identical at runtime. RequestId is accepted where str is wanted, but not vice versa. No wrapper, no allocation, no validation. It cannot hide a representation — Handle.opaque is public bytes and what makes it opaque is the documented contract, not the type.

Final

PROTOCOL_VERSION: Final = 1

"Never reassigned." Checker-only, and it also infers the narrowest type.

Type aliases

Frame = Annotated[Request | Response | ..., Field(discriminator="kind")]  # plain name
type Thing = dict[str, int]  # PEP 695

3.14 has the type statement for lazily-evaluated aliases. Do not use it for anything pydantic touches — laziness and TypeAdapter interact badly. Plain assignment for anything that becomes a model field or a TypeAdapter.

match and exhaustiveness

match event:
    case MessageReceived():
        ...
    case DeliveryReceipt():
        ...
    case _:
        assert_never(event)

case ClassName() is a class pattern: it tests the type. You can destructure too: case DeliveryReceipt(state="failed"):.

typing.assert_never takes a value whose type must be Never. If every case is handled, the narrowed type in the final branch is Never and it type-checks. Miss a variant and pyright reports that you passed a real type where Never was required. That is how you get a non-exhaustive-match error in Python, and it is the only way.

A note on from __future__ import annotations

You will see it at the top of every module here. It makes annotations plain strings rather than evaluated expressions, which historically avoided forward-reference and circular-import problems. Python 3.14 evaluates annotations lazily anyway (PEP 649), so on this interpreter the line is largely vestigial. Harmless, consistent, leave it.

vs OCaml — inference is local and shallow, so you annotate boundaries by hand. The checker is a separate program with no runtime consequence: nothing is erased because nothing was ever compiled with types in the first place. Literal unions plus match plus assert_never is the closest available thing to a variant with a checked match, and assert_never is the only part that gives you exhaustiveness.

5. Pydantic

The library that turns annotations into runtime validation. Everything crossing the wire in this project is a pydantic model.

A model

class Message(BaseModel):
    model_config = ConfigDict(frozen=True, extra="forbid")

    native_id: NativeMessageId
    text: str
    claimed_at: UtcTimestamp

Annotations become validated fields. Construction validates:

Message(native_id=..., text="hi", claimed_at=...)     # keyword args, validated
Message.model_validate({"text": "hi", ...})           # from a dict
m.model_dump()                                        # back to a dict

Two config settings used everywhere here:

Discriminated unions

The central idiom. Each variant carries a literal tag:

class Ok(BaseModel):
    kind: Literal["ok"] = "ok"
    value: ResponseValue


class Failure(BaseModel):
    kind: Literal["failure"] = "failure"
    code: ErrorCode

and the union names the tag field:

ResponseResult = Annotated[Ok | Failure, Field(discriminator="kind")]

The kind: Literal["ok"] = "ok" line looks redundant and is not. The annotation constrains it to exactly that string; the default means you never pass it. The discriminator then lets pydantic read kind and validate exactly one branch.

Without the discriminator, pydantic tries each member in turn and a failure reports a composite error across all of them — useless for debugging.

Unions nest. Ok.value is itself a union with its own discriminator, resolved independently.

TypeAdapter

Frame is an Annotated[...] alias, not a class, so it has no .model_validate. For a bare annotation:

_FRAMES: TypeAdapter[Frame] = TypeAdapter(Frame)

_FRAMES.validate_python(raw)  # dict -> Frame, raises ValidationError
_FRAMES.dump_python(frame)  # Frame -> dict

Expensive to build, cheap to use: construct once at module level.

Validators

def _to_utc(value: datetime) -> datetime:
    return value.astimezone(UTC)


UtcTimestamp = Annotated[AwareDatetime, AfterValidator(_to_utc)]

AfterValidator runs after pydantic's own parsing, so _to_utc is guaranteed a real datetime. AwareDatetime rejects naive ones before it gets there.

6. Exceptions

Python's idiom for failure. Not Result.

class HermesProtocolError(Exception): ...


class FramingError(HermesProtocolError): ...


raise FramingError("stream lost sync")

try:
    raw = msgpack.unpackb(data)
except (msgpack.UnpackException, ValueError) as exc:
    raise FramingError(f"not msgpack: {exc}") from exc

Four things to know:

f-string formatting shows up in these messages: f"{x}" interpolates, f"{x!r}" uses repr() — quoted and unambiguous, which is what you want for ids in an error message.

A function's signature says nothing about what it raises; the docstring does. That is a real loss compared to a Result return type, and the compensation is that you are not converting at every call site forever.

vs OCaml — closest to OCaml exceptions, but they are the default idiom rather than the escape hatch, and there is no effect tracking at all. Building a Result type in Python means fighting every library you call.

7. Generators and iterators

Needed for FrameStream.drain.

Protocol

An iterable has __iter__. An iterator has __next__ and raises StopIteration when exhausted. for x in thing calls iter(thing) then next() until StopIteration.

Generators

Any function containing yield is a generator function. Calling it runs none of the body — it returns a generator object. The body advances only as the caller pulls:

def drain(self) -> Iterator[Frame]:
    while True:
        raw = next(self._unpacker)
        yield validate(raw)

Consequences that matter here:

PEP 479, the trap

A StopIteration that escapes a generator body is converted into a RuntimeError. So this is wrong:

def drain(self):
    while True:
        yield validate(next(self._unpacker))  # StopIteration -> RuntimeError

and this is right:

while True:
    try:
        raw = next(self._unpacker)
    except StopIteration:
        return  # ends the generator cleanly
    yield validate(raw)

return in a generator means "I am done", not "here is a value". Catch StopIteration explicitly, which is also why you use next() rather than a for loop — the for would swallow it and you would have nowhere to hook the distinction between "out of bytes" and "corrupt bytes".

8. Odds and ends you will hit

Keyword-only parameters. Everything after a bare * must be passed by name:

def __init__(self, *, max_buffer_size: int = DEFAULT) -> None: ...


FrameStream(max_buffer_size=4096)  # required
FrameStream(4096)  # TypeError

Use it for anything that would otherwise read as a mystery number at the call site.

tuple vs list. list is mutable, tuple is immutable and hashable. tuple[Capability, ...] means "any length, all that type"; tuple[int, str] means exactly two, of those types. On a frozen model, always tuple.

Mutable default arguments. def f(x: list = []) creates one list, shared across every call, forever. Same bug as the dataclass field. Use None and build inside, or default_factory in a dataclass.

get_args. typing.get_args(DeliveryState) returns the tuple of a Literal's members. The tests use it to generate from the type itself, so adding a variant extends the generator automatically.

Underscore names. _x is private by convention. __x on a class attribute triggers name mangling — a different mechanism you do not need here.

9. Testing

pytest

Plain functions, bare assert:

def test_version_mismatch_names_both() -> None:
    with pytest.raises(VersionMismatch) as exc_info:
        check_protocol_version(99)
    assert exc_info.value.remote == 99

pytest rewrites assert so a failure shows both sides. No assertion library.

hypothesis

Property-based testing. You describe how to generate inputs; it searches for a counterexample and then shrinks it to the smallest failing one.

_handles = st.builds(
    Handle,
    driver=st.text(min_size=1, max_size=16).map(DriverId),
    opaque=st.binary(max_size=64),
)


@given(_frames)
def test_decode_undoes_encode(frame: Frame) -> None:
    assert decode(encode(frame)) == frame

Failures are saved in .hypothesis/ and replayed first on subsequent runs, so a counterexample found once stays tested.

vs OCaml — QCheck, except shrinking works on composed strategies without you writing a shrinker, because it shrinks the underlying choice sequence rather than the value.

10. Order of attack

  1. Read _ids.py, _errors.py, _handle.py, _message.py, _status.py. They are intact and contain one example of nearly everything above.
  2. _handshake.py — section 3 and 4.
  3. _outbound.py — the tag pattern, section 5.
  4. _inbound.py — discriminated unions, section 5. The one to get properly.
  5. _frames.py models — more of the same, nested.
  6. _codec.py — TypeAdapter and exceptions, sections 5 and 6.
  7. _stream.py and Correlator — generators and dataclasses, sections 3 and 7.

Run pyright as often as the tests. While a module is still a stub, it is the better error message.