In short

A Python library is described twice: once in the source, where the author writes docstrings for the reader of the code, and once in a documentation site, written for the reader of the library. The second description is a copy of the first, and a copy drifts. This reads the source and writes the site, so that only one description exists.

What it produces is Markdown, and HTML built from that Markdown. The HTML is not a page of its own invention: it is written for Whakerexa, which is what makes the pages accessible.

Where it comes from

Documentation tools ask the source to be complete before they say anything about it: a missing type, an unknown field, a docstring written in a style the tool does not know, and the entry is dropped or the run fails. A library that grows is never in that state, and the author is left choosing between writing for the tool and writing for the reader.

The choice made here is the other one: flexibility rather than completeness. Whatever the docstring holds is documented, in the shape it was written. Nothing is required, and nothing is rejected.

What it does not do

It documents classes. A function written at module level and named in __all__ is skipped in silence: ClamsPack keeps only what passes inspect.isclass(), and ClammingClassParser raises a TypeError for anything else. This is a gap, not a decision, and it is written here as such.

It does not read inheritance: a class is documented with what its own source holds, and an inherited method appears in the page of the class that defines it. It does not resolve cross-references between classes, does not read type annotations to build a signature, and does not compare one version of a library to another.

It does not check the accessibility of what it writes. The HTML is built to be accessible, and it is Whakerexa that carries that property; nothing in the generator verifies it.

The problem, and what the tool is for

A Python library carries its own description: docstrings, written where the code is written, by whoever wrote it. Published documentation is a second description of the same thing, and two descriptions of one thing diverge — the second one being the one nobody rereads.

The tools that close that gap ask for something in exchange. They ask the docstring to be complete, and drop what is not; they ask it to be written in the markup they parse, and fail on the rest; and what they output is a site whose accessibility is whatever their default theme happens to give. An author who writes for the reader of the code has to write a second time for the tool.

The question this answers is therefore: how is a publishable documentation produced from the docstrings as they are already written — whatever their style, however incomplete — in more than one format, one of which is accessible, without asking the author to write anything twice?

What follows from that question, as objectives:

• read the docstrings as they stand, and document whatever they hold;
• accept the field styles that Python authors actually use, and the variants they write;
• produce Markdown for what is read in a repository, and HTML for what is read on a site;
• make the HTML accessible by construction rather than by inspection, which is what building it for Whakerexa buys;
• document a whole library — modules, and the classes of each — in one run;
• let prose written by hand into the generated pages, for what a docstring is not the place for;
• remain a plain object-oriented Python library, so that what it does not do can be added to it.

What it is not: a replacement for the docstrings, a linter that judges them, or a site generator with a documentation feature. It documents; it does not rate what it documents.

The needs, expressed

Reading what is there. The first need is not a feature but a posture: the docstring is the author's, not the tool's. Whatever is written is documented, in the shape it was written, and what is not written raises nothing. A library halfway through its documentation effort is a normal library, and it is documented as it stands.

Understanding the conventions anyway. Reading what is there does not mean reading it flatly. The conventions Python authors use — the reST field lists, the Epydoc ones, the variants they write without thinking — carry meaning: a parameter, a returned value, an exception, an example. What is recognized is laid out as such. What is not recognized is kept, not dropped.

One run, one library. The unit of work is not the class. An author publishes a library, so the run takes a list of modules and writes everything: a page per class, an index per module, an index for the whole, and the links between them.

Two formats, one truth. A repository reads Markdown, a site reads HTML, and both are needed. What must not happen is two conversions saying two different things, so one conversion feeds both.

Accessibility as a function. The HTML output is not accessible as a matter of good will: the structure and the stylesheets come from Whakerexa, whose conformity to WCAG 2.1 is its own subject. What the generator owes is to produce the markup that framework expects, and nothing that undoes it.

Two axes of adjustment, not one. What a reader adjusts falls in two independent places. The first is the text itself: the contrast switch of Whakerexa raises the font size from 16 to 18 px, the line height from 1.2 to 1.5, the letter spacing from 0.04 to 0.12 rem, and thickens the outlines — it touches no colour but that of the links in light mode. The second is the palette: flat background, maximum ratio, no gradient, no rounded corner. A reader who needs the second is not served by the first, and the colours of the documentation are the same whether the contrast switch is on or off.

Prose that is not a docstring. A library has things to say that no function docstring can hold: what it is for, how it is installed, where it comes from. That prose is written in Markdown, kept beside the code, and it belongs in the generated pages.

The pages belong to the documented software. Name, logo, address, copyright, favicon, language, and the paths where the statics live: the author sets them. The generator is not a brand.

Saying what was not done. A generator that silently omits is worse than one that fails: the author discovers the hole in the published site. What could not be documented is said during the run.

Costing nothing to the reader of the library. Whoever installs the documented library installs a library, not a documentation toolchain. What the generator needs, the generator needs — not its user's users.

Elicitation: the concepts

The needs above become, in the application, this closed set of concepts. They are what the code names, and what the rest of this file talks about.

• Docstring: the text the author wrote under a class, a constructor or a function. The one source.
• Field: a line of a docstring opening with : or @, carrying a name and a text. Recognized or not, it is a field.
• Documented entity: what is known of one thing — a name, a list of argument names, a source code, a docstring. A class, a constructor and a function are all of them one of these.
• Class: one documented entity, one constructor, and a set of functions, split by what their names say — public, private, protected, overload.
• Module: the classes its __all__ declares, and the README found beside it.
• Set of modules: what one run documents, and what the general index lists.
• Output format: Markdown, or HTML built from it.
• Page: one file of the HTML output — a class, the index of a module, the general index.
• Theme: a whole stylesheet, holding the palette and nothing structural. Themes are exclusive: one is active, and activating another replaces it entirely.
• Export options: what surrounds the documentation and belongs to the documented software rather than to its code.

Elicitation: the pathway

The needs are met along one path, and each step of it is a place where something can be given up without the next step failing:

Source → entities → Markdown → HTML → files. Reading the source gives entities; a docstring that says nothing gives an entity that says nothing. Converting gives Markdown; an unrecognized field is kept as prose. Rendering gives HTML; a missing renderer gives no HTML and a warning, and the Markdown produced before it stands. Writing gives files; the files that were written before an error are written.

Elicitation: what the needs left to settle

Collected as they are, the needs are neither precise nor compatible. What follows is what had to be settled before a single functional item could be written.

Elicitation: the inferred needs

Two things in the output were asked for by nobody. They come from watching what reading documentation actually costs.

The source code, folded. A reader who doubts the prose goes to the source, and going there means leaving the page for the repository. Putting the source in the page answers that; putting it folded answers the reader who does not want it. The details element does both, and asks nothing of the author.

The table of contents, set aside. The same reasoning as the source code, one level up: a panel that is always shown takes a fifth of the width from whoever is reading a long docstring, and a documentation is read far more often than it is navigated. Folding it behind a button gives the width back and costs a click to whoever navigates. It is the author's choice rather than the reader's, because it decides the shape of the pages.

The table of contents, built when the page opens. A generated page knows its own headings, so nothing needs to write them a second time into a navigation panel. It is built in the reader's browser, from the page itself, and it cannot fall out of step with what it lists.

The needs, organized

Each item below is one thing the application does, stated so that it can be verified on its own. They are grouped by domain, and the numbering leaves room inside each group.

[010] Starting a generation

[011] The author can document one class, given the class object.
[012] The author can document one module, given the module object.
[013] The author can document a list of modules in a single run.
[014] The author can run the generator as a Python library or from the command line.
[015] The system returns the documentation as a string when no output path is given, and writes files when one is.

[100] The sources to be documented

[101] The author designates the modules to be documented.
[102] The system documents the classes named in the __all__ of a module.
[103] The system skips what is named there and is not a class.
[104] The system reports a module without __all__, documents it as empty, and goes on.
[105] The system refuses a built-in class, and a class whose source cannot be read.

[200] Reading the code

[201] The system reads the text of the source; it does not call what it documents.
[202] The system extracts the name and the docstring of the class.
[203] The system extracts the constructor: its arguments, its source, its docstring.
[204] The system extracts each function: its name, its arguments, its source, its docstring.
[205] The system keeps __init__ and __new__ out of the list of functions.
[206] The system classifies a function from its name alone: public, private, protected, overload.

[300] Reading the docstrings

[301] The system reads both field styles, :field: and @field:, in any case.
[302] The system normalizes the variants of a field name to one field.
[303] The system recognizes four fields as sections: parameters, returns, raises, example.
[304] The system reads :code: as an example.
[305] The system keeps an unrecognized field as an item of a description list.
[306] The system ignores type and rtype, the source annotations being the truth.
[307] The system opens an example section on a line starting with >>>, even when no field announced it.
[308] The system takes a first line ending with a dot as the short description.
[309] The system requires no field: an absent or empty docstring produces an entry all the same.

[400] The Markdown output

[401] The system produces the Markdown of a class, of a module, and of a set of modules.
[402] The Markdown output depends on nothing beyond the standard library.
[403] The recognized fields become headed sections.
[404] The source code is written in a fenced code block.

[500] The HTML output

[501] The system builds the HTML from the Markdown it produced.
[502] The system colours the source code it shows.
[503] The system folds the source code into a details element.
[504] The system disables the HTML output, with a warning, when a renderer is missing.
[505] The system writes the markup Whakerexa expects, and links its stylesheets and its loader.

[600] The files, and their organization

[601] The system writes one file per documented class.
[602] The system writes an index for each module.
[603] The system writes a general index for the set of modules.
[604] The system creates the output folder when it does not exist.
[605] Each page links to the previous and next class, to the previous and next module, and to the index.
[606] Each page builds its own table of contents from its headings when it is opened.
[607] The author chooses the table of contents to be a panel always shown, or one folded behind a button.

[700] The prose written by hand

[701] The author can have a README inserted into the general index.
[702] The system inserts the README found inside a module into the page of that module.
[703] The author can refuse the insertion of the READMEs.
[704] The system reports a README that cannot be read, and writes the page without it.

[800] The identity of the documented software

[801] The author sets the name, the copyright, the address, the logo, the favicon, the title and the language.
[802] The author sets the paths where the statics and Whakerexa are found.
[803] The author sets the colour mode the pages open in.
[804] The system refuses an option that is not of the expected type, and says which one.
[805] The author provides the theme the pages open with, which carries the colours of the documented software, and names its file.
[806] The system declares no theme when the author names none, and the pages are then what they were before themes existed.

[900] What the run reports

[901] The system reports a module without __all__.
[902] The system reports a class it cannot reach.
[903] The system reports a disabled output format, and what is missing for it.
[904] The system reports a README it could not read.
[905] No report of this kind stops the run.

[1000] Reading the result

[1001] The pages are HTML-5 built for Whakerexa.
[1002] Each page offers the contrast and colour switches.
[1003] Each page opens with a link that skips to its main content.
[1004] Each page carries a stylesheet for printing.
[1005] Each page declares its language.
[1006] The reader can switch to a high-contrast theme: flat palette, maximum ratio, no gradient, no rounded corner.
[1007] A theme chosen on one page is still the one in force on the next.
[1008] Switching a theme leaves the text adjustments untouched, and adjusting the text leaves the theme untouched.

What is not met yet

The items above are what the application has to do; four of them it does not do, and two needs are met only in part. They are listed here rather than left for a reader to discover in the code.

The source is read, not run

The structure of a class is taken from its text: inspect.getsource() gives the source, textwrap.dedent() makes it parsable wherever it was written, and ast walks it. Names, arguments, source and docstrings all come from that tree.

The module itself is imported — ClamsPack needs __all__ and importlib to reach the classes it names — but nothing of what is documented is called. What a decorator would return, what a metaclass would build, what a property computes: none of it is asked, so none of it is in the page.

Markdown is the pivot

A docstring is converted once, into Markdown. The HTML is that Markdown given to markdown2, and the source code is given to Pygments. Two formats, one conversion: what is read of a docstring cannot differ between them, because it is read once.

The consequence is accepted: what Markdown cannot express does not reach the HTML either.

What a field is worth

A line opening with : or @ is a candidate field. Its name is normalized — lowercased, then mapped through the table of variants — and four names become sections of the page: param, return, raise and example. A candidate whose name is none of these is not an error: it is written as an item of a description list, which is what a reader would take it for.

A line opening with >>> opens an example section on its own, for the author who wrote the example and forgot to announce it.

The first line, twice

When the first line of a docstring ends with a dot, it is the short description: it is written in italics at the head of the entry, and for a class it also becomes the description of the page, cut to 160 characters — the length a search engine shows.

Four kinds of functions

Functions are grouped by what their name says of them, and the vocabulary is the one of this project:

__init__ and __new__ are not in any of them: the constructor has a section of its own, before the functions.

A missing dependency disables a format

ClamUtils holds the import of markdown2 in a try: on failure it keeps the message and leaves the converter at None. Every conversion asks that value first, warns, and returns an empty string. Markdown output needs nothing but the standard library, so a library without markdown2 is still documented — in one format instead of two.

What carries the data

What does the work

The way, and why it is written here

The models below are those of Merise, kept in the order the method gives them: who exchanges what, what the data are made of, which operations exist, where and when they happen, how the data are laid out, where they live, and where the code goes. A generator has no database, so two of the models say less than they would elsewhere; they are kept all the same, because what they would hold is what a reader looks for when something is to be changed.

MCC: who exchanges what

The data handled, and the values they take

MCD: what the data are made of

A documented entity is the only thing the generator stores. Everything above it — class, module, set — is a way of grouping those entities and of deciding which file they are written to.

MCT: the operations

MOT: where, when, and by whom

The pages are written once and served as files: what happens at the reader's request happens in their browser and changes nothing on disk. Two of these three operations are theirs to ask for, and the third — the table of contents — is asked for by nobody.

MLD: the structures

There is no database. What the tables of a logical model would hold is held, for the time of one run, by these structures:

MPD: where each thing exists

MOpT: where the code goes

UML: the sequence of one export

What happens when ClamsModules.html_export_packages() is called:

ClamsModules            ClamsPack             ClamsClass          ClammingClassParser
     |                       |                     |                       |
     |-- html_export_index ->|                     |                       |
     |   (index.html)        |                     |                       |
     |                       |                     |                       |
     |-- html_export_clams ->|                     |                       |
     |                       |-- for each class -->|                       |
     |                       |                     |-- parse ------------->|
     |                       |                     |<-- ClamInfo x n ------|
     |                       |                     |                       |
     |                       |                     |-- markdown / html --> (ClamInfoMarkdown)
     |                       |<-- content ---------|                       |
     |                       |                     |                       |
     |                       |-- ExportOptions: head, header, nav, footer   |
     |                       |-- write <Class>.html                         |
     |<-- list of files -----|                     |                       |

The export ends there. Nothing of this library runs again afterwards: the objects that act on a page being read belong to Whakerexa.

UML: the classes

Attributes are private throughout — the language mangles them, and what is readable is read through an operation. Everything below is what the code declares.

+--------------------------------+          +-------------------------------+
|          ClamInfo              |          |      ClamInfoMarkdown         |
+--------------------------------+          +-------------------------------+
| - name: str                    |<>--1..1--| - clam: ClamInfo              |
| - args: list                   |          +-------------------------------+
| - source: str                  |          | + name(): str                 |
| - docstring: str               |          | + args(): list                |
+--------------------------------+          | + source(): str               |
| + get_name() / set_name()      |          | + docstring(): str            |
| + get_args() / set_args()      |          | + convert_name(): str         |
| + get_source() / set_source()  |          | + convert_source(): str       |
| + get_docstring()              |          | + convert_docstring(): str    |
| + set_docstring()              |          +-------------------------------+
+--------------------------------+

+--------------------------------+          +-------------------------------+
|     ClammingClassParser        |          |          ClamsClass           |
+--------------------------------+          +-------------------------------+
| - obj: type                    |--1..1---<| - utils: ClamUtils            |
| - obj_src: str                 |          | - info_class_name: str        |
| - obj_clams: ClamInfo          |          | - info_constructor: ClamInfo  |
| - init_clams: ClamInfo         |          | - info_public_fcts: list      |
| - fct_clams: dict              |          | - info_private_fcts: list     |
+--------------------------------+          | - info_protected_fcts: list   |
| + get_obj_clams(): ClamInfo    |          | - info_overloads: list        |
| + get_init_clams(): ClamInfo   |          +-------------------------------+
| + get_fct_clams(): dict        |          | + get_name(): str             |
+--------------------------------+          | + get_short_description(): str|
                                            | + markdown(): str             |
                                            | + html(): str                 |
                                            +-------------------------------+

+--------------------------------+          +-------------------------------+
|          ClamsPack             |          |         ClamsModules          |
+--------------------------------+          +-------------------------------+
| - name: str                    |>--1..*---| - clams_packs: list           |
| - pack: module                 |          +-------------------------------+
| - clams: list<ClamsClass>      |          | + markdown_export_packages()  |
+--------------------------------+          | + html_export_index(): str    |
| + get_name(): str              |          | + html_export_packages(): list|
| + get_readme(): str            |          +-------------------------------+
| + markdown(): str              |
| + html(): str                  |          +-------------------------------+
| + html_index(): str            |          |          ClamUtils            |
| + html_export_clams(): list    |          +-------------------------------+
+--------------------------------+          | + markdowner: Markdown        |
              |                             | + lexer: PythonLexer          |
              | uses 1..1                   | + formatter: HtmlFormatter    |
              v                             +-------------------------------+
+--------------------------------+          | + get_class(): Any            |
|        ExportOptions           |          | + markdown_convert(): str     |
+--------------------------------+          | + markdown_to_html(): str     |
| - software, copyright, url     |          | + source_to_html(): str       |
| - icon, favicon, title, lang   |          +-------------------------------+
| - theme, statics, wexa_statics |
| - readme: bool                 |
| - aside_toc: bool              |
| - next/prev class and module   |
+--------------------------------+
| + get_root_class(): str        |
| + get_head(): str              |
| + get_header(): str            |
| + get_nav(): str               |
| + get_footer(): str            |
+--------------------------------+

Two aggregations carry the whole structure: a set of modules holds its modules, a module holds its classes. The parser and the options are used, not held: a ClamsClass is built from a parser it does not keep alive, and the same ExportOptions instance travels through every export of a run, its neighbouring-page fields being rewritten between two files.

UML: the sequence when a page is read

Same objects whichever way they were loaded — modules over HTTP, bundle over file://.

Reader        Browser         Book       ThemeManager   AccessibilityManager
  |              |              |              |              |
  |-- open ----->|              |              |              |
  |              |-- create --->|              |              |
  |              |              |-- read the headings         |
  |              |              |-- build the table of contents
  |              |              |  [ if the table is set aside ]
  |              |              |-- make the panel out of reach
  |              |              |-- add the button that opens it
  |              |              |              |              |
  |              |-- create ------------------>|              |
  |              |              |     register clamming       |
  |              |              |     register highcontrast   |
  |              |              |     read the address        |
  |              |              |     activate a theme        |
  |              |              |              |              |
  |              |-- create ----------------------------------|
  |              |              |              |     read the address
  |              |              |              |     apply the adjustments
  |              |              |              |              |
  |-- switch the theme ------------------------>|             |
  |              |<-- replace the stylesheet ---|              |
  |              |                                            |
  |-- adjust the text ------------------------------------->  |
  |              |<-- set the class on the root element -------|
  |                                                            |
  |-- open the table of contents -->|            |             |
  |              |<-- show the panel, and put it in reach -----|

UML: the states of the presentation

The presentation of a page is one composite state with two orthogonal regions: one holds the palette, the other the text. A transition in one region leaves the other where it was, which is what makes the four combinations reachable.

+-- Presentation ------------------------------------------------+
|                                                                |
|  region: palette   (exists only if a theme is declared)        |
|    (o)--> [ of the software ] -- switch --> [ high contrast ]   |
|                ^                                    |          |
|                +------------- switch ---------------+          |
|                                                                |
| ---------------------------------------------------------------|
|                                                                |
|  region: text                                                  |
|    (o)--> [ as written ] --- switch contrast ---> [ adjusted ]  |
|                ^                                       |       |
|                +--------- switch contrast -------------+       |
|                                                                |
| ---------------------------------------------------------------|
|                                                                |
|  region: table of contents  (set aside by the author)          |
|    (o)--> [ closed ] ------- press the button ---> [ open ]     |
|                ^                                       |       |
|                +--------- press the button ------------+       |
|                                                                |
|    a table of contents that is not set aside has no region:    |
|    it is shown, and there is nothing to reach                  |
|                                                                |
+----------------------------------------------------------------+

Neither state is a degraded form of another. The third region is the one that does not survive a page: a table of contents opens closed on every page, since nothing carries its state and nothing should — a panel left open on the previous page would hide the beginning of the next one. The two others keep their state across pages the same way — in the address, under wexa_theme for the palette, wexa_contrast and wexa_color for the text and the colour mode — because pages that are files have nowhere else to keep them.

The error policy

Three things can go wrong, and each kind has one answer. The line between them is the one this whole file is built on: what the author of the generator got wrong stops; what the author of the documented library did not write does not.

Nothing is caught and hidden: the two exceptions the library raises are its own, and everything it swallows is written to the log before being swallowed.

Invariants

What holds at every point of a run, whatever was given to it:

What a run guarantees

A run writes files one after another, and nothing undoes what it wrote.

Where the tests are

What no test covers

Nothing verifies what a browser does with the pages: the table of contents built at load time, the folded source code, the navigation from page to page, the contrast and colour switches. Nothing verifies the accessibility of the result.

Nothing covers the documentation of functions written outside a class, because nothing implements it.

ClamsClass and ClamInfoMarkdown have no test file of their own: they are covered from the tests of the classes that use them, which leaves their own errors — a docstring at None, a class without a constructor — verified only where another class happens to produce them.