Transit: understanding transit data

A personal civic data project designed for different transit networks, currently using data from Montréal and Ottawa.

A delay shown on a map looks straightforward. Understanding it takes a few more questions. Where did the information come from? When was it received? Does it describe a reported position, a predicted arrival or an observation saved for a daily summary? The answer changes what the number can tell us.

I’m developing Transit to make those questions easier to explore. The site brings public transit data into a French and English interface. Visitors can look at network conditions, open a route or stop, explore a map and read the explanations behind the measures. The architecture is designed to accommodate different transit data providers. Montréal’s STM and Ottawa’s OC Transpo are the two networks currently configured and presented on the site, with the information available from each.

Transit is an independent project. Its purpose is to make transit operations easier to inspect and understand. The transit agencies’ own services remain the reference for planning a journey or checking an urgent service notice.

Starting with a question#

The site starts with everyday questions: how is the network running, which route deserves a closer look, and what has been observed over time? That gives people a way in without requiring them to know the name of a metric or how the database is organized.

The map places reported positions in context and opens up their details. Route and stop pages bring together schedules, available predictions and reliability observations. Network views and daily summaries offer a broader perspective. A methodology page explains what the numbers measure, while data health shows source freshness and known gaps.

These different views complement each other. Someone can start with a place they recognize, look at a pattern and then open the explanation behind a number. The detail is available without making everyone work through it first.

Two networks, different contexts#

Working with Montréal and Ottawa makes the differences between sources tangible. Both networks provide schedules and updates, but their information is not interchangeable. Switching cities also needs to change the map context, routes, stops and links to the relevant agency.

Ottawa offers a useful example. Transit has no connected OC Transpo service-alert feed. The interface says so and links to the official alerts. Displaying zero alerts would suggest that there were no disruptions, which the available data cannot establish.

The wording on the screen is part of the product’s code. This short excerpt from the English version connects the agency name, the explanation and the next available step:

title: 'Service alerts unavailable',
body: (operator: string) =>
	`No ${operator} alert feed is connected here. This does not mean there are no disruptions.`,
link: 'Check official service alerts',

operator is replaced with the agency’s name. The message distinguishes missing information from an absence of disruptions, then points to the official source. Even at this small scale, writing an interface means considering what a reader can reasonably conclude.

Another example involves arrival predictions. A prediction can exist for a trip even when the map has no vehicle associated with that trip. Work on the Ottawa pages preserved that information where it belongs. A vehicle marker and an arrival prediction answer different questions.

Continuing the work#

The interface uses an identified release of my shared design foundation, yesid.dev-design. Colours, shared components and interface behaviours have a common source. Transit still has its own reading needs: a dense map, tables, filters, unknown states and explanations that need to work on a small screen.

The project is deployed and continues to develop. The public site includes pages for Montréal and Ottawa, but that does not guarantee that every source is always available or that its history is complete. Repair and consolidation work is ongoing. Existing checks retain their specific scope; they do not stand in for every real-device or assistive-technology test.

I’m developing Transit with the support of AI tools, while also working to understand the system more deeply. I’m taking time to follow the code, understand where each behaviour belongs and explain the project’s decisions more clearly. That learning is still in progress. I want to connect what the site shows to the code and data that produce it, then make changes with a stronger understanding of their consequences.

The project brings together several parts of my work as a Freelance Digital Solutions Developer: starting with a useful question, organizing imperfect sources, building a readable interface and making limitations visible. The next steps concern the reliability and simplicity of the system as much as the clarity of what people can learn from it.

Explore Transit

From the screen to the underlying system#

The map is Transit’s most visible part. The following sections explain the work connecting a prediction, a measure and a published version. Each can be read independently, with the calculations and examples needed to understand that part of the system.

The repository contains three applications with different responsibilities. Their boundaries help trace a problem to the place where it can be explained and corrected.

• apps/db receives data, organizes it in PostgreSQL and prepares the public files. Python handles ingestion, processing, checks and publication.
• apps/data-proxy is a Cloudflare Worker that serves those files from object storage. It handles HTTP transport: reads, metadata, caching and partial responses.
• apps/web is the SvelteKit application. It reads published files, validates their structure and presents maps, tables and explanations. It does not need direct access to the processing database.

Consider a fictional update to follow the full journey. A feed predicts a stop arrival at 2:07 p.m., while the matching timetable gives 2:03 p.m. This is an illustrative example, not an observed trip in Montréal or Ottawa.

The first step is to retain the received response. The layer called Bronze keeps the source and its identity. A hash can identify its contents, but content and observation are different: receiving identical bytes at two different times can represent two captures. Deduplicating files without keeping that distinction would remove part of the timeline.

Silver then turns the response into relational records. The loading step receives the exact capture and the bytes just collected. It does not simply ask for “the latest” capture, which might change between those steps. The code checks that the receipt matches the expected provider, endpoint and capture identifier.

Gold brings the records together and prepares measures. In this example, it needs to match the network, the relevant timetable edition, the trip and the stop sequence. If that match succeeds and both times use the same time reference, the predicted delay is 240 seconds. If the timetable cannot be linked to the prediction, the predicted arrival can remain available while its delay remains unknown.

Published files make the information available to readers. A page can then show the arrival prediction, its context and, when it can be calculated, the associated delay. The presence of a prediction does not require a corresponding vehicle to appear on the map.

Each stage keeps its own execution evidence. Capture can succeed while loading fails; loading can succeed while publication fails. Those differences matter when deciding where to resume work. A single green light for the entire pipeline would hide the last stage that was actually completed.

A trip identifier only makes sense in context. Two providers can use the same value. One provider can publish a new timetable edition. A capture contains multiple entities, and a trip update contains multiple stop updates. Joins therefore need the provider, relevant edition, capture, entity and stop sequence at the appropriate level.

The same precision applies to the public catalogue. A network can be active for ingestion without being ready to appear in the interface. The catalogue builder checks the published identity, time zone, geographic boundaries and required map objects. This excerpt compares three manifest values with the configuration:

if (manifest.provider, manifest.tz, manifest.bbox) != (
    provider_id,
    identity.timezone,
    identity.bounds.bbox(),
):
    raise ValueError("Published identity/geography differs from configuration")

A mismatch stops the catalogue build with an error. The check does not try to guess which source is right. It prevents an incompatible identity and geography from being presented together.

This structure is intended to make Transit independent of any particular provider. The registry loads YAML files from the configuration directory and distinguishes active networks from those ready to appear publicly. Each manifest describes an identity, time zone, geographic bounds, sources, formats, refresh intervals and access methods. A GTFS timetable is required; positions, predictions and alerts are optional inputs. Declaring a network therefore does not make every feature available.

Shared collection and normalization processes receive that identity; the database and published files retain the network to which the data belongs. Publication contracts, the catalogue and the interface allow the same reading journey to be reused. Some sources still need specific handling. The STM geographic package and its i3 alerts have their own formats, while OC Transpo includes route shapes in its GTFS timetable and uses a different authentication header for realtime data. Those differences belong in the components that handle them.

Adding a city requires checking the actual sources, their reuse conditions and attribution, then configuring access without publishing secrets. Trip and stop identifiers also need to match correctly between the timetable and updates. An unsupported format or authentication method may require additional code. Operation needs a prepared database, storage, access permissions and scheduled collection. Configuration alone does not provide those resources.

Public presentation follows: a consistent published identity, an available basemap, suitable framing, French and English labels, official links and a valid catalogue. Repository tests include a synthetic provider distinct from the two current networks and publication rejection when its map or identity is inconsistent. They make that extension testable; their presence does not establish that another city has been integrated or that these tests have just been run.

Storage is still configured for the deployment, with separate paths for each provider. The catalogue does not let each contributor connect their own storage and credentials to the same hosted interface. That federation would require additional design. Other limitations remain: some shared time-display functions still use America/Toronto, and the default operating configuration retains STM-specific settings and a pruning service. The STM endpoint discussed later is another current dependency.

Eventually, I want other people to be able to help add cities, sources and the resources needed to operate them. That work could involve an API, storage, adaptations or verification. It is a direction for the project’s development, not an automatic signup service.

Geography also serves several purposes. A position locates a reported vehicle. Timetable shapes describe route geometry. A basemap supplies visual context. Bounds and framing determine the area to display. Ottawa has shapes within its timetable without requiring a separate geographic feed for each of these purposes. Treating the basemap as the source of the vehicle positions would erase that distinction.

Five time references are useful: the trip’s service date, the feed timestamp, the measurement time of a position, Transit’s capture time and the file’s publication time. A response captured at noon can contain a position measured at 11:56 a.m. Collection has just happened, but the position is already four minutes old. Reloading the page does not change its age.

Service timetables introduce another complication. A value such as 25:10:00 refers to a time beyond the first 24 hours of a service day. Reducing it to 01:10:00 without retaining the service date can move the trip to the wrong day. Processing keeps these times as elapsed offsets and uses the provider’s time zone. For derived delays, the code builds its time origin from local noon minus twelve hours to handle daylight-saving transition days under that convention.

The appropriate date also depends on the question. Some summaries group observations by local capture date. Other joins require the service day. Those calendars can differ after midnight. Naming the period being displayed is therefore part of defining the measure.

Transit’s on-time measure describes predicted-delay readings. For routes and daily receipts, an observation belongs to the on-time band when its delay is at least −60 seconds and less than 300 seconds. The denominator contains readings with a known delay.

The formula is on-time percentage = 100 × readings in [−60, 300) / readings with a known delay. The left bracket includes −60; the right parenthesis excludes 300. The publication calculation leaves the result unknown if the numerator is missing or the denominator is not positive:

def otp_pct(on_time: SqlNumber | None, known: SqlNumber | None) -> int | None:
    if on_time is None or not known:
        return None
    known_obs = float(known)
    if known_obs <= 0:
        return None
    return int(round_half_away(100.0 * float(on_time) / known_obs, 0))

Consider seven fictional readings: −60, 0, 120, 300, 600 and 7,200 seconds, plus one unknown value. They are used only to explain the calculations, not to report either network’s performance.

Six readings have a known delay. The first three belong to the on-time band, giving 100 × 3 / 6 = 50%. The unknown value is neither zero delay nor an on-time observation. The 7,200-second reading remains in this measure’s denominator.

Mean delay uses a different population: known delays from −3,600 to +3,600 seconds, including both limits. In this example, 7,200 and the unknown value are excluded. The five remaining values sum to 960 seconds: 960 / 5 = 192 seconds, or 3.2 minutes. Extreme values are excluded, not replaced with 3,600 seconds. An excluded value could still reflect a real disruption: the filter defines the calculation’s population, rather than certifying the source.

Severe-delay share introduces a third distinction. Its numerator counts delays strictly above 300 seconds and no greater than 3,600. For the route, one severe reading out of six known delays gives 16.7%. For a stop statistic using only the eligible range, one out of five gives 20%. The difference does not come from another disruption; it comes from the denominator.

Exactly 300 seconds belongs to neither the on-time band nor the severe-delay numerator. The two percentages should not be added with the expectation that they cover every case.

Combining groups also requires the right inputs. Suppose group A has 600 seconds of accumulated delay across two observations, and group B has 300 seconds across three. The pooled result is 900 / 5 = 180 seconds, or three minutes. Averaging the two group means, 300 and 100 seconds, would give 200 seconds by assigning equal weight to groups of different sizes.

That is why the summaries retain sums and counts, then convert and round the final result. An older, already-rounded mean is not enough to recover an exact sum. If a required contributing hour cannot be reconstructed, a daily summary may remain unavailable.

The same discipline applies to cancellations. Their rate concerns trip-days reported by the feed, grouped by trip identifier and service date, some of which were explicitly marked cancelled. It does not cover every trip in the timetable. Comparing observed and scheduled volumes is another calculation: a shortfall in counts does not identify which trips are missing.

A mean does not describe a whole distribution. Transit also uses percentiles. The median, or p50, locates the middle of the observations; p90 describes another position in their distribution. It does not promise that the next journey will stay below that value.

For daily percentiles, known delays within −3,600 to +3,600 seconds are sorted, then the calculation interpolates between values. With four fictional values of 0, 60, 180 and 600 seconds, the median lies halfway between 60 and 180: 120 seconds. For p90, the position is (4 − 1) × 0.9 = 2.7, counting from zero. Moving 70% of the way between the third and fourth values gives 180 + 0.7 × (600 − 180) = 474 seconds, or 7.9 minutes.

For longer windows, pooled histograms support estimates from ranges of values. That differs from calculating an exact percentile over all the raw observations. Averaging daily p90 values cannot reconstruct a period’s p90 either. The period, population and method need to stay attached to the displayed value.

Gaps between trips address a different question. The system uses first eligible captures within the same route, direction, service day and shift. Eligibility requires a known predicted delay between −3,600 and +3,600 seconds, inclusive. Windowed series exclude weekend service. The scheduled reference comes from the current GTFS timetable on a representative weekday. Gaps must be positive and below 240 minutes. These are appearances in the feed, not physically measured arrivals at a stop. Interrupted collection or a trip that never appears can change the sample.

The wait model uses gap sums and squared-gap sums: AWT = sum(gap²) / (2 × sum(gap)), with gaps in minutes. It then subtracts half the scheduled gap and clamps a negative result to zero.

A second illustrative example shows the effect. Two regular gaps of 10 and 10 minutes give 200 / 40 = 5 minutes. Two gaps of 5 and 15 minutes have the same ten-minute mean but give 250 / 40 = 6.25 minutes. Against a ten-minute scheduled reference, the second case has a modelled excess of 1.25 minutes, published as 1.3 minutes after rounding.

Squaring the gaps gives longer intervals more weight. The model assumes uniform rider arrivals during those intervals, without ridership data. It illustrates the effect of irregularity; it does not measure anyone’s actual wait. Older rows without these moments may use a different proxy based on the difference between medians. A summary that averages available shifts is also different from pooling every gap across a day.

Sampling remains central. If one trip supplies nine readings and another supplies one, the first carries nine times the weight in an observation-based measure. A percentage of 90% could therefore describe nine similar predictions for one trip. It describes neither nine passengers in ten nor a proportion of trips that arrived on time.

Rankings also use sample qualifications and, in some views, Wilson bounds. The code uses z = 1.96; some rankings require at least 30 eligible observations. These methods account for observation volume in the calculation, but repeated predictions are not independent trials. They do not establish 95% statistical coverage for the real network. Missing occupancy data, meanwhile, remains unknown occupancy rather than evidence of an empty vehicle.

The site reads three families of data. Static data describes networks, routes, stops and geographic objects. Live data carries recent positions, predictions and states. History serves retained periods and summaries. These families have different rates of change and different ways of becoming visible.

Contracts describe the files’ structure. Other metadata identifies the methodology and publication generation. A schema version tells a consumer how to read a document; a methodology version identifies the meaning of its calculation; a generation identifier associates results with a publication. Combining those roles would make changes harder to interpret.

On the PostgreSQL side, publication uses a transaction with REPEATABLE READ isolation. It can read a consistent database state while preparing the output. An advisory lock for each provider and publication family prevents competing publishers from taking the same lane simultaneously. This does not turn database writes and object-storage writes into one distributed transaction.

For live data, child files are written before the manifest, which acts as a completion marker. Imagine five files to write: four are replaced, then the fifth fails. The previous manifest may remain, but some direct addresses can already return the new files. Writing the manifest last therefore does not mean that all files are replaced atomically.

Live checks are recorded without systematically blocking availability. In the normal static path, data is checked before upload. Those are different operating policies that need to remain visible when simplifying the code.

History uses another boundary. Child files are immutable, and their reference graph is prepared before its root is activated. The new root is activated only if the previous version still matches the expected version. This is a conditional write: if another publisher changed the root in the meantime, the operation encounters a conflict instead of silently overwriting that work.

A reader following a coherent root and its immutable children can therefore use a stable generation while another is being prepared. Newly written objects may exist without belonging to the announced version. Compatibility addresses retain their own behaviour; the graph’s guarantee does not automatically extend to every URL.

This approach lets the site serve files without asking the database to repeat its calculations for each visitor. In exchange, the system must manage publication, retained objects, failures and active references. The consistency available to a reader depends on the data family and how its references are followed.

The data Worker serves published objects. It does not repeat the whole normalization process on each request. That narrower responsibility allows data errors and transport errors to be handled separately.

An ETag identifies an object version for conditional requests. A HEAD request can retrieve metadata without the contents. A range request can retrieve only the bytes needed from a map archive. The code distinguishes partial responses, failed conditions, missing objects and invalid ranges. An unrelated storage error must not be disguised as a badly requested range.

Caching introduces several clocks: the source observation time, the publication generation time, the time intermediate work was computed and the time the page receives it. An observation from 11:56 a.m. received at noon does not become a noon observation just because an HTTP request succeeded.

One specific endpoint illustrates the distinction. In the reviewed code, the /api/v1/kpis aggregate is fixed to the STM. It is not a universal backend for both cities. Its response can carry no-store while being assembled from internally cached work. Vehicle data anchors availability, while the ages of trip and network data independently qualify certain derived fields. Work caching and source freshness answer different questions.

In the web application, the first render can receive server-prepared data. The browser then continues refreshing and changing selections. Responses are validated at runtime; TypeScript types alone cannot guarantee the structure of a file received over the network.

Switching cities creates a concrete risk: a request started for Montréal may finish after someone selects Ottawa. Cancelling the first request helps, but a late response can still arrive. The resource checks a sequence token before accepting the result:

.then((value) => {
	if (token !== seq) return;
	data = value;
	dataKey = activeKey;
})

If a new request has superseded the old one, its token is no longer current and the response is ignored. Data also belongs to a context key. A Montréal value is not retained as though it belonged to Ottawa; a refresh of the same context can keep its last usable value while loading.

The catalogue, URL, city selection and page links need to retain the same provider. A discovery failure must not silently display STM data under an explicit Ottawa selection. These rules give interface consistency a practical meaning: the chosen question and the visible data must continue to match, even when responses arrive out of order.

The most useful checks target a behaviour that could plausibly break. In Transit, a selection test protects route predictions without an associated vehicle. A rendering test protects the unavailable-alert message. HTTP tests check conditional responses and ranges. They address different risks.

Publication contracts also cross language boundaries. Python models produce JSON schemas, while the web consumer uses its own parsers. Conformance checks compare those representations. A generated file is not an independent second specification on its own: the producer, its export and the consumer’s acceptance or rejection of data all need attention.

For transactions, constraints and recovery, tests against a real database provide evidence that a simulation cannot. The repository includes a case where historical projection fails on its second step: facts, serving state and invalidations must return together to their previous state. Another checks that historical replay does not rewind the version used for live data.

That distinction between history and current state matters. Correcting a past day may require rebuilding its derived data without replacing the vehicle positions currently being served. The code keeps selected-capture identities and processing states. Its timetable-matching policy also matters: the reviewed replay uses the current static edition and reports that choice, rather than claiming to reconstruct everything that was known at the time.

Daily summaries retain their own completion dates for each metric family. One family being current does not establish that every other family is current. A correction can invalidate affected periods; exact sums, retained observations and available dates determine what can be recalculated.

Retention is therefore a functional decision as well as a storage decision. Keeping more source material helps investigation and replay, but consumes resources. Removing it reduces cost and also reduces some reconstruction options. A dashboard can continue displaying summaries after the data needed for a more detailed correction is no longer available.

Operations need to distinguish an absent feed, a failed capture, incomplete loading, delayed calculations and publication that has stopped advancing. Restarting the wrong stage may add work without addressing the cause. Processing receipts, timestamps and health checks help locate that boundary.

Backups follow the same logic. An existing backup file proves that an object exists. A tested restoration into a separate temporary database provides different evidence. Recovering production is another result again. The project has separate procedures for these operations; their presence in the repository does not mean that a complete restoration has just been demonstrated.

Finally, code tests, checks on the deployed site and tests on real devices or with assistive technologies remain complementary. They make it possible to describe what was checked without turning a local result into a general promise. They also help contain complexity: retain checks that protect a real boundary, and simplify without losing the reason those checks exist.