Deployed Services and Sites#
Many services and websites are deployed as part of the Cal-ITP ecosystem, maintained directly by Cal-ITP contributors and orchestrated via Google Kubernetes Engine or published via Netlify or GitHub Pages.
With the exception of Airflow, which is managed via Google Cloud Composer, changes to the services discussed here are deployed via CI/CD processes that run automatically when new code is merged to the relevant Cal-ITP repository. These CI/CD processes are not all identical - different services have different testing steps that run when a pull request is opened against the services’s code. Some services undergo a full test deployment when a PR is opened, some report the changes that a subject Helm chart will undergo upon merge, and some just perform basic linting.
READMEs describing the individual testing and deployment process for each service are linked in the below table, and the CI README provides some more general context for Kubernetes-based deployments. Many services are monitored via Sentry, discussed in a later section.
Name |
Function |
URL |
Source code and README (if present) |
K8s namespace |
Development/test environment? |
Service Type |
|---|---|---|---|---|---|---|
Airflow |
General orchestation/automation platform; downloads non-GTFS Realtime data and orchestrates data transformations outside of dbt; executes stateless jobs such as dbt and data publishing |
https://b2062ffca77d44a28b4e05f8f5bf4996-dot-us-west2.composer.googleusercontent.com/home |
n/a |
Yes (local) |
Infrastructure / Ingestion |
|
BigQuery |
Data warehouse used for data modeling and analysis (managed via Google Cloud Composer) |
https://console.cloud.google.com/bigquery?project=cal-itp-data-infra |
cal-itp/data-infra (infra managed via Google Cloud Composer) |
n/a |
Yes (cal-itp-data-infra-staging) |
Infrastructure |
Cal-ITP Docs Site |
Public-facing documentation for various Cal-ITP repositories, services, and related technical resources |
n/a |
Yes (Netlify deploy on PRs) |
Documentation |
||
Dask |
Parallelization infrastructure used by JupyerHub and Prometheus |
n/a |
dask |
No |
Infrastructure |
|
Grafana |
Application observability (i.e. monitoring and alerting on metrics) |
JarvusInnovations/cluster-template (via hologit) |
monitoring-grafana |
No |
Infrastructure |
|
GTFS-RT Archiver |
Downloads GTFS Realtime data (more rapidly than Airflow can handle) |
n/a |
gtfs-rt-v3 |
Yes (gtfs-rt-v3-test) |
Ingestion |
|
JupyterHub |
Kubernetes-driven Jupyter workspace provider |
jupyterhub |
No |
Analysis |
||
Metabase |
Web-hosted BI tool |
metabase |
Yes (metabase-test) |
Analysis |
||
Netlify |
Web publishing infrastructure for Cal-ITP Analysis Portfolio, dbt docs site, and California GTFS Quality Dashboard, managed via GitHub interface |
TODO |
Varies by specific deployed site (infra managed via Netlify GitHub interface) |
n/a |
Yes (varies by deployed site) |
Documentation / Infrastructure |
Sentry |
Application error observability (i.e. collecting errors for investigation) |
sentry |
No |
Infrastructure |
Code and deployments (unless otherwise specified, deployments occur via GitHub Actions)#
The above services are managed by code and deployed according to the following processes.
Monitoring running services#
Error monitoring through Sentry#
A subset of our services and sites send error information to the Cal-ITP Sentry instance, which groups errors based on criteria we define in order to identify and track new errors, regressions, and intermittent service issues. A runbook is available here which discusses daily triage of events logged in Sentry, and general documentation for self-hosted instances of Sentry like ours is available here.
Cost and performance monitoring#
In addition to standard Google Cloud tooling for monitoring specific services like Kubernetes Engine, Composer, and BigQuery, we maintain a couple dashboards that can make it easier to glean at-at-glance insights and report topline information to stakeholders who don’t have the time, access, or knowledge to dig into GCP monitoring directly.
The BigQuery overview dashboard gives a detailed view of daily and monthly BigQuery costs, and helps identify tables and dbt models that are producing greater build costs, reference costs, and query costs than others.
In a prior staffing configuration, a person was assigned to fill the Cal-ITP System Performance and Outcomes Monitoring dashboard and use it to populate the metrics in the spreadsheet linked within the dashboard each Friday. Checks were recommended to occur on Fridays because for one of the metrics, Google auto-bins 7 day periods to Friday-Thursday. This dashboard and spreadsheet could be re-activated if desired.