Internal Services
This page provides some documenation on other services used within burble.dn42 that are not directly available for public use.
burble.dn42
burble.dn42 nodes are virtualised using Incus.
Most services in the environment are stateless containers built using HashiCorp Packer.
ansible and OpenTofu are used as configration and deployment tools.
Ansible is mainly used for the host node configuration, and for configuration of individual services.
OpenTofu is used to deploy:
- Incus resources (profiles, containers, volumes etc)
- Nomad volumes
- Garage buckets
- OpenBao and Nomad Configuration
dnscontrol is used to manage DNS.
Most configuration is maintained within the burble.dn42 git server and OpenTofu state is stored (encrypted, using OpenBao) in the S3 service.
burble.dn42 uses Borg for backing up nodes and services.
There is a 3 node backup strategy; most nodes back up to OVH dedicated servers in either Europe or North America depending on location. OVH servers backup to a HostHatch storage node.
The 3 node strategy ensures that nodes are always backed up to storage that sits with an independent provider.
burble.dn42 runs a global traefik cluster which acts as a reverse proxy and load balancer for burble.dn42 web services.
The traefik instances are anycast globally (traefik.burble.dn42), but also have regional load balancing groups for Europe (traefik-eu.burble.dn42) and North America (traefik-na.burble.dn42). This regional split helps to direct users to local services where possible.
The burble.dn42 git has an associated CI/CD service based on woodpecker.
The CI/CD service is used to build and publish applications and the burble.dn42 website.
elburb.dn42
elburb.dn42 is an internal services network that provides core infrastructure functionality to burble.dn42.
elburb.dn42 operates separate resources, with its own configuration and isolated network.
OpenBao is used as a central secret store across networks.
OpenBao is deployed as a 5 node cluster across the Europe core nodes and uses the internal raft database as a back end.
OpenTofu is used to manage the OpenBao configuration.
OpenBao acts as the main certificate authority for burble.dn42 PKI.
OpenBao allows for regular, automated renewal of certificates on short timeframes (typically a rolling week or monthly basis).
OpenBao also acts as an SSH certificate authority, verifying both users and servers within the network.
Server certificates are generated during deployment, whilst user (or role) certificates are short lived and generated on demand.
OpenBao holds secrets used during node and service deployments.
Most burble.dn42 are built as stateless container images and secrets are pushed from OpenBao in to the live containers at runtime. This ensures the container images do not contain secrets and that secrets can be applied per instance even when using a common image.
OpenBao also manages database credentials (using the mysql/mariadb integration), and these are also automatically generated and pushed in to container instances on deployment.
The authority to access deployment secrets is inherited, on demand, from the user token during the deployment process. This ensures that even if access was gained to the deployment server, secrets could still not be accessed without also having access to a live user token.
elburb.dn42 runs two, globally federated HashiCorp Nomad clusters, one in Europe and the other in North America.
Nomad workloads run in containerd environments using the OpenBao and Traefik services to provide resilient, globally available services. Services are partitioned in to namespaces for isolation.
OpenTofu is used to deploy nomad volumes and to manage the nomad configuration.
A small Garage cluster provides S3 storage capability for resilient configuration or other data.