felixalb/pvv-nixos-config
1
0
Fork 0
mirror of https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git synced 2025-12-29 20:08:22 +01:00
Code Issues Projects Releases Wiki Activity

docs/{development, development-misc}: init, README: clean

Browse Source
This commit is contained in:
h7x4
2025-12-29 23:13:02 +09:00
parent 5158b39ec8
commit ba36db33b8
3 changed files with 322 additions and 33 deletions
Download Patch File Download Diff File Expand all files Collapse all files

62
README.md
View File

@@ -1,40 +1,36 @@
# PVV NixOS configs
# PVV NixOS config
## Hvordan endre på ting
This repository contains the NixOS configurations for Programvareverkstedet's server closet.
In addition to machine configurations, it also contains a bunch of shared modules, packages, and
more.
Før du endrer på ting husk å ikke putte ting som skal være hemmelig uten å først lese seksjonen for hemmeligheter!
## Machines
Etter å ha klonet prosjektet ned og gjort endringer kan du evaluere configene med:
| Name | Type | Description |
|----------------------------|----------|-----------------------------------------------------------|
| [bekkalokk][bek] | Physical | Our main web host, webmail, wiki, idp, minecraft map, ... |
| [bicep][bic] | Virtual | Database host, matrix, git mirrors, ... |
| bikkje | Virtual | Experimental login box |
| [brzeczyszczykiewicz][brz] | Physical | Shared music player |
| [georg][geo] | Physical | Shared music player |
| [ildkule][ild] | Virtual | Logging and monitoring host, prometheus, grafana, ... |
| [kommode][kom] | Virtual | Gitea + Gitea pages |
| [lupine][lup] | Physical | Gitea CI/CD runners |
| shark | Virtual | Test host for authentication, absolutely horrendous |
| [wenche][wen] | Virtual | Nix-builders, general purpose compute |
`nix flake check --keep-going`
før du bygger en maskin med:
`nix build .#<maskinnavn>`
hvis du vil være ekstra sikker på at alt bygger så kan du kjøre:
`nix build .` for å bygge alle de viktige maskinene.
NB: Dette kan ta opp til 30 minutter avhengig av hva som ligger i caches
Husk å hvertfall stage nye filer om du har laget dem!
Om alt bygger fint commit det og push til git repoet.
Det er sikkert lurt å lage en PR først om du ikke er vandt til nix enda.
Innen 24h skal alle systemene hente ned den nye konfigurasjonen og deploye den.
Du kan tvinge en maskin til å oppdatere seg før dette ved å kjøre:
`nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --upgrade --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git`
som root på maskinen.
Hvis du ikke har lyst til å oppdatere alle pakkene (og kanskje måtte vente en stund!) kan du kjøre
`nixos-rebuild switch --override-input nixpkgs nixpkgs --override-input nixpkgs-unstable nixpkgs-unstable --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git`
## Annen dokumentasjon
## Documentation
- [Development - working on the PVV machines](./docs/development.md)
- [Miscellaneous development notes](./docs/development-misc.md)
- [User management](./docs/users.md)
- [Secret management and `sops-nix`](./docs/secret-management.md)
[bek]: https://wiki.pvv.ntnu.no/wiki/Maskiner/bekkalokk
[bic]: https://wiki.pvv.ntnu.no/wiki/Maskiner/bicep
[brz]: https://wiki.pvv.ntnu.no/wiki/Maskiner/brzęczyszczykiewicz
[geo]: https://wiki.pvv.ntnu.no/wiki/Maskiner/georg
[ild]: https://wiki.pvv.ntnu.no/wiki/Maskiner/ildkule
[kom]: https://wiki.pvv.ntnu.no/wiki/Maskiner/kommode
[lup]: https://wiki.pvv.ntnu.no/wiki/Maskiner/lupine
[wen]: https://wiki.pvv.ntnu.no/wiki/Maskiner/wenche

103
docs/development-misc.md Normal file
View File

@@ -0,0 +1,103 @@
# Miscellaneous development notes
This document contains a bunch of information that is not particularly specific to the pvv nixos config,
but concerns technologies we use often or gotchas to be aware of when working with NixOS. A lot of the information
here is already public information spread around the internet, but we've collected some of the items we use often
here.
## The firewall
`networking.firewall` is a NixOS module that configures `iptables` rules on the machine. It is enabled by default on
all of our machines, and it can be easy to forget about it when setting up new services, especially when we are the
ones creating the NixOS module.
When setting up a new service that listens on a TCP or UDP port, make sure to add the appropriate ports to either
`networking.firewall.allowedTCPPorts` or `networking.firewall.allowedUDPPorts`.
You can list out the current firewall rules by running `sudo iptables -L -n -v` on the machine.
## Finding stuff
Finding stuff, both underlying implementation and usage is absolutely crucial when working on nix.
Oftentimes, the documentation will be outdated, lacking or just plain out wrong. These are some of
the techniques we have found to be quite good when working with nix.
### [ripgrep](https://github.com/BurntSushi/ripgrep)
ripgrep (or `rg` for short) is a tool that lets you recursively grep for regex patters in a directory.
It is great for finding references to configuration, and where and how certain things are used. It is
especially great when working with [nixpkgs](https://github.com/NixOS/nixpkgs), which is quite large.
### GitHub Search
When trying to set up a new service or reconfigure something, it is very common that someone has done it
before you, but it has never been documented anywhere. A lot of Nix code exists on GitHub, and you can
easily query it by using the `lang:nix` filter in the search bar.
For example: https://github.com/search?q=lang%3Anix+dibbler&type=code
## rsync
`rsync` is a tool for synchronizing files between machines. It is very useful when transferring large
amounts of data from a to b. We use it for multiple things, often when data is produced or stored on
one machine, and we want to process or convert it on another. For example, we use it to transfer gitea
artifacts, to transfer gallery pictures, to transfer minecraft world data for map rendering, and more.
Along with `rsync`, we often use a lesser known tool called `rrsync`, which you can use inside an ssh
configuration (`authorized_keys` file) to restrict what paths a user can access when connecting over ssh.
This is useful both as a security measure, but also to avoid accidental overwrites of files outside the intended
path. `rrsync` will use chroot to restrict what paths the user can access, as well as refuse to run arbitrary commands.
## `nix repl`
`nix repl` is an interactive REPL for the Nix language. It is very useful for experimenting with Nix code,
and testing out small snippets of code to make sure it behaves as expected. You can also use it to explore
NixOS machine configurations, to interactively see that the configuration evaluates to what you expect.
```
# While in the pvv-nixos-config directory
nix repl .
# Upon writing out the config path and clickin [Tab], you will get autocompletion suggestions:
nix-repl> nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts._
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.bekkalokk.pvv.ntnu.no-nixos-metrics
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.idp.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.minecraft.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pvv.org
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pw.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.roundcubeplaceholder.example.com
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.snappymail.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.webmail.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.wiki.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.www.pvv.ntnu.no
nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.www.pvv.org
```
## `nix why-depends`
If you ever wonder why a certain package is being used as a dependency of another package,
or another machine, you can use `nix why-depends` to find the dependency path from one package to another.
This is often useful after updating nixpkgs and finding an error saying that a certain package is insecure,
broken or whatnot. You can do something like the following
```bash
# Why does bekkalokk depend on openssl?
nix why-depends .#nixosConfigurations.bekkalokk.config.system.build.toplevel .#nixosConfigurations.bekkalokk.pkgs.openssl
# Why does bekkalokk's minecraft-server depend on zlib? (this is not real)
nix why-depends .#nixosConfigurations.bekkalokk.pkgs.minecraft-server .#nixosConfigurations.bekkalokk.pkgs.zlib
```
## php-fpm
php-fpm (FastCGI Process Manager) is a PHP implementation that is designed for speed and production use. We host a bunch
of different PHP applications (including our own website), and so we use php-fpm quite a bit. php-fpm typically exposes a
unix socket that nginx will connect to, and php-fpm will then render php upon web requests forwarded from nginx and return
it.
php-fpm has a tendency to be a bit hard to debug. It is not always very willing to spit out error messages and logs, and so
it can be a bit hard to figure out what's up when something goes wrong. You should see some of the commented stuff laying around
in the website code on bekkalokk for examples of how to configure php-fpm for better logging and error reporting.

190
docs/development.md Normal file
View File

@@ -0,0 +1,190 @@
# Development - working on the PVV machines
This document outlines the process of editing our NixOS configurations, and testing and deploying said changes
to the machines. Most of the information written here is specific to the PVV NixOS configuration, and the topics
will not really cover the nix code itself in detail. You can find some more resources for that by either following
the links from the *Upstream documentation* section below, or in [Miscellaneous development notes](./development-misc.md).
## Editing nix files
> [!WARN]
> Before editing any nix files, make sure to read [Secret management and `sops-nix`](./secret-management.md)!
> We do not want to add any secrets in plaintext to the nix files, and certainly not commit and publish
> them into the common public.
The files are plaintext code, written in the [`Nix` language](https://nix.dev/manual/nix/stable/language/).
Below is a list of important files and directories, and a description of what they contain.
### `flake.nix`
The `flake.nix` file is a [nix flake](https://wiki.nixos.org/wiki/Flakes) and makes up the entrypoint of the
entire configuration. It declares what inputs are used (similar to dependencies), as well as what outputs the
flake exposes. In our case, the most important outputs are the `nixosConfigurations` (our machine configs), but
we also expose custom modules, packages, devshells, and more. You can run `nix flake show` to get an overview of
the outputs (however you will need to [enable the `nix-flakes` experimental option](https://wiki.nixos.org/wiki/Flakes#Setup)).
You will find that a lot of the flake inputs are the different PVV projects that we develop, imported to be hosted
on the NixOS machines. This makes it easy to deploy changes to these projects, as we can just update the flake input
to point to a new commit or version, and then rebuild the machines.
A NixOS configuration is usually made with the `nixpkgs.lib.nixosSystem` function, however we have a few custom wrapper
functions named `nixosConfig` and `stableNixosConfig` that abstracts away some common configuration we want on all our machines.
### `values.nix`
`values.nix` is a somewhat rare pattern in NixOS configurations around the internet. It contains a bunch of constant values
that we use throughout the configuration, such as IP addresses, DNS names, paths and more. This not only makes it easier to
change the values should we need to, but it also makes the configuration more readable. Instead of caring what exact IP any
machine has, you can write `values.machines.name.ipv4` and abstract the details away.
### `base`
The `base` directory contains a bunch of NixOS configuration that is common for all or most machines. Some of the config
you will find here sets defaults for certain services without enabling them, so that when they are enabled in a machine config,
we don't need to repeat the same defaults over again. Other parts actually enable certain services that we want on all machines,
such as `openssh` or the auto upgrade timer.
### Vendoring `modules` and `packages`
Sometimes, we either find that the packages or modules provided by `nixpkgs` is not sufficient for us,
or that they are bugged in some way that can not be easily overrided. There are also cases where the
modules or packages does not exist. In these cases, we tend to either copy and modify the modules and
packages from nixpkgs, or create our own. These modules and packages end up in the top-level `modules`
and `packages` directories. They are usually exposed in `flake.nix` as flake outputs `nixosModules.<name>`
and `packages.<platform>.<name>`, and they are usually also added to the machines that need them in the flake.
In order to override or add an extra package, the easiest way is to use an [`overlay`](https://wiki.nixos.org/wiki/Overlays).
This makes it so that the package from `pkgs.<name>` now refers to the modified variant of the package.
In order to add a module, you can just register it in the modules of the nixos machine.
In order to override a module, you also have to use `disabledModules = [ "<path-relative-to-nixpkgs/modules>" ];`.
Use `rg` to find examples of the latter.
Do note that if you believe a new module to be of high enough quality, or the change you are making to be
relevant for every nix user, you should strongly consider also creating a PR towards nixpkgs. However,
getting changes made there has a bit higher threshold and takes more time than making changes in the PVV config,
so feel free to make the changes here first. We can always remove the changes again once the upstreaming is finished.
### `users`, `secrets` and `keys`
For `users`, see [User management](./users.md)
For `secrets` and `keys`, see [Secret management and `sops-nix`](./secret-management.md)
### Collaboration
We use our gitea to collaborate on changes to the nix configuration. Every PVV maintenance member should have
access to the repository. The usual workflow is that we create a branch for the change we want to make, do a bunch
of commits and changes, and then open a merge request for review (or just rebase on master if you know what you are doing).
### Upstream documentation
Here are different sources of documentation and stuff that you might find useful while
writing, editing and debugging nix code.
- [nixpkgs repository](https://github.com/NixOS/nixpkgs)
This is particularly useful to read the source code, as well as upstreaming pieces of code that we think
everyone would want
- [NixOS search](https://search.nixos.org/)
This is useful for searching for both packages and NixOS options.
- [nixpkgs documentation](https://nixos.org/manual/nixpkgs/stable/)
- [NixOS documentation](https://nixos.org/manual/nixos/stable/)
- [nix (the tool) documentation](https://nix.dev/manual/nix/stable/)
All of the three above make up the official documentation with all technical
details about the different pieces that makes up NixOS.
- [The official NixOS wiki](https://wiki.nixos.org)
User-contributed guides, tips and tricks, and whatever else.
- [nix.dev](https://nix.dev)
Additional stuff
- [Noogle](https://noogle.dev)
This is useful when looking for nix functions and packaging helpers.
## Testing and deploying changes
After editing the nix files on a certain branch, you will want to test and deploy the changes to the machines.
Unfortunately, we don't really have a good setup for testing for runtime correctness locally, but we can at least
make sure that the code evaluates and builds correctly before deploying.
To just check that the code evaluates without errors, you can run:
```bash
nix flake check
# Or if you want to keep getting all errors before it quits:
nix flake check --keep-going
```
> [!NOTE]
> If you are making changes that involves creating new nix files, remember to `git add` those files before running
> any nix commands. Nix refuses to acknowledge files that are not either commited or at least staged. It will spit
> out an error message about not finding the file in question.
### Building machine configurations
To build any specific machine configuration and look at the output, you can run:
```bash
nix build .#nixosConfigurations.<machine-name>.config.system.build.toplevel
# or just
nix build .#<machine-name>
```
This will create a symlink name `./result` to a directory containing the built NixOS system. It is oftentimes
the case that config files for certain services only end up in the nix store without being put into `/etc`. If you wish
to read those files, you can often find them by looking at the systemd unit files in `./result/etc/systemd/system/`.
(if you are using vim, `gf` or go-to-file while the cursor is over a file path is a useful trick while doing this).
If you have edited something that affects multiple machines, you can also build all important machines at once by running:
```bash
nix build .#
```
> [!NOTE]
> Building all machines at once can take a long time, depending on what has changed and whether you have already
> built some of the machines recently. Be prepared to wait for up to an hour to build all machines from scratch
> if this is the first time.
### Deploying to machines
> [!WARN]
> Be careful to think about state when testing changes against the machines. Sometimes, a certain change
> can lead to irreversible changes to the data stored on the machine. An example would be a set of database
> migrations applied when testing a newer version of a service. Unless that service also comes with downwards
> migrations, you can not go back to the previous version without losing data.
To deploy the changes to a machine, you should first SSH into the machine, and clone the pvv-nixos-config
repository unless you have already done so. After that, checkout the branch you want to deploy from, and rebuild:
```bash
# Run this while in the pvv-nixos-config directory
sudo nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --flake .# --upgrade
```
This will rebuild the NixOS system on the current branch and switch the system configuration to reflect the new changes.
Note that unless you eventually merge the current changes into `main`, the machine will rebuild itself automatically and
revert the changes on the next nightly rebuild (tends to happen when everybody is asleep).
### Forcefully reset to `main`
If you ever want to reset a machine to the `main` branch, you can do so by running:
```bash
nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --upgrade --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git
```
This will ignore the current branch and just pull the latest `main` from the git repository directly from gitea.
You can also use this command if there are updates on the `main` branch that you want to deploy to the machine without
waiting for the nightly rebuild.