diff --git a/README.md b/README.md index dd395f1..9962eff 100644 --- a/README.md +++ b/README.md @@ -1,40 +1,36 @@ -# PVV NixOS configs +# PVV NixOS config -## Hvordan endre på ting +This repository contains the NixOS configurations for Programvareverkstedet's server closet. +In addition to machine configurations, it also contains a bunch of shared modules, packages, and +more. -Før du endrer på ting husk å ikke putte ting som skal være hemmelig uten å først lese seksjonen for hemmeligheter! +## Machines -Etter å ha klonet prosjektet ned og gjort endringer kan du evaluere configene med: +| Name | Type | Description | +|----------------------------|----------|-----------------------------------------------------------| +| [bekkalokk][bek] | Physical | Our main web host, webmail, wiki, idp, minecraft map, ... | +| [bicep][bic] | Virtual | Database host, matrix, git mirrors, ... | +| bikkje | Virtual | Experimental login box | +| [brzeczyszczykiewicz][brz] | Physical | Shared music player | +| [georg][geo] | Physical | Shared music player | +| [ildkule][ild] | Virtual | Logging and monitoring host, prometheus, grafana, ... | +| [kommode][kom] | Virtual | Gitea + Gitea pages | +| [lupine][lup] | Physical | Gitea CI/CD runners | +| shark | Virtual | Test host for authentication, absolutely horrendous | +| [wenche][wen] | Virtual | Nix-builders, general purpose compute | -`nix flake check --keep-going` - -før du bygger en maskin med: - -`nix build .#` - -hvis du vil være ekstra sikker på at alt bygger så kan du kjøre: - -`nix build .` for å bygge alle de viktige maskinene. - -NB: Dette kan ta opp til 30 minutter avhengig av hva som ligger i caches - -Husk å hvertfall stage nye filer om du har laget dem! - -Om alt bygger fint commit det og push til git repoet. -Det er sikkert lurt å lage en PR først om du ikke er vandt til nix enda. - -Innen 24h skal alle systemene hente ned den nye konfigurasjonen og deploye den. - -Du kan tvinge en maskin til å oppdatere seg før dette ved å kjøre: -`nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --upgrade --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git` - -som root på maskinen. - -Hvis du ikke har lyst til å oppdatere alle pakkene (og kanskje måtte vente en stund!) kan du kjøre - -`nixos-rebuild switch --override-input nixpkgs nixpkgs --override-input nixpkgs-unstable nixpkgs-unstable --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git` - -## Annen dokumentasjon +## Documentation +- [Development - working on the PVV machines](./docs/development.md) +- [Miscellaneous development notes](./docs/development-misc.md) - [User management](./docs/users.md) - [Secret management and `sops-nix`](./docs/secret-management.md) + +[bek]: https://wiki.pvv.ntnu.no/wiki/Maskiner/bekkalokk +[bic]: https://wiki.pvv.ntnu.no/wiki/Maskiner/bicep +[brz]: https://wiki.pvv.ntnu.no/wiki/Maskiner/brzęczyszczykiewicz +[geo]: https://wiki.pvv.ntnu.no/wiki/Maskiner/georg +[ild]: https://wiki.pvv.ntnu.no/wiki/Maskiner/ildkule +[kom]: https://wiki.pvv.ntnu.no/wiki/Maskiner/kommode +[lup]: https://wiki.pvv.ntnu.no/wiki/Maskiner/lupine +[wen]: https://wiki.pvv.ntnu.no/wiki/Maskiner/wenche diff --git a/docs/development-misc.md b/docs/development-misc.md new file mode 100644 index 0000000..bb1eeb1 --- /dev/null +++ b/docs/development-misc.md @@ -0,0 +1,103 @@ +# Miscellaneous development notes + +This document contains a bunch of information that is not particularly specific to the pvv nixos config, +but concerns technologies we use often or gotchas to be aware of when working with NixOS. A lot of the information +here is already public information spread around the internet, but we've collected some of the items we use often +here. + +## The firewall + +`networking.firewall` is a NixOS module that configures `iptables` rules on the machine. It is enabled by default on +all of our machines, and it can be easy to forget about it when setting up new services, especially when we are the +ones creating the NixOS module. + +When setting up a new service that listens on a TCP or UDP port, make sure to add the appropriate ports to either +`networking.firewall.allowedTCPPorts` or `networking.firewall.allowedUDPPorts`. + +You can list out the current firewall rules by running `sudo iptables -L -n -v` on the machine. + +## Finding stuff + +Finding stuff, both underlying implementation and usage is absolutely crucial when working on nix. +Oftentimes, the documentation will be outdated, lacking or just plain out wrong. These are some of +the techniques we have found to be quite good when working with nix. + +### [ripgrep](https://github.com/BurntSushi/ripgrep) + +ripgrep (or `rg` for short) is a tool that lets you recursively grep for regex patters in a directory. + +It is great for finding references to configuration, and where and how certain things are used. It is +especially great when working with [nixpkgs](https://github.com/NixOS/nixpkgs), which is quite large. + +### GitHub Search + +When trying to set up a new service or reconfigure something, it is very common that someone has done it +before you, but it has never been documented anywhere. A lot of Nix code exists on GitHub, and you can +easily query it by using the `lang:nix` filter in the search bar. + +For example: https://github.com/search?q=lang%3Anix+dibbler&type=code + +## rsync + +`rsync` is a tool for synchronizing files between machines. It is very useful when transferring large +amounts of data from a to b. We use it for multiple things, often when data is produced or stored on +one machine, and we want to process or convert it on another. For example, we use it to transfer gitea +artifacts, to transfer gallery pictures, to transfer minecraft world data for map rendering, and more. + +Along with `rsync`, we often use a lesser known tool called `rrsync`, which you can use inside an ssh +configuration (`authorized_keys` file) to restrict what paths a user can access when connecting over ssh. +This is useful both as a security measure, but also to avoid accidental overwrites of files outside the intended +path. `rrsync` will use chroot to restrict what paths the user can access, as well as refuse to run arbitrary commands. + +## `nix repl` + +`nix repl` is an interactive REPL for the Nix language. It is very useful for experimenting with Nix code, +and testing out small snippets of code to make sure it behaves as expected. You can also use it to explore +NixOS machine configurations, to interactively see that the configuration evaluates to what you expect. + +``` +# While in the pvv-nixos-config directory +nix repl . + +# Upon writing out the config path and clickin [Tab], you will get autocompletion suggestions: +nix-repl> nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts. +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts._ +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.bekkalokk.pvv.ntnu.no-nixos-metrics +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.idp.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.minecraft.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pvv.org +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.pw.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.roundcubeplaceholder.example.com +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.snappymail.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.webmail.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.wiki.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.www.pvv.ntnu.no +nixosConfigurations.bekkalokk.config.services.nginx.virtualHosts.www.pvv.org +``` + +## `nix why-depends` + +If you ever wonder why a certain package is being used as a dependency of another package, +or another machine, you can use `nix why-depends` to find the dependency path from one package to another. +This is often useful after updating nixpkgs and finding an error saying that a certain package is insecure, +broken or whatnot. You can do something like the following + +```bash +# Why does bekkalokk depend on openssl? +nix why-depends .#nixosConfigurations.bekkalokk.config.system.build.toplevel .#nixosConfigurations.bekkalokk.pkgs.openssl + +# Why does bekkalokk's minecraft-server depend on zlib? (this is not real) +nix why-depends .#nixosConfigurations.bekkalokk.pkgs.minecraft-server .#nixosConfigurations.bekkalokk.pkgs.zlib +``` + +## php-fpm + +php-fpm (FastCGI Process Manager) is a PHP implementation that is designed for speed and production use. We host a bunch +of different PHP applications (including our own website), and so we use php-fpm quite a bit. php-fpm typically exposes a +unix socket that nginx will connect to, and php-fpm will then render php upon web requests forwarded from nginx and return +it. + +php-fpm has a tendency to be a bit hard to debug. It is not always very willing to spit out error messages and logs, and so +it can be a bit hard to figure out what's up when something goes wrong. You should see some of the commented stuff laying around +in the website code on bekkalokk for examples of how to configure php-fpm for better logging and error reporting. diff --git a/docs/development.md b/docs/development.md new file mode 100644 index 0000000..ab9e02e --- /dev/null +++ b/docs/development.md @@ -0,0 +1,190 @@ +# Development - working on the PVV machines + +This document outlines the process of editing our NixOS configurations, and testing and deploying said changes +to the machines. Most of the information written here is specific to the PVV NixOS configuration, and the topics +will not really cover the nix code itself in detail. You can find some more resources for that by either following +the links from the *Upstream documentation* section below, or in [Miscellaneous development notes](./development-misc.md). + +## Editing nix files + +> [!WARN] +> Before editing any nix files, make sure to read [Secret management and `sops-nix`](./secret-management.md)! +> We do not want to add any secrets in plaintext to the nix files, and certainly not commit and publish +> them into the common public. + +The files are plaintext code, written in the [`Nix` language](https://nix.dev/manual/nix/stable/language/). + +Below is a list of important files and directories, and a description of what they contain. + +### `flake.nix` + +The `flake.nix` file is a [nix flake](https://wiki.nixos.org/wiki/Flakes) and makes up the entrypoint of the +entire configuration. It declares what inputs are used (similar to dependencies), as well as what outputs the +flake exposes. In our case, the most important outputs are the `nixosConfigurations` (our machine configs), but +we also expose custom modules, packages, devshells, and more. You can run `nix flake show` to get an overview of +the outputs (however you will need to [enable the `nix-flakes` experimental option](https://wiki.nixos.org/wiki/Flakes#Setup)). + +You will find that a lot of the flake inputs are the different PVV projects that we develop, imported to be hosted +on the NixOS machines. This makes it easy to deploy changes to these projects, as we can just update the flake input +to point to a new commit or version, and then rebuild the machines. + +A NixOS configuration is usually made with the `nixpkgs.lib.nixosSystem` function, however we have a few custom wrapper +functions named `nixosConfig` and `stableNixosConfig` that abstracts away some common configuration we want on all our machines. + +### `values.nix` + +`values.nix` is a somewhat rare pattern in NixOS configurations around the internet. It contains a bunch of constant values +that we use throughout the configuration, such as IP addresses, DNS names, paths and more. This not only makes it easier to +change the values should we need to, but it also makes the configuration more readable. Instead of caring what exact IP any +machine has, you can write `values.machines.name.ipv4` and abstract the details away. + +### `base` + +The `base` directory contains a bunch of NixOS configuration that is common for all or most machines. Some of the config +you will find here sets defaults for certain services without enabling them, so that when they are enabled in a machine config, +we don't need to repeat the same defaults over again. Other parts actually enable certain services that we want on all machines, +such as `openssh` or the auto upgrade timer. + +### Vendoring `modules` and `packages` + +Sometimes, we either find that the packages or modules provided by `nixpkgs` is not sufficient for us, +or that they are bugged in some way that can not be easily overrided. There are also cases where the +modules or packages does not exist. In these cases, we tend to either copy and modify the modules and +packages from nixpkgs, or create our own. These modules and packages end up in the top-level `modules` +and `packages` directories. They are usually exposed in `flake.nix` as flake outputs `nixosModules.` +and `packages..`, and they are usually also added to the machines that need them in the flake. + +In order to override or add an extra package, the easiest way is to use an [`overlay`](https://wiki.nixos.org/wiki/Overlays). +This makes it so that the package from `pkgs.` now refers to the modified variant of the package. + +In order to add a module, you can just register it in the modules of the nixos machine. +In order to override a module, you also have to use `disabledModules = [ "" ];`. +Use `rg` to find examples of the latter. + +Do note that if you believe a new module to be of high enough quality, or the change you are making to be +relevant for every nix user, you should strongly consider also creating a PR towards nixpkgs. However, +getting changes made there has a bit higher threshold and takes more time than making changes in the PVV config, +so feel free to make the changes here first. We can always remove the changes again once the upstreaming is finished. + +### `users`, `secrets` and `keys` + +For `users`, see [User management](./users.md) + +For `secrets` and `keys`, see [Secret management and `sops-nix`](./secret-management.md) + +### Collaboration + +We use our gitea to collaborate on changes to the nix configuration. Every PVV maintenance member should have +access to the repository. The usual workflow is that we create a branch for the change we want to make, do a bunch +of commits and changes, and then open a merge request for review (or just rebase on master if you know what you are doing). + +### Upstream documentation + +Here are different sources of documentation and stuff that you might find useful while +writing, editing and debugging nix code. + + - [nixpkgs repository](https://github.com/NixOS/nixpkgs) + +This is particularly useful to read the source code, as well as upstreaming pieces of code that we think +everyone would want + + - [NixOS search](https://search.nixos.org/) + +This is useful for searching for both packages and NixOS options. + + - [nixpkgs documentation](https://nixos.org/manual/nixpkgs/stable/) + - [NixOS documentation](https://nixos.org/manual/nixos/stable/) + - [nix (the tool) documentation](https://nix.dev/manual/nix/stable/) + +All of the three above make up the official documentation with all technical +details about the different pieces that makes up NixOS. + + - [The official NixOS wiki](https://wiki.nixos.org) + +User-contributed guides, tips and tricks, and whatever else. + + - [nix.dev](https://nix.dev) + +Additional stuff + + - [Noogle](https://noogle.dev) + +This is useful when looking for nix functions and packaging helpers. + +## Testing and deploying changes + +After editing the nix files on a certain branch, you will want to test and deploy the changes to the machines. +Unfortunately, we don't really have a good setup for testing for runtime correctness locally, but we can at least +make sure that the code evaluates and builds correctly before deploying. + +To just check that the code evaluates without errors, you can run: + +```bash +nix flake check +# Or if you want to keep getting all errors before it quits: +nix flake check --keep-going +``` + +> [!NOTE] +> If you are making changes that involves creating new nix files, remember to `git add` those files before running +> any nix commands. Nix refuses to acknowledge files that are not either commited or at least staged. It will spit +> out an error message about not finding the file in question. + +### Building machine configurations + +To build any specific machine configuration and look at the output, you can run: + +```bash +nix build .#nixosConfigurations..config.system.build.toplevel +# or just +nix build .# +``` + +This will create a symlink name `./result` to a directory containing the built NixOS system. It is oftentimes +the case that config files for certain services only end up in the nix store without being put into `/etc`. If you wish +to read those files, you can often find them by looking at the systemd unit files in `./result/etc/systemd/system/`. +(if you are using vim, `gf` or go-to-file while the cursor is over a file path is a useful trick while doing this). + +If you have edited something that affects multiple machines, you can also build all important machines at once by running: + +```bash +nix build .# +``` + +> [!NOTE] +> Building all machines at once can take a long time, depending on what has changed and whether you have already +> built some of the machines recently. Be prepared to wait for up to an hour to build all machines from scratch +> if this is the first time. + +### Deploying to machines + +> [!WARN] +> Be careful to think about state when testing changes against the machines. Sometimes, a certain change +> can lead to irreversible changes to the data stored on the machine. An example would be a set of database +> migrations applied when testing a newer version of a service. Unless that service also comes with downwards +> migrations, you can not go back to the previous version without losing data. + +To deploy the changes to a machine, you should first SSH into the machine, and clone the pvv-nixos-config +repository unless you have already done so. After that, checkout the branch you want to deploy from, and rebuild: + +```bash +# Run this while in the pvv-nixos-config directory +sudo nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --flake .# --upgrade +``` + +This will rebuild the NixOS system on the current branch and switch the system configuration to reflect the new changes. + +Note that unless you eventually merge the current changes into `main`, the machine will rebuild itself automatically and +revert the changes on the next nightly rebuild (tends to happen when everybody is asleep). + +### Forcefully reset to `main` + +If you ever want to reset a machine to the `main` branch, you can do so by running: + +```bash +nixos-rebuild switch --update-input nixpkgs --update-input nixpkgs-unstable --no-write-lock-file --refresh --upgrade --flake git+https://git.pvv.ntnu.no/Drift/pvv-nixos-config.git +``` + +This will ignore the current branch and just pull the latest `main` from the git repository directly from gitea. +You can also use this command if there are updates on the `main` branch that you want to deploy to the machine without +waiting for the nightly rebuild.