diff options
Diffstat (limited to 'blog')
| -rw-r--r-- | blog/2022-08-29-blog-start.md | 9 | ||||
| -rw-r--r-- | blog/2022-08-29-blog-test.md | 41 | ||||
| -rw-r--r-- | blog/2022-08-30-blog-tools.md | 104 | ||||
| -rw-r--r-- | blog/2022-09-19-gctf-readysetaction.md | 84 | ||||
| -rw-r--r-- | blog/2022-09-25-ductf-file-magic.md | 260 | ||||
| -rw-r--r-- | blog/2022-11-07-programming-language-design.md | 70 | ||||
| -rw-r--r-- | blog/2022-11-10-artist-correlation.md | 78 | ||||
| -rw-r--r-- | blog/2023-02-13-new-website-using-gnix.md | 103 | ||||
| -rw-r--r-- | blog/2024-05-03-replaced-infra.md | 107 | ||||
| -rw-r--r-- | blog/2024-11-28-optimal-computer-part-1.md | 137 |
10 files changed, 993 insertions, 0 deletions
diff --git a/blog/2022-08-29-blog-start.md b/blog/2022-08-29-blog-start.md new file mode 100644 index 0000000..2f17e6c --- /dev/null +++ b/blog/2022-08-29-blog-start.md @@ -0,0 +1,9 @@ +# Starting my blog + +This will be the place where I will post short articles about my thoughts and projects in the future, whenever I feel like it. + +A list of topics of which I already have an idea in mind are: + +- The tools I used to create this blog +- My dream operating system (maybe even logs about me trying (aka failing) to implement it) +- In search of a perfect text editor diff --git a/blog/2022-08-29-blog-test.md b/blog/2022-08-29-blog-test.md new file mode 100644 index 0000000..1c4a28a --- /dev/null +++ b/blog/2022-08-29-blog-test.md @@ -0,0 +1,41 @@ +# Blog test + +Hello world! + +## heading + +_italic_ **bold** `code` [Link](https://metamuffin.org/) + +### sub heading + +Look at this code here. + +```rs +#[derive(Default)] +struct Impossible<T> { + something: HashMap<usize, T>, + // ^- look here, a keyword! --the grammar regex + itself: Impossible, +} + +pub async fn useless(s: Box<dyn Write>) -> impl Future<usize> { + todo!() // this panics +} +fn main() { + 1 + "blub" + for (x, y) in [(1,2),(3,4)] { + println!("{x} and {y}"); + Impossible::default(); + } +} +``` + +As you can see, its pointless i.e. no points + +- list +- list +- list + +1. first this +2. then that +3. done diff --git a/blog/2022-08-30-blog-tools.md b/blog/2022-08-30-blog-tools.md new file mode 100644 index 0000000..81d1ff9 --- /dev/null +++ b/blog/2022-08-30-blog-tools.md @@ -0,0 +1,104 @@ +# Tools I use to create this blog + +As you might expect, this blog uses some overengineered tools, after all, it +wouldn't be so much fun otherwise. This article is probably not too interesting +on its own. It's just "documentation". + +// TODO upload source code and link it here. + +## "Build system" + +```tree +. +├── ... (readme 'n stuff) +├── code +│ ├── Cargo.toml +│ └── src +│ └── ... (rust source) +└── content + ├── articles + │ ├── 2022-08-29-blog-start.md + │ ├── 2022-08-29-blog-test.md + │ └── 2022-08-30-blog-tools.md + ├── makefile + ├── out + │ ├── ... (generated files) + └── style.css +``` + +The entry point to this "build system" is `content/makefile`, it has rules for +generating HTM from the markdown sources, the index and the atom feed. The +"compiler" here is a small rust program. (`code`). + +```makefile +# oversimplifed, doesnt work +TOOL := blog-tool + +out/index: $(ALL_ARTICLES) $(TOOL) + $(TOOL) render-index > $@ +out/feed.atom: $(ALL_ARTICLES) $(TOOL) + $(TOOL) generate-atom > $@ +out/%: articles/%.md $(TOOL) + $(TOOL) render-article $< > $@ +``` + +A small trick here is, to make everything depend on the compiler (`$(TOOL)`) +too, so that when it changes, re-generation of all articles if triggered. + +## Rust tools + +I use the [laby](https://crates.io/crates/laby) crate for templating the HTM. + +// TODO what is important here?! + +### File server + +I wanted to serve all files without the file name extension (`.html`), but my +previous http server ([http-server](https://github.com/http-party/http-server)) +exclusively inferred the MIME type from the extension, which makes it impossible +to do that. The obvious solution was to reinvent the wheel. + +This had the great side-effect of making my website blazing fast 🚀🚀! :) + +#### http-server (node.js) + +``` +Running 10s test @ http://127.0.0.1:8080/test-file + 2 threads and 100 connections +Requests/sec: 2314.00 +Transfer/sec: 725.38KB +``` + +#### fileserver (rust, no caching) + +``` +Running 10s test @ http://127.0.0.1:8080/test-file + 2 threads and 100 connections +Requests/sec: 24464.69 +Transfer/sec: 3.10MB +``` + +// TODO also upload source code and link it + +### Syntax highlighing + +For that, I chose the crate [synoptic](https://crates.io/crates/synoptic). It +provides a some functions for defining tokens with a regex, then returning +start- and end-points of those for further processing. For example, this is how +i defined comments and types: + +```rs +let rust_grammar = &[ + (&["(?m)(//.*)$"], "comment"), + (&["[A-Z][a-z]*", "bool", "usize", /* simplified */ ], "type"), + /* more rules */ +] +``` + +The library finds all the tokens and lets me serialize them to HTM. + +## End + +I still need to learn, how to write this blog well and what is the most +interesting to read. Please give me some advice or commentary via matrix, mail +or fedi. (see [contact](https://metamuffin.org/contact)) diff --git a/blog/2022-09-19-gctf-readysetaction.md b/blog/2022-09-19-gctf-readysetaction.md new file mode 100644 index 0000000..0d5b719 --- /dev/null +++ b/blog/2022-09-19-gctf-readysetaction.md @@ -0,0 +1,84 @@ +# Google Beginner CTF: ReadySetAction + +**SPOILER WARNING: Go play the CTF at +[capturetheflag.withgoogle.com](https://capturetheflag.withgoogle.com/) first** + +The challenge was a short python script with the following code: + +```py +from Crypto.Util.number import * + +flag = b"REDACTED" + +p = getPrime(1024) +q = getPrime(1024) +n = p*q + +m = bytes_to_long(flag) + + +c = pow(m,3,n) + +print(c) +print(n) +#154780 ... 6709 +#210348 ... 4477 +#(i removed both 617 digit long numbers here) +``` + +It was immediatly obvious to me that this has to do with RSA cryptography +because it multiplies large primes and then does some powmod magic. Because I +didn't remember exactly how RSA worked, I quickly read the Wikipedia article +again - specifically section +[4.1 Attacks against plain RSA](https://en.wikipedia.org/wiki/RSA_%29cryptosystem%29#Attacks_against_plain_RSA) + +The first item already seems like the solution: + +> When encrypting with low encryption exponents (e.g., e = 3) and small values +> of the m (i.e., m < n1/e), the result of me is strictly less than the modulus +> n. In this case, ciphertexts can be decrypted easily by taking the eth root of +> the ciphertext over the integers. + +Which would mean that $\sqrt[3]{c}$ is a solution _if_ $m$ is small enough. +However + +$c = m^3 \mod n \\$ + +so the exponentiation could yield a much higher result that we dont get because +it is wrapped at $n$. So we can try out how often it wrapped: + +$m = \sqrt[3]{c + x*n} \quad x \in \N\\$ + +where $x$ can be brute-forced if $m$ is sufficiently small. Back in python it +looks like this: + +```py +c = ... +n = ... + +# simple but bad binary search for n-th root +def nth_root(c,e): + high = 1 + while high**e < c: + high *= 2 + low = 0 + mid = 0 + last_mid = 0 + while mid**e != c: + mid = (low + high) // 2 + if last_mid == mid: + return 0 # i use 0 to indicate a failure + if mid**e > c: + high = mid + else: + low = mid + last_mid = mid + return mid + +for x in range(100000): + m = nth_root(c + n*x, e) + # the probability of finding a perfect cube number is low, so any result is what we want + if m != 0: print(long_to_bytes(m)) +``` + +Thats it! With $x=1831$ we get our flag: `CTF{34sy_RS4_1s_e4sy_us3}` diff --git a/blog/2022-09-25-ductf-file-magic.md b/blog/2022-09-25-ductf-file-magic.md new file mode 100644 index 0000000..f7db40f --- /dev/null +++ b/blog/2022-09-25-ductf-file-magic.md @@ -0,0 +1,260 @@ +# DownUnderCTF 2022: File Magic + +A short writeup about my favorite challenge from DUCTF. It took me approximatly +12h to solve. I found it was the most creative and challenging one that i +solved. + +## Task + +The challenge consists of a python script and an ip-port pair which appears to +be running that script. Also the path of the flag is given: `./flag.txt` + +```py +#!/usr/bin/env python3 + +from Crypto.Cipher import AES +from PIL import Image +from tempfile import NamedTemporaryFile +from io import BytesIO +import subprocess, os + +KEY = b'downunderctf2022' + +iv = bytes.fromhex(input('iv (hex): ')) +assert len(iv) == 16 and b'DUCTF' in iv, 'Invalid IV' + +data = bytes.fromhex(input('file (hex): ')) +assert len(data) % 16 == 0, 'Misaligned file length' +assert len(data) < 1337, 'Oversized file length' + +data_buf = BytesIO(data) +img = Image.open(data_buf, formats=['jpeg']) +assert img.width == 13 and img.height == 37, 'Invalid image size' +assert img.getpixel((7, 7)) == (7, 7, 7), 'Invalid image contents' + +aes = AES.new(KEY, iv=iv, mode=AES.MODE_CBC) +dec = aes.decrypt(data) +assert dec.startswith(b'\x7fELF'), 'Not an ELF' + +f = NamedTemporaryFile(delete=False) +f.write(dec) +f.close() + +os.chmod(f.name, 0o777) +pipes = subprocess.Popen([f.name], stdout=subprocess.PIPE) +stdout, _ = pipes.communicate() +print(stdout.decode()) + +os.remove(f.name) +``` + +So, for anything to make it past these checks and be executed it must: + +1. be a valid 13x37 JPEG image with the pixel at 7,7 set to `#070707` +2. be a valid ELF binary that reads `./flag.txt` after decrypting with AES CBC, + fixed key (`downunderctf2022`) and the provided IV +3. The IV must contain `DUCTF` + +During the competition I discovered the information in the next three headings +in parallel but internally in-order. + +## 1. AES CBC "flaw" + +We need to generate a file that is a sort-of polyglot with JPEG and ELF, +converted with AES CBC (Cipher block chaining). + +AES itself operates on 16-byte (for 128-bit AES) blocks, so bigger files are +split and then encrypted separately. To ensure that identical blocks don't +result in identical blocks in ciphertext, each block is first xor'd with +something that won't be identical. In the case of CBC, the last ciphertext block +or the initialisation vector (IV) is used. Here is a diagram for encryption + +``` + ___plaintext____|___plaintext____|___plaintext____|... + v v v +IV--->XOR ,---------->XOR ,--------->XOR ,---- ... + v | v | v | + AES | AES | AES | + v---' v---' v---' + ___ciphertext___|___ciphertext___|___ciphertext___|... +``` + +For decryption we can just flip the diagram and replace AES with reverse AES. + +``` + ___ciphertext___|___ciphertext___|___ciphertext___|... + v---, v---, v---, + ∀EZ | ∀EZ | ∀EZ | + v | v | v | +IV--->XOR '---------->XOR '--------->XOR '---- ... + v v v + ___plaintext____|___plaintext____|___plaintext____|... +``` + +This does make sense, however i noticed that all but the first block do not +depend on IV. This turns out useful since we can turn any block into any other +block by applying a chosen value with XOR. So we can control the ciphertext with +the IV directly as follows: + +- $m$: first plaintext block +- $c$: first ciphertext block + +$$ + +c = AES(m \oplus IV) \\ + +AES^{-1}(c) = m \oplus IV \\ + +AES^{-1}(c) \oplus m = IV + +$$ + +All blocks in ciphertext after the first are now "uncontrollable" because IV and +plaintext are set. + +## 2. JPEG + +JPEG consists of a list of _segments_. Each starts with a marker byte (`ff`) +followed by a identifier and the length of the segment (if non-zero). + +| Identifier | Name | +| ---------- | ----------------------------------------------- | +| `d8` | Start of Image | +| `fe` | Comment | +| `d9` | End of Image | +| ... | _a bunch more that you dont need to know about_ | + +The comment segment is perfect for embedding our ELF binary into JPEG. We can +first generate a JPEG image, then insert a _comment_ somewhere containing any +data we want. + +## 3. ELF Payload + +The binary needs to be super small so creating it "manually" was required. I +followed the guide +[Creating a minimal ELF-64 file by tchajed](https://github.com/tchajed/minimal-elf/) +and recreated it for my needs. Like in the guide i also wrote the assembly with +a rust EDSL. + +```rs +let mut str_end = a.create_label(); +let mut filename = a.create_label(); +a.jmp(str_end)?; // jump over the string +a.set_label(&mut filename)?; +a.db(b"flag.txt\0")?; +a.set_label(&mut str_end)?; + +// open("flag.txt", O_RDONLY) +a.mov(eax, 2)?; +a.lea(rdi, ptr(filename))?; +a.mov(rsi, 0u64)?; +a.syscall()?; // fd -> rax + +// sendfile(1, rax, 0, 0xff) +a.mov(rsi, rax)?; +a.mov(eax, 40)?; +a.mov(rdi, 1u64)?; +a.mov(rdx, 0u64)?; +a.mov(r10, 0xffu64)?; +a.syscall()?; +``` + +I was able to produce a 207 byte long executable. + +## 4. _magic!_ + +Here is how we align the files now (single quotes `'` indicate ASCII +representation for clarity, question marks `?` represent data that is implicitly +defined): + +``` +plaintext: 7f 'E 'L 'F 02 01 01 00 00 00 00 00 00 00 00 00 +iv: ?? ?? ?? ?? ?? ?? 'D 'U 'C 'T 'F 00 00 00 00 00 +ciphertext: ff d8 ff fe {len} ?? ?? ?? ?? ?? ?? ?? ?? ?? ?? +``` + +This scenario is a little more complicated because in some places we define +ciphertext and plaintext and in some we define ciphertext and IV. This is not a +problem because XOR operates on every byte individually. + +All-in-all, the file looks like this now: + +- First block: + - overlapping headers (6 bytes) + - `DUCTF` string (5 bytes) + - padding (5 bytes) +- Rest of the ELF binary +- _end of the jpeg comment_ +- Rest of the JPEG image + +All this information should be enough to solve this challenge! + +I here attach the implementation that i created _during_ the CTF here. I kept it +as messy as it was, just removed not-so-interesting code and added comments. + +```rs +let mut i = image::RgbImage::new(13, 37); +// jpeg is lossy so filling guarantees the pixel actually has the *exact* color +i.pixels_mut().for_each(|p| *p = Rgb([7, 7, 7])); + +let mut imgbuf = Cursor::new(vec![]); +image::codecs::jpeg::JpegEncoder::new(&mut imgbuf) + .encode_image(&i) + .unwrap(); + +let key = Aes128::new(&GenericArray::from(*b"downunderctf2022")); +let mut payload = create_elf_payload(); +let mut buf = BytesMut::new(); + +// pad payload so AES works +while payload.len() % 16 != 0 { + payload.put_u8(0); +} +assert!(payload.len() % 16 == 0); + +let prefix_len = 6; +// write the JPEG headers to start a comment +buf.put_u16(0xffd8); +buf.put_u16(0xfffe); +buf.put_u16(payload.len() as u16 + 2 /*seg len*/ - prefix_len as u16); + +// find a good IV +let iv = { + let mut fblock_target = vec![0u8; 16]; + fblock_target[0..prefix_len].copy_from_slice(&buf[0..prefix_len]); + key.decrypt_block(GenericArray::from_mut_slice(&mut fblock_target)); + let mut iv = xor_slice(&fblock_target, &payload[0..16]); + iv[prefix_len..prefix_len + 5].copy_from_slice(b"DUCTF"); + iv +}; + +// fill the first block up with zeroes +for _ in prefix_len..16 { + buf.put_u8(0); +} + +// encrypt starting with the 2nd block +buf.put(&payload[16..]); +let block_range = |n: usize| (n) * 16..(n + 1) * 16; +for i in 1..buf.len() / 16 { + let tb = Vec::from(&buf[block_range(i - 1)]); + xor_assign(&mut buf[block_range(i)], &tb); + key.encrypt_block(GenericArray::from_mut_slice(&mut buf[block_range(1)])); +} + +// append the rest of the image, stripping the first segment +buf.put(&imgbuf.into_inner()[2..]); + +// pad the final buffer again because the image might not be aligned +while buf.len() % 16 != 0 { + buf.put_u8(0); +} +assert!(buf.len() < 1337); + +File::create("image").unwrap().write_all(&buf).unwrap(); +File::create("iv").unwrap().write_all(&iv).unwrap(); +``` + +I am also still looking for team mates for upcoming CTF events and would be +happy to hack together with you! Just [contact](https://metamuffin.org/contact) +me. diff --git a/blog/2022-11-07-programming-language-design.md b/blog/2022-11-07-programming-language-design.md new file mode 100644 index 0000000..674e696 --- /dev/null +++ b/blog/2022-11-07-programming-language-design.md @@ -0,0 +1,70 @@ +# Some Thoughts on Programming Language Design + +This is a collection of ideas to look at when inventing new langauges. + +## Other Ideas + +- The Language pushes abstraction to the limit by not noting any + hardware-related issues like memory-allocations, parallelism, heterogenous + computer architecture (CPU, GPU, …) + - requires a very "intellegent" compiler and a way to describe unknowns like + possible inputs, etc. in the language itself +- Start with assembly but add a very flexible macro system + +## Type System + +```diff +# Haskell +data LinkedList a = Nil | Cons a (Box (LinkedList a)) +data Test = Empty | Blub Int | State { x :: Int, y :: Int } +# Rust +enum LinkedList<T> { Nil, Cons(T, LinkedList<T>) } +``` + +## Memory Management + +- **Drop when out-of-scope** +- Garbage collections +- Reference counting + +## Compile-time logic + +- Annotation when calling function to be run as-far-as-possible at comptime + +```diff +fn format(template: String, args: [String]) -> String { + template.replace("@", (match, i) => args[i]) +} + +fun add(x, y) x + y + +fun main() print(format!("@ ist @; @", ["1+1", 1+1, x])) +# should expand to +fun main() print("1+1 ist 2; " ~ x)) +``` + +## Examples + +### Fizz-Buzz + +```diff +for (n in 0..100) { + if (n % (3*5) == 0) print("FizzBuzz") + else if (n % 3 == 0) print("Fizz") + else if (n % 5 == 0) print("Buzz") + else print(n) +} + + +if (true) x = 1 +if (true) { x = 1 } +``` + +``` +f(x) = 10 + g(x) +f x = 10 + g x + +main = { + +} +``` diff --git a/blog/2022-11-10-artist-correlation.md b/blog/2022-11-10-artist-correlation.md new file mode 100644 index 0000000..d52e613 --- /dev/null +++ b/blog/2022-11-10-artist-correlation.md @@ -0,0 +1,78 @@ +# Correlating music artists + +I listen to a lot of music and so every few months my music collection gets +boring again. So far I have asked friends to recommend me music but I am running +out of friend too now. Therefore I came up with a new solution during a few +days. + +I want to find new music that i might like too. After some research I found that +there is [Musicbrainz](https://musicbrainz.org/) (a database of all artists and +recordings ever made) and [Listenbrainz](https://listenbrainz.org/) (a service +to which you can submit what you are listening to). Both databases are useful +for this project. The high-level goal is to know, what people that have a lot of +music in common with me, like to listen to. For that the shared number of +listeners for each artist is relevant. I use the word 'a listen', to refer to +one playthrough of a track. + +## The Procedure + +### Parse data & drop unnecessary detail + +All of the JSON files of listenbrainz are parsed and only information about how +many listens each user has submitted for what artist are kept. The result is +stored in a B-tree map on my disk (the +[sled library](https://crates.io/crates/sledg) is great for that). + +- First mapping created: `(user, artist) -> shared listens`. +- (Also created a name lookup: `artist -> artist name`) + +The B-Tree stores values ordered, such that i can iterate through all artists of +a user, by scanning the prefix `(user, …`. + +### Create a graph + +Next an undirected graph with weighted edges is generated where nodes are +artists and edges are shared listens. For each user, each edge connecting +artists they listen to, the weight is incremented by the sum of the logarhythms +of either one's playthrough count for that user. This means that artists that +share listeners are connected and because of the logarhythms, users that listen +to an artist _a lot_ won't be weighted proportionally. + +Mapping: `(artist, artist) -> weight`. (Every key `(x, y)` is identical with +`(y, x)` so that edges are undirectional.) + +### Query artists + +The graph tree can now be queried by scanning with a prefix of one artist +(`("The Beatles", …`) and all correlated artists are returned with a weight. The +top-weighted results are kept and saved. + +### Notes + +Two issues appeared during this project that lead to the following fixes: + +- Limit one identity to 32 artists at most because the edge count grows + quadratically (100 artists -> 10000 edges) +- When parsing data the user id is made dependent of the time to seperate arists + when music tastes changing over time. Every 10Ms (~4 months) the user ids + change. + +## Results + +In a couple of minutes I rendered about 2.2 million HTML documents with my +results. They are available at `https://metamuffin.org/artist-correl/{name}`. +Some example links: + +- [The Beatles](https://metamuffin.org/artist-correl/The%20Beatles) +- [Aimer](https://metamuffin.org/artist-correl/Aimer) +- [Rammstein](https://metamuffin.org/artist-correl/Rammstein) +- [Mitski](https://metamuffin.org/artist-correl/Mitski) + +## Numbers + +- Musicbrainz: 15GB +- Listenbrainz: 350GB +- Extracted listening data: 23GB +- Graph: 56GB +- Rendered HTML: 2.3GB +- Compressed HTML (squashfs with zstd): 172MB diff --git a/blog/2023-02-13-new-website-using-gnix.md b/blog/2023-02-13-new-website-using-gnix.md new file mode 100644 index 0000000..04cdc02 --- /dev/null +++ b/blog/2023-02-13-new-website-using-gnix.md @@ -0,0 +1,103 @@ +# New Website and gnix + +_fixme: probably contains a lot of errors, shouldn't have written this late_ + +Last weekend I started a new attempt writing a reverse proxy: This time, with +success! I have been able to finally replace nginx for all services. +Additionally I now have a wildcard TLS certificate for all of +`*.metamuffin.org`. + +The cgit instance is no longer available since it used CGI, which gnix does not +support nor I like. + +## The reverse-proxy + +Nginx was not optimal because I found it was hard to configure, required certbot +automatically chaning the config and was also just _too much_ for my use case. +(Who needs a http server that can also serve SMTP?!) + +My new solution ([gnix](https://codeberg.org/metamuffin/gnix)) has very limited +configuration abilities for now but just enough to work. I simplified about 540 +lines of `/etc/nginx/nginx.conf` to only 20 lines of `/etc/gnix.toml` (yesss. +TOML. of course it is.). The Proxy now only acts as a "Hostname Demultiplexer". +A configuration could look like this: + +```toml +[http] +bind = "0.0.0.0:80" + +[https] +bind = "0.0.0.0:443" +tls_cert = "/path/to/cert.pem" +tls_key = "/path/to/key.pem" + +[hosts] +"domain.tld" = { backend = "127.0.0.1:18000" } +"www.domain.tld" = { backend = "127.0.0.1:18000" } +"keksmeet.domain.tld" = { backend = "127.0.0.1:18001" } +"otherdomain.tld" = { backend = "example.org:80" } +``` + +I am running two gnix instances now, one for `:80`+`:443` and another for matrix +federation on `:8448`. Additionally this required me to move my matrix +homeserver from `https://metamuffin.org/_matrix` to +`https://matrix.metamuffin.org/_matrix` via the `.well-known/matrix/server` +file. And that intern required me to host a file there, which was nginx' job +previously. At this point I started rewriting my main website. + +## Wildcard Certificates + +Another inconvinience was that I would need `certbot` to aquire one certificate +for each subdomain. Letsencrypt offers wildcard certificates; These can be +obtained by solving an ACME challenge that requires changing a DNS record (to +prove you own the domain). My current registrar (Namecheap) does not offer me an +API for automatically applying these though. They do however (through a very +very confusing, badly designed user interface) allow me to set a custom +nameserver. By setting the nameserver to `144.91.114.82` (IP address of my VPS) +the server can run its own nameserver that has authority over resolving +`metamuffin.org`. I used BIND9's `named` to do that and also dynamically update +records. + +```conf +# /etc/named.conf (-rw-------; owned by named) +zone "metamuffin.org" IN { + type master; + # the zone file is trivial to configure, look it up somewhere else. :) + file "metamuffin.org.zone"; + update-policy { + # only allow certbot to change TXT records of _acme-challenge.metamuffin.org + grant certbot. name _acme-challenge.metamuffin.org. TXT; + }; +}; + +# generated with `tsig-keygen -a HMAC-SHA512 -n HOST certbot` +key "certbot" { + algorithm hmac-sha512; + secret "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX" +} +``` + +Then certbot can be configured to use these credentials for solving challenges: + +```ini +# /etc/certbot/rfc2136.ini (-rw-------; owned by root) +dns_rfc2136_server = 127.0.0.1 +dns_rfc2136_port = 53 +dns_rfc2136_name = certbot +dns_rfc2136_secret = XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX +dns_rfc2136_algorithm = HMAC-SHA512 +``` + +Now you can automatically request new wildcard certificates by running +`doas certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/rfc2136.ini -d '*.domain.tld' -d 'domain.tld' --server https://acme-v02.api.letsencrypt.org/directory` + +## Rewrite of my website + +As mentioned above, I replace my former Deno + pug.js + static file server setup +with a custom rust application (using Rocket and Markup and 253 other +dependencies). I rewrote my blog rendering system too, that why you don't see +syntax highlighting right now. + +## End + +In case of questions, ask me. Have fun suffering with the modern web! diff --git a/blog/2024-05-03-replaced-infra.md b/blog/2024-05-03-replaced-infra.md new file mode 100644 index 0000000..1177d7b --- /dev/null +++ b/blog/2024-05-03-replaced-infra.md @@ -0,0 +1,107 @@ +# infra - my homebrew package and configuration manager + +The last 8 months I was configuring my server with my own little package manager +[`infra`](https://codeberg.org/metamuffin/infra) before just recently replacing +it again a few days ago for reasons I will mention later. This had multiple +reasons; primarily that I want to version control all configuration of all my +servers (the VPS and a Raspi) from one source-controlled repository. + +## infra good + +My "package manager" was actually more like a library where one would write code +to specify how the machines were set up. This is similar to how Nix or Guix do +it. My implementation however was written in Rust and initially hacked together +in a few hours. Let's look at an example: + +```rs +// You would pass credentials to the machine here +let m = Machine::new(...); + +// Load a minimal gnix config without any hosts. +let mut gnix = Gnix::load(include_str!("gnix_base.yaml")) + +m.install(Fastbangs { + // Here gnix automatically allocates a new port for internal proxying. + bind_addr: gnix.new_host("search.metamuffin.org"), + data_dir: "/var/local/fastbangs".into(), + ..Default::default() +}.service()); + +m.eu1.install(MetamuffinWebsite { + bind_addr: eu1_gnix.new_host("metamuffin.org"), +}.service()); // `service` transforms the mere software configuration to a systemd service that uses it. + +m.install(gnix); +``` + +Infra would for this example automatically download, compile, install, configure +and start/enable required services. It was a total solution. The current state +of each machine (i.e. installed packages) was tracked locally. Whenever the +deployment specification changes, a diff is calculated and applied +incrementally. Applying an installation is very general term in infra, it could +mean placing some configs somewhere but also starting and stopping systemd +services or whatever. + +Procedurally generating the server configuration had more advantages than not +struggling with port numbers, I could also create fully meshed Wireguard VPNs +with ease. Something that is considerably more pain if done manually. The +following excerpt creates such a network, automatically assigns IP addresses and +even writes hostnames to `/etc/hosts`. + +```rs +let mut hosts = HostsFile::default(); +let mut wg_myservers_subnet = IpSubnet([10, 123, 0, 0].into(), 16); +let wg_myservers = WireguardMesh::new("myservers", 16); +server1.install(wg_myservers.add_peer( + hosts.register(wg_myservers_subnet.next(), "server1"), + "server1.example.org:12345".into(), + "SECRET VALUE GOES HERE" +)); +server2.install(wg_myservers.add_peer( + hosts.register(wg_myservers_subnet.next(), "server2"), + "server2.example.org:12345".into(), + "SECRET VALUE GOES HERE" +)); +server3.install(wg_myservers.add_peer( + hosts.register(wg_myservers_subnet.next(), "server2"), + "server3.example.org:12345".into(), + "SECRET VALUE GOES HERE" +)) +for s in [server1, server2, server3] { + s.install(hosts); +} +``` + +## infra bad + +Like mentioned the state was tracked locally, which meant I was restricted to a +single machine for maintaining my server. + +Also, and this is the worst part, infra would manage the server by connecting +via SSH and executing random commands through a simple shell wrapper. This +process will not take any feedback about the success about what it did. +Obviously all commands are executed intelligently but if the connection broke or +something else destabilized the system, there was no way to fix it trivially. + +Another problem was transparency. Although infra allows exporting an overview of +your deployment as a directed graph, it still didn't suffice for a good +understanding about what was configured where. Sometimes I would install two +packages that wrote to the same configuration - something that infra does not +worry about - and get unexpected results. + +It could also be considered a minor problem, that some configuration files just +don't prettier when written with a Rust DSL. + +## infra useless? + +`infra` is flawed. It just doesn't work if you have anything serious to +maintain. However the basic concept of generating configuration from code is +quite nice and were somewhat elegant to use sometimes. + +In the last week I replaced infra in my deployments with a different system that +I will describe in the next article. Going forward though I am thinking of +taking the best from infra and turning it into a static configuration and script +generator to be used in the new system. + +That's that. If you have any interesting feedback or thoughts on this topic, +please [reach out](/contact). diff --git a/blog/2024-11-28-optimal-computer-part-1.md b/blog/2024-11-28-optimal-computer-part-1.md new file mode 100644 index 0000000..27f1ce1 --- /dev/null +++ b/blog/2024-11-28-optimal-computer-part-1.md @@ -0,0 +1,137 @@ +# My dream computer (Part 1: Processor architecture) + +## Preamble + +For multiple years now I have had a vision for how a perfect operating system +should be designed. The specifics transformed slightly over the years but have +generally stood the same. However I have never actually gotten to implement this +design that envisioned, because of some or another problem that was hard to +solve at the time. I have still learned many things with my failed attempts and +considered more radical ideas too. In my continuing efforts in implementing my +ideas some day I want to write my most important thoughts on this topic in the +next few days because I realized that a problem of such great complexity is hard +to solve just within my head. These notes should be helpful for me but may also +interest others and so are published here. + +During my time of failing to implement the thing I found myself questioning +lower-level things a lot and so reevaluated these too and came to the conclusion +that these should be redesigned aswell. For my overview I will therefore start +from the bottom. + +In the rest of this article I am also just writing what I think is true or close +to the truth even for topics that I know less about. So view the text below as +pure speculation. I expect some things to be wrong. In that case, please tell +me. + +## Content + +My goal is it to design a _Computer system_ (You should know what that means to +you.). The heart of the physical part of this machine is a _Processor_, the +remaining parts store data or interface with the outside world or user and are +of less interest to me. Software is also important and I will write more about +this in the next posts. + +For performing the actual computations many design principles have been found in +the past: You could use a mechanical contraption, DNA/Protein's biochemical +reactions or anything really. Electric circuits however are most useful and +widely deployed today and since I do not have the expertise, I am not +reconsidering at this level. + +With electronic semiconductor logic gates you can arrange circuits in form of a +silicon die that perform most arithmetic and logic operations. The arrangement +of those gates on such a die is immutable however, meaning it is not possible to +be reprogrammed in that way. It is therefore required to build a circuit that is +general-purpose and can be programmed with data ("software"). + +This problem has been approached from multiple sides before. An FPGA is where +relatively small clusters of gates can be arranged to circuits by switching +connecting gates (that's a simplification). [Write how "traditional" CPUs work] +In the end there are many ways to achieve the required semi-turing-completeness +required for a computer. The performance of these does differ though. From +observing optimization in current CPUs it seems that the **useful computation +per space per time** matters most. The minimum time interval for computations +(cpu clock) does primarily depend on the size of the die (which is constraint by +the manufacturing technology which I consider fixed) and distance of components +on the die. A major part in optimizing _computational density_ of a processor is +keeping all parts as busy with computation as possible. + +I see a tradeoff between having small reconfigurable parts (like FPGAs) where +the overhead of reconfiguration is large (as in: many "useless" gates for +programming) and big parts where you lack the flexibility optimize the software. +In an unoptimized basic CPU for example, only one part would active at a time. +(The ADD instruction only performs an addition while other destinct components +like the multiplier, logic unit or memory are idle). Modern CPU design avoid +this to some extent with pipelining, speculative execution and simultaneous +multithreading which however introduce new complexity and also overhead for this +coordination. Also the program execution timing varies at runtime which +introduces additional problems. The concept of _Very Long Instruction Word_ +architectures appeals to me. It describes architectures where an instruction +specifies the configuration of most (usually all) components of the processor at +once. + +[Skipping a few steps because its already 4 am...] Next I am describing my +design: Firstly the processor is made up of a number of _modules_ that have +_inputs_, _outputs_ and _parameters_. These modules' inputs are then connected +with data multiplexers to allow abitrary cross-module data flow. Parameters of +modules are bits that configure the modules operating within one clock cycle of +the processor. These parameters also include an output index for each input of +the module to obtain the data for this cycle from. The processor operates in +clock cycles where each cycle a new instruction is loaded and executed. This +module system is very flexible; Here are a few examples of useful modules for a +processor: + +- Add/Sub unit: Two inputs and one output. A flag to switch subtract mode. +- Literal loader: One output set to a value set by 32 or so parameter bits. + Essentially providing a way to introduce new constants from program code +- Memory loader: One input (address) and an output that takes on the value of + the RAM at the provided address. +- Flow controller: Two inputs, a condition and a program counter value, that set + the PC when a condition is met, otherwise increments the PC as normal. +- Memory storer: (left as an exercise to the reader) +- Logic unit +- Mul/Div unit +- Floating-point unit + +To balance the load between modules, the most busy units can be contained +multiple times within the processor, allowing for example to perform two +addition per cycle. + +Additionally, since typical RAM is rather slow, it is also possible to introduce +a _latency_ to some modules like that memory loader that need main memory where +there is internal uninterupted pipelining for memory fetches such that, for +example memory fetches arrive with 4 cycles delay and jumps happen 4 cycles +after beeing issued. This always assumes the worst case memory latency and can +not utilise a cache. You could however reimplement a cache manually be adding +secondary "fast" memory that is local but smaller where you load data into with +software. + +Still current superscalar CPUs can be smarter about module load balancing and +memory caching because of their runtime knowledge, that a compiler does not +have. In general, writing a _good_ compiler for this architecture (assuming a +traditional language) is relatively hard. + +I am already visualizing code for such a processor in my head in a table with +time and module axis. For some later post or development I might generate such +diagrams from code to think about about. (will probably look super cool) + +## End + +Thats it for today. The writing of this article has been motivated by me +"participating" in +[Macoy Madson's Simple Useful System Challenge](https://macoy.me/blog/programming/SimpleUsefulSystem). +As usual I am always happy to receive feedback about what I am writing here. +[Send your thoughts!](https://metamuffin.org/contact) (any method is fine) + +Next article will be more about my ideas about software. The overview is: + +- Compiler/Language design + - Memory safety + - Type systems + - Representing hardware execution/performance characteristics + - Memory management + - Data flow oriented programming +- Software design + - Permission checks at compile-time + - Cooperative multithreading asserted by compiler + - No driver-program boundary + - Global Safety |