blob: c62e7908cd21c6d6133561bf730739a89bbdff53 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
|
# Correlating music artists
A hear a lot of music and so every few months my music collection gets boring
again. So far I have asked friends to recommend me music but I am running out of
friend too now. Therefore I came up with a new solution during a few days.
I want to find new music that i might like too. After some research I found that
there is [Musicbrainz](https://musicbrainz.org/) (a database of all artists and
recordings ever made) and [Listenbrainz](https://listenbrainz.org/) (a service
to which you can submit what you are listening to). Both databases are useful
for this project. The high-level goal is to know, what people that have a lot of
listens in common, like to listen to. For that the shared number of listeners
for each artist is relevant.
## The Procedure
### Parse data & drop unnecessary detail
I parse all of the JSON files of listenbrainz and only keep information about
how many listens each user has submitted for what artist. The result is stored
in a B-tree map on my disk (the [sled library](https://crates.io/crates/sledg)
is great for that).
- First mapping created: `(user, artist) -> shared listens`.
- (Also created a name lookup: `artist -> artist name`)
The B-Tree stores values ordered, such that i can iterate through all artists of
a user, by scanning the prefix `(user, …`.
### Create a graph
Next an undirected graph with weighted edges is generated where nodes are
artists and edges are shared listens. For every user, each pair of artists they
listen to, receives the sum of listens to either one's listens. This means that
artists that share listeners are connected.
Mapping: `(artist, artist) -> weight`. (Every key `(x, y)` is identical with
`(y, x)` so that edges are undirectional.)
### Query artists
The graph tree can now be queried by scanning with a prefix of one artist
(`("The Beatles", …`) and all correlated artists are returned with a weight. The
top-weighted results are kept and saved.
## Results
In a couple of minutes I rendered about 2.2 million HTML documents with my
results. They are available at `https://metamuffin.org/artist-correl/{name}`.
Some example links:
- [The Beatles](https://metamuffin.org/artist-correl/The%20Beatles)
- [Aimer](https://metamuffin.org/artist-correl/Aimer)
- [Rammstein](https://metamuffin.org/artist-correl/Rammstein)
- [Mitski](https://metamuffin.org/artist-correl/Mitski)
## Numbers
- Musicbrainz: 15GB
- Listenbrainz: 350GB
- Extracted listening data: 11GB
- Graph: 24GB
- Rendered HTML: 8.4GB
- Compressed HTML (squashfs with zstd): 105MB
|
