blob: c7d10708475638d1e4a269952b29036895d26240 (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
|
# isda
Generic task queue server with scripts for downloading with different tools.
## Architecture
There is a central task queue server (isdad) and multiple workers that connect
to that server to complete tasks.
Tasks have a _key_ which is a string used to identify it and _data_ which is a
arbitrary JSON object. Keys are in the format `kind:id`; The kind is used by the
queue server to determine which workers are able to process the task.
When workers connect to the queue server they register which tasks kinds they
can process. After that they can accept tasks. Whenever a supported task exists,
it will be assigned to that worker. The worker will then carry out the work and
mark the task as completed. If workers disconnect (crash) while working, all
assigned tasks will be moved back to the queue.
## Usage
First start the queue daemon process. It will save its state to JSON files in
its working directory. As its first argument supply the global configuration
file.
```
cargo run -- config.yaml
```
The configuration file contains two keys read be the daemon `bind_port` and
`bind_addr`. All other options are passed as-is to the workers.
Then you can start workers linked to the queue server. Most workers take the URI
to the worker websocket endpoint of the queue server as their first argument.
```
deno run -A scripts/<name>.ts ws://127.0.0.1:44794/worker_ws
```
Each workers usage and configuration is documented in the markdown file next to
the script. These are also linked below.
## Worker Scripts
- [enqueuer](./scripts/enqueue.md)
- [yt-dlp media downloader](./scripts/ytdlp_download.md)
- [yt-dlp playlist flattener](./scripts/ytdlp_flatten.md)
- [yt-dlp channel info generator](./scripts/ytdlp_channel_info.md)
- [complete tasks based on existing files](./scripts/complete_from_files.md)
## License
AGPL-3.0-only; See [COPYING](./COPYING).
|
