isda

Generic task queue server with scripts for downloading with different tools.

Architecture

There is a central task queue server (isdad) and multiple workers that connect to that server to complete tasks.

Tasks have a key which is a string used to identify it and data which is a arbitrary JSON object. Keys are in the format kind:id; The kind is used by the queue server to determine which workers are able to process the task.

When workers connect to the queue server they register which tasks kinds they can process. After that they can accept tasks. Whenever a supported task exists, it will be assigned to that worker. The worker will then carry out the work and mark the task as completed. If workers disconnect (crash) while working, all assigned tasks will be moved back to the queue.

Installation

Install the queue server with cargo install --path .. The worker scripts dont need installation.

For ArchLinux there exist PKGBUILDs for both workers and the queue server; Both also available pre-built on pkg.metamuffin.org.

Usage

First start the queue daemon process. It will save its state to JSON files in its working directory. As its first argument supply the global configuration file.

isdad config.yaml

The configuration file contains two keys read be the daemon bind_port and bind_addr. All other options are passed as-is to the workers.

Then you can start workers linked to the queue server. Most workers take the URI to the worker websocket endpoint of the queue server as their first argument.

deno run -A scripts/<name>.ts ws://127.0.0.1:44794/worker_ws

Each workers usage and configuration is documented in the markdown file next to the script. These are also linked below.

Worker Scripts

License

AGPL-3.0-only; See COPYING.