From 5382b10eb9fe41ead8d9a7c2548debb50d08b3e8 Mon Sep 17 00:00:00 2001 From: CodeX Date: Tue, 7 Apr 2026 02:12:52 +0200 Subject: [PATCH] Correct consumer model: Cleanuparr is the only direct consumer --- Consumers.md | 213 +++++++++++++++++++++++++++------------------------ Lists.md | 30 ++++---- 2 files changed, 132 insertions(+), 111 deletions(-) diff --git a/Consumers.md b/Consumers.md index 60540a6..1c8bc79 100644 --- a/Consumers.md +++ b/Consumers.md @@ -1,19 +1,25 @@ # Consumers -The blocklists are consumed by two tools in the ARR stack: +The blocklists are consumed by [Cleanuparr](https://github.com/Cleanuparr/Cleanuparr). +qBittorrent itself does not read any URL from this repository -- the only +direct consumer is Cleanuparr, which exposes two independent features that +each pull a different file: -| Tool | Role | File consumed | Mode | -|---|---|---|---| -| qBittorrent | Download client | `blacklist` | Excluded file names | -| Cleanuparr | Media cleanup / malware blocker | `blacklist` or `whitelist` | Blacklist or whitelist mode | +| Cleanuparr feature | File consumed | What it does | +|---|---|---| +| Blocklist Sync | `blacklist` | Pushes the full pattern set to qBittorrent's "Excluded file names" via the qBittorrent Web API | +| Malware Blocker | `whitelist` (whitelist mode) or `blacklist` (blacklist mode) | Inspects torrents in the Sonarr/Radarr queue and removes ones that violate the rules | -Both tools read a remote text file over HTTPS, one glob pattern per line. -They refresh on their own schedule (qBittorrent on restart or manual -refresh; Cleanuparr on its configured interval). +The two features are configured independently in Cleanuparr's settings, +each with its own URL and refresh schedule. Use both, either, or neither. +This repository is set up so that **both** can be used together: Blocklist +Sync against `blacklist`, Malware Blocker against `whitelist` in whitelist +mode. ## Raw URLs -Point consumers at the raw file URLs, not the Gitea blob viewer URLs: +Point Cleanuparr's features at the raw file URLs, not the Gitea blob viewer +URLs: ``` https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist @@ -24,129 +30,128 @@ The `raw/branch/main/` path serves the file contents directly with the correct `text/plain` content type. Using `src/branch/main/` instead serves the HTML viewer page and will break the consumer. -## qBittorrent +## Blocklist Sync (push to qBittorrent) -qBittorrent has an **excluded file names** feature that skips files -matching any of the configured glob patterns when downloading a torrent. -There is no "included file names" or whitelist mode -- qBittorrent only -supports exclusion. This is why it consumes the merged `blacklist` and not -the `whitelist`. +Cleanuparr's **Blocklist Sync** fetches a blocklist URL on its configured +interval, then pushes the full pattern list into qBittorrent's "Excluded +file names" setting via the qBittorrent Web API. The user does not +configure qBittorrent's exclusion list directly -- everything happens +through Cleanuparr. + +### What it supports + +- **Blacklists only.** There is no whitelist mode for Blocklist Sync. +- **Full replacement** of qBittorrent's exclusion list on every push. Any + patterns set manually in qBittorrent will be overwritten. +- **All connected qBittorrent instances** receive the same list (one push + per configured download client in Cleanuparr). ### Configuration -1. Open **Options** (Tools -> Options, or Ctrl+,). -2. Go to **Downloads**. -3. Scroll to **Excluded file names**. -4. Enable the checkbox. -5. Set the URL to: +In Cleanuparr's settings, enable Blocklist Sync and point it at: ``` https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist ``` -qBittorrent fetches the list on startup and whenever you click **Reload** -next to the field. There is no automatic refresh interval -- a restart or -manual reload is required to pick up changes. +Cleanuparr fetches on its configured interval, then writes the patterns +to qBittorrent. Check Cleanuparr's logs to confirm the push succeeded +after each sync. -### What qBittorrent does with the list +### How qBittorrent uses the pushed list -When a torrent is added, qBittorrent iterates the files inside it and -checks each filename against the excluded patterns. Matching files are -marked as "do not download" and will not be written to disk. The rest of -the torrent downloads normally. +Once Cleanuparr has pushed the patterns, qBittorrent treats them exactly +the same as if they had been set manually. When a torrent is added, +qBittorrent iterates the files inside it and checks each filename against +the excluded patterns. Matching files are marked as "do not download" and +will not be written to disk. The rest of the torrent downloads normally. This means the list operates at the **file level within a torrent**, not the torrent level. A torrent containing `movie.mkv` and `movie.nor.srt` -would download both files if `*.srt` is in the whitelist (and thus not in -the blacklist), or just `movie.mkv` if `*.srt` were in the blacklist. +downloads both files because `*.srt` is in the whitelist (and therefore +not in the pushed pattern list). -### Refreshing after a whitelist change +### Why this URL -qBittorrent does not auto-refresh the list. After updating `whitelist`: +The `blacklist` file in this repository is the upstream Cleanuparr blacklist +with the locally-maintained `whitelist` already subtracted. Anything in +`whitelist` is therefore *not* pushed to qBittorrent, so subtitles, AV1 +webm releases, and other extensions you want kept will not be excluded +when qBittorrent downloads a torrent. -1. Wait for the next sync run (or dispatch the workflow manually). -2. In qBittorrent, open the excluded file names setting and click - **Reload**, or restart qBittorrent. -3. New torrents added from this point on will use the updated list. - Torrents already in the client are not retroactively changed. +## Malware Blocker (Sonarr/Radarr queue) -## Cleanuparr +Cleanuparr's **Malware Blocker** inspects torrents in the Sonarr and Radarr +download queues, examines the files they contain, and removes any torrent +that violates the configured rules. It runs independently from Blocklist +Sync and uses its own URL and refresh schedule. -Cleanuparr supports two modes for its Malware Blocker and Blacklist Sync -features. The repository provides files suitable for both. +### What it supports -### Blacklist mode +- **Whitelist mode** -- only torrents whose contained files all match the + whitelist patterns are kept. Everything else is removed from the queue. +- **Blacklist mode** -- torrents containing any file matching the blacklist + patterns are removed. +- **Sonarr and Radarr queues** -- both supported, configured per-arr in + Cleanuparr's settings. -In blacklist mode, Cleanuparr deletes any file matching a pattern in the -configured list. +### Recommended mode and URL -Point it at the same URL as qBittorrent: +This repository is configured for **whitelist mode**: a small explicit list +of allowed extensions is easier to reason about than the much larger +upstream blacklist, especially when the goal is "only let through legitimate +video and subtitle files." -``` -https://git.hisp.no/arr/blocklists/raw/branch/main/blacklist -``` - -Because the whitelist has already been subtracted, this file will not -cause Cleanuparr to delete anything you have marked as "keep" in the -whitelist. Consistent behaviour between the two tools without any -per-tool customisation. - -### Whitelist mode - -In whitelist mode, Cleanuparr keeps only files matching a pattern in the -configured list and deletes everything else. - -Point it at: +Point Malware Blocker at: ``` https://git.hisp.no/arr/blocklists/raw/branch/main/whitelist ``` -This is the more conservative choice: only the extensions explicitly -listed (video containers and subtitles) are allowed. Anything else -- -including extensions that upstream has not yet flagged as malicious -- -is deleted. +In whitelist mode, only torrents whose contained files all have whitelisted +extensions are allowed to remain in the queue. A torrent with `movie.mkv` +and `movie.nor.srt` passes (both extensions are whitelisted). A torrent +with `movie.mkv` and `setup.exe` is removed from the queue before it +reaches qBittorrent. -### Which mode to use +If you would rather use blacklist mode, point Malware Blocker at the +`blacklist` URL instead -- the same one Blocklist Sync uses. Both modes +use the same data files; only the matching logic differs. -| Use case | Mode | Why | -|---|---|---| -| You trust upstream Cleanuparr's coverage and want to keep everything except known-bad | Blacklist | Lets through unusual-but-legitimate file types (e.g. exotic subtitle formats) | -| You only want a strict set of video + subtitle files on disk | Whitelist | Much stricter; deletes anything not explicitly listed | -| You want behaviour consistent with qBittorrent | Blacklist | Same source file, same semantics | +### Why two layers are useful -Blacklist mode is the recommended default because it matches the -qBittorrent side and avoids unexpected deletions of legitimate but -non-listed files. +Blocklist Sync and Malware Blocker operate at different layers: -## Keeping both consumers in sync +- **Malware Blocker (queue layer)** stops bad torrents from being added to + qBittorrent in the first place. A torrent that fails the whitelist check + never reaches the download client. +- **Blocklist Sync (file layer)** keeps qBittorrent from writing individual + junk files (samples, NFOs, mixed-in executables) to disk while still + allowing the legitimate files in a release through. -Both consumers ultimately read the whitelist (directly in Cleanuparr -whitelist mode, indirectly via subtraction in blacklist mode and in -qBittorrent). This means maintenance is centralised: - -1. Add a line to `whitelist`. -2. Wait for the next sync run (or dispatch manually). -3. Both consumers honour the change after their next refresh. - -There is no per-tool configuration drift because there is no per-tool -configuration to drift. +Used together, the two layers give you a queue-level allow list and a +file-level deny list, both maintained from the same `whitelist` source of +truth. ## Troubleshooting -### A file I whitelisted is still being blocked / deleted +### A file I whitelisted is still being blocked Check each layer in order: -1. **Sync ran successfully?** Open the Gitea Actions page for the - repository and verify the most recent run is green and newer than - your whitelist commit. +1. **Sync ran successfully?** Open the Gitea Actions page for the repository + and verify the most recent run is green and newer than your whitelist + commit. 2. **Blacklist was updated?** Read `blacklist` in Gitea and confirm your - whitelisted entry is not present. -3. **Consumer refreshed?** qBittorrent requires a manual reload or - restart. Cleanuparr refreshes on its own interval -- check its logs - to confirm it picked up the new file. -4. **Exact string match?** Whitelist entries must match blacklist entries + whitelisted entry is no longer present. +3. **Cleanuparr refreshed?** Blocklist Sync runs on its configured interval, + not on demand. Check Cleanuparr's logs to confirm it picked up the new + file. Trigger a manual run from Cleanuparr's UI if available. +4. **Cleanuparr pushed to qBittorrent?** After fetching the new file, + Cleanuparr writes it to qBittorrent. Check qBittorrent's + Options -> Downloads -> Excluded file names and confirm the new list + is in place. +5. **Exact string match?** Whitelist entries must match blacklist entries exactly. `*.srt` in whitelist does not strip `*sample.srt` from blacklist. See [Lists](Lists) for pattern semantics. @@ -155,11 +160,11 @@ Check each layer in order: Check whether the pattern is in the blacklist at all: 1. Open `blacklist` in Gitea and search for the extension. -2. If it is not there, upstream does not block it either. You can add +2. If it is not there, upstream Cleanuparr does not block it. You can add it to `blacklist` directly (manual local addition, preserved by the three-way merge) or file an upstream issue. -### Consumer returns 404 +### Cleanuparr returns 404 for the blocklist URL Verify the URL uses `raw/branch/main/`, not `src/branch/main/`: @@ -174,8 +179,20 @@ https://git.hisp.no/arr/blocklists/src/branch/main/blacklist Also check the repository name and branch are correct (`arr/blocklists`, `main`). -### Cleanuparr deletes subtitle files +### Malware Blocker is removing legitimate torrents -Cleanuparr is running in whitelist mode against `blacklist`, which is -the wrong combination. Either switch it to blacklist mode (keep the URL), -or keep whitelist mode and point it at `whitelist` instead. +Make sure Malware Blocker is in whitelist mode and pointed at `whitelist`, +not `blacklist`. In blacklist mode it uses the much larger upstream +blacklist and may remove a torrent that contains a single junk file +inside an otherwise legitimate release. + +### Blocklist Sync did not push to qBittorrent + +Check Cleanuparr's connection to qBittorrent: + +1. Cleanuparr settings -> Download clients -- verify the qBittorrent host, + port, username, and password. +2. Cleanuparr logs -- look for Blocklist Sync errors after the most recent + scheduled run. Connection failures are logged at error level. +3. qBittorrent Web API enabled -- the push uses the Web API, which must + be enabled in qBittorrent's options under Web UI. diff --git a/Lists.md b/Lists.md index 76d8462..fdf8615 100644 --- a/Lists.md +++ b/Lists.md @@ -15,8 +15,9 @@ the three-way merge baseline used by the sync script. Never edit it. ## `blacklist` -The blacklist is the output file consumed by qBittorrent and (optionally) -Cleanuparr. It is regenerated on every sync as: +The blacklist is the output file consumed by Cleanuparr's Blocklist Sync +feature, which pushes the patterns to qBittorrent's excluded file names. +It is regenerated on every sync as: ``` upstream_new | custom_local_additions - whitelist @@ -137,8 +138,10 @@ You want qBittorrent to download them, not strip them. *.srt ``` -After the next sync, `*.srt` is gone from `blacklist`. qBittorrent now -accepts `.srt` files from within torrents. `*sample.srt` remains blocked. +After the next sync, `*.srt` is gone from `blacklist`. Cleanuparr's +Blocklist Sync pushes the new list to qBittorrent, which then accepts +`.srt` files from within torrents. `*sample.srt` remains blocked because +exact-string subtraction does not touch it. ### Supporting AV1 in `.webm` containers @@ -150,8 +153,9 @@ currently blocked because the upstream blacklist treats `*.webm` as junk. *.webm ``` -After the next sync, `*.webm` is gone from `blacklist`. `.webm` torrents -download normally. `*sample.webm` remains blocked. +After the next sync, `*.webm` is gone from `blacklist`. Cleanuparr's +Blocklist Sync pushes the updated list to qBittorrent and `.webm` +torrents download normally. `*sample.webm` remains blocked. ### Adding a site-specific junk extension @@ -194,13 +198,13 @@ check the file in Gitea for the current state. ## Consumer consequences -Changes to either file affect what consumers see: +Changes to either file affect what Cleanuparr's two features see: -| Change | Effect on qBittorrent | Effect on Cleanuparr (blacklist mode) | Effect on Cleanuparr (whitelist mode) | -|---|---|---|---| -| Add to `whitelist` | Stops blocking this extension | Stops deleting this extension | Starts allowing this extension | -| Remove from `whitelist` | Resumes blocking (if upstream has it) | Resumes deleting (if upstream has it) | Stops allowing this extension | -| Add to `blacklist` directly | Starts blocking this extension | Starts deleting this extension | No effect | -| Remove from `blacklist` directly | No effect (sync re-adds) | No effect (sync re-adds) | No effect | +| Change | Effect on Blocklist Sync (pushed to qBittorrent) | Effect on Malware Blocker (whitelist mode) | +|---|---|---| +| Add to `whitelist` | Stops being pushed to qBittorrent's excluded list | Starts allowing this extension in queue | +| Remove from `whitelist` | Resumes being pushed (if upstream has it) | Stops allowing this extension in queue | +| Add to `blacklist` directly | Starts being pushed to qBittorrent's excluded list | No effect (Malware Blocker reads `whitelist`) | +| Remove from `blacklist` directly | No effect (sync re-adds from upstream) | No effect | See [Consumers](Consumers) for configuration details.