Compare commits

...

64 Commits

Author SHA1 Message Date
Flaminel 2b766300ff fixed processing files marked as skipped 2025-03-26 23:51:29 +02:00
Flaminel 2d3ff04172 fixed Deluge failing when WebUI is not connected 2025-03-26 22:15:37 +02:00
Flaminel 874351aed7 added dry run for category creation 2025-03-26 21:50:02 +02:00
Flaminel 1a89822bad fixed some logs; updated test files; fixed deluge not working after fresh start 2025-03-26 20:49:23 +02:00
Flaminel ac086fcd47 fixed missing download location 2025-03-26 02:05:05 +02:00
Flaminel 4b38a6fee1 Merge branch 'main' into add_cleanup_on_no_hardlinks 2025-03-26 00:28:59 +02:00
Flaminel 9f770473e5 Remove Transmission downloads cache (#105) 2025-03-26 00:26:10 +02:00
Flaminel 60e838cba7 fixed Deluge processing for changing category 2025-03-25 22:15:04 +02:00
Flaminel 7b95ec579c fixed category change notification 2025-03-25 22:14:50 +02:00
Flaminel ab8fbc4b6e fixed Deluge get labels 2025-03-25 22:13:57 +02:00
Flaminel f2130ad734 fixed label being init only 2025-03-25 22:13:45 +02:00
Flaminel c86e9c97b8 removed commented and unused code 2025-03-25 16:21:31 +02:00
Flaminel 7639b0787e added set torrent label for Deluge 2025-03-25 16:13:46 +02:00
Flaminel 5e362d4af8 added Deluge label creation 2025-03-25 16:11:53 +02:00
Flaminel 3b63d1b7e5 fixed some logs and comments 2025-03-25 10:55:37 +02:00
Flaminel 4a1e0f6896 renamed vars; added root dir; fixed root dir file counts; fixed qbit flow 2025-03-24 18:05:50 +02:00
Flaminel a83809eef7 Merge branch 'main' into add_cleanup_on_no_hardlinks 2025-03-24 16:34:11 +02:00
Flaminel d993cd30a7 qbit test 2025-03-24 15:51:00 +02:00
Flaminel 6bc59c8389 added implementation for Deluge and Transmission 2025-03-24 15:50:56 +02:00
Flaminel 5fe0f5750a Fix qBit queued items being processed (#102) 2025-03-21 23:06:31 +02:00
Flaminel b8ce225ccc Fix Deluge service crashing when download is not found (#97) 2025-03-20 00:09:58 +02:00
Flaminel f21f7388b7 Add download client customizable url base (#43) 2025-03-20 00:09:24 +02:00
Flaminel a1354f231a Add base path support for arrs (#96) 2025-03-20 00:08:51 +02:00
Flaminel 4bc1c33e81 Add option to explicitly disable the download client (#93) 2025-03-19 16:02:46 +02:00
Flaminel 32bcbab523 added docs for FreeBSD 2025-03-19 01:26:04 +02:00
Flaminel b94ae21e11 update permissive blacklist 2025-03-13 10:16:52 +02:00
Flaminel a92ebd75c2 Update docs (#88) 2025-03-11 23:42:21 +02:00
Flaminel e6d3929fc9 Restrict max strikes to a minimum value (#87) 2025-03-11 23:35:07 +02:00
Flaminel a68e13af35 Fix notifications when poster is not found (#89) 2025-03-11 23:34:44 +02:00
Flaminel 324c3ace8f Fix multiple runs on queue cleaner when download cleaner is enabled (#90) 2025-03-11 23:34:27 +02:00
Flaminel 3a9d5d9085 Fix patterns being loaded for disabled arrs (#80) 2025-03-11 23:18:34 +02:00
Flaminel 89a6eaf0ce Disable cleanup on torrent items if download client is not configured (#85) 2025-03-10 00:13:40 +02:00
Flaminel 027c4a0f4d Add option to ignore specific downloads (#79) 2025-03-09 23:38:27 +02:00
Flaminel 81990c6768 fixed missing README link 2025-03-03 22:37:22 +02:00
Flaminel ba02aa0e49 Fix notifications failing when poster image is not set (#78) 2025-03-02 22:48:21 +02:00
Flaminel b1d98c2b62 fixed category changing 2025-02-27 22:23:59 +02:00
Flaminel 46ac50c393 fixed hard links check 2025-02-27 22:01:59 +02:00
Flaminel b834a8bc01 fixed downloads to change category value 2025-02-27 21:52:46 +02:00
Flaminel 3c8ef3db91 fixed service type 2025-02-27 21:51:27 +02:00
Flaminel bc642d8f80 fixed typo 2025-02-27 21:44:54 +02:00
Flaminel a994bc4526 fixed missing methods 2025-02-27 21:41:07 +02:00
Flaminel 1243da3d22 streamlined downloads processing after category changed 2025-02-26 23:39:08 +02:00
Flaminel 6b33075a21 fixed merge conflicts 2025-02-26 22:29:24 +02:00
Flaminel c27ee326f7 updated test data 2025-02-26 22:20:23 +02:00
Flaminel d27562a889 explicit dispose and clean on some objects 2025-02-26 22:20:23 +02:00
Flaminel e8d287de84 fixed missing notification configuration 2025-02-26 22:20:23 +02:00
Flaminel c65c85a0c5 fixed categories not being filtered when fetching downloads; fixed inconsistent var naming 2025-02-26 22:20:22 +02:00
Flaminel bd81f2ffca fixed using root directory instead of root path 2025-02-26 22:20:22 +02:00
Flaminel 5bd2a9cbea added trace logs 2025-02-26 22:20:06 +02:00
Flaminel e006521dc9 refactored method names; fixed qbit category creation 2025-02-26 22:20:05 +02:00
Flaminel 1ad07b1f51 refactored names; fixed return values for hard links service 2025-02-26 22:19:09 +02:00
Flaminel 9b68792ea9 added handling for windows files 2025-02-26 22:17:31 +02:00
Flaminel 8c8d412ef1 refactored hardlink service 2025-02-26 22:17:31 +02:00
Flaminel 029f255351 added debug logs 2025-02-26 22:17:30 +02:00
Flaminel 19ac8cbd28 trying to account for cross-seed 2025-02-26 22:17:30 +02:00
Flaminel f91e85651f removed debug logs 2025-02-26 22:17:30 +02:00
Flaminel fbe6ebaa6b trying to fix Unix stat yet again 2025-02-26 22:17:30 +02:00
Flaminel 268ede8a9c trying to fix Unix stat again 2025-02-26 22:17:30 +02:00
Flaminel a63bae0bb9 added debug logs 2025-02-26 22:17:29 +02:00
Flaminel d454a094a0 fixed qbit file path 2025-02-26 22:17:29 +02:00
Flaminel 1650b0e5a4 trying to fix Unix stat #2 2025-02-26 22:17:29 +02:00
Flaminel 2d6f16692c trying to fix Unix stat 2025-02-26 22:17:29 +02:00
Flaminel 017e25fb06 added category change for downloads with no additional hardlinks 2025-02-26 22:17:28 +02:00
Flaminel 5adbdbd920 Fix weird time zone display name on startup (#70) 2025-02-25 21:32:19 +02:00
67 changed files with 2366 additions and 581 deletions
+135 -34
View File
@@ -16,6 +16,7 @@ cleanuperr was created primarily to address malicious files, such as `*.lnk` or
> - Trigger a search for downloads removed from the *arrs.
> - Clean up downloads that have been seeding for a certain amount of time.
> - Notify on strike or download removal.
> - Ignore certain torrent hashes, categories, tags or trackers from processing.
cleanuperr supports both qBittorrent's built-in exclusion features and its own blocklist-based system. Binaries for all platforms are provided, along with Docker images for easy deployment.
@@ -25,15 +26,22 @@ cleanuperr supports both qBittorrent's built-in exclusion features and its own b
> https://discord.gg/sWggpnmGNY
## Table of contents:
- [Naming choice](README.md#naming-choice)
- [Quick Start](README.md#quick-start)
- [How it works](README.md#how-it-works)
- [Setup](README.md#setup)
- [Usage](README.md#usage)
- [Docker Compose](README.md#docker-compose-yaml)
- [Environment Variables](README.md#environment-variables)
- [Binaries](README.md#binaries-if-youre-not-using-docker)
- [Credits](README.md#credits)
- [Naming choice](#naming-choice)
- [Quick Start](#quick-start)
- [How it works](#how-it-works)
- [Content blocker](#1-content-blocker-will)
- [Queue cleaner](#2-queue-cleaner-will)
- [Download cleaner](#3-download-cleaner-will)
- [Setup](#setup-examples)
- [Usage](#usage)
- [Docker](#docker)
- [Environment Variables](#environment-variables)
- [Docker Compose](#docker-compose-example)
- [Windows](#windows)
- [Linux](#linux)
- [MacOS](#macos)
- [FreeBSD](#freebsd)
- [Credits](#credits)
## Naming choice
@@ -52,10 +60,10 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
> Use the Unraid Community App.
>
> 3. **Manual Installation (if you're not using Docker)**
> More details [here](#binaries-if-youre-not-using-docker).
> Go to [Windows](#windows), [Linux](#linux) or [MacOS](#macos).
> [!TIP]
> Refer to the [Environment variables](#Environment-variables) section for detailed configuration instructions and the [Setup](#Setup) section for an in-depth explanation of the cleanup process.
> Refer to the [Environment variables](#environment-variables) section for detailed configuration instructions and the [Setup examples](#setup-examples) section for an in-depth explanation of the cleanup process.
> [!IMPORTANT]
@@ -69,7 +77,7 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
# How it works
1. **Content blocker** will:
#### 1. **Content blocker** will:
- Run every 5 minutes (or configured cron).
- Process all items in the *arr queue.
- Find the corresponding item from the download client for each queue item.
@@ -80,7 +88,7 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
- It will be removed from the *arr's queue and blocked.
- It will be deleted from the download client.
- A new search will be triggered for the *arr item.
2. **Queue cleaner** will:
#### 2. **Queue cleaner** will:
- Run every 5 minutes (or configured cron, or right after `content blocker`).
- Process all items in the *arr queue.
- Check each queue item if it is **stalled (download speed is 0)**, **stuck in metadata downloading** or **failed to be imported**.
@@ -93,11 +101,11 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
- It will be removed from the *arr's queue and blocked.
- It will be deleted from the download client.
- A new search will be triggered for the *arr item.
3. **Download cleaner** will:
#### 3. **Download cleaner** will:
- Run every hour (or configured cron).
- Automatically clean up downloads that have been seeding for a certain amount of time.
# Setup
# Setup examples
## Using qBittorrent's built-in feature (works only with qBittorrent)
@@ -112,7 +120,7 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
## Using cleanuperr's blocklist (works with all supported download clients)
1. Set both `QUEUECLEANER__ENABLED` and `CONTENTBLOCKER__ENABLED` to `true` in your environment variables.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](#Arr-variables) section.
2. Configure and enable either a **blacklist** or a **whitelist** as described in the [Arr variables](variables.md#Arr-settings) section.
3. Once configured, cleanuperr will perform the following tasks:
- Execute the **content blocker** job, as explained in the [How it works](#how-it-works) section.
- Execute the **queue cleaner** job, as explained in the [How it works](#how-it-works) section.
@@ -129,7 +137,26 @@ I've seen a few discussions on this type of naming and I've decided that I didn'
## Usage
### Docker compose yaml
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/docker.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Docker</span>
### **Environment variables**
**Jump to:**
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
### Docker compose example
> [!NOTE]
>
> This example contains all settings and should be modified to fit your needs.
```
version: "3.3"
@@ -139,6 +166,7 @@ services:
restart: unless-stopped
volumes:
- ./cleanuperr/logs:/var/logs
- ./cleanuperr/ignored.txt:/ignored.txt
environment:
- TZ=America/New_York
- DRY_RUN=false
@@ -153,6 +181,7 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0 0 * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=false
@@ -165,10 +194,12 @@ services:
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- CONTENTBLOCKER__IGNORE_PRIVATE=false
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored.txt
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
@@ -183,15 +214,18 @@ services:
# OR
# - DOWNLOAD_CLIENT=qBittorrent
# - QBITTORRENT__URL=http://localhost:8080
# - QBITTORRENT__URL_BASE=myCustomPath
# - QBITTORRENT__USERNAME=user
# - QBITTORRENT__PASSWORD=pass
# OR
# - DOWNLOAD_CLIENT=deluge
# - DELUGE__URL_BASE=myCustomPath
# - DELUGE__URL=http://localhost:8112
# - DELUGE__PASSWORD=testing
# OR
# - DOWNLOAD_CLIENT=transmission
# - TRANSMISSION__URL=http://localhost:9091
# - TRANSMISSION__URL_BASE=myCustomPath
# - TRANSMISSION__USERNAME=test
# - TRANSMISSION__PASSWORD=testing
@@ -228,28 +262,96 @@ services:
- NOTIFIARR__CHANNEL_ID=discord_channel_id
```
## Environment variables
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/windows.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Windows</span>
Jump to:
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
### Binaries (if you're not using Docker)
1. Download the binaries from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract them from the zip file.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [above](#environment-variables).
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `C:\example\directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Execute `cleanuperr.exe`.
> [!TIP]
> ### Run as a Windows Service
> Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/linux.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">Linux</span>
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `/example/directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Open a terminal and execute these commands:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/apple.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">MacOS</span>
1. Download the zip file from [releases](https://github.com/flmorg/cleanuperr/releases).
2. Extract the zip file into `/example/directory`.
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Open a terminal and execute these commands:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
> [!IMPORTANT]
> Some people have experienced problems when trying to execute cleanuperr on MacOS because the system actively blocked the file for not being signed.
> As per [this](), you may need to also execute this command:
> ```
> codesign --sign - --force --preserve-metadata=entitlements,requirements,flags,runtime /example/directory/cleanuperr
> ```
### <img src="https://raw.githubusercontent.com/FortAwesome/Font-Awesome/6.x/svgs/brands/freebsd.svg" height="20" style="vertical-align: middle;"> <span style="vertical-align: middle;">FreeBSD</span>
1. Installation:
```
# install dependencies
pkg install -y git icu libinotify libunwind wget
# set up the dotnet SDK
cd ~
wget -q https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/dotnet-sdk-9.0.104-freebsd-x64.tar.gz
export DOTNET_ROOT=$(pwd)/.dotnet
mkdir -p "$DOTNET_ROOT" && tar zxf dotnet-sdk-9.0.104-freebsd-x64.tar.gz -C "$DOTNET_ROOT"
export PATH=$PATH:$DOTNET_ROOT:$DOTNET_ROOT/tools
# download NuGet dependencies
mkdir -p /tmp/nuget
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.AspNetCore.App.Runtime.freebsd-x64.9.0.3.nupkg
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.NETCore.App.Host.freebsd-x64.9.0.3.nupkg
wget -q -P /tmp/nuget/ https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/download/v9.0.104-amd64-freebsd-14/Microsoft.NETCore.App.Runtime.freebsd-x64.9.0.3.nupkg
# add NuGet source
dotnet nuget add source /tmp/nuget --name tmp
# add GitHub NuGet source
# a PAT (Personal Access Token) can be generated here https://github.com/settings/tokens
dotnet nuget add source --username <YOUR_USERNAME> --password <YOUR_PERSONAL_ACCESS_TOKEN> --store-password-in-clear-text --name flmorg https://nuget.pkg.github.com/flmorg/index.json
```
2. Building:
```
# clone the project
git clone https://github.com/flmorg/cleanuperr.git
cd cleanuperr
# build and publish the app
dotnet publish code/Executable/Executable.csproj -c Release --self-contained -o artifacts /p:PublishSingleFile=true
# move the files to permanent destination
mv artifacts/cleanuperr /example/directory/
mv artifacts/appsettings.json /example/directory/
```
3. Edit **appsettings.json**. The paths from this json file correspond with the docker env vars, as described [here](#environment-variables).
4. Run the app:
```
cd /example/directory
chmod +x cleanuperr
./cleanuperr
```
# Credits
Special thanks for inspiration go to:
- [ThijmenGThN/swaparr](https://github.com/ThijmenGThN/swaparr)
@@ -261,4 +363,3 @@ Special thanks for inspiration go to:
If I made your life just a tiny bit easier, consider buying me a coffee!
<a href="https://buymeacoffee.com/flaminel" target="_blank"><img src="https://www.buymeacoffee.com/assets/img/custom_images/orange_img.png" alt="Buy Me A Coffee" style="height: 41px !important;width: 174px !important;box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;-webkit-box-shadow: 0px 3px 2px 0px rgba(190, 190, 190, 0.5) !important;" ></a>
+1
View File
@@ -1,4 +1,5 @@
*.apk
*.arj
*.bat
*.bin
*.bmp
@@ -2,7 +2,7 @@
namespace Common.Configuration.ContentBlocker;
public sealed record ContentBlockerConfig : IJobConfig
public sealed record ContentBlockerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "ContentBlocker";
@@ -13,6 +13,9 @@ public sealed record ContentBlockerConfig : IJobConfig
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
public void Validate()
{
@@ -3,7 +3,7 @@ using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadCleaner;
public sealed record Category : IConfig
public sealed record CleanCategory : IConfig
{
public required string Name { get; init; }
@@ -3,16 +3,28 @@ using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadCleaner;
public sealed record DownloadCleanerConfig : IJobConfig
public sealed record DownloadCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "DownloadCleaner";
public bool Enabled { get; init; }
public List<Category>? Categories { get; init; }
public List<CleanCategory>? Categories { get; init; }
[ConfigurationKeyName("DELETE_PRIVATE")]
public bool DeletePrivate { get; set; }
public bool DeletePrivate { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
[ConfigurationKeyName("UNLINKED_TARGET_CATEGORY")]
public string UnlinkedTargetCategory { get; init; } = "cleanuperr-unlinked";
[ConfigurationKeyName("UNLINKED_IGNORED_ROOT_DIR")]
public string UnlinkedIgnoredRootDir { get; init; } = string.Empty;
[ConfigurationKeyName("UNLINKED_CATEGORIES")]
public List<string>? UnlinkedCategories { get; init; }
public void Validate()
{
@@ -28,9 +40,34 @@ public sealed record DownloadCleanerConfig : IJobConfig
if (Categories?.GroupBy(x => x.Name).Any(x => x.Count() > 1) is true)
{
throw new ValidationException("duplicated categories found");
throw new ValidationException("duplicated clean categories found");
}
Categories?.ForEach(x => x.Validate());
if (string.IsNullOrEmpty(UnlinkedTargetCategory))
{
return;
}
if (UnlinkedCategories?.Count is null or 0)
{
throw new ValidationException("no unlinked categories configured");
}
if (UnlinkedCategories.Contains(UnlinkedTargetCategory))
{
throw new ValidationException($"{SectionName.ToUpperInvariant()}__UNLINKED_TARGET_CATEGORY should not be present in {SectionName.ToUpperInvariant()}__UNLINKED_CATEGORIES");
}
if (UnlinkedCategories.Any(string.IsNullOrEmpty))
{
throw new ValidationException("empty unlinked category filter found");
}
if (!string.IsNullOrEmpty(UnlinkedIgnoredRootDir) && !Directory.Exists(UnlinkedIgnoredRootDir))
{
throw new ValidationException($"{UnlinkedIgnoredRootDir} root directory does not exist");
}
}
}
@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed record DelugeConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Password { get; init; }
public void Validate()
@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public sealed class QBitConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = string.Empty;
public string? Username { get; init; }
public string? Password { get; init; }
@@ -1,4 +1,5 @@
using Common.Exceptions;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.DownloadClient;
@@ -8,6 +9,9 @@ public record TransmissionConfig : IConfig
public Uri? Url { get; init; }
[ConfigurationKeyName("URL_BASE")]
public string UrlBase { get; init; } = "transmission";
public string? Username { get; init; }
public string? Password { get; init; }
@@ -0,0 +1,6 @@
namespace Common.Configuration;
public interface IIgnoredDownloadsConfig
{
string? IgnoredDownloadsPath { get; }
}
@@ -15,8 +15,16 @@ public abstract record NotificationConfig
[ConfigurationKeyName("ON_DOWNLOAD_CLEANED")]
public bool OnDownloadCleaned { get; init; }
[ConfigurationKeyName("ON_CATEGORY_CHANGED")]
public bool OnCategoryChanged { get; init; }
public bool IsEnabled => OnImportFailedStrike || OnStalledStrike || OnQueueItemDeleted || OnDownloadCleaned;
public bool IsEnabled =>
OnImportFailedStrike ||
OnStalledStrike ||
OnQueueItemDeleted ||
OnDownloadCleaned ||
OnCategoryChanged;
public abstract bool IsValid();
}
@@ -1,8 +1,9 @@
using Microsoft.Extensions.Configuration;
using Common.Exceptions;
using Microsoft.Extensions.Configuration;
namespace Common.Configuration.QueueCleaner;
public sealed record QueueCleanerConfig : IJobConfig
public sealed record QueueCleanerConfig : IJobConfig, IIgnoredDownloadsConfig
{
public const string SectionName = "QueueCleaner";
@@ -10,6 +11,9 @@ public sealed record QueueCleanerConfig : IJobConfig
public required bool RunSequentially { get; init; }
[ConfigurationKeyName("IGNORED_DOWNLOADS_PATH")]
public string? IgnoredDownloadsPath { get; init; }
[ConfigurationKeyName("IMPORT_FAILED_MAX_STRIKES")]
public ushort ImportFailedMaxStrikes { get; init; }
@@ -36,5 +40,14 @@ public sealed record QueueCleanerConfig : IJobConfig
public void Validate()
{
if (ImportFailedMaxStrikes is > 0 and < 3)
{
throw new ValidationException("the minimum value for IMPORT_FAILED_MAX_STRIKES must be 3");
}
if (StalledMaxStrikes is > 0 and < 3)
{
throw new ValidationException("the minimum value for STALLED_MAX_STRIKES must be 3");
}
}
}
+2 -1
View File
@@ -5,5 +5,6 @@ public enum DownloadClient
QBittorrent,
Deluge,
Transmission,
None
None,
Disabled
}
+12
View File
@@ -0,0 +1,12 @@
namespace Common.Exceptions;
public class FatalException : Exception
{
public FatalException()
{
}
public FatalException(string message) : base(message)
{
}
}
+5 -1
View File
@@ -2,7 +2,11 @@
public enum DeleteReason
{
None,
Stalled,
ImportFailed,
AllFilesBlocked
DownloadingMetadata,
AllFilesSkipped,
AllFilesSkippedByQBit,
AllFilesBlocked,
}
+1
View File
@@ -3,5 +3,6 @@
public enum StrikeType
{
Stalled,
DownloadingMetadata,
ImportFailed
}
@@ -17,10 +17,20 @@ public sealed record TorrentStatus
[JsonProperty("total_done")]
public long TotalDone { get; init; }
public string? Label { get; init; }
public string? Label { get; set; }
[JsonProperty("seeding_time")]
public long SeedingTime { get; init; }
public float Ratio { get; init; }
public required IReadOnlyList<Tracker> Trackers { get; init; }
[JsonProperty("download_location")]
public required string DownloadLocation { get; init; }
}
public sealed record Tracker
{
public required Uri Url { get; init; }
}
@@ -17,7 +17,9 @@ public static class MainDI
.AddLogging(builder => builder.ClearProviders().AddConsole())
.AddHttpClients(configuration)
.AddConfiguration(configuration)
.AddMemoryCache()
.AddMemoryCache(options => {
options.ExpirationScanFrequency = TimeSpan.FromMinutes(1);
})
.AddServices()
.AddQuartzServices(configuration)
.AddNotifications(configuration)
@@ -27,6 +29,7 @@ public static class MainDI
config.AddConsumer<NotificationConsumer<StalledStrikeNotification>>();
config.AddConsumer<NotificationConsumer<QueueItemDeletedNotification>>();
config.AddConsumer<NotificationConsumer<DownloadCleanedNotification>>();
config.AddConsumer<NotificationConsumer<CategoryChangedNotification>>();
config.UsingInMemory((context, cfg) =>
{
@@ -36,6 +39,7 @@ public static class MainDI
e.ConfigureConsumer<NotificationConsumer<StalledStrikeNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<QueueItemDeletedNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<DownloadCleanedNotification>>(context);
e.ConfigureConsumer<NotificationConsumer<CategoryChangedNotification>>(context);
e.ConcurrentMessageLimit = 1;
e.PrefetchCount = 1;
});
@@ -55,7 +55,7 @@ public static class QuartzDI
if (contentBlockerConfig?.Enabled is true && queueCleanerConfig is { Enabled: true, RunSequentially: true })
{
q.AddJob<QueueCleaner>(queueCleanerConfig, string.Empty);
q.AddJobListener(new JobChainingListener(nameof(QueueCleaner)));
q.AddJobListener(new JobChainingListener(nameof(ContentBlocker), nameof(QueueCleaner)));
}
else
{
@@ -1,4 +1,8 @@
using Infrastructure.Interceptors;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadCleaner;
@@ -6,6 +10,7 @@ using Infrastructure.Verticals.DownloadClient;
using Infrastructure.Verticals.DownloadClient.Deluge;
using Infrastructure.Verticals.DownloadClient.QBittorrent;
using Infrastructure.Verticals.DownloadClient.Transmission;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.QueueCleaner;
@@ -23,6 +28,9 @@ public static class ServicesDI
.AddTransient<ContentBlocker>()
.AddTransient<DownloadCleaner>()
.AddTransient<IFilenameEvaluator, FilenameEvaluator>()
.AddTransient<IHardLinkFileService, HardLinkFileService>()
.AddTransient<UnixHardLinkFileService>()
.AddTransient<WindowsHardLinkFileService>()
.AddTransient<DummyDownloadService>()
.AddTransient<QBitService>()
.AddTransient<DelugeService>()
@@ -30,5 +38,8 @@ public static class ServicesDI
.AddTransient<ArrQueueIterator>()
.AddTransient<DownloadServiceFactory>()
.AddSingleton<BlocklistProvider>()
.AddSingleton<IStriker, Striker>();
.AddSingleton<IStriker, Striker>()
.AddSingleton<IgnoredDownloadsProvider<QueueCleanerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<ContentBlockerConfig>>()
.AddSingleton<IgnoredDownloadsProvider<DownloadCleanerConfig>>();
}
+1 -1
View File
@@ -16,7 +16,7 @@ public static class HostExtensions
: $"cleanuperr v{version.Major}.{version.Minor}.{version.Build}"
);
logger.LogInformation("timezone: {tz}", TimeZoneInfo.Local.Id);
logger.LogInformation("timezone: {tz}", TimeZoneInfo.Local.DisplayName);
return host;
}
+18 -6
View File
@@ -3,7 +3,7 @@
"HTTP_MAX_RETRIES": 0,
"HTTP_TIMEOUT": 10,
"Logging": {
"LogLevel": "Debug",
"LogLevel": "Verbose",
"Enhanced": true,
"File": {
"Enabled": false,
@@ -18,18 +18,20 @@
"ContentBlocker": {
"Enabled": true,
"IGNORE_PRIVATE": true,
"DELETE_PRIVATE": false
"DELETE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IMPORT_FAILED_MAX_STRIKES": 5,
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads",
"IMPORT_FAILED_MAX_STRIKES": 3,
"IMPORT_FAILED_IGNORE_PRIVATE": true,
"IMPORT_FAILED_DELETE_PRIVATE": false,
"IMPORT_FAILED_IGNORE_PATTERNS": [
"file is a sample"
],
"STALLED_MAX_STRIKES": 5,
"STALLED_MAX_STRIKES": 3,
"STALLED_RESET_STRIKES_ON_PROGRESS": true,
"STALLED_IGNORE_PRIVATE": true,
"STALLED_DELETE_PRIVATE": false
@@ -42,22 +44,32 @@
"Name": "tv-sonarr",
"MAX_RATIO": -1,
"MIN_SEED_TIME": 0,
"MAX_SEED_TIME": -1
"MAX_SEED_TIME": 240
}
]
],
"UNLINKED_TARGET_CATEGORY": "cleanuperr-unlinked",
"UNLINKED_IGNORED_ROOT_DIR": "",
"UNLINKED_CATEGORIES": [
"tv-sonarr",
"radarr"
],
"IGNORED_DOWNLOADS_PATH": "../test/data/cleanuperr/ignored_downloads"
},
"DOWNLOAD_CLIENT": "qbittorrent",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "test",
"Password": "testing"
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},
+11 -2
View File
@@ -17,11 +17,13 @@
},
"ContentBlocker": {
"Enabled": false,
"IGNORE_PRIVATE": false
"IGNORE_PRIVATE": false,
"IGNORED_DOWNLOADS_PATH": ""
},
"QueueCleaner": {
"Enabled": true,
"RunSequentially": true,
"IGNORED_DOWNLOADS_PATH": "",
"IMPORT_FAILED_MAX_STRIKES": 0,
"IMPORT_FAILED_IGNORE_PRIVATE": false,
"IMPORT_FAILED_DELETE_PRIVATE": false,
@@ -34,20 +36,27 @@
"DownloadCleaner": {
"Enabled": false,
"DELETE_PRIVATE": false,
"CATEGORIES": []
"CATEGORIES": [],
"UNLINKED_TARGET_CATEGORY": "cleanuperr-unlinked",
"UNLINKED_IGNORED_ROOT_DIR": "",
"UNLINKED_CATEGORIES": [],
"IGNORED_DOWNLOADS_PATH": ""
},
"DOWNLOAD_CLIENT": "none",
"qBittorrent": {
"Url": "http://localhost:8080",
"URL_BASE": "",
"Username": "",
"Password": ""
},
"Deluge": {
"Url": "http://localhost:8112",
"URL_BASE": "",
"Password": "testing"
},
"Transmission": {
"Url": "http://localhost:9091",
"URL_BASE": "transmission",
"Username": "test",
"Password": "testing"
},
@@ -1,9 +1,10 @@
using Common.Configuration.ContentBlocker;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -56,6 +57,7 @@ public class DownloadServiceFixture : IDisposable
var filenameEvaluator = Substitute.For<IFilenameEvaluator>();
var notifier = Substitute.For<INotificationPublisher>();
var dryRunInterceptor = Substitute.For<IDryRunInterceptor>();
var hardlinkFileService = Substitute.For<IHardLinkFileService>();
return new TestDownloadService(
Logger,
@@ -66,7 +68,8 @@ public class DownloadServiceFixture : IDisposable
filenameEvaluator,
Striker,
notifier,
dryRunInterceptor
dryRunInterceptor,
hardlinkFileService
);
}
@@ -105,13 +105,14 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
// Arrange
const string hash = "test-hash";
const string itemName = "test-item";
_fixture.Striker.StrikeAndCheckLimit(hash, itemName, 3, StrikeType.Stalled)
StrikeType strikeType = StrikeType.Stalled;
_fixture.Striker.StrikeAndCheckLimit(hash, itemName, 3, strikeType)
.Returns(true);
TestDownloadService sut = _fixture.CreateSut();
// Act
bool result = await sut.StrikeAndCheckLimit(hash, itemName);
bool result = await sut.StrikeAndCheckLimit(hash, itemName, strikeType);
// Assert
result.ShouldBeTrue();
@@ -132,7 +133,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public void WhenRatioAndMinSeedTimeReached_ShouldReturnTrue()
{
// Arrange
Category category = new()
CleanCategory category = new()
{
Name = "test",
MaxRatio = 1.0,
@@ -158,7 +159,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public void WhenRatioReachedAndMinSeedTimeNotReached_ShouldReturnFalse()
{
// Arrange
Category category = new()
CleanCategory category = new()
{
Name = "test",
MaxRatio = 1.0,
@@ -184,7 +185,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public void WhenMaxSeedTimeReached_ShouldReturnTrue()
{
// Arrange
Category category = new()
CleanCategory category = new()
{
Name = "test",
MaxRatio = -1,
@@ -210,7 +211,7 @@ public class DownloadServiceTests : IClassFixture<DownloadServiceFixture>
public void WhenNeitherConditionMet_ShouldReturnFalse()
{
// Arrange
Category category = new()
CleanCategory category = new()
{
Name = "test",
MaxRatio = 2.0,
@@ -1,11 +1,13 @@
using System.Collections.Concurrent;
using System.Collections.Concurrent;
using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.DownloadClient;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -25,25 +27,29 @@ public class TestDownloadService : DownloadService
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
filenameEvaluator, striker, notifier, dryRunInterceptor, hardLinkFileService
)
{
}
public override void Dispose() { }
public override Task LoginAsync() => Task.CompletedTask;
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash) => Task.FromResult(new StalledResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes) => Task.FromResult(new BlockFilesResult());
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new StalledResult());
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType,
ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads) => Task.FromResult(new BlockFilesResult());
public override Task DeleteDownload(string hash) => Task.CompletedTask;
public override Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories) => Task.FromResult<List<object>?>(null);
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes) => Task.CompletedTask;
public override Task CreateCategoryAsync(string name) => Task.CompletedTask;
public override Task<List<object>?> GetSeedingDownloads() => Task.FromResult<List<object>?>(null);
public override List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories) => null;
public override List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories) => null;
public override Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads) => Task.CompletedTask;
public override Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads) => Task.CompletedTask;
// Expose protected methods for testing
public new void ResetStrikesOnProgress(string hash, long downloaded) => base.ResetStrikesOnProgress(hash, downloaded);
public new Task<bool> StrikeAndCheckLimit(string hash, string itemName) => base.StrikeAndCheckLimit(hash, itemName);
public new SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category) => base.ShouldCleanDownload(ratio, seedingTime, category);
public new Task<bool> StrikeAndCheckLimit(string hash, string itemName, StrikeType strikeType) => base.StrikeAndCheckLimit(hash, itemName, strikeType);
public new SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, CleanCategory category) => base.ShouldCleanDownload(ratio, seedingTime, category);
}
@@ -0,0 +1,29 @@
using Domain.Models.Deluge.Response;
namespace Infrastructure.Extensions;
public static class DelugeExtensions
{
public static bool ShouldIgnore(this TorrentStatus download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Label?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.Trackers.Any(x => x.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase)))
{
return true;
}
}
return false;
}
}
@@ -0,0 +1,42 @@
using QBittorrent.Client;
namespace Infrastructure.Extensions;
public static class QBitExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.Hash.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Category.Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
if (download.Tags.Contains(value, StringComparer.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
public static bool ShouldIgnore(this TorrentTracker tracker, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (tracker.Url.Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
}
return false;
}
}
@@ -0,0 +1,42 @@
using Transmission.API.RPC.Entity;
namespace Infrastructure.Extensions;
public static class TransmissionExtensions
{
public static bool ShouldIgnore(this TorrentInfo download, IReadOnlyList<string> ignoredDownloads)
{
foreach (string value in ignoredDownloads)
{
if (download.HashString?.Equals(value, StringComparison.InvariantCultureIgnoreCase) ?? false)
{
return true;
}
if (download.GetCategory().Equals(value, StringComparison.InvariantCultureIgnoreCase))
{
return true;
}
bool? hasIgnoredTracker = download.Trackers?
.Any(x => new Uri(x.Announce!).Host.EndsWith(value, StringComparison.InvariantCultureIgnoreCase));
if (hasIgnoredTracker is true)
{
return true;
}
}
return false;
}
public static string GetCategory(this TorrentInfo download)
{
if (string.IsNullOrEmpty(download.DownloadDir))
{
return string.Empty;
}
return Path.GetFileName(Path.TrimEndingDirectorySeparator(download.DownloadDir));
}
}
+2
View File
@@ -11,4 +11,6 @@ public static class CacheKeys
public static string BlocklistRegexes(InstanceType instanceType) => $"{instanceType.ToString()}_regexes";
public static string Item(string hash) => $"item_{hash}";
public static string IgnoredDownloads(string name) => $"{name}_ignored";
}
+2 -1
View File
@@ -13,11 +13,12 @@
<ItemGroup>
<PackageReference Include="FLM.QBittorrent" Version="1.0.0" />
<PackageReference Include="FLM.Transmission" Version="1.0.2" />
<PackageReference Include="FLM.Transmission" Version="1.0.3" />
<PackageReference Include="Mapster" Version="7.4.0" />
<PackageReference Include="MassTransit" Version="8.3.6" />
<PackageReference Include="Microsoft.Extensions.Caching.Memory" Version="9.0.2" />
<PackageReference Include="Microsoft.Extensions.Http" Version="9.0.2" />
<PackageReference Include="Mono.Posix.NETStandard" Version="1.0.0" />
<PackageReference Include="Quartz" Version="3.13.1" />
<PackageReference Include="Scrutor" Version="6.0.1" />
</ItemGroup>
@@ -0,0 +1,82 @@
using Common.Configuration;
using Infrastructure.Helpers;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
namespace Infrastructure.Providers;
public sealed class IgnoredDownloadsProvider<T>
where T : IIgnoredDownloadsConfig
{
private readonly ILogger<IgnoredDownloadsProvider<T>> _logger;
private IIgnoredDownloadsConfig _config;
private readonly IMemoryCache _cache;
private DateTime _lastModified = DateTime.MinValue;
public IgnoredDownloadsProvider(ILogger<IgnoredDownloadsProvider<T>> logger, IOptionsMonitor<T> config, IMemoryCache cache)
{
_config = config.CurrentValue;
config.OnChange((newValue) => _config = newValue);
_logger = logger;
_cache = cache;
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return;
}
if (!File.Exists(_config.IgnoredDownloadsPath))
{
throw new FileNotFoundException("file not found", _config.IgnoredDownloadsPath);
}
}
public async Task<IReadOnlyList<string>> GetIgnoredDownloads()
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
FileInfo fileInfo = new(_config.IgnoredDownloadsPath);
if (fileInfo.LastWriteTime > _lastModified ||
!_cache.TryGetValue(CacheKeys.IgnoredDownloads(typeof(T).Name), out IReadOnlyList<string>? ignoredDownloads) ||
ignoredDownloads is null)
{
_lastModified = fileInfo.LastWriteTime;
return await LoadFile();
}
return ignoredDownloads;
}
private async Task<IReadOnlyList<string>> LoadFile()
{
try
{
if (string.IsNullOrEmpty(_config.IgnoredDownloadsPath))
{
return Array.Empty<string>();
}
string[] ignoredDownloads = (await File.ReadAllLinesAsync(_config.IgnoredDownloadsPath))
.Where(x => !string.IsNullOrWhiteSpace(x))
.ToArray();
_cache.Set(CacheKeys.IgnoredDownloads(typeof(T).Name), ignoredDownloads);
_logger.LogInformation("ignored downloads reloaded");
return ignoredDownloads;
}
catch (Exception exception)
{
_logger.LogError(exception, "error while reading ignored downloads file | {file}", _config.IgnoredDownloadsPath);
}
return Array.Empty<string>();
}
}
+26 -12
View File
@@ -43,9 +43,11 @@ public abstract class ArrClient : IArrClient
public virtual async Task<QueueListResponse> GetQueueItemsAsync(ArrInstance arrInstance, int page)
{
Uri uri = new(arrInstance.Url, GetQueueUrlPath(page));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueUrlPath().TrimStart('/')}";
uriBuilder.Query = GetQueueUrlQuery(page);
using HttpRequestMessage request = new(HttpMethod.Get, uri);
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -56,7 +58,7 @@ public abstract class ArrClient : IArrClient
}
catch
{
_logger.LogError("queue list failed | {uri}", uri);
_logger.LogError("queue list failed | {uri}", uriBuilder.Uri);
throw;
}
@@ -65,7 +67,7 @@ public abstract class ArrClient : IArrClient
if (queueResponse is null)
{
throw new Exception($"unrecognized queue list response | {uri} | {responseBody}");
throw new Exception($"unrecognized queue list response | {uriBuilder.Uri} | {responseBody}");
}
return queueResponse;
@@ -112,13 +114,20 @@ public abstract class ArrClient : IArrClient
return false;
}
public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient)
public virtual async Task DeleteQueueItemAsync(
ArrInstance arrInstance,
QueueRecord record,
bool removeFromClient,
DeleteReason deleteReason
)
{
Uri uri = new(arrInstance.Url, GetQueueDeleteUrlPath(record.Id, removeFromClient));
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/{GetQueueDeleteUrlPath(record.Id).TrimStart('/')}";
uriBuilder.Query = GetQueueDeleteUrlQuery(removeFromClient);
try
{
using HttpRequestMessage request = new(HttpMethod.Delete, uri);
using HttpRequestMessage request = new(HttpMethod.Delete, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
HttpResponseMessage? response = await _dryRunInterceptor.InterceptAsync<HttpResponseMessage>(SendRequestAsync, request);
@@ -126,15 +135,16 @@ public abstract class ArrClient : IArrClient
_logger.LogInformation(
removeFromClient
? "queue item deleted | {url} | {title}"
: "queue item removed from arr | {url} | {title}",
? "queue item deleted with reason {reason} | {url} | {title}"
: "queue item removed from arr with reason {reason} | {url} | {title}",
deleteReason.ToString(),
arrInstance.Url,
record.Title
);
}
catch
{
_logger.LogError("queue delete failed | {uri} | {title}", uri, record.Title);
_logger.LogError("queue delete failed | {uri} | {title}", uriBuilder.Uri, record.Title);
throw;
}
}
@@ -152,9 +162,13 @@ public abstract class ArrClient : IArrClient
return true;
}
protected abstract string GetQueueUrlPath(int page);
protected abstract string GetQueueUrlPath();
protected abstract string GetQueueDeleteUrlPath(long recordId, bool removeFromClient);
protected abstract string GetQueueUrlQuery(int page);
protected abstract string GetQueueDeleteUrlPath(long recordId);
protected abstract string GetQueueDeleteUrlQuery(bool removeFromClient);
protected virtual void SetApiKey(HttpRequestMessage request, string apiKey)
{
@@ -11,7 +11,7 @@ public interface IArrClient
Task<bool> ShouldRemoveFromQueue(InstanceType instanceType, QueueRecord record, bool isPrivateDownload);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient);
Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord record, bool removeFromClient, DeleteReason deleteReason);
Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items);
@@ -27,29 +27,42 @@ public class LidarrClient : ArrClient, ILidarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v1/queue?page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
return "/api/v1/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v1/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownArtistItems=true&includeArtist=true&includeAlbum=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v1/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
{
if (items?.Count is null or 0) return;
if (items?.Count is null or 0)
{
return;
}
Uri uri = new(arrInstance.Url, "/api/v1/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/command";
foreach (var command in GetSearchCommands(items))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -132,8 +145,11 @@ public class LidarrClient : ArrClient, ILidarrClient
private async Task<List<Album>?> GetAlbumsAsync(ArrInstance arrInstance, List<long> albumIds)
{
Uri uri = new(arrInstance.Url, $"api/v1/album?{string.Join('&', albumIds.Select(x => $"albumIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v1/album";
uriBuilder.Query = string.Join('&', albumIds.Select(x => $"albumIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using var response = await _httpClient.SendAsync(request);
@@ -27,18 +27,27 @@ public class RadarrClient : ArrClient, IRadarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return $"page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true";
}
return path;
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -50,14 +59,16 @@ public class RadarrClient : ArrClient, IRadarrClient
List<long> ids = items.Select(item => item.Id).ToList();
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
RadarrCommand command = new()
{
Name = "MoviesSearch",
MovieIds = ids,
};
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command),
Encoding.UTF8,
@@ -135,8 +146,10 @@ public class RadarrClient : ArrClient, IRadarrClient
private async Task<Movie?> GetMovie(ArrInstance arrInstance, long movieId)
{
Uri uri = new(arrInstance.Url, $"api/v3/movie/{movieId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/movie/{movieId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -28,18 +28,27 @@ public class SonarrClient : ArrClient, ISonarrClient
{
}
protected override string GetQueueUrlPath(int page)
protected override string GetQueueUrlPath()
{
return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
return "/api/v3/queue";
}
protected override string GetQueueDeleteUrlPath(long recordId, bool removeFromClient)
protected override string GetQueueUrlQuery(int page)
{
string path = $"/api/v3/queue/{recordId}?blocklist=true&skipRedownload=true&changeCategory=false";
return $"page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true&includeEpisode=true";
}
path += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
protected override string GetQueueDeleteUrlPath(long recordId)
{
return $"/api/v3/queue/{recordId}";
}
return path;
protected override string GetQueueDeleteUrlQuery(bool removeFromClient)
{
string query = "blocklist=true&skipRedownload=true&changeCategory=false";
query += removeFromClient ? "&removeFromClient=true" : "&removeFromClient=false";
return query;
}
public override async Task RefreshItemsAsync(ArrInstance arrInstance, HashSet<SearchItem>? items)
@@ -49,11 +58,12 @@ public class SonarrClient : ArrClient, ISonarrClient
return;
}
Uri uri = new(arrInstance.Url, "/api/v3/command");
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/command";
foreach (SonarrCommand command in GetSearchCommands(items.Cast<SonarrSearchItem>().ToHashSet()))
{
using HttpRequestMessage request = new(HttpMethod.Post, uri);
using HttpRequestMessage request = new(HttpMethod.Post, uriBuilder.Uri);
request.Content = new StringContent(
JsonConvert.SerializeObject(command, new JsonSerializerSettings { NullValueHandling = NullValueHandling.Ignore }),
Encoding.UTF8,
@@ -199,8 +209,11 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<List<Episode>?> GetEpisodesAsync(ArrInstance arrInstance, List<long> episodeIds)
{
Uri uri = new(arrInstance.Url, $"api/v3/episode?{string.Join('&', episodeIds.Select(x => $"episodeIds={x}"))}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/episode";
uriBuilder.Query = string.Join('&', episodeIds.Select(x => $"episodeIds={x}"));
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -212,8 +225,10 @@ public class SonarrClient : ArrClient, ISonarrClient
private async Task<Series?> GetSeriesAsync(ArrInstance arrInstance, long seriesId)
{
Uri uri = new(arrInstance.Url, $"api/v3/series/{seriesId}");
using HttpRequestMessage request = new(HttpMethod.Get, uri);
UriBuilder uriBuilder = new(arrInstance.Url);
uriBuilder.Path = $"{uriBuilder.Path.TrimEnd('/')}/api/v3/series/{seriesId}";
using HttpRequestMessage request = new(HttpMethod.Get, uriBuilder.Uri);
SetApiKey(request, arrInstance.ApiKey);
using HttpResponseMessage response = await _httpClient.SendAsync(request);
@@ -43,15 +43,15 @@ public sealed class BlocklistProvider
{
if (_initialized)
{
_logger.LogDebug("blocklists already loaded");
_logger.LogTrace("blocklists already loaded");
return;
}
try
{
await LoadPatternsAndRegexesAsync(_sonarrConfig.Block.Type, _sonarrConfig.Block.Path, InstanceType.Sonarr);
await LoadPatternsAndRegexesAsync(_radarrConfig.Block.Type, _radarrConfig.Block.Path, InstanceType.Radarr);
await LoadPatternsAndRegexesAsync(_lidarrConfig.Block.Type, _lidarrConfig.Block.Path, InstanceType.Lidarr);
await LoadPatternsAndRegexesAsync(_sonarrConfig, InstanceType.Sonarr);
await LoadPatternsAndRegexesAsync(_radarrConfig, InstanceType.Radarr);
await LoadPatternsAndRegexesAsync(_lidarrConfig, InstanceType.Lidarr);
_initialized = true;
}
@@ -83,14 +83,19 @@ public sealed class BlocklistProvider
return regexes ?? [];
}
private async Task LoadPatternsAndRegexesAsync(BlocklistType blocklistType, string? blocklistPath, InstanceType instanceType)
private async Task LoadPatternsAndRegexesAsync(ArrConfig arrConfig, InstanceType instanceType)
{
if (string.IsNullOrEmpty(blocklistPath))
if (!arrConfig.Enabled)
{
return;
}
string[] filePatterns = await ReadContentAsync(blocklistPath);
if (string.IsNullOrEmpty(arrConfig.Block.Path))
{
return;
}
string[] filePatterns = await ReadContentAsync(arrConfig.Block.Path);
long startTime = Stopwatch.GetTimestamp();
ParallelOptions options = new() { MaxDegreeOfParallelism = 5 };
@@ -121,13 +126,13 @@ public sealed class BlocklistProvider
TimeSpan elapsed = Stopwatch.GetElapsedTime(startTime);
_cache.Set(CacheKeys.BlocklistType(instanceType), blocklistType);
_cache.Set(CacheKeys.BlocklistType(instanceType), arrConfig.Block.Type);
_cache.Set(CacheKeys.BlocklistPatterns(instanceType), patterns);
_cache.Set(CacheKeys.BlocklistRegexes(instanceType), regexes);
_logger.LogDebug("loaded {count} patterns", patterns.Count);
_logger.LogDebug("loaded {count} regexes", regexes.Count);
_logger.LogDebug("blocklist loaded in {elapsed} ms | {path}", elapsed.TotalMilliseconds, blocklistPath);
_logger.LogDebug("blocklist loaded in {elapsed} ms | {path}", elapsed.TotalMilliseconds, arrConfig.Block.Path);
}
private async Task<string[]> ReadContentAsync(string path)
@@ -6,6 +6,7 @@ using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -22,7 +23,8 @@ public sealed class ContentBlocker : GenericHandler
{
private readonly ContentBlockerConfig _config;
private readonly BlocklistProvider _blocklistProvider;
private readonly IgnoredDownloadsProvider<ContentBlockerConfig> _ignoredDownloadsProvider;
public ContentBlocker(
ILogger<ContentBlocker> logger,
IOptions<ContentBlockerConfig> config,
@@ -36,7 +38,8 @@ public sealed class ContentBlocker : GenericHandler
ArrQueueIterator arrArrQueueIterator,
BlocklistProvider blocklistProvider,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<ContentBlockerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -47,13 +50,14 @@ public sealed class ContentBlocker : GenericHandler
{
_config = config.Value;
_blocklistProvider = blocklistProvider;
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None or Common.Enums.DownloadClient.Disabled)
{
_logger.LogWarning("download client is set to none");
_logger.LogWarning("download client is not set");
return;
}
@@ -73,6 +77,8 @@ public sealed class ContentBlocker : GenericHandler
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -106,13 +112,19 @@ public sealed class ContentBlocker : GenericHandler
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
_logger.LogDebug("searching unwanted files for {title}", record.Title);
BlockFilesResult result = await _downloadService
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes);
.BlockUnwantedFilesAsync(record.DownloadId, blocklistType, patterns, regexes, ignoredDownloads);
if (!result.ShouldRemove)
{
@@ -130,7 +142,7 @@ public sealed class ContentBlocker : GenericHandler
removeFromClient = false;
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, DeleteReason.AllFilesBlocked);
await _notifier.NotifyQueueItemDeleted(removeFromClient, DeleteReason.AllFilesBlocked);
}
});
@@ -3,6 +3,7 @@ using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Domain.Enums;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.DownloadClient;
@@ -17,8 +18,11 @@ namespace Infrastructure.Verticals.DownloadCleaner;
public sealed class DownloadCleaner : GenericHandler
{
private readonly DownloadCleanerConfig _config;
private readonly IgnoredDownloadsProvider<DownloadCleanerConfig> _ignoredDownloadsProvider;
private readonly HashSet<string> _excludedHashes = [];
private static bool _hardLinkCategoryCreated;
public DownloadCleaner(
ILogger<DownloadCleaner> logger,
IOptions<DownloadCleanerConfig> config,
@@ -31,7 +35,8 @@ public sealed class DownloadCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<DownloadCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,13 +47,14 @@ public sealed class DownloadCleaner : GenericHandler
{
_config = config.Value;
_config.Validate();
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
public override async Task ExecuteAsync()
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None or Common.Enums.DownloadClient.Disabled)
{
_logger.LogWarning("download client is set to none");
_logger.LogWarning("download client is not set");
return;
}
@@ -58,14 +64,23 @@ public sealed class DownloadCleaner : GenericHandler
return;
}
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
await _downloadService.LoginAsync();
List<object>? downloads = await _downloadService.GetAllDownloadsToBeCleaned(_config.Categories);
if (downloads?.Count is null or 0)
List<object>? downloads = await _downloadService.GetSeedingDownloads();
List<object>? downloadsToChangeCategory = null;
if (!string.IsNullOrEmpty(_config.UnlinkedTargetCategory) && _config.UnlinkedCategories?.Count > 0)
{
_logger.LogDebug("no downloads found in the download client");
return;
if (!_hardLinkCategoryCreated)
{
_logger.LogDebug("creating category {cat}", _config.UnlinkedTargetCategory);
await _downloadService.CreateCategoryAsync(_config.UnlinkedTargetCategory);
_hardLinkCategoryCreated = true;
}
downloadsToChangeCategory = _downloadService.FilterDownloadsToChangeCategoryAsync(downloads, _config.UnlinkedCategories);
}
// wait for the downloads to appear in the arr queue
@@ -75,7 +90,16 @@ public sealed class DownloadCleaner : GenericHandler
await ProcessArrConfigAsync(_radarrConfig, InstanceType.Radarr, true);
await ProcessArrConfigAsync(_lidarrConfig, InstanceType.Lidarr, true);
await _downloadService.CleanDownloads(downloads, _config.Categories, _excludedHashes);
_logger.LogTrace("looking for downloads to change category");
await _downloadService.ChangeCategoryForNoHardLinksAsync(downloadsToChangeCategory, _excludedHashes, ignoredDownloads);
List<object>? downloadsToClean = _downloadService.FilterDownloadsToBeCleanedAsync(downloads, _config.Categories);
// release unused objects
downloads = null;
_logger.LogTrace("looking for downloads to clean");
await _downloadService.CleanDownloadsAsync(downloadsToClean, _config.Categories, _excludedHashes, ignoredDownloads);
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
@@ -2,6 +2,7 @@
using System.Text.Json.Serialization;
using Common.Configuration;
using Common.Configuration.DownloadClient;
using Common.Exceptions;
using Domain.Models.Deluge.Exceptions;
using Domain.Models.Deluge.Request;
using Domain.Models.Deluge.Response;
@@ -16,9 +17,25 @@ public sealed class DelugeClient
private readonly DelugeConfig _config;
private readonly HttpClient _httpClient;
private static readonly IReadOnlyList<string> Fields =
[
"hash",
"state",
"name",
"eta",
"private",
"total_done",
"label",
"seeding_time",
"ratio",
"trackers",
"download_location"
];
public DelugeClient(IOptions<DelugeConfig> config, IHttpClientFactory httpClientFactory)
{
_config = config.Value;
_config.Validate();
_httpClient = httpClientFactory.CreateClient(nameof(DelugeService));
}
@@ -27,11 +44,42 @@ public sealed class DelugeClient
return await SendRequest<bool>("auth.login", _config.Password);
}
public async Task<bool> IsConnected()
{
return await SendRequest<bool>("web.connected");
}
public async Task<bool> Connect()
{
string? firstHost = await GetHost();
if (string.IsNullOrEmpty(firstHost))
{
return false;
}
var result = await SendRequest<List<string>?>("web.connect", firstHost);
return result?.Count > 0;
}
public async Task<bool> Logout()
{
return await SendRequest<bool>("auth.delete_session");
}
public async Task<string?> GetHost()
{
var hosts = await SendRequest<List<List<string>?>?>("web.get_hosts");
if (hosts?.Count > 1)
{
throw new FatalException("multiple Deluge hosts found - please connect to only one host");
}
return hosts?.FirstOrDefault()?.FirstOrDefault();
}
public async Task<List<DelugeTorrent>> ListTorrents(Dictionary<string, string>? filters = null)
{
filters ??= new Dictionary<string, string>();
@@ -65,11 +113,24 @@ public sealed class DelugeClient
public async Task<TorrentStatus?> GetTorrentStatus(string hash)
{
return await SendRequest<TorrentStatus?>(
"web.get_torrent_status",
hash,
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
);
try
{
return await SendRequest<TorrentStatus?>(
"web.get_torrent_status",
hash,
Fields
);
}
catch (DelugeClientException e)
{
// Deluge returns an error when the torrent is not found
if (e.Message == "AttributeError: 'NoneType' object has no attribute 'call'")
{
return null;
}
throw;
}
}
public async Task<List<TorrentStatus>?> GetStatusForAllTorrents()
@@ -77,7 +138,7 @@ public sealed class DelugeClient
Dictionary<string, TorrentStatus>? downloads = await SendRequest<Dictionary<string, TorrentStatus>?>(
"core.get_torrents_status",
"",
new[] { "hash", "state", "name", "eta", "private", "total_done", "label", "seeding_time", "ratio" }
Fields
);
return downloads?.Values.ToList();
@@ -107,15 +168,19 @@ public sealed class DelugeClient
{
StringContent content = new StringContent(json);
content.Headers.ContentType = new MediaTypeWithQualityHeaderValue("application/json");
var responseMessage = await _httpClient.PostAsync(new Uri(_config.Url, "/json"), content);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/json"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/json";
var responseMessage = await _httpClient.PostAsync(uriBuilder.Uri, content);
responseMessage.EnsureSuccessStatusCode();
var responseJson = await responseMessage.Content.ReadAsStringAsync();
return responseJson;
}
private DelugeRequest CreateRequest(string method, params object[] parameters)
private static DelugeRequest CreateRequest(string method, params object[] parameters)
{
if (String.IsNullOrWhiteSpace(method))
{
@@ -161,4 +226,19 @@ public sealed class DelugeClient
return webResponse.Result;
}
public async Task<IReadOnlyList<string>> GetLabels()
{
return await SendRequest<IReadOnlyList<string>>("label.get_labels");
}
public async Task CreateLabel(string label)
{
await SendRequest<DelugeResponse<object>>("label.add", label);
}
public async Task SetTorrentLabel(string hash, string newLabel)
{
await SendRequest<DelugeResponse<object>>("label.set_torrent", hash, newLabel);
}
}
@@ -5,11 +5,14 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.Exceptions;
using Domain.Enums;
using Domain.Models.Deluge.Response;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -33,10 +36,11 @@ public class DelugeService : DownloadService, IDelugeService
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
filenameEvaluator, striker, notifier, dryRunInterceptor, hardLinkFileService
)
{
config.Value.Validate();
@@ -46,23 +50,36 @@ public class DelugeService : DownloadService, IDelugeService
public override async Task LoginAsync()
{
await _client.LoginAsync();
if (!await _client.IsConnected() && !await _client.Connect())
{
throw new FatalException("Deluge WebUI is not connected to the daemon");
}
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
DelugeContents? contents = null;
StalledResult result = new();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
TorrentStatus? download = await _client.GetTorrentStatus(hash);
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
result.IsPrivate = download.Private;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
try
{
@@ -72,6 +89,7 @@ public class DelugeService : DownloadService, IDelugeService
{
_logger.LogDebug(exception, "failed to find torrent {hash} in the download client", hash);
}
bool shouldRemove = contents?.Contents?.Count > 0;
@@ -85,45 +103,47 @@ public class DelugeService : DownloadService, IDelugeService
if (shouldRemove)
{
result.DeleteReason = DeleteReason.AllFilesBlocked;
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(status);
result.IsPrivate = status.Private;
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await IsItemStuckAndShouldRemove(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
hash = hash.ToLowerInvariant();
TorrentStatus? status = await _client.GetTorrentStatus(hash);
TorrentStatus? download = await _client.GetTorrentStatus(hash);
BlockFilesResult result = new();
if (status?.Hash is null)
if (download?.Hash is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
result.IsPrivate = status.Private;
if (_contentBlockerConfig.IgnorePrivate && status.Private)
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
result.IsPrivate = download.Private;
if (_contentBlockerConfig.IgnorePrivate && download.Private)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", status.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -194,19 +214,39 @@ public class DelugeService : DownloadService, IDelugeService
return result;
}
public override async Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories)
public override async Task<List<object>?> GetSeedingDownloads()
{
return (await _client.GetStatusForAllTorrents())
?.Where(x => !string.IsNullOrEmpty(x.Hash))
.Where(x => x.State?.Equals("seeding", StringComparison.InvariantCultureIgnoreCase) is true)
.Where(x => categories.Any(cat => cat.Name.Equals(x.Label, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
}
public override List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories) =>
downloads
?.Cast<TorrentStatus>()
.Where(x => categories.Any(cat => cat.Name.Equals(x.Label, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
public override List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories) =>
downloads
?.Cast<TorrentStatus>()
.Where(x => !string.IsNullOrEmpty(x.Hash))
.Where(x => categories.Any(cat => cat.Equals(x.Label, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override async Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean, HashSet<string> excludedHashes,
IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
foreach (TorrentStatus download in downloads)
{
if (string.IsNullOrEmpty(download.Hash))
@@ -214,19 +254,25 @@ public class DelugeService : DownloadService, IDelugeService
continue;
}
Category? category = categoriesToClean
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
CleanCategory? category = categoriesToClean
.FirstOrDefault(x => x.Name.Equals(download.Label, StringComparison.InvariantCultureIgnoreCase));
if (category is null)
{
continue;
}
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
if (!_downloadCleanerConfig.DeletePrivate && download.Private)
{
@@ -258,7 +304,107 @@ public class DelugeService : DownloadService, IDelugeService
await _notifier.NotifyDownloadCleaned(download.Ratio, seedingTime, category.Name, result.Reason);
}
}
public override async Task CreateCategoryAsync(string name)
{
IReadOnlyList<string> existingLabels = await _client.GetLabels();
if (existingLabels.Contains(name, StringComparer.InvariantCultureIgnoreCase))
{
return;
}
await _dryRunInterceptor.InterceptAsync(CreateLabel, name);
}
public override async Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
if (!string.IsNullOrEmpty(_downloadCleanerConfig.UnlinkedIgnoredRootDir))
{
_hardLinkFileService.PopulateFileCounts(_downloadCleanerConfig.UnlinkedIgnoredRootDir);
}
foreach (TorrentStatus download in downloads.Cast<TorrentStatus>())
{
if (string.IsNullOrEmpty(download.Hash) || string.IsNullOrEmpty(download.Name) || string.IsNullOrEmpty(download.Label))
{
continue;
}
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
ContextProvider.Set("downloadName", download.Name);
ContextProvider.Set("hash", download.Hash);
DelugeContents? contents = null;
try
{
contents = await _client.GetTorrentFiles(download.Hash);
}
catch (Exception exception)
{
_logger.LogDebug(exception, "failed to find torrent files for {name}", download.Name);
continue;
}
bool hasHardlinks = false;
ProcessFiles(contents?.Contents, (_, file) =>
{
string filePath = string.Join(Path.DirectorySeparatorChar, Path.Combine(download.DownloadLocation, file.Path).Split(['\\', '/']));
if (file.Priority <= 0)
{
_logger.LogDebug("skip | file is not downloaded | {file}", filePath);
return;
}
long hardlinkCount = _hardLinkFileService.GetHardLinkCount(filePath, !string.IsNullOrEmpty(_downloadCleanerConfig.UnlinkedIgnoredRootDir));
if (hardlinkCount < 0)
{
_logger.LogDebug("skip | could not get file properties | {file}", filePath);
hasHardlinks = true;
return;
}
if (hardlinkCount > 0)
{
hasHardlinks = true;
}
});
if (hasHardlinks)
{
_logger.LogDebug("skip | download has hardlinks | {name}", download.Name);
continue;
}
await _dryRunInterceptor.InterceptAsync(ChangeLabel, download.Hash, _downloadCleanerConfig.UnlinkedTargetCategory);
_logger.LogInformation("category changed for {name}", download.Name);
await _notifier.NotifyCategoryChanged(download.Label, _downloadCleanerConfig.UnlinkedTargetCategory);
download.Label = _downloadCleanerConfig.UnlinkedTargetCategory;
}
}
/// <inheritdoc/>
[DryRunSafeguard]
public override async Task DeleteDownload(string hash)
@@ -267,6 +413,12 @@ public class DelugeService : DownloadService, IDelugeService
await _client.DeleteTorrents([hash]);
}
[DryRunSafeguard]
protected async Task CreateLabel(string name)
{
await _client.CreateLabel(name);
}
[DryRunSafeguard]
protected virtual async Task ChangeFilesPriority(string hash, List<int> sortedPriorities)
@@ -274,33 +426,39 @@ public class DelugeService : DownloadService, IDelugeService
await _client.ChangeFilesPriority(hash, sortedPriorities);
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentStatus status)
[DryRunSafeguard]
protected virtual async Task ChangeLabel(string hash, string newLabel)
{
await _client.SetTorrentLabel(hash, newLabel);
}
private async Task<(bool, DeleteReason)> IsItemStuckAndShouldRemove(TorrentStatus status)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, default);
}
if (_queueCleanerConfig.StalledIgnorePrivate && status.Private)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", status.Name);
return false;
return (false, default);
}
if (status.State is null || !status.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase))
{
return false;
return (false, default);
}
if (status.Eta > 0)
{
return false;
return (false, default);
}
ResetStrikesOnProgress(status.Hash!, status.TotalDone);
return await StrikeAndCheckLimit(status.Hash!, status.Name!);
return (await StrikeAndCheckLimit(status.Hash!, status.Name!, StrikeType.Stalled), DeleteReason.Stalled);
}
private static void ProcessFiles(Dictionary<string, DelugeFileOrDirectory>? contents, Action<string, DelugeFileOrDirectory> processFile)
@@ -1,4 +1,4 @@
using System.Collections.Concurrent;
using System.Collections.Concurrent;
using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
@@ -10,6 +10,7 @@ using Infrastructure.Helpers;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -30,6 +31,7 @@ public abstract class DownloadService : IDownloadService
protected readonly MemoryCacheEntryOptions _cacheOptions;
protected readonly INotificationPublisher _notifier;
protected readonly IDryRunInterceptor _dryRunInterceptor;
protected readonly IHardLinkFileService _hardLinkFileService;
protected DownloadService(
ILogger<DownloadService> logger,
@@ -40,7 +42,8 @@ public abstract class DownloadService : IDownloadService
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
)
{
_logger = logger;
@@ -52,6 +55,7 @@ public abstract class DownloadService : IDownloadService
_striker = striker;
_notifier = notifier;
_dryRunInterceptor = dryRunInterceptor;
_hardLinkFileService = hardLinkFileService;
_cacheOptions = new MemoryCacheEntryOptions()
.SetSlidingExpiration(StaticConfiguration.TriggerValue + Constants.CacheLimitBuffer);
}
@@ -60,25 +64,35 @@ public abstract class DownloadService : IDownloadService
public abstract Task LoginAsync();
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
public abstract Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public abstract Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
);
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task DeleteDownload(string hash);
/// <inheritdoc/>
public abstract Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
public abstract Task<List<object>?> GetSeedingDownloads();
/// <inheritdoc/>
public abstract List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories);
/// <inheritdoc/>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
public abstract List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories);
/// <inheritdoc/>
public abstract Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads);
/// <inheritdoc/>
public abstract Task CreateCategoryAsync(string name);
protected void ResetStrikesOnProgress(string hash, long downloaded)
{
if (!_queueCleanerConfig.StalledResetStrikesOnProgress)
@@ -101,13 +115,14 @@ public abstract class DownloadService : IDownloadService
/// </summary>
/// <param name="hash">The torrent hash.</param>
/// <param name="itemName">The name or title of the item.</param>
/// <param name="strikeType"></param>
/// <returns>True if the limit has been reached; otherwise, false.</returns>
protected async Task<bool> StrikeAndCheckLimit(string hash, string itemName)
protected async Task<bool> StrikeAndCheckLimit(string hash, string itemName, StrikeType strikeType)
{
return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled);
return await _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, strikeType);
}
protected SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, Category category)
protected SeedingCheckResult ShouldCleanDownload(double ratio, TimeSpan seedingTime, CleanCategory category)
{
// check ratio
if (DownloadReachedRatio(ratio, seedingTime, category))
@@ -132,7 +147,27 @@ public abstract class DownloadService : IDownloadService
return new();
}
private bool DownloadReachedRatio(double ratio, TimeSpan seedingTime, Category category)
protected string? GetRootWithFirstDirectory(string path)
{
if (string.IsNullOrWhiteSpace(path))
{
return null;
}
string? root = Path.GetPathRoot(path);
if (root is null)
{
return null;
}
string relativePath = path[root.Length..].TrimStart(Path.DirectorySeparatorChar);
string[] parts = relativePath.Split(Path.DirectorySeparatorChar, StringSplitOptions.RemoveEmptyEntries);
return parts.Length > 0 ? Path.Combine(root, parts[0]) : root;
}
private bool DownloadReachedRatio(double ratio, TimeSpan seedingTime, CleanCategory category)
{
if (category.MaxRatio < 0)
{
@@ -158,7 +193,7 @@ public abstract class DownloadService : IDownloadService
return true;
}
private bool DownloadReachedMaxSeedTime(TimeSpan seedingTime, Category category)
private bool DownloadReachedMaxSeedTime(TimeSpan seedingTime, CleanCategory category)
{
if (category.MaxSeedTime < 0)
{
@@ -25,6 +25,7 @@ public sealed class DownloadServiceFactory
Common.Enums.DownloadClient.Deluge => _serviceProvider.GetRequiredService<DelugeService>(),
Common.Enums.DownloadClient.Transmission => _serviceProvider.GetRequiredService<TransmissionService>(),
Common.Enums.DownloadClient.None => _serviceProvider.GetRequiredService<DummyDownloadService>(),
Common.Enums.DownloadClient.Disabled => _serviceProvider.GetRequiredService<DummyDownloadService>(),
_ => throw new ArgumentOutOfRangeException()
};
}
@@ -1,10 +1,11 @@
using System.Collections.Concurrent;
using System.Collections.Concurrent;
using System.Text.RegularExpressions;
using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.QueueCleaner;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -15,7 +16,21 @@ namespace Infrastructure.Verticals.DownloadClient;
public class DummyDownloadService : DownloadService
{
public DummyDownloadService(ILogger<DownloadService> logger, IOptions<QueueCleanerConfig> queueCleanerConfig, IOptions<ContentBlockerConfig> contentBlockerConfig, IOptions<DownloadCleanerConfig> downloadCleanerConfig, IMemoryCache cache, IFilenameEvaluator filenameEvaluator, IStriker striker, INotificationPublisher notifier, IDryRunInterceptor dryRunInterceptor) : base(logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache, filenameEvaluator, striker, notifier, dryRunInterceptor)
public DummyDownloadService(
ILogger<DownloadService> logger,
IOptions<QueueCleanerConfig> queueCleanerConfig,
IOptions<ContentBlockerConfig> contentBlockerConfig,
IOptions<DownloadCleanerConfig> downloadCleanerConfig,
IMemoryCache cache,
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig,
cache, filenameEvaluator, striker, notifier, dryRunInterceptor, hardLinkFileService
)
{
}
@@ -28,22 +43,43 @@ public class DummyDownloadService : DownloadService
return Task.CompletedTask;
}
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns, ConcurrentBag<Regex> regexes)
public override Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash, BlocklistType blocklistType, ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories)
public override Task<List<object>?> GetSeedingDownloads()
{
throw new NotImplementedException();
}
public override Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories)
{
throw new NotImplementedException();
}
public override List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories)
{
throw new NotImplementedException();
}
public override Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
throw new NotImplementedException();
}
public override Task CreateCategoryAsync(string name)
{
throw new NotImplementedException();
}
@@ -14,7 +14,8 @@ public interface IDownloadService : IDisposable
/// Checks whether the download should be removed from the *arr queue.
/// </summary>
/// <param name="hash">The download hash.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash);
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
public Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Blocks unwanted files from being fully downloaded.
@@ -23,31 +24,62 @@ public interface IDownloadService : IDisposable
/// <param name="blocklistType">The <see cref="BlocklistType"/>.</param>
/// <param name="patterns">The patterns to test the files against.</param>
/// <param name="regexes">The regexes to test the files against.</param>
/// <param name="ignoredDownloads">Downloads to ignore from processing.</param>
/// <returns>True if all files have been blocked; otherwise false.</returns>
public Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
);
/// <summary>
/// Fetches all downloads.
/// Fetches all seeding downloads.
/// </summary>
/// <returns>A list of downloads that are seeding.</returns>
Task<List<object>?> GetSeedingDownloads();
/// <summary>
/// Filters downloads that should be cleaned.
/// </summary>
/// <param name="downloads">The downloads to filter.</param>
/// <param name="categories">The categories by which to filter the downloads.</param>
/// <returns>A list of downloads for the provided categories.</returns>
Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories);
List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories);
/// <summary>
/// Filters downloads that should have their category changed.
/// </summary>
/// <param name="downloads">The downloads to filter.</param>
/// <param name="categories">The categories by which to filter the downloads.</param>
/// <returns>A list of downloads for the provided categories.</returns>
List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories);
/// <summary>
/// Cleans the downloads.
/// </summary>
/// <param name="downloads"></param>
/// <param name="downloads">The downloads to clean.</param>
/// <param name="categoriesToClean">The categories that should be cleaned.</param>
/// <param name="excludedHashes">The hashes that should not be cleaned.</param>
public abstract Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes);
/// <param name="ignoredDownloads">The downloads to ignore from processing.</param>
Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Changes the category for downloads that have no hardlinks.
/// </summary>
/// <param name="downloads">The downloads to change.</param>
/// <param name="excludedHashes">The hashes that should not be cleaned.</param>
/// <param name="ignoredDownloads">The downloads to ignore from processing.</param>
Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads);
/// <summary>
/// Deletes a download item.
/// </summary>
public Task DeleteDownload(string hash);
/// <summary>
/// Creates a category.
/// </summary>
/// <param name="name">The category name.</param>
public Task CreateCategoryAsync(string name);
}
@@ -5,18 +5,20 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.Exceptions;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using QBittorrent.Client;
using Category = Common.Configuration.DownloadCleaner.Category;
namespace Infrastructure.Verticals.DownloadClient.QBittorrent;
@@ -36,15 +38,20 @@ public class QBitService : DownloadService, IQBitService
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
filenameEvaluator, striker, notifier, dryRunInterceptor, hardLinkFileService
)
{
_config = config.Value;
_config.Validate();
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), _config.Url);
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? uriBuilder.Path
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/')}";
_client = new(httpClientFactory.CreateClient(Constants.HttpClientWithRetryName), uriBuilder.Uri);
}
public override async Task LoginAsync()
@@ -58,18 +65,27 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -82,52 +98,57 @@ public class QBitService : DownloadService, IQBitService
bool.TryParse(dictValue?.ToString(), out bool boolValue)
&& boolValue;
// if all files were blocked by qBittorrent
if (torrent is { CompletionOn: not null, Downloaded: null or 0 })
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
IReadOnlyList<TorrentContent>? files = await _client.GetTorrentContentsAsync(hash);
// if all files are marked as skip
if (files?.Count is > 0 && files.All(x => x.Priority is TorrentContentPriority.Skip))
{
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
// if all files were blocked by qBittorrent
if (download is { CompletionOn: not null, Downloaded: null or 0 })
{
result.DeleteReason = DeleteReason.AllFilesSkippedByQBit;
return result;
}
// remove if all files are unwanted
result.DeleteReason = DeleteReason.AllFilesSkipped;
return result;
}
result.ShouldRemove = await IsItemStuckAndShouldRemove(torrent, result.IsPrivate);
if (result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await IsItemStuckAndShouldRemove(download, result.IsPrivate);
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
ConcurrentBag<Regex> regexes,
IReadOnlyList<string> ignoredDownloads
)
{
TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
TorrentInfo? download = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] }))
.FirstOrDefault();
BlockFilesResult result = new();
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads)) is true))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
return result;
}
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(hash);
if (torrentProperties is null)
@@ -145,7 +166,7 @@ public class QBitService : DownloadService, IQBitService
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -207,19 +228,42 @@ public class QBitService : DownloadService, IQBitService
}
/// <inheritdoc/>
public override async Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories) =>
public override async Task<List<object>?> GetSeedingDownloads() =>
(await _client.GetTorrentListAsync(new()
{
Filter = TorrentListFilter.Seeding
}))
?.Where(x => !string.IsNullOrEmpty(x.Hash))
.Where(x => categories.Any(cat => cat.Name.Equals(x.Category, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories) =>
downloads
?.Cast<TorrentInfo>()
.Where(x => !string.IsNullOrEmpty(x.Hash))
.Where(x => categories.Any(cat => cat.Name.Equals(x.Category, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
/// <inheritdoc/>
public override List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories) =>
downloads
?.Cast<TorrentInfo>()
.Where(x => !string.IsNullOrEmpty(x.Hash))
.Where(x => categories.Any(cat => cat.Equals(x.Category, StringComparison.InvariantCultureIgnoreCase)))
.Cast<object>()
.ToList();
/// <inheritdoc/>
public override async Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean,
HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
foreach (TorrentInfo download in downloads)
{
if (string.IsNullOrEmpty(download.Hash))
@@ -227,7 +271,22 @@ public class QBitService : DownloadService, IQBitService
continue;
}
Category? category = categoriesToClean
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(download.Hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads))))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
CleanCategory? category = categoriesToClean
.FirstOrDefault(x => download.Category.Equals(x.Name, StringComparison.InvariantCultureIgnoreCase));
if (category is null)
@@ -235,12 +294,6 @@ public class QBitService : DownloadService, IQBitService
continue;
}
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
if (!_downloadCleanerConfig.DeletePrivate)
{
TorrentProperties? torrentProperties = await _client.GetTorrentPropertiesAsync(download.Hash);
@@ -286,12 +339,125 @@ public class QBitService : DownloadService, IQBitService
}
}
public override async Task CreateCategoryAsync(string name)
{
IReadOnlyDictionary<string, Category>? existingCategories = await _client.GetCategoriesAsync();
if (existingCategories.Any(x => x.Value.Name.Equals(name, StringComparison.InvariantCultureIgnoreCase)))
{
return;
}
await _dryRunInterceptor.InterceptAsync(CreateCategory, name);
}
public override async Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
if (!string.IsNullOrEmpty(_downloadCleanerConfig.UnlinkedIgnoredRootDir))
{
_hardLinkFileService.PopulateFileCounts(_downloadCleanerConfig.UnlinkedIgnoredRootDir);
}
foreach (TorrentInfo download in downloads)
{
if (string.IsNullOrEmpty(download.Hash))
{
continue;
}
if (excludedHashes.Any(x => x.Equals(download.Hash, StringComparison.InvariantCultureIgnoreCase)))
{
_logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
continue;
}
IReadOnlyList<TorrentTracker> trackers = await GetTrackersAsync(download.Hash);
if (ignoredDownloads.Count > 0 &&
(download.ShouldIgnore(ignoredDownloads) || trackers.Any(x => x.ShouldIgnore(ignoredDownloads))))
{
_logger.LogInformation("skip | download is ignored | {name}", download.Name);
continue;
}
IReadOnlyList<TorrentContent>? files = await _client.GetTorrentContentsAsync(download.Hash);
if (files is null)
{
_logger.LogDebug("failed to find files for {name}", download.Name);
continue;
}
ContextProvider.Set("downloadName", download.Name);
ContextProvider.Set("hash", download.Hash);
bool hasHardlinks = false;
foreach (TorrentContent file in files)
{
if (!file.Index.HasValue)
{
_logger.LogDebug("skip | file index is null for {name}", download.Name);
hasHardlinks = true;
break;
}
string filePath = string.Join(Path.DirectorySeparatorChar, Path.Combine(download.SavePath, file.Name).Split(['\\', '/']));
if (file.Priority is TorrentContentPriority.Skip)
{
_logger.LogDebug("skip | file is not downloaded | {file}", filePath);
continue;
}
long hardlinkCount = _hardLinkFileService.GetHardLinkCount(filePath, !string.IsNullOrEmpty(_downloadCleanerConfig.UnlinkedIgnoredRootDir));
if (hardlinkCount < 0)
{
_logger.LogDebug("skip | could not get file properties | {file}", filePath);
hasHardlinks = true;
break;
}
if (hardlinkCount > 0)
{
hasHardlinks = true;
break;
}
}
if (hasHardlinks)
{
_logger.LogDebug("skip | download has hardlinks | {name}", download.Name);
continue;
}
await _dryRunInterceptor.InterceptAsync(ChangeCategory, download.Hash, _downloadCleanerConfig.UnlinkedTargetCategory);
_logger.LogInformation("category changed for {name}", download.Name);
await _notifier.NotifyCategoryChanged(download.Category, _downloadCleanerConfig.UnlinkedTargetCategory);
download.Category = _downloadCleanerConfig.UnlinkedTargetCategory;
}
}
/// <inheritdoc/>
[DryRunSafeguard]
public override async Task DeleteDownload(string hash)
{
await _client.DeleteAsync(hash, deleteDownloadedData: true);
}
[DryRunSafeguard]
protected async Task CreateCategory(string name)
{
await _client.AddCategoryAsync(name);
}
[DryRunSafeguard]
protected virtual async Task SkipFile(string hash, int fileIndex)
@@ -299,34 +465,52 @@ public class QBitService : DownloadService, IQBitService
await _client.SetFilePriorityAsync(hash, fileIndex, TorrentContentPriority.Skip);
}
[DryRunSafeguard]
protected virtual async Task ChangeCategory(string hash, string newCategory)
{
await _client.SetTorrentCategoryAsync([hash], newCategory);
}
public override void Dispose()
{
_client.Dispose();
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent, bool isPrivate)
private async Task<(bool, DeleteReason)> IsItemStuckAndShouldRemove(TorrentInfo torrent, bool isPrivate)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, default);
}
if (_queueCleanerConfig.StalledIgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
return (false, default);
}
if (torrent.State is not TorrentState.StalledDownload and not TorrentState.FetchingMetadata
and not TorrentState.ForcedFetchingMetadata)
{
// ignore other states
return false;
return (false, default);
}
ResetStrikesOnProgress(torrent.Hash, torrent.Downloaded ?? 0);
return await StrikeAndCheckLimit(torrent.Hash, torrent.Name);
if (torrent.State is TorrentState.StalledDownload)
{
return (await StrikeAndCheckLimit(torrent.Hash, torrent.Name, StrikeType.Stalled), DeleteReason.Stalled);
}
return (await StrikeAndCheckLimit(torrent.Hash, torrent.Name, StrikeType.DownloadingMetadata), DeleteReason.DownloadingMetadata);
}
private async Task<IReadOnlyList<TorrentTracker>> GetTrackersAsync(string hash)
{
return (await _client.GetTorrentTrackersAsync(hash))
.Where(x => !x.Url.ToString().Contains("**"))
.ToList();
}
}
@@ -5,11 +5,14 @@ using Common.Configuration.ContentBlocker;
using Common.Configuration.DownloadCleaner;
using Common.Configuration.DownloadClient;
using Common.Configuration.QueueCleaner;
using Common.Exceptions;
using Common.Helpers;
using Domain.Enums;
using Infrastructure.Extensions;
using Infrastructure.Interceptors;
using Infrastructure.Verticals.ContentBlocker;
using Infrastructure.Verticals.Context;
using Infrastructure.Verticals.Files;
using Infrastructure.Verticals.ItemStriker;
using Infrastructure.Verticals.Notifications;
using Microsoft.Extensions.Caching.Memory;
@@ -25,7 +28,23 @@ public class TransmissionService : DownloadService, ITransmissionService
{
private readonly TransmissionConfig _config;
private readonly Client _client;
private TorrentInfo[]? _torrentsCache;
private static readonly string[] Fields =
[
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER,
TorrentFields.DOWNLOAD_DIR,
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO,
TorrentFields.TRACKERS
];
public TransmissionService(
IHttpClientFactory httpClientFactory,
@@ -38,17 +57,22 @@ public class TransmissionService : DownloadService, ITransmissionService
IFilenameEvaluator filenameEvaluator,
IStriker striker,
INotificationPublisher notifier,
IDryRunInterceptor dryRunInterceptor
IDryRunInterceptor dryRunInterceptor,
IHardLinkFileService hardLinkFileService
) : base(
logger, queueCleanerConfig, contentBlockerConfig, downloadCleanerConfig, cache,
filenameEvaluator, striker, notifier, dryRunInterceptor
filenameEvaluator, striker, notifier, dryRunInterceptor, hardLinkFileService
)
{
_config = config.Value;
_config.Validate();
UriBuilder uriBuilder = new(_config.Url);
uriBuilder.Path = string.IsNullOrEmpty(_config.UrlBase)
? $"{uriBuilder.Path.TrimEnd('/')}/rpc"
: $"{uriBuilder.Path.TrimEnd('/')}/{_config.UrlBase.TrimStart('/').TrimEnd('/')}/rpc";
_client = new(
httpClientFactory.CreateClient(Constants.HttpClientWithRetryName),
new Uri(_config.Url, "/transmission/rpc").ToString(),
uriBuilder.Uri.ToString(),
login: _config.Username,
password: _config.Password
);
@@ -60,21 +84,27 @@ public class TransmissionService : DownloadService, ITransmissionService
}
/// <inheritdoc/>
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash)
public override async Task<StalledResult> ShouldRemoveFromArrQueueAsync(string hash, IReadOnlyList<string> ignoredDownloads)
{
StalledResult result = new();
TorrentInfo? torrent = await GetTorrentAsync(hash);
TorrentInfo? download = await GetTorrentAsync(hash);
if (torrent is null)
if (download is null)
{
_logger.LogDebug("failed to find torrent {hash} in the download client", hash);
return result;
}
bool shouldRemove = torrent.FileStats?.Length > 0;
result.IsPrivate = torrent.IsPrivate ?? false;
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool shouldRemove = download.FileStats?.Length > 0;
result.IsPrivate = download.IsPrivate ?? false;
foreach (TransmissionTorrentFileStats? stats in torrent.FileStats ?? [])
foreach (TransmissionTorrentFileStats? stats in download.FileStats ?? [])
{
if (!stats.Wanted.HasValue)
{
@@ -91,43 +121,45 @@ public class TransmissionService : DownloadService, ITransmissionService
if (shouldRemove)
{
// remove if all files are unwanted
result.ShouldRemove = true;
result.DeleteReason = DeleteReason.AllFilesBlocked;
return result;
}
// remove if all files are unwanted or download is stuck
result.ShouldRemove = shouldRemove || await IsItemStuckAndShouldRemove(torrent);
// remove if download is stuck
(result.ShouldRemove, result.DeleteReason) = await IsItemStuckAndShouldRemove(download);
if (!shouldRemove && result.ShouldRemove)
{
result.DeleteReason = DeleteReason.Stalled;
}
return result;
}
/// <inheritdoc/>
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(
string hash,
public override async Task<BlockFilesResult> BlockUnwantedFilesAsync(string hash,
BlocklistType blocklistType,
ConcurrentBag<string> patterns,
ConcurrentBag<Regex> regexes
)
ConcurrentBag<Regex> regexes, IReadOnlyList<string> ignoredDownloads)
{
TorrentInfo? torrent = await GetTorrentAsync(hash);
TorrentInfo? download = await GetTorrentAsync(hash);
BlockFilesResult result = new();
if (torrent?.FileStats is null || torrent.Files is null)
if (download?.FileStats is null || download.Files is null)
{
return result;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
return result;
}
bool isPrivate = torrent.IsPrivate ?? false;
bool isPrivate = download.IsPrivate ?? false;
result.IsPrivate = isPrivate;
if (_contentBlockerConfig.IgnorePrivate && isPrivate)
{
// ignore private trackers
_logger.LogDebug("skip files check | download is private | {name}", torrent.Name);
_logger.LogDebug("skip files check | download is private | {name}", download.Name);
return result;
}
@@ -135,27 +167,27 @@ public class TransmissionService : DownloadService, ITransmissionService
long totalFiles = 0;
long totalUnwantedFiles = 0;
for (int i = 0; i < torrent.Files.Length; i++)
for (int i = 0; i < download.Files.Length; i++)
{
if (torrent.FileStats?[i].Wanted == null)
if (download.FileStats?[i].Wanted == null)
{
continue;
}
totalFiles++;
if (!torrent.FileStats[i].Wanted.Value)
if (!download.FileStats[i].Wanted.Value)
{
totalUnwantedFiles++;
continue;
}
if (_filenameEvaluator.IsValid(torrent.Files[i].Name, blocklistType, patterns, regexes))
if (_filenameEvaluator.IsValid(download.Files[i].Name, blocklistType, patterns, regexes))
{
continue;
}
_logger.LogInformation("unwanted file found | {file}", torrent.Files[i].Name);
_logger.LogInformation("unwanted file found | {file}", download.Files[i].Name);
unwantedFiles.Add(i);
totalUnwantedFiles++;
}
@@ -175,13 +207,12 @@ public class TransmissionService : DownloadService, ITransmissionService
_logger.LogDebug("changing priorities | torrent {hash}", hash);
await _dryRunInterceptor.InterceptAsync(SetUnwantedFiles, torrent.Id, unwantedFiles.ToArray());
await _dryRunInterceptor.InterceptAsync(SetUnwantedFiles, download.Id, unwantedFiles.ToArray());
return result;
}
/// <inheritdoc/>
public override async Task<List<object>?> GetAllDownloadsToBeCleaned(List<Category> categories)
public override async Task<List<object>?> GetSeedingDownloads()
{
string[] fields = [
TorrentFields.FILES,
@@ -197,11 +228,21 @@ public class TransmissionService : DownloadService, ITransmissionService
TorrentFields.SECONDS_SEEDING,
TorrentFields.UPLOAD_RATIO
];
return (await _client.TorrentGetAsync(fields))
?.Torrents
?.Where(x => !string.IsNullOrEmpty(x.HashString))
.Where(x => x.Status is 5 or 6)
.Cast<object>()
.ToList();
}
/// <inheritdoc/>
public override List<object>? FilterDownloadsToBeCleanedAsync(List<object>? downloads, List<CleanCategory> categories)
{
return downloads
?
.Cast<TorrentInfo>()
.Where(x => categories
.Any(cat =>
{
@@ -218,17 +259,34 @@ public class TransmissionService : DownloadService, ITransmissionService
.ToList();
}
/// <inheritdoc/>
public override async Task CleanDownloads(List<object> downloads, List<Category> categoriesToClean, HashSet<string> excludedHashes)
public override List<object>? FilterDownloadsToChangeCategoryAsync(List<object>? downloads, List<string> categories)
{
throw new NotImplementedException();
}
/// <inheritdoc/>
public override async Task CleanDownloadsAsync(List<object>? downloads, List<CleanCategory> categoriesToClean,
HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
foreach (TorrentInfo download in downloads)
{
if (string.IsNullOrEmpty(download.HashString))
{
continue;
}
if (ignoredDownloads.Count > 0 && download.ShouldIgnore(ignoredDownloads))
{
_logger.LogDebug("skip | download is ignored | {name}", download.Name);
continue;
}
Category? category = categoriesToClean
CleanCategory? category = categoriesToClean
.FirstOrDefault(x =>
{
if (download.DownloadDir is null)
@@ -282,6 +340,138 @@ public class TransmissionService : DownloadService, ITransmissionService
}
}
public override async Task CreateCategoryAsync(string name)
{
throw new NotImplementedException();
}
public override async Task ChangeCategoryForNoHardLinksAsync(List<object>? downloads, HashSet<string> excludedHashes, IReadOnlyList<string> ignoredDownloads)
{
if (downloads?.Count is null or 0)
{
return;
}
// TODO ignored downloads
throw new NotImplementedException();
// if (_downloadCleanerConfig.NoHardLinksIgnoreRootDir)
// {
// downloads
// .Cast<TorrentInfo>()
// .Select(x =>
// {
// if (x.DownloadDir == null)
// {
// return string.Empty;
// }
//
// string? firstDir = GetRootWithFirstDirectory(x.DownloadDir);
//
// if (string.IsNullOrEmpty(firstDir))
// {
// return string.Empty;
// }
//
// if (firstDir == Path.GetPathRoot(x.DownloadDir))
// {
// return string.Empty;
// }
//
// return firstDir;
// })
// .Where(x => !string.IsNullOrEmpty(x))
// .Distinct()
// .ToList()
// .ForEach(x =>
// {
// _logger.LogTrace("populating file counts from {dir}", x);
//
// if (!Directory.Exists(x))
// {
// throw new ValidationException($"directory \"{x}\" does not exist");
// }
//
// _hardLinkFileService.PopulateFileCounts(x);
// });
// }
//
// foreach (TorrentInfo download in downloads.Cast<TorrentInfo>())
// {
// if (string.IsNullOrEmpty(download.HashString) || download.DownloadDir == null)
// {
// _logger.LogDebug("skip | download hash or download directory is null for {name}", download.Name);
// continue;
// }
//
// if (excludedHashes.Any(x => x.Equals(download.HashString, StringComparison.InvariantCultureIgnoreCase)))
// {
// _logger.LogDebug("skip | download is used by an arr | {name}", download.Name);
// continue;
// }
//
// ContextProvider.Set("downloadName", download.Name);
// ContextProvider.Set("hash", download.HashString);
//
// bool hasHardlinks = false;
//
// if (download.Files != null)
// {
// foreach (TransmissionTorrentFiles file in download.Files)
// {
// string filePath = Path.Combine(download.DownloadDir, file.Name);
//
// long hardlinkCount = _hardLinkFileService.GetHardLinkCount(filePath, _downloadCleanerConfig.NoHardLinksIgnoreRootDir);
//
// if (hardlinkCount < 0)
// {
// _logger.LogDebug("skip | could not get file properties | {name}", download.Name);
// hasHardlinks = true;
// break;
// }
//
// if (hardlinkCount > 0)
// {
// hasHardlinks = true;
// break;
// }
// }
// }
//
// if (hasHardlinks)
// {
// _logger.LogDebug("skip | download has hardlinks | {name}", download.Name);
// continue;
// }
//
// // Get the current category (directory name)
// string currentCategory = Path.GetFileName(Path.TrimEndingDirectorySeparator(download.DownloadDir));
//
// // Create the new location path
// string newLocation = Path.Combine(
// Path.GetDirectoryName(Path.TrimEndingDirectorySeparator(download.DownloadDir)) ?? string.Empty,
// _downloadCleanerConfig.NoHardLinksCategory
// );
//
// await _dryRunInterceptor.InterceptAsync(MoveDownload, download.Id, newLocation);
//
// _logger.LogInformation("category changed for {name}", download.Name);
//
// await _notifier.NotifyCategoryChanged(currentCategory, _downloadCleanerConfig.NoHardLinksCategory);
// }
}
[DryRunSafeguard]
protected virtual async Task MoveDownload(long downloadId, string newLocation)
{
await _client.TorrentSetAsync(new TorrentSettings
{
Ids = [downloadId],
Location = newLocation,
// Move = true
});
}
public override async Task DeleteDownload(string hash)
{
TorrentInfo? torrent = await GetTorrentAsync(hash);
@@ -314,72 +504,38 @@ public class TransmissionService : DownloadService, ITransmissionService
});
}
private async Task<bool> IsItemStuckAndShouldRemove(TorrentInfo torrent)
private async Task<(bool, DeleteReason)> IsItemStuckAndShouldRemove(TorrentInfo torrent)
{
if (_queueCleanerConfig.StalledMaxStrikes is 0)
{
return false;
return (false, default);
}
if (_queueCleanerConfig.StalledIgnorePrivate && (torrent.IsPrivate ?? false))
{
// ignore private trackers
_logger.LogDebug("skip stalled check | download is private | {name}", torrent.Name);
return false;
return (false, default);
}
if (torrent.Status is not 4)
{
// not in downloading state
return false;
return (false, default);
}
if (torrent.Eta > 0)
{
return false;
return (false, default);
}
ResetStrikesOnProgress(torrent.HashString!, torrent.DownloadedEver ?? 0);
return await StrikeAndCheckLimit(torrent.HashString!, torrent.Name!);
return (await StrikeAndCheckLimit(torrent.HashString!, torrent.Name!, StrikeType.Stalled), DeleteReason.Stalled);
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash)
{
TorrentInfo? torrent = _torrentsCache?
.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (_torrentsCache is null || torrent is null)
{
string[] fields = [
TorrentFields.FILES,
TorrentFields.FILE_STATS,
TorrentFields.HASH_STRING,
TorrentFields.ID,
TorrentFields.ETA,
TorrentFields.NAME,
TorrentFields.STATUS,
TorrentFields.IS_PRIVATE,
TorrentFields.DOWNLOADED_EVER
];
// refresh cache
_torrentsCache = (await _client.TorrentGetAsync(fields))
?.Torrents;
}
if (_torrentsCache?.Length is null or 0)
{
_logger.LogDebug("could not list torrents | {url}", _config.Url);
}
torrent = _torrentsCache?.FirstOrDefault(x => x.HashString.Equals(hash, StringComparison.InvariantCultureIgnoreCase));
if (torrent is null)
{
_logger.LogDebug("could not find torrent | {hash} | {url}", hash, _config.Url);
}
return torrent;
}
private async Task<TorrentInfo?> GetTorrentAsync(string hash) =>
(await _client.TorrentGetAsync(Fields, hash))
?.Torrents
?.FirstOrDefault();
}
@@ -0,0 +1,51 @@
using System.Runtime.InteropServices;
using Microsoft.Extensions.Logging;
namespace Infrastructure.Verticals.Files;
public class HardLinkFileService : IHardLinkFileService
{
private readonly ILogger<HardLinkFileService> _logger;
private readonly UnixHardLinkFileService _unixHardLinkFileService;
private readonly WindowsHardLinkFileService _windowsHardLinkFileService;
public HardLinkFileService(
ILogger<HardLinkFileService> logger,
UnixHardLinkFileService unixHardLinkFileService,
WindowsHardLinkFileService windowsHardLinkFileService
)
{
_logger = logger;
_unixHardLinkFileService = unixHardLinkFileService;
_windowsHardLinkFileService = windowsHardLinkFileService;
}
public void PopulateFileCounts(string directoryPath)
{
_logger.LogTrace("populating file counts from {dir}", directoryPath);
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
_windowsHardLinkFileService.PopulateFileCounts(directoryPath);
return;
}
_unixHardLinkFileService.PopulateFileCounts(directoryPath);
}
public long GetHardLinkCount(string filePath, bool ignoreRootDir)
{
if (!File.Exists(filePath))
{
_logger.LogDebug("file {file} does not exist", filePath);
return -1;
}
if (RuntimeInformation.IsOSPlatform(OSPlatform.Windows))
{
return _windowsHardLinkFileService.GetHardLinkCount(filePath, ignoreRootDir);
}
return _unixHardLinkFileService.GetHardLinkCount(filePath, ignoreRootDir);
}
}
@@ -0,0 +1,19 @@
namespace Infrastructure.Verticals.Files;
public interface IHardLinkFileService
{
/// <summary>
/// Populates the inode counts for Unix and the file index counts for Windows.
/// Needs to be called before <see cref="GetHardLinkCount"/> to populate the inode counts.
/// </summary>
/// <param name="directoryPath">The root directory where to search for hardlinks.</param>
void PopulateFileCounts(string directoryPath);
/// <summary>
/// Get the hardlink count of a file.
/// </summary>
/// <param name="filePath">File path.</param>
/// <param name="ignoreRootDir">Whether to ignore hardlinks found in the same root dir.</param>
/// <returns>-1 on error, 0 if there are no hardlinks and 1 otherwise.</returns>
long GetHardLinkCount(string filePath, bool ignoreRootDir);
}
@@ -0,0 +1,87 @@
using System.Collections.Concurrent;
using Microsoft.Extensions.Logging;
using Mono.Unix.Native;
namespace Infrastructure.Verticals.Files;
public class UnixHardLinkFileService : IHardLinkFileService, IDisposable
{
private readonly ILogger<UnixHardLinkFileService> _logger;
private readonly ConcurrentDictionary<ulong, int> _inodeCounts = new();
public UnixHardLinkFileService(ILogger<UnixHardLinkFileService> logger)
{
_logger = logger;
}
/// <inheritdoc/>
public long GetHardLinkCount(string filePath, bool ignoreRootDir)
{
try
{
if (Syscall.stat(filePath, out Stat stat) != 0)
{
_logger.LogDebug("failed to stat file {file}", filePath);
return -1;
}
if (!ignoreRootDir)
{
_logger.LogDebug("stat file | hardlinks: {nlink} | {file}", stat.st_nlink, filePath);
return (long)stat.st_nlink == 1 ? 0 : 1;
}
// get the number of hardlinks in the same root directory
int linksInIgnoredDir = _inodeCounts.TryGetValue(stat.st_ino, out int count)
? count
: 1; // default to 1 if not found
_logger.LogDebug("stat file | hardlinks: {nlink} | ignored: {ignored} | {file}", stat.st_nlink, linksInIgnoredDir, filePath);
return (long)stat.st_nlink - linksInIgnoredDir;
}
catch (Exception exception)
{
_logger.LogError(exception, "failed to stat file {file}", filePath);
return -1;
}
}
/// <inheritdoc/>
public void PopulateFileCounts(string directoryPath)
{
try
{
// traverse all files in the ignored path and subdirectories
foreach (string file in Directory.EnumerateFiles(directoryPath, "*", SearchOption.AllDirectories))
{
AddInodeToCount(file);
}
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to populate inode counts from {dir}", directoryPath);
throw;
}
}
private void AddInodeToCount(string path)
{
try
{
if (Syscall.stat(path, out Stat stat) == 0)
{
_inodeCounts.AddOrUpdate(stat.st_ino, 1, (_, count) => count + 1);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "could not stat {path} during inode counting", path);
throw;
}
}
public void Dispose()
{
_inodeCounts.Clear();
}
}
@@ -0,0 +1,113 @@
using System.Collections.Concurrent;
using System.Runtime.InteropServices;
using Microsoft.Extensions.Logging;
using Microsoft.Win32.SafeHandles;
namespace Infrastructure.Verticals.Files;
public class WindowsHardLinkFileService : IHardLinkFileService, IDisposable
{
private readonly ILogger<WindowsHardLinkFileService> _logger;
private readonly ConcurrentDictionary<ulong, int> _fileIndexCounts = new();
public WindowsHardLinkFileService(ILogger<WindowsHardLinkFileService> logger)
{
_logger = logger;
}
/// <inheritdoc/>
public long GetHardLinkCount(string filePath, bool ignoreRootDir)
{
try
{
using SafeFileHandle fileStream = File.OpenHandle(filePath);
if (!GetFileInformationByHandle(fileStream, out var file))
{
_logger.LogDebug("failed to get file handle {file}", filePath);
return -1;
}
if (!ignoreRootDir)
{
_logger.LogDebug("stat file | hardlinks: {nlink} | {file}", file.NumberOfLinks, filePath);
return file.NumberOfLinks == 1 ? 0 : 1;
}
// Get unique file ID (combination of high and low indices)
ulong fileIndex = ((ulong)file.FileIndexHigh << 32) | file.FileIndexLow;
// get the number of hardlinks in the same root directory
int linksInIgnoredDir = _fileIndexCounts.TryGetValue(fileIndex, out int count)
? count
: 1; // default to 1 if not found
_logger.LogDebug("stat file | hardlinks: {links} | ignored: {ignored} | {file}", file.NumberOfLinks, linksInIgnoredDir, filePath);
return file.NumberOfLinks - linksInIgnoredDir;
}
catch (Exception exception)
{
_logger.LogError(exception, "failed to stat file {file}", filePath);
return -1;
}
}
/// <inheritdoc/>
public void PopulateFileCounts(string directoryPath)
{
try
{
// traverse all files in the ignored path and subdirectories
foreach (string file in Directory.EnumerateFiles(directoryPath, "*", SearchOption.AllDirectories))
{
AddFileIndexToCount(file);
}
}
catch (Exception ex)
{
_logger.LogError(ex, "Failed to populate file index counts from {dir}", directoryPath);
}
}
private void AddFileIndexToCount(string path)
{
try
{
using SafeFileHandle fileStream = File.OpenHandle(path);
if (GetFileInformationByHandle(fileStream, out var file))
{
ulong fileIndex = ((ulong)file.FileIndexHigh << 32) | file.FileIndexLow;
_fileIndexCounts.AddOrUpdate(fileIndex, 1, (_, count) => count + 1);
}
}
catch (Exception ex)
{
_logger.LogWarning(ex, "Couldn't stat {path} during file index counting", path);
}
}
[DllImport("kernel32.dll", SetLastError = true)]
private static extern bool GetFileInformationByHandle(
SafeFileHandle hFile,
out BY_HANDLE_FILE_INFORMATION lpFileInformation
);
private struct BY_HANDLE_FILE_INFORMATION
{
public uint FileAttributes;
public System.Runtime.InteropServices.ComTypes.FILETIME CreationTime;
public System.Runtime.InteropServices.ComTypes.FILETIME LastAccessTime;
public System.Runtime.InteropServices.ComTypes.FILETIME LastWriteTime;
public uint VolumeSerialNumber;
public uint FileSizeHigh;
public uint FileSizeLow;
public uint NumberOfLinks;
public uint FileIndexHigh;
public uint FileIndexLow;
}
public void Dispose()
{
_fileIndexCounts.Clear();
}
}
@@ -4,10 +4,12 @@ namespace Infrastructure.Verticals.Jobs;
public class JobChainingListener : IJobListener
{
private readonly string _firstJobName;
private readonly string _nextJobName;
public JobChainingListener(string nextJobName)
public JobChainingListener(string firstJobName, string nextJobName)
{
_firstJobName = firstJobName;
_nextJobName = nextJobName;
}
@@ -19,7 +21,7 @@ public class JobChainingListener : IJobListener
public async Task JobWasExecuted(IJobExecutionContext context, JobExecutionException? jobException, CancellationToken cancellationToken)
{
if (string.IsNullOrEmpty(_nextJobName) || context.JobDetail.Key.Name == _nextJobName)
if (string.IsNullOrEmpty(_nextJobName) || context.JobDetail.Key.Name == _nextJobName || context.JobDetail.Key.Name != _firstJobName)
{
return;
}
@@ -33,6 +33,9 @@ public sealed class NotificationConsumer<T> : IConsumer<T> where T : Notificatio
case DownloadCleanedNotification downloadCleanedNotification:
await _notificationService.Notify(downloadCleanedNotification);
break;
case CategoryChangedNotification categoryChangedNotification:
await _notificationService.Notify(categoryChangedNotification);
break;
default:
throw new NotImplementedException();
}
@@ -9,4 +9,6 @@ public interface INotificationFactory
List<INotificationProvider> OnQueueItemDeletedEnabled();
List<INotificationProvider> OnDownloadCleanedEnabled();
List<INotificationProvider> OnCategoryChangedEnabled();
}
@@ -16,4 +16,6 @@ public interface INotificationProvider
Task OnQueueItemDeleted(QueueItemDeletedNotification notification);
Task OnDownloadCleaned(DownloadCleanedNotification notification);
Task OnCategoryChanged(CategoryChangedNotification notification);
}
@@ -9,4 +9,6 @@ public interface INotificationPublisher
Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason);
Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason);
Task NotifyCategoryChanged(string oldCategory, string newCategory);
}
@@ -0,0 +1,5 @@
namespace Infrastructure.Verticals.Notifications.Models;
public sealed record CategoryChangedNotification : Notification
{
}
@@ -41,6 +41,11 @@ public class NotifiarrProvider : NotificationProvider
{
await _proxy.SendNotification(BuildPayload(notification), _config);
}
public override async Task OnCategoryChanged(CategoryChangedNotification notification)
{
await _proxy.SendNotification(BuildPayload(notification), _config);
}
private NotifiarrPayload BuildPayload(ArrNotification notification, string color)
{
@@ -105,4 +110,32 @@ public class NotifiarrProvider : NotificationProvider
return payload;
}
private NotifiarrPayload BuildPayload(CategoryChangedNotification notification)
{
NotifiarrPayload payload = new()
{
Discord = new()
{
Color = WarningColor,
Text = new()
{
Title = notification.Title,
Icon = Logo,
Description = notification.Description,
Fields = notification.Fields?.Adapt<List<Field>>() ?? []
},
Ids = new Ids
{
Channel = _config.ChannelId
},
Images = new()
{
Thumbnail = new Uri(Logo)
}
}
};
return payload;
}
}
@@ -1,5 +1,6 @@
using System.Text;
using Common.Helpers;
using Microsoft.Extensions.Logging;
using Newtonsoft.Json;
using Newtonsoft.Json.Serialization;
@@ -7,12 +8,14 @@ namespace Infrastructure.Verticals.Notifications.Notifiarr;
public class NotifiarrProxy : INotifiarrProxy
{
private readonly ILogger<NotifiarrProxy> _logger;
private readonly HttpClient _httpClient;
private const string Url = "https://notifiarr.com/api/v1/notification/passthrough/";
public NotifiarrProxy(IHttpClientFactory httpClientFactory)
public NotifiarrProxy(ILogger<NotifiarrProxy> logger, IHttpClientFactory httpClientFactory)
{
_logger = logger;
_httpClient = httpClientFactory.CreateClient(Constants.HttpClientWithRetryName);
}
@@ -25,6 +28,8 @@ public class NotifiarrProxy : INotifiarrProxy
ContractResolver = new CamelCasePropertyNamesContractResolver()
});
_logger.LogTrace("sending notification to Notifiarr: {content}", content);
using HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, $"{Url}{config.ApiKey}");
request.Method = HttpMethod.Post;
request.Content = new StringContent(content, Encoding.UTF8, "application/json");
@@ -34,4 +34,9 @@ public class NotificationFactory : INotificationFactory
ActiveProviders()
.Where(n => n.Config.OnDownloadCleaned)
.ToList();
public List<INotificationProvider> OnCategoryChangedEnabled() =>
ActiveProviders()
.Where(n => n.Config.OnCategoryChanged)
.ToList();
}
@@ -22,4 +22,6 @@ public abstract class NotificationProvider : INotificationProvider
public abstract Task OnQueueItemDeleted(QueueItemDeletedNotification notification);
public abstract Task OnDownloadCleaned(DownloadCleanedNotification notification);
public abstract Task OnCategoryChanged(CategoryChangedNotification notification);
}
@@ -32,7 +32,7 @@ public class NotificationPublisher : INotificationPublisher
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
Uri? imageUrl = GetImageFromContext(record, instanceType);
ArrNotification notification = new()
{
@@ -48,10 +48,10 @@ public class NotificationPublisher : INotificationPublisher
switch (strikeType)
{
case StrikeType.Stalled:
await _dryRunInterceptor.InterceptAsync(Notify<StalledStrikeNotification>, notification.Adapt<StalledStrikeNotification>());
await NotifyInternal(notification.Adapt<StalledStrikeNotification>());
break;
case StrikeType.ImportFailed:
await _dryRunInterceptor.InterceptAsync(Notify<FailedImportStrikeNotification>, notification.Adapt<FailedImportStrikeNotification>());
await NotifyInternal(notification.Adapt<FailedImportStrikeNotification>());
break;
}
}
@@ -63,42 +63,82 @@ public class NotificationPublisher : INotificationPublisher
public virtual async Task NotifyQueueItemDeleted(bool removeFromClient, DeleteReason reason)
{
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
try
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await _dryRunInterceptor.InterceptAsync(Notify<QueueItemDeletedNotification>, notification);
QueueRecord record = ContextProvider.Get<QueueRecord>(nameof(QueueRecord));
InstanceType instanceType = (InstanceType)ContextProvider.Get<object>(nameof(InstanceType));
Uri instanceUrl = ContextProvider.Get<Uri>(nameof(ArrInstance) + nameof(ArrInstance.Url));
Uri? imageUrl = GetImageFromContext(record, instanceType);
QueueItemDeletedNotification notification = new()
{
InstanceType = instanceType,
InstanceUrl = instanceUrl,
Hash = record.DownloadId.ToLowerInvariant(),
Title = $"Deleting item from queue with reason: {reason}",
Description = record.Title,
Image = imageUrl,
Fields = [new() { Title = "Removed from download client?", Text = removeFromClient ? "Yes" : "No" }]
};
await NotifyInternal(notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify queue item deleted");
}
}
public virtual async Task NotifyDownloadCleaned(double ratio, TimeSpan seedingTime, string categoryName, CleanReason reason)
{
DownloadCleanedNotification notification = new()
try
{
Title = $"Cleaned item from download client with reason: {reason}",
DownloadCleanedNotification notification = new()
{
Title = $"Cleaned item from download client with reason: {reason}",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new()
{
Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h"
}
],
Level = NotificationLevel.Important
};
await NotifyInternal(notification);
}
catch (Exception ex)
{
_logger.LogError(ex, "failed to notify download cleaned");
}
}
public virtual async Task NotifyCategoryChanged(string oldCategory, string newCategory)
{
CategoryChangedNotification notification = new()
{
Title = "Category changed",
Description = ContextProvider.Get<string>("downloadName"),
Fields =
[
new() { Title = "Hash", Text = ContextProvider.Get<string>("hash").ToLowerInvariant() },
new() { Title = "Category", Text = categoryName.ToLowerInvariant() },
new() { Title = "Ratio", Text = $"{ratio.ToString(CultureInfo.InvariantCulture)}%" },
new() { Title = "Seeding hours", Text = $"{Math.Round(seedingTime.TotalHours, 0).ToString(CultureInfo.InvariantCulture)}h" }
new() { Title = "Old category", Text = oldCategory },
new() { Title = "New category", Text = newCategory }
],
Level = NotificationLevel.Important
};
await _dryRunInterceptor.InterceptAsync(Notify<DownloadCleanedNotification>, notification);
await NotifyInternal(notification);
}
private Task NotifyInternal<T>(T message) where T: notnull
{
return _dryRunInterceptor.InterceptAsync(Notify<T>, message);
}
[DryRunSafeguard]
@@ -107,12 +147,21 @@ public class NotificationPublisher : INotificationPublisher
return _messageBus.Publish(message);
}
private static Uri GetImageFromContext(QueueRecord record, InstanceType instanceType) =>
instanceType switch
private Uri? GetImageFromContext(QueueRecord record, InstanceType instanceType)
{
Uri? image = instanceType switch
{
InstanceType.Sonarr => record.Series!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Radarr => record.Movie!.Images.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Lidarr => record.Album!.Images.FirstOrDefault(x => x.CoverType == "cover")?.Url,
InstanceType.Sonarr => record.Series?.Images?.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Radarr => record.Movie?.Images?.FirstOrDefault(x => x.CoverType == "poster")?.RemoteUrl,
InstanceType.Lidarr => record.Album?.Images?.FirstOrDefault(x => x.CoverType == "cover")?.Url,
_ => throw new ArgumentOutOfRangeException(nameof(instanceType))
} ?? throw new Exception("failed to get image url from context");
};
if (image is null)
{
_logger.LogWarning("no poster found for {title}", record.Title);
}
return image;
}
}
@@ -73,4 +73,19 @@ public class NotificationService
}
}
}
public async Task Notify(CategoryChangedNotification notification)
{
foreach (INotificationProvider provider in _notificationFactory.OnCategoryChangedEnabled())
{
try
{
await provider.OnCategoryChanged(notification);
}
catch (Exception exception)
{
_logger.LogWarning(exception, "failed to send notification | provider {provider}", provider.Name);
}
}
}
}
@@ -4,6 +4,7 @@ using Common.Configuration.QueueCleaner;
using Domain.Enums;
using Domain.Models.Arr;
using Domain.Models.Arr.Queue;
using Infrastructure.Providers;
using Infrastructure.Verticals.Arr;
using Infrastructure.Verticals.Arr.Interfaces;
using Infrastructure.Verticals.Context;
@@ -19,7 +20,8 @@ namespace Infrastructure.Verticals.QueueCleaner;
public sealed class QueueCleaner : GenericHandler
{
private readonly QueueCleanerConfig _config;
private readonly IgnoredDownloadsProvider<QueueCleanerConfig> _ignoredDownloadsProvider;
public QueueCleaner(
ILogger<QueueCleaner> logger,
IOptions<QueueCleanerConfig> config,
@@ -32,7 +34,8 @@ public sealed class QueueCleaner : GenericHandler
LidarrClient lidarrClient,
ArrQueueIterator arrArrQueueIterator,
DownloadServiceFactory downloadServiceFactory,
INotificationPublisher notifier
INotificationPublisher notifier,
IgnoredDownloadsProvider<QueueCleanerConfig> ignoredDownloadsProvider
) : base(
logger, downloadClientConfig,
sonarrConfig, radarrConfig, lidarrConfig,
@@ -42,10 +45,14 @@ public sealed class QueueCleaner : GenericHandler
)
{
_config = config.Value;
_config.Validate();
_ignoredDownloadsProvider = ignoredDownloadsProvider;
}
protected override async Task ProcessInstanceAsync(ArrInstance instance, InstanceType instanceType)
{
IReadOnlyList<string> ignoredDownloads = await _ignoredDownloadsProvider.GetIgnoredDownloads();
using var _ = LogContext.PushProperty("InstanceName", instanceType.ToString());
HashSet<SearchItem> itemsToBeRefreshed = [];
@@ -75,15 +82,27 @@ public sealed class QueueCleaner : GenericHandler
continue;
}
if (ignoredDownloads.Contains(record.DownloadId, StringComparer.InvariantCultureIgnoreCase))
{
_logger.LogInformation("skip | {title} | ignored", record.Title);
continue;
}
// push record to context
ContextProvider.Set(nameof(QueueRecord), record);
StalledResult stalledCheckResult = new();
if (_downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.None && record.Protocol is "torrent")
if (record.Protocol is "torrent" && _downloadClientConfig.DownloadClient is not Common.Enums.DownloadClient.Disabled)
{
if (_downloadClientConfig.DownloadClient is Common.Enums.DownloadClient.None)
{
_logger.LogWarning("skip | download client is not configured | {title}", record.Title);
continue;
}
// stalled download check
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId);
stalledCheckResult = await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId, ignoredDownloads);
}
// failed import check
@@ -113,7 +132,7 @@ public sealed class QueueCleaner : GenericHandler
}
}
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient);
await arrClient.DeleteQueueItemAsync(instance, record, removeFromClient, deleteReason);
await _notifier.NotifyQueueItemDeleted(removeFromClient, deleteReason);
}
});
@@ -0,0 +1 @@
ignored
+14 -4
View File
@@ -178,7 +178,7 @@ services:
- TZ=Europe/Bucharest
- DRY_RUN=false
- LOGGING__LOGLEVEL=Debug
- LOGGING__LOGLEVEL=Verbose
- LOGGING__FILE__ENABLED=true
- LOGGING__FILE__PATH=/var/logs
- LOGGING__ENHANCED=true
@@ -191,6 +191,7 @@ services:
- TRIGGERS__DOWNLOADCLEANER=0/30 * * * * ?
- QUEUECLEANER__ENABLED=true
- QUEUECLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- QUEUECLEANER__RUNSEQUENTIALLY=true
- QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5
- QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE=true
@@ -201,19 +202,25 @@ services:
- QUEUECLEANER__STALLED_DELETE_PRIVATE=false
- CONTENTBLOCKER__ENABLED=true
- CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH=/ignored
- CONTENTBLOCKER__IGNORE_PRIVATE=true
- CONTENTBLOCKER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__ENABLED=true
- DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH=/ignored
- DOWNLOADCLEANER__DELETE_PRIVATE=false
- DOWNLOADCLEANER__CATEGORIES__0__NAME=tv-sonarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO=-1
- DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME=0
- DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME=0.01
- DOWNLOADCLEANER__CATEGORIES__1__NAME=radarr
- DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME=99999
- DOWNLOADCLEANER__CATEGORIES__1__NAME=nohardlink
- DOWNLOADCLEANER__CATEGORIES__1__MAX_RATIO=-1
- DOWNLOADCLEANER__CATEGORIES__1__MIN_SEED_TIME=0
- DOWNLOADCLEANER__CATEGORIES__1__MAX_SEED_TIME=0.01
- DOWNLOADCLEANER__CATEGORIES__1__MAX_SEED_TIME=99999
- DOWNLOADCLEANER__UNLINKED_TARGET_CATEGORY=cleanuperr-unlinked
- DOWNLOADCLEANER__UNLINKED_IGNORED_ROOT_DIR=/downloads
- DOWNLOADCLEANER__UNLINKED_CATEGORIES__0=tv-sonarr
- DOWNLOADCLEANER__UNLINKED_CATEGORIES__1=radarr
- DOWNLOAD_CLIENT=qbittorrent
- QBITTORRENT__URL=http://qbittorrent:8080
@@ -252,10 +259,13 @@ services:
# - NOTIFIARR__ON_STALLED_STRIKE=true
# - NOTIFIARR__ON_QUEUE_ITEM_DELETED=true
# - NOTIFIARR__ON_DOWNLOAD_CLEANED=true
# - NOTIFIARR__ON_CATEGORY_CHANGED=true
# - NOTIFIARR__API_KEY=notifiarr_secret
# - NOTIFIARR__CHANNEL_ID=discord_channel_id
volumes:
- ./data/cleanuperr/logs:/var/logs
- ./data/cleanuperr/ignored_downloads:/ignored
- ./data/qbittorrent/downloads:/downloads
restart: unless-stopped
depends_on:
- qbittorrent
+303 -177
View File
@@ -1,52 +1,52 @@
## Table of contents
- [General settings](variables.md#general-settings)
- [Queue Cleaner settings](variables.md#queue-cleaner-settings)
- [Content Blocker settings](variables.md#content-blocker-settings)
- [Download Cleaner settings](variables.md#download-cleaner-settings)
- [Download Client settings](variables.md#download-client-settings)
- [Arr settings](variables.md#arr-settings)
- [Notification settings](variables.md#notification-settings)
- [Advanced settings](variables.md#advanced-settings)
- [General settings](#general-settings)
- [Queue Cleaner settings](#queue-cleaner-settings)
- [Content Blocker settings](#content-blocker-settings)
- [Download Cleaner settings](#download-cleaner-settings)
- [Download Client settings](#download-client-settings)
- [Arr settings](#arr-settings)
- [Notification settings](#notification-settings)
- [Advanced settings](#advanced-settings)
#
### General settings
**`TZ`**
#### **`TZ`**
- The time zone to use.
- Type: String.
- Possible values: Any valid timezone.
- Default: `UTC`.
- Required: No.
**`DRY_RUN`**
#### **`DRY_RUN`**
- When enabled, simulates irreversible operations (like deletions and notifications) without making actual changes.
- Type: Boolean.
- Possible values: `true`, `false`.
- Default: `false`.
- Required: No.
**`LOGGING__LOGLEVEL`**
#### **`LOGGING__LOGLEVEL`**
- Controls the detail level of application logs.
- Type: String.
- Possible values: `Verbose`, `Debug`, `Information`, `Warning`, `Error`, `Fatal`.
- Default: `Information`.
- Required: No.
**`LOGGING__FILE__ENABLED`**
#### **`LOGGING__FILE__ENABLED`**
- Enables logging to a file.
- Type: Boolean.
- Possible values: `true`, `false`.
- Default: `false`.
- Required: No.
**`LOGGING__FILE__PATH`**
#### **`LOGGING__FILE__PATH`**
- Directory where log files will be saved.
- Type: String.
- Default: Empty.
- Required: No.
**`LOGGING__ENHANCED`**
#### **`LOGGING__ENHANCED`**
- Provides more detailed descriptions in logs whenever possible.
- Type: Boolean.
- Possible values: `true`, `false`.
@@ -58,7 +58,7 @@
### Queue Cleaner settings
**`TRIGGERS__QUEUECLEANER`**
#### **`TRIGGERS__QUEUECLEANER`**
- Cron schedule for the queue cleaner job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0/5 * * * ?` (every 5 minutes).
@@ -66,9 +66,9 @@
> [!NOTE]
> - Maximum interval is 6 hours.
> - Ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`.
> - Is ignored if `QUEUECLEANER__RUNSEQUENTIALLY=true` and `CONTENTBLOCKER__ENABLED=true`.
**`QUEUECLEANER__ENABLED`**
#### **`QUEUECLEANER__ENABLED`**
- Enables or disables the queue cleaning functionality.
- When enabled, processes all items in the *arr queue.
- Type: Boolean
@@ -76,7 +76,32 @@
- Default: `true`
- Required: No.
**`QUEUECLEANER__RUNSEQUENTIALLY`**
#### **`QUEUECLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`QUEUECLEANER__RUNSEQUENTIALLY`**
- Controls whether queue cleaner runs after content blocker instead of in parallel.
- When `true`, streamlines the cleaning process by running immediately after content blocker.
- Type: Boolean
@@ -84,23 +109,24 @@
- Default: `true`
- Required: No.
**`QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES`**
#### **`QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES`**
- Number of strikes before removing a failed import.
- Set to `0` to never remove failed imports.
- A strike is given when an item is stalled, stuck in metadata downloading, or failed to be imported.
- A strike is given when an item fails to be imported.
- Type: Integer
- Possible values: `0` or greater
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE`**
#### **`QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE`**
- Controls whether to ignore failed imports from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE`**
#### **`QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE`**
- Controls whether to delete failed imports from private trackers from the download client.
- Has no effect if `QUEUECLEANER__IMPORT_FAILED_IGNORE_PRIVATE` is `true`.
- Type: Boolean
@@ -109,44 +135,45 @@
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `QUEUECLEANER__IMPORT_FAILED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
**`QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS`**
#### **`QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS`**
- Patterns to look for in failed import messages that should be ignored.
- Multiple patterns can be specified using incrementing numbers starting from 0.
- Type: String array
- Default: Empty.
- Required: No.
- Example:
```yaml
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0: "title mismatch"
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
```
```yaml
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__0: "title mismatch"
QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
```
**`QUEUECLEANER__STALLED_MAX_STRIKES`**
#### **`QUEUECLEANER__STALLED_MAX_STRIKES`**
- Number of strikes before removing a stalled download.
- Set to `0` to never remove stalled downloads.
- A strike is given when download speed is 0.
- A strike is given when an item is stalled (not downloading) or stuck while downloading metadata.
- Type: Integer
- Possible values: `0` or greater
- Default: `0`
- Required: No.
> [!NOTE]
> If not set to `0`, the minimum value is `3`.
**`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`**
#### **`QUEUECLEANER__STALLED_RESET_STRIKES_ON_PROGRESS`**
- Controls whether to remove strikes if any download progress was made since last checked.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__STALLED_IGNORE_PRIVATE`**
#### **`QUEUECLEANER__STALLED_IGNORE_PRIVATE`**
- Controls whether to ignore stalled downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`QUEUECLEANER__STALLED_DELETE_PRIVATE`**
#### **`QUEUECLEANER__STALLED_DELETE_PRIVATE`**
- Controls whether to delete stalled private downloads from the download client.
- Has no effect if `QUEUECLEANER__STALLED_IGNORE_PRIVATE` is `true`.
- Type: Boolean
@@ -155,13 +182,13 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `QUEUECLEANER__STALLED_DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#
### Content Blocker settings
**`TRIGGERS__CONTENTBLOCKER`**
#### **`TRIGGERS__CONTENTBLOCKER`**
- Cron schedule for the content blocker job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0/5 * * * ?` (every 5 minutes).
@@ -170,7 +197,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> - Maximum interval is 6 hours.
**`CONTENTBLOCKER__ENABLED`**
#### **`CONTENTBLOCKER__ENABLED`**
- Enables or disables the content blocker functionality.
- When enabled, processes all items in the *arr queue and marks unwanted files.
- Type: Boolean
@@ -178,14 +205,39 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`CONTENTBLOCKER__IGNORE_PRIVATE`**
#### **`CONTENTBLOCKER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`CONTENTBLOCKER__IGNORE_PRIVATE`**
- Controls whether to ignore downloads from private trackers.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`CONTENTBLOCKER__DELETE_PRIVATE`**
#### **`CONTENTBLOCKER__DELETE_PRIVATE`**
- Controls whether to delete private downloads that have all files blocked from the download client.
- Has no effect if `CONTENTBLOCKER__IGNORE_PRIVATE` is `true`.
- Type: Boolean
@@ -194,13 +246,13 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `CONTENTBLOCKER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `CONTENTBLOCKER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#
### Download Cleaner settings
**`TRIGGERS__DOWNLOADCLEANER`**
#### **`TRIGGERS__DOWNLOADCLEANER`**
- Cron schedule for the download cleaner job.
- Type: String - [Quartz cron format](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html).
- Default: `0 0 * * * ?` (every hour).
@@ -209,7 +261,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> - Maximum interval is 6 hours.
**`DOWNLOADCLEANER__ENABLED`**
#### **`DOWNLOADCLEANER__ENABLED`**
- Enables or disables the download cleaner functionality.
- When enabled, automatically cleans up downloads that have been seeding for a certain amount of time.
- Type: Boolean.
@@ -217,7 +269,32 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `false`
- Required: No.
**`DOWNLOADCLEANER__DELETE_PRIVATE`**
#### **`DOWNLOADCLEANER__IGNORED_DOWNLOADS_PATH`**
- Local path to the file containing ignored downloads.
- If the contents of the file are changed, they will be reloaded on the next job run.
- Accepted values:
- torrent hash
- qBitTorrent tag or category
- Deluge label
- Transmission category (last directory from the save location)
- torrent tracker domain
- Each value needs to be on a new line.
- Type: String.
- Default: Empty.
- Required: No.
- Example: `/ignored.txt`.
- Example of file contents:
```
fa800a7d7c443a2c3561d1f8f393c089036dade1
tv-sonarr
qbit-tag
mytracker.com
...
```
>[!IMPORTANT]
> Some people have experienced problems using Docker where the mounted file would not update inside the container if it was modified on the host. This is a Docker configuration problem and can not be solved by cleanuperr.
#### **`DOWNLOADCLEANER__DELETE_PRIVATE`**
- Controls whether to delete private downloads.
- Type: Boolean.
- Possible values: `true`, `false`
@@ -225,9 +302,9 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Required: No.
> [!WARNING]
> Setting `DOWNLOADCLEANER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your tracker account.
> Setting `DOWNLOADCLEANER__DELETE_PRIVATE=true` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
**`DOWNLOADCLEANER__CATEGORIES__0__NAME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__NAME`**
- Name of the category to clean.
- Type: String.
- Default: Empty.
@@ -235,18 +312,21 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> [!NOTE]
> The category name must match the category that was set in the *arr.
>
> For qBittorrent, the category name is the name of the download category.
>
> For Deluge, the category name is the name of the label.
> For Transmission, the category name is the name of the download location.
>
> For Transmission, the category name is the last directory from the save location.
**`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MAX_RATIO`**
- Maximum ratio to reach before removing a download.
- Type: Decimal.
- Possible values: `-1` or greater (`-1` means no limit or disabled).
- Default: `-1`
- Required: No.
**`DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MIN_SEED_TIME`**
- Minimum number of hours to seed before removing a download, if the ratio has been met.
- Used with `MAX_RATIO` to ensure a minimum seed time.
- Type: Decimal.
@@ -254,7 +334,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `0`
- Required: No.
**`DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME`**
#### **`DOWNLOADCLEANER__CATEGORIES__0__MAX_SEED_TIME`**
- Maximum number of hours to seed before removing a download.
- Type: Decimal.
- Possible values: `-1` or greater (`-1` means no limit or disabled).
@@ -277,59 +357,82 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Download Client settings
**`DOWNLOAD_CLIENT`**
#### **`DOWNLOAD_CLIENT`**
- Specifies which download client is used by *arrs.
- Type: String.
- Possible values: `none`, `qbittorrent`, `deluge`, `transmission`.
- Possible values: `none`, `qbittorrent`, `deluge`, `transmission`, `disabled`.
- Default: `none`
- Required: No.
> [!NOTE]
> Only one download client can be enabled at a time. If you have more than one download client, you should deploy multiple instances of cleanuperr.
**`QBITTORRENT__URL`**
> [!IMPORTANT]
> When the download client is set to `disabled`, the queue cleaner will be able to remove items that are failed to be imported even if there is no download client configured. This means that all downloads, including private ones, will be completely removed.
>
> Setting `DOWNLOAD_CLIENT=disabled` means you don't care about seeding, ratio, H&R and potentially losing your private tracker account.
#### **`QBITTORRENT__URL`**
- URL of the qBittorrent instance.
- Type: String.
- Default: `http://localhost:8080`.
- Required: No.
**`QBITTORRENT__USERNAME`**
#### **`QBITTORRENT__URL_BASE`**
- Adds a prefix to the qBittorrent url, such as `[QBITTORRENT__URL]/[QBITTORRENT__URL_BASE]/api`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`QBITTORRENT__USERNAME`**
- Username for qBittorrent authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`QBITTORRENT__PASSWORD`**
#### **`QBITTORRENT__PASSWORD`**
- Password for qBittorrent authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`DELUGE__URL`**
#### **`DELUGE__URL`**
- URL of the Deluge instance.
- Type: String.
- Default: `http://localhost:8112`.
- Required: No.
**`DELUGE__PASSWORD`**
#### **`DELUGE__URL_BASE`**
- Adds a prefix to the deluge json url, such as `[DELUGE__URL]/[DELUGE__URL_BASE]/json`.
- Type: String.
- Default: Empty.
- Required: No.
#### **`DELUGE__PASSWORD`**
- Password for Deluge authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`TRANSMISSION__URL`**
#### **`TRANSMISSION__URL`**
- URL of the Transmission instance.
- Type: String.
- Default: `http://localhost:9091`.
- Required: No.
**`TRANSMISSION__USERNAME`**
#### **`TRANSMISSION__URL_BASE`**
- Adds a prefix to the Transmission rpc url, such as `[TRANSMISSION__URL]/[TRANSMISSION__URL_BASE]/rpc`.
- Type: String.
- Default: `transmission`.
- Required: No.
#### **`TRANSMISSION__USERNAME`**
- Username for Transmission authentication.
- Type: String.
- Default: Empty.
- Required: No.
**`TRANSMISSION__PASSWORD`**
#### **`TRANSMISSION__PASSWORD`**
- Password for Transmission authentication.
- Type: String.
- Default: Empty.
@@ -339,112 +442,6 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Arr settings
**`SONARR__ENABLED`**
- Enables or disables Sonarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`SONARR__BLOCK__TYPE`**
- Determines how file blocking works for Sonarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`SONARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`SONARR__SEARCHTYPE`**
- Determines what to search for after removing a queue item.
- Type: String
- Possible values: `Episode`, `Season`, `Series`
- Default: `Episode`
- Required: No.
**`SONARR__INSTANCES__0__URL`**
- URL of the Sonarr instance.
- Type: String
- Default: `http://localhost:8989`
- Required: No.
**`SONARR__INSTANCES__0__APIKEY`**
- API key for the Sonarr instance.
- Type: String
- Default: Empty.
- Required: No.
**`RADARR__ENABLED`**
- Enables or disables Radarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`RADARR__BLOCK__TYPE`**
- Determines how file blocking works for Radarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`RADARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`RADARR__INSTANCES__0__URL`**
- URL of the Radarr instance.
- Type: String
- Default: `http://localhost:7878`
- Required: No.
**`RADARR__INSTANCES__0__APIKEY`**
- API key for the Radarr instance.
- Type: String
- Default: Empty.
- Required: No.
**`LIDARR__ENABLED`**
- Enables or disables Lidarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`LIDARR__BLOCK__TYPE`**
- Determines how file blocking works for Lidarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
**`LIDARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
**`LIDARR__INSTANCES__0__URL`**
- URL of the Lidarr instance.
- Type: String
- Default: `http://localhost:8686`
- Required: No.
**`LIDARR__INSTANCES__0__APIKEY`**
- API key for the Lidarr instance.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> Multiple instances can be specified for each *arr using this format, where `<NUMBER>` starts from 0:
> ```yaml
@@ -452,8 +449,29 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> <ARR>__INSTANCES__<NUMBER>__APIKEY
> ```
#### **`SONARR__ENABLED`**
- Enables or disables Sonarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`SONARR__BLOCK__TYPE`**
- Determines how file blocking works for Sonarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`SONARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists (blacklist/whitelist) support the following patterns:
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
@@ -463,47 +481,155 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
> ```
> [!NOTE]
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr, but they are not suitable for other *arrs.
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist), [this permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr.
#### **`SONARR__SEARCHTYPE`**
- Determines what to search for after removing a queue item.
- Type: String
- Possible values: `Episode`, `Season`, `Series`
- Default: `Episode`
- Required: No.
#### **`SONARR__INSTANCES__0__URL`**
- URL of the Sonarr instance.
- Type: String
- Default: `http://localhost:8989`
- Required: No.
#### **`SONARR__INSTANCES__0__APIKEY`**
- API key for the Sonarr instance.
- Type: String
- Default: Empty.
- Required: No.
#### **`RADARR__ENABLED`**
- Enables or disables Radarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`RADARR__BLOCK__TYPE`**
- Determines how file blocking works for Radarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`RADARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
> *example* // file name has "example" in the name
> example // file name is exactly the word "example"
> regex:<ANY_REGEX> // regex that needs to be marked at the start of the line with "regex:"
> ```
> [!NOTE]
> [This blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist), [this permissive blacklist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/blacklist_permissive) and [this whitelist](https://raw.githubusercontent.com/flmorg/cleanuperr/refs/heads/main/whitelist) can be used for Sonarr and Radarr.
#### **`RADARR__INSTANCES__0__URL`**
- URL of the Radarr instance.
- Type: String
- Default: `http://localhost:7878`
- Required: No.
#### **`RADARR__INSTANCES__0__APIKEY`**
- API key for the Radarr instance.
- Type: String
- Default: Empty.
- Required: No.
#### **`LIDARR__ENABLED`**
- Enables or disables Lidarr cleanup.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
#### **`LIDARR__BLOCK__TYPE`**
- Determines how file blocking works for Lidarr.
- Type: String
- Possible values: `blacklist`, `whitelist`
- Default: `blacklist`
- Required: No.
#### **`LIDARR__BLOCK__PATH`**
- Path to the blocklist file (local file or URL).
- Must be JSON compatible.
- Type: String
- Default: Empty.
- Required: No.
> [!NOTE]
> The blocklists support the following patterns:
> ```
> *example // file name ends with "example"
> example* // file name starts with "example"
> *example* // file name has "example" in the name
> example // file name is exactly the word "example"
> regex:<ANY_REGEX> // regex that needs to be marked at the start of the line with "regex:"
> ```
#### **`LIDARR__INSTANCES__0__URL`**
- URL of the Lidarr instance.
- Type: String
- Default: `http://localhost:8686`
- Required: No.
#### **`LIDARR__INSTANCES__0__APIKEY`**
- API key for the Lidarr instance.
- Type: String
- Default: Empty.
- Required: No.
#
### Notification settings
**`NOTIFIARR__API_KEY`**
#### **`NOTIFIARR__API_KEY`**
- Notifiarr API key for sending notifications.
- Requires Notifiarr's [`Passthrough`](https://notifiarr.wiki/en/Website/Integrations/Passthrough) integration to work.
- Type: String
- Default: Empty.
- Required: No.
**`NOTIFIARR__CHANNEL_ID`**
#### **`NOTIFIARR__CHANNEL_ID`**
- Discord channel ID where notifications will be sent.
- Type: String
- Default: Empty.
- Required: No.
**`NOTIFIARR__ON_IMPORT_FAILED_STRIKE`**
#### **`NOTIFIARR__ON_IMPORT_FAILED_STRIKE`**
- Controls whether to notify when an item receives a failed import strike.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_STALLED_STRIKE`**
#### **`NOTIFIARR__ON_STALLED_STRIKE`**
- Controls whether to notify when an item receives a stalled download strike.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_QUEUE_ITEM_DELETED`**
#### **`NOTIFIARR__ON_QUEUE_ITEM_DELETED`**
- Controls whether to notify when a queue item is deleted.
- Type: Boolean
- Possible values: `true`, `false`
- Default: `false`
- Required: No.
**`NOTIFIARR__ON_DOWNLOAD_CLEANED`**
#### **`NOTIFIARR__ON_DOWNLOAD_CLEANED`**
- Controls whether to notify when a download is cleaned.
- Type: Boolean
- Possible values: `true`, `false`
@@ -514,7 +640,7 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
### Advanced settings
**`HTTP_MAX_RETRIES`**
#### **`HTTP_MAX_RETRIES`**
- The number of times to retry a failed HTTP call.
- Applies to calls to *arrs, download clients, and other services.
- Type: Integer
@@ -522,10 +648,10 @@ QUEUECLEANER__IMPORT_FAILED_IGNORE_PATTERNS__1: "manual import required"
- Default: `0`
- Required: No.
**`HTTP_TIMEOUT`**
#### **`HTTP_TIMEOUT`**
- The number of seconds to wait before failing an HTTP call.
- Applies to calls to *arrs, download clients, and other services.
- Type: Integer
- Possible values: Greater than `0`
- Possible values: Greater than `0`.
- Default: `100`
- Required: No.