diff --git a/README.md b/README.md index 4e97240..6348fa2 100644 --- a/README.md +++ b/README.md @@ -8,9 +8,19 @@ The tool supports both qBittorrent's built-in exclusion features and its own blo Refer to the [Environment variables](#Environment-variables) section for detailed configuration instructions and the [Setup](#Setup) section for an in-depth explanation of the cleanup process. +## Key features +- Marks unwanted files as skip/unwanted in the download client. +- Automatically strikes stalled or stuck downloads. +- Removes and blocks downloads that reached the maximum number of strikes or are marked as unwanted by the download client or by cleanuperr and triggers a search for removed downloads. + ## Important note -Only the **latest versions** of qBittorrent, Deluge, Sonarr etc. are supported, or earlier versions that have the same API as the latest version. +Only the **latest versions** of the following apps are supported, or earlier versions that have the same API as the latest version: +- qBittorrent +- Deluge +- Transmission +- Sonarr +- Radarr This tool is actively developed and still a work in progress. Join the Discord server if you want to reach out to me quickly (or just stay updated on new releases) so we can squash those pesky bugs together: @@ -28,12 +38,14 @@ This tool is actively developed and still a work in progress. Join the Discord s 2. **Queue cleaner** will: - Run every 5 minutes (or configured cron). - Process all items in the *arr queue. + - Check each queue item if it is **stalled (download speed is 0)**, **stuck in matadata downloading** or **failed to be imported**. + - If it is, the item receives a **strike** and will continue to accumulate strikes every time it meets any of these conditions. - Check each queue item if it meets one of the following condition in the download client: - **Marked as completed, but 0 bytes have been downloaded** (due to files being blocked by qBittorrent or the **content blocker**). - All associated files of are marked as **unwanted/skipped**. - If the item **DOES NOT** match the above criteria, it will be skipped. - - If the item **DOES** match the criteria: - - It will be removed from the *arr's queue. + - If the item **DOES** match the criteria or has received the **maximum number of strikes**: + - It will be removed from the *arr's queue and blocked. - It will be deleted from the download client. - A new search will be triggered for the *arr item. @@ -78,6 +90,8 @@ services: - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true + - QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5 + - QUEUECLEANER__STALLED_MAX_STRIKES=5 - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__BLACKLIST__ENABLED=true @@ -91,21 +105,25 @@ services: - QBITTORRENT__USERNAME=user - QBITTORRENT__PASSWORD=pass # OR + # - DOWNLOAD_CLIENT=deluge # - DELUGE__URL=http://localhost:8112 # - DELUGE__PASSWORD=testing # OR + # - DOWNLOAD_CLIENT=transmission # - TRANSMISSION__URL=http://localhost:9091 # - TRANSMISSION__USERNAME=test # - TRANSMISSION__PASSWORD=testing - SONARR__ENABLED=true - SONARR__SEARCHTYPE=Episode + - SONARR__STALLED_MAX_STRIKES=5 - SONARR__INSTANCES__0__URL=http://localhost:8989 - SONARR__INSTANCES__0__APIKEY=secret1 - SONARR__INSTANCES__1__URL=http://localhost:8990 - SONARR__INSTANCES__1__APIKEY=secret2 - RADARR__ENABLED=true + - RADARR__STALLED_MAX_STRIKES=5 - RADARR__INSTANCES__0__URL=http://localhost:7878 - RADARR__INSTANCES__0__APIKEY=secret3 - RADARR__INSTANCES__1__URL=http://localhost:7879 @@ -123,11 +141,13 @@ services: | LOGGING__FILE__PATH | No | Directory where to save the log files | empty | | LOGGING__ENHANCED | No | Enhance logs whenever possible
A more detailed description is provided [here](variables.md#LOGGING__ENHANCED) | true | ||||| -| TRIGGERS__QUEUECLEANER | Yes if queue cleaner is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) | 0 0/5 * * * ? | -| TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html) | 0 0/5 * * * ? | +| TRIGGERS__QUEUECLEANER | Yes if queue cleaner is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html)
Can be a max of 1h interval | 0 0/5 * * * ? | +| TRIGGERS__CONTENTBLOCKER | Yes if content blocker is enabled | [Quartz cron trigger](https://www.quartz-scheduler.org/documentation/quartz-2.3.0/tutorials/crontrigger.html)
Can be a max of 1h interval | 0 0/5 * * * ? | ||||| | QUEUECLEANER__ENABLED | No | Enable or disable the queue cleaner | true | | QUEUECLEANER__RUNSEQUENTIALLY | No | If set to true, the queue cleaner will run after the content blocker instead of running in parallel, streamlining the cleaning process | true | +| QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES | No | After how many strikes should a failed import be removed
0 means never | 0 | +| QUEUECLEANER__STALLED_MAX_STRIKES | No | After how many strikes should a stalled download be removed
0 means never | 0 | ||||| | CONTENTBLOCKER__ENABLED | No | Enable or disable the content blocker | false | | CONTENTBLOCKER__BLACKLIST__ENABLED | Yes if content blocker is enabled and whitelist is not enabled | Enable or disable the blacklist | false | @@ -149,12 +169,12 @@ services: ||||| | SONARR__ENABLED | No | Enable or disable Sonarr cleanup | true | | SONARR__SEARCHTYPE | No | What to search for after removing a queue item
Can be `Episode`, `Season` or `Series` | `Episode` | -| SONARR__INSTANCES__0__URL | Yes | First Sonarr instance url | http://localhost:8989 | -| SONARR__INSTANCES__0__APIKEY | Yes | First Sonarr instance API key | empty | +| SONARR__INSTANCES__0__URL | No | First Sonarr instance url | http://localhost:8989 | +| SONARR__INSTANCES__0__APIKEY | No | First Sonarr instance API key | empty | ||||| | RADARR__ENABLED | No | Enable or disable Radarr cleanup | false | -| RADARR__INSTANCES__0__URL | Yes | First Radarr instance url | http://localhost:8989 | -| RADARR__INSTANCES__0__APIKEY | Yes | First Radarr instance API key | empty | +| RADARR__INSTANCES__0__URL | No | First Radarr instance url | http://localhost:8989 | +| RADARR__INSTANCES__0__APIKEY | No | First Radarr instance API key | empty | # ### To be noted @@ -186,4 +206,10 @@ SONARR__INSTANCES____APIKEY ### Run as a Windows Service -Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678 \ No newline at end of file +Check out this stackoverflow answer on how to do it: https://stackoverflow.com/a/15719678 + +## Credits +Special thanks for inspiration go to: +- [ThijmenGThN/swaparr](https://github.com/ThijmenGThN/swaparr) +- [ManiMatter/decluttarr](https://github.com/ManiMatter/decluttarr) +- [PaeyMoopy/sonarr-radarr-queue-cleaner](https://github.com/PaeyMoopy/sonarr-radarr-queue-cleaner) \ No newline at end of file diff --git a/code/Common/Common.csproj b/code/Common/Common.csproj index 395c626..3290e70 100644 --- a/code/Common/Common.csproj +++ b/code/Common/Common.csproj @@ -7,7 +7,8 @@ - + + diff --git a/code/Common/Configuration/Arr/ArrConfig.cs b/code/Common/Configuration/Arr/ArrConfig.cs index 4ecf19a..51f539b 100644 --- a/code/Common/Configuration/Arr/ArrConfig.cs +++ b/code/Common/Configuration/Arr/ArrConfig.cs @@ -1,4 +1,4 @@ -namespace Common.Configuration.Arr; +namespace Common.Configuration.Arr; public abstract record ArrConfig { diff --git a/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs b/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs index c382cf9..c353979 100644 --- a/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs +++ b/code/Common/Configuration/QueueCleaner/QueueCleanerConfig.cs @@ -1,4 +1,6 @@ -namespace Common.Configuration.QueueCleaner; +using Microsoft.Extensions.Configuration; + +namespace Common.Configuration.QueueCleaner; public sealed record QueueCleanerConfig : IJobConfig { @@ -7,6 +9,12 @@ public sealed record QueueCleanerConfig : IJobConfig public required bool Enabled { get; init; } public required bool RunSequentially { get; init; } + + [ConfigurationKeyName("IMPORT_FAILED_MAX_STRIKES")] + public ushort ImportFailedMaxStrikes { get; init; } + + [ConfigurationKeyName("STALLED_MAX_STRIKES")] + public ushort StalledMaxStrikes { get; init; } public void Validate() { diff --git a/code/Domain/Domain.csproj b/code/Domain/Domain.csproj index 75d3466..de7d774 100644 --- a/code/Domain/Domain.csproj +++ b/code/Domain/Domain.csproj @@ -7,7 +7,11 @@ - + + + + + diff --git a/code/Domain/Enums/StrikeType.cs b/code/Domain/Enums/StrikeType.cs new file mode 100644 index 0000000..1a04c15 --- /dev/null +++ b/code/Domain/Enums/StrikeType.cs @@ -0,0 +1,7 @@ +namespace Domain.Enums; + +public enum StrikeType +{ + Stalled, + ImportFailed +} \ No newline at end of file diff --git a/code/Domain/Models/Arr/Queue/QueueListResponse.cs b/code/Domain/Models/Arr/Queue/QueueListResponse.cs index 685bb4e..7198e57 100644 --- a/code/Domain/Models/Arr/Queue/QueueListResponse.cs +++ b/code/Domain/Models/Arr/Queue/QueueListResponse.cs @@ -1,4 +1,4 @@ -namespace Domain.Arr.Queue; +namespace Domain.Models.Arr.Queue; public record QueueListResponse { diff --git a/code/Domain/Models/Arr/Queue/QueueRecord.cs b/code/Domain/Models/Arr/Queue/QueueRecord.cs index 298fad0..0158cda 100644 --- a/code/Domain/Models/Arr/Queue/QueueRecord.cs +++ b/code/Domain/Models/Arr/Queue/QueueRecord.cs @@ -1,4 +1,4 @@ -namespace Domain.Arr.Queue; +namespace Domain.Models.Arr.Queue; public record QueueRecord { diff --git a/code/Domain/Models/Arr/SonarrSearchItem.cs b/code/Domain/Models/Arr/SonarrSearchItem.cs index 4ac7eea..541d028 100644 --- a/code/Domain/Models/Arr/SonarrSearchItem.cs +++ b/code/Domain/Models/Arr/SonarrSearchItem.cs @@ -1,9 +1,13 @@ -namespace Domain.Models.Arr; +using Common.Configuration.Arr; + +namespace Domain.Models.Arr; public sealed class SonarrSearchItem : SearchItem { public long SeriesId { get; set; } + public SonarrSearchType SearchType { get; set; } + public override bool Equals(object? obj) { if (obj is not SonarrSearchItem other) diff --git a/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs b/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs deleted file mode 100644 index 220a6fd..0000000 --- a/code/Domain/Models/Deluge/Response/DelugeMinimalStatus.cs +++ /dev/null @@ -1,6 +0,0 @@ -namespace Domain.Models.Deluge.Response; - -public sealed record DelugeMinimalStatus -{ - public string? Hash { get; set; } -} \ No newline at end of file diff --git a/code/Domain/Models/Deluge/Response/TorrentStatus.cs b/code/Domain/Models/Deluge/Response/TorrentStatus.cs new file mode 100644 index 0000000..97d22a4 --- /dev/null +++ b/code/Domain/Models/Deluge/Response/TorrentStatus.cs @@ -0,0 +1,12 @@ +namespace Domain.Models.Deluge.Response; + +public sealed record TorrentStatus +{ + public string? Hash { get; set; } + + public string? State { get; set; } + + public string? Name { get; set; } + + public ulong Eta { get; set; } +} \ No newline at end of file diff --git a/code/Domain/Models/Sonarr/SonarrCommand.cs b/code/Domain/Models/Sonarr/SonarrCommand.cs index 0ff1f3a..a0f8cc4 100644 --- a/code/Domain/Models/Sonarr/SonarrCommand.cs +++ b/code/Domain/Models/Sonarr/SonarrCommand.cs @@ -1,4 +1,6 @@ -namespace Domain.Models.Sonarr; +using Common.Configuration.Arr; + +namespace Domain.Models.Sonarr; public sealed record SonarrCommand { @@ -9,4 +11,6 @@ public sealed record SonarrCommand public long? SeasonNumber { get; set; } public List? EpisodeIds { get; set; } + + public SonarrSearchType SearchType { get; set; } } \ No newline at end of file diff --git a/code/Executable/DependencyInjection/ConfigurationDI.cs b/code/Executable/DependencyInjection/ConfigurationDI.cs index d955f3b..f237985 100644 --- a/code/Executable/DependencyInjection/ConfigurationDI.cs +++ b/code/Executable/DependencyInjection/ConfigurationDI.cs @@ -3,6 +3,7 @@ using Common.Configuration.Arr; using Common.Configuration.ContentBlocker; using Common.Configuration.DownloadClient; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Enums; namespace Executable.DependencyInjection; @@ -11,6 +12,7 @@ public static class ConfigurationDI { public static IServiceCollection AddConfiguration(this IServiceCollection services, IConfiguration configuration) => services + .Configure(configuration.GetSection(QueueCleanerConfig.SectionName)) .Configure(configuration.GetSection(ContentBlockerConfig.SectionName)) .Configure(configuration.GetSection(QBitConfig.SectionName)) .Configure(configuration.GetSection(DelugeConfig.SectionName)) diff --git a/code/Executable/DependencyInjection/LoggingDI.cs b/code/Executable/DependencyInjection/LoggingDI.cs index 5efbd6c..0de177b 100644 --- a/code/Executable/DependencyInjection/LoggingDI.cs +++ b/code/Executable/DependencyInjection/LoggingDI.cs @@ -52,6 +52,7 @@ public static class LoggingDI Log.Logger = logConfig .MinimumLevel.Is(level) .MinimumLevel.Override("Microsoft.Hosting.Lifetime", LogEventLevel.Information) + .MinimumLevel.Override("Microsoft.Extensions.Http", LogEventLevel.Warning) .MinimumLevel.Override("Quartz", LogEventLevel.Warning) .MinimumLevel.Override("System.Net.Http.HttpClient", LogEventLevel.Error) .WriteTo.Console(new ExpressionTemplate(consoleOutputTemplate.Replace("PAD", padding.ToString()))) diff --git a/code/Executable/DependencyInjection/MainDI.cs b/code/Executable/DependencyInjection/MainDI.cs index fe8bd46..0e6d9c5 100644 --- a/code/Executable/DependencyInjection/MainDI.cs +++ b/code/Executable/DependencyInjection/MainDI.cs @@ -19,6 +19,7 @@ public static class MainDI .AddLogging(builder => builder.ClearProviders().AddConsole()) .AddHttpClients() .AddConfiguration(configuration) + .AddMemoryCache() .AddServices() .AddQuartzServices(configuration); diff --git a/code/Executable/DependencyInjection/QuartzDI.cs b/code/Executable/DependencyInjection/QuartzDI.cs index 32a18bf..534d9a8 100644 --- a/code/Executable/DependencyInjection/QuartzDI.cs +++ b/code/Executable/DependencyInjection/QuartzDI.cs @@ -6,6 +6,7 @@ using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.Jobs; using Infrastructure.Verticals.QueueCleaner; using Quartz; +using Quartz.Spi; namespace Executable.DependencyInjection; @@ -94,7 +95,20 @@ public static class QuartzDI { return; } + + var triggerObj = (IOperableTrigger)TriggerBuilder.Create() + .WithIdentity("ExampleTrigger") + .StartNow() + .WithCronSchedule(trigger) + .Build(); + var nextFireTimes = TriggerUtils.ComputeFireTimes(triggerObj, null, 2); + + if (nextFireTimes[1] - nextFireTimes[0] > TimeSpan.FromHours(1)) + { + throw new Exception($"{trigger} should have a fire time of maximum 1 hour"); + } + q.AddTrigger(opts => { opts.ForJob(typeName) diff --git a/code/Executable/DependencyInjection/ServicesDI.cs b/code/Executable/DependencyInjection/ServicesDI.cs index a5a62bb..ed9ec24 100644 --- a/code/Executable/DependencyInjection/ServicesDI.cs +++ b/code/Executable/DependencyInjection/ServicesDI.cs @@ -1,10 +1,10 @@ -using Executable.Jobs; -using Infrastructure.Verticals.Arr; +using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.ContentBlocker; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.DownloadClient.Deluge; using Infrastructure.Verticals.DownloadClient.QBittorrent; using Infrastructure.Verticals.DownloadClient.Transmission; +using Infrastructure.Verticals.ItemStriker; using Infrastructure.Verticals.QueueCleaner; namespace Executable.DependencyInjection; @@ -23,5 +23,6 @@ public static class ServicesDI .AddTransient() .AddTransient() .AddTransient() - .AddSingleton(); + .AddSingleton() + .AddSingleton(); } \ No newline at end of file diff --git a/code/Executable/Executable.csproj b/code/Executable/Executable.csproj index 3a1ecc4..023e980 100644 --- a/code/Executable/Executable.csproj +++ b/code/Executable/Executable.csproj @@ -9,15 +9,15 @@ - - + + - + - - + + diff --git a/code/Executable/Jobs/GenericJob.cs b/code/Executable/Jobs/GenericJob.cs index 8fc8adc..48dc727 100644 --- a/code/Executable/Jobs/GenericJob.cs +++ b/code/Executable/Jobs/GenericJob.cs @@ -11,7 +11,6 @@ public sealed class GenericJob : IJob private readonly ILogger> _logger; private readonly T _handler; - public GenericJob(ILogger> logger, T handler) { _logger = logger; diff --git a/code/Executable/appsettings.Development.json b/code/Executable/appsettings.Development.json index aea9045..f0c76fa 100644 --- a/code/Executable/appsettings.Development.json +++ b/code/Executable/appsettings.Development.json @@ -24,7 +24,9 @@ }, "QueueCleaner": { "Enabled": true, - "RunSequentially": true + "RunSequentially": true, + "IMPORT_FAILED_MAX_STRIKES": 5, + "STALLED_MAX_STRIKES": 5 }, "DOWNLOAD_CLIENT": "qbittorrent", "qBittorrent": { diff --git a/code/Executable/appsettings.json b/code/Executable/appsettings.json index 4c1011a..cd298b7 100644 --- a/code/Executable/appsettings.json +++ b/code/Executable/appsettings.json @@ -24,7 +24,9 @@ }, "QueueCleaner": { "Enabled": true, - "RunSequentially": true + "RunSequentially": true, + "IMPORT_FAILED_MAX_STRIKES": 5, + "STALLED_MAX_STRIKES": 5 }, "DOWNLOAD_CLIENT": "qbittorrent", "qBittorrent": { diff --git a/code/Infrastructure/Infrastructure.csproj b/code/Infrastructure/Infrastructure.csproj index 69df953..9e876b0 100644 --- a/code/Infrastructure/Infrastructure.csproj +++ b/code/Infrastructure/Infrastructure.csproj @@ -13,7 +13,8 @@ - + + diff --git a/code/Infrastructure/Verticals/Arr/ArrClient.cs b/code/Infrastructure/Verticals/Arr/ArrClient.cs index a95bc46..ac03ce4 100644 --- a/code/Infrastructure/Verticals/Arr/ArrClient.cs +++ b/code/Infrastructure/Verticals/Arr/ArrClient.cs @@ -1,8 +1,10 @@ -using Common.Configuration; -using Common.Configuration.Arr; +using Common.Configuration.Arr; using Common.Configuration.Logging; -using Domain.Arr.Queue; +using Common.Configuration.QueueCleaner; +using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,17 +16,28 @@ public abstract class ArrClient protected readonly ILogger _logger; protected readonly HttpClient _httpClient; protected readonly LoggingConfig _loggingConfig; + protected readonly QueueCleanerConfig _queueCleanerConfig; + protected readonly Striker _striker; - protected ArrClient(ILogger logger, IHttpClientFactory httpClientFactory, IOptions loggingConfig) + protected ArrClient( + ILogger logger, + IHttpClientFactory httpClientFactory, + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) { _logger = logger; + _striker = striker; _httpClient = httpClientFactory.CreateClient(); _loggingConfig = loggingConfig.Value; + _queueCleanerConfig = queueCleanerConfig.Value; + _striker = striker; } public virtual async Task GetQueueItemsAsync(ArrInstance arrInstance, int page) { - Uri uri = new(arrInstance.Url, $"/api/v3/queue?page={page}&pageSize=200&sortKey=timeleft"); + Uri uri = new(arrInstance.Url, GetQueueUrlPath(page)); using HttpRequestMessage request = new(HttpMethod.Get, uri); SetApiKey(request, arrInstance.ApiKey); @@ -52,6 +65,28 @@ public abstract class ArrClient return queueResponse; } + public virtual bool ShouldRemoveFromQueue(QueueRecord record) + { + bool hasWarn() => record.TrackedDownloadStatus + .Equals("warning", StringComparison.InvariantCultureIgnoreCase); + bool isImportBlocked() => record.TrackedDownloadState + .Equals("importBlocked", StringComparison.InvariantCultureIgnoreCase); + bool isImportPending() => record.TrackedDownloadState + .Equals("importPending", StringComparison.InvariantCultureIgnoreCase); + + if (hasWarn() && (isImportBlocked() || isImportPending())) + { + return _striker.StrikeAndCheckLimit( + record.DownloadId, + record.Title, + _queueCleanerConfig.ImportFailedMaxStrikes, + StrikeType.ImportFailed + ); + } + + return false; + } + public virtual async Task DeleteQueueItemAsync(ArrInstance arrInstance, QueueRecord queueRecord) { Uri uri = new(arrInstance.Url, $"/api/v3/queue/{queueRecord.Id}?removeFromClient=true&blocklist=true&skipRedownload=true&changeCategory=false"); @@ -76,6 +111,25 @@ public abstract class ArrClient public abstract Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items); + public virtual bool IsRecordValid(QueueRecord record) + { + if (string.IsNullOrEmpty(record.DownloadId)) + { + _logger.LogDebug("skip | download id is null for {title}", record.Title); + return false; + } + + if (record.DownloadId.Equals(record.Title, StringComparison.InvariantCultureIgnoreCase)) + { + _logger.LogDebug("skip | item is not ready yet | {title}", record.Title); + return false; + } + + return true; + } + + protected abstract string GetQueueUrlPath(int page); + protected virtual void SetApiKey(HttpRequestMessage request, string apiKey) { request.Headers.Add("x-api-key", apiKey); diff --git a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs index d5c01de..9f87a6c 100644 --- a/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs +++ b/code/Infrastructure/Verticals/Arr/ArrQueueIterator.cs @@ -1,6 +1,6 @@ using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; +using Domain.Models.Arr.Queue; using Microsoft.Extensions.Logging; namespace Infrastructure.Verticals.Arr; diff --git a/code/Infrastructure/Verticals/Arr/RadarrClient.cs b/code/Infrastructure/Verticals/Arr/RadarrClient.cs index a768a35..a5bc757 100644 --- a/code/Infrastructure/Verticals/Arr/RadarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/RadarrClient.cs @@ -1,8 +1,12 @@ using System.Text; using Common.Configuration.Arr; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Domain.Models.Radarr; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,11 +18,18 @@ public sealed class RadarrClient : ArrClient public RadarrClient( ILogger logger, IHttpClientFactory httpClientFactory, - IOptions loggingConfig - ) : base(logger, httpClientFactory, loggingConfig) + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } + protected override string GetQueueUrlPath(int page) + { + return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownMovieItems=true&includeMovie=true"; + } + public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) { if (items?.Count is null or 0) @@ -59,6 +70,17 @@ public sealed class RadarrClient : ArrClient } } + public override bool IsRecordValid(QueueRecord record) + { + if (record.MovieId is 0) + { + _logger.LogDebug("skip | item information missing | {title}", record.Title); + return false; + } + + return base.IsRecordValid(record); + } + private static string GetSearchLog(Uri instanceUrl, RadarrCommand command, bool success, string? logContext) { string status = success ? "triggered" : "failed"; diff --git a/code/Infrastructure/Verticals/Arr/SonarrClient.cs b/code/Infrastructure/Verticals/Arr/SonarrClient.cs index 307eb15..2f6f119 100644 --- a/code/Infrastructure/Verticals/Arr/SonarrClient.cs +++ b/code/Infrastructure/Verticals/Arr/SonarrClient.cs @@ -1,8 +1,12 @@ using System.Text; using Common.Configuration.Arr; using Common.Configuration.Logging; +using Common.Configuration.QueueCleaner; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Domain.Models.Sonarr; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Caching.Memory; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Newtonsoft.Json; @@ -14,10 +18,17 @@ public sealed class SonarrClient : ArrClient public SonarrClient( ILogger logger, IHttpClientFactory httpClientFactory, - IOptions loggingConfig - ) : base(logger, httpClientFactory, loggingConfig) + IOptions loggingConfig, + IOptions queueCleanerConfig, + Striker striker + ) : base(logger, httpClientFactory, loggingConfig, queueCleanerConfig, striker) { } + + protected override string GetQueueUrlPath(int page) + { + return $"/api/v3/queue?page={page}&pageSize=200&includeUnknownSeriesItems=true&includeSeries=true"; + } public override async Task RefreshItemsAsync(ArrInstance arrInstance, ArrConfig config, HashSet? items) { @@ -26,11 +37,9 @@ public sealed class SonarrClient : ArrClient return; } - SonarrConfig sonarrConfig = (SonarrConfig)config; - Uri uri = new(arrInstance.Url, "/api/v3/command"); - foreach (SonarrCommand command in GetSearchCommands(sonarrConfig.SearchType, items)) + foreach (SonarrCommand command in GetSearchCommands(items.Cast().ToHashSet())) { using HttpRequestMessage request = new(HttpMethod.Post, uri); request.Content = new StringContent( @@ -41,22 +50,33 @@ public sealed class SonarrClient : ArrClient SetApiKey(request, arrInstance.ApiKey); using HttpResponseMessage response = await _httpClient.SendAsync(request); - string? logContext = await ComputeCommandLogContextAsync(arrInstance, command, sonarrConfig.SearchType); + string? logContext = await ComputeCommandLogContextAsync(arrInstance, command, command.SearchType); try { response.EnsureSuccessStatusCode(); - _logger.LogInformation("{log}", GetSearchLog(sonarrConfig.SearchType, arrInstance.Url, command, true, logContext)); + _logger.LogInformation("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, true, logContext)); } catch { - _logger.LogError("{log}", GetSearchLog(sonarrConfig.SearchType, arrInstance.Url, command, false, logContext)); + _logger.LogError("{log}", GetSearchLog(command.SearchType, arrInstance.Url, command, false, logContext)); throw; } } } + public override bool IsRecordValid(QueueRecord record) + { + if (record.EpisodeId is 0 || record.SeriesId is 0) + { + _logger.LogDebug("skip | item information missing | {title}", record.Title); + return false; + } + + return base.IsRecordValid(record); + } + private static string GetSearchLog( SonarrSearchType searchType, Uri instanceUrl, @@ -191,7 +211,7 @@ public sealed class SonarrClient : ArrClient return JsonConvert.DeserializeObject(responseBody); } - private List GetSearchCommands(SonarrSearchType searchType, HashSet items) + private List GetSearchCommands(HashSet items) { const string episodeSearch = "EpisodeSearch"; const string seasonSearch = "SeasonSearch"; @@ -199,13 +219,13 @@ public sealed class SonarrClient : ArrClient List commands = new(); - foreach (SearchItem item in items) + foreach (SonarrSearchItem item in items) { - SonarrCommand command = searchType is SonarrSearchType.Episode + SonarrCommand command = item.SearchType is SonarrSearchType.Episode ? commands.FirstOrDefault() ?? new() { Name = episodeSearch, EpisodeIds = new() } : new(); - switch (searchType) + switch (item.SearchType) { case SonarrSearchType.Episode when command.EpisodeIds is null: command.EpisodeIds = [item.Id]; @@ -227,15 +247,16 @@ public sealed class SonarrClient : ArrClient break; default: - throw new ArgumentOutOfRangeException(nameof(searchType), searchType, null); + throw new ArgumentOutOfRangeException(nameof(item.SearchType), item.SearchType, null); } - if (searchType is SonarrSearchType.Episode && commands.Count > 0) + if (item.SearchType is SonarrSearchType.Episode && commands.Count > 0) { // only one command will be generated for episodes search continue; } + command.SearchType = item.SearchType; commands.Add(command); } diff --git a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs index 880530f..7c5d1d9 100644 --- a/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs +++ b/code/Infrastructure/Verticals/ContentBlocker/ContentBlocker.cs @@ -1,7 +1,7 @@ using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; using Domain.Enums; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -58,12 +58,4 @@ public sealed class ContentBlocker : GenericHandler } }); } - - private ArrClient GetClient(InstanceType type) => - type switch - { - InstanceType.Sonarr => _sonarrClient, - InstanceType.Radarr => _radarrClient, - _ => throw new NotImplementedException($"instance type {type} is not yet supported") - }; } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs index 484fd8f..6e7134a 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Deluge/DelugeService.cs @@ -1,44 +1,46 @@ -using Common.Configuration; using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Domain.Models.Deluge.Response; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; namespace Infrastructure.Verticals.DownloadClient.Deluge; -public sealed class DelugeService : IDownloadService +public sealed class DelugeService : DownloadServiceBase { - private readonly ILogger _logger; private readonly DelugeClient _client; - private readonly FilenameEvaluator _filenameEvaluator; public DelugeService( ILogger logger, IOptions config, IHttpClientFactory httpClientFactory, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; config.Value.Validate(); _client = new (config, httpClientFactory); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { await _client.LoginAsync(); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { hash = hash.ToLowerInvariant(); DelugeContents? contents = null; + + TorrentStatus? status = await GetTorrentStatus(hash); - if (!await HasMinimalStatus(hash)) + if (status?.Hash is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } @@ -51,13 +53,7 @@ public sealed class DelugeService : IDownloadService _logger.LogDebug(exception, "failed to find torrent {hash} in the download client", hash); } - // if no files found, torrent might be stuck in Downloading metadata - if (contents?.Contents?.Count is null or 0) - { - return false; - } - - bool shouldRemove = true; + bool shouldRemove = contents?.Contents?.Count > 0; ProcessFiles(contents.Contents, (_, file) => { @@ -67,15 +63,18 @@ public sealed class DelugeService : IDownloadService } }); - return shouldRemove; + return shouldRemove || IsItemStuckAndShouldRemove(status); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { hash = hash.ToLowerInvariant(); - if (!await HasMinimalStatus(hash)) + TorrentStatus? status = await GetTorrentStatus(hash); + + if (status?.Hash is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return; } @@ -126,22 +125,29 @@ public sealed class DelugeService : IDownloadService await _client.ChangeFilesPriority(hash, sortedPriorities); } - - private async Task HasMinimalStatus(string hash) + + private bool IsItemStuckAndShouldRemove(TorrentStatus status) { - DelugeMinimalStatus? status = await _client.SendRequest( - "web.get_torrent_status", - hash, - new[] { "hash" } - ); - - if (status?.Hash is null) + if (status.State is null || !status.State.Equals("Downloading", StringComparison.InvariantCultureIgnoreCase)) { - _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } - return true; + if (status.Eta > 0) + { + return false; + } + + return StrikeAndCheckLimit(status.Hash!, status.Name!); + } + + private async Task GetTorrentStatus(string hash) + { + return await _client.SendRequest( + "web.get_torrent_status", + hash, + new[] { "hash", "state", "name", "eta" } + ); } private static void ProcessFiles(Dictionary contents, Action processFile) @@ -161,7 +167,7 @@ public sealed class DelugeService : IDownloadService } } - public void Dispose() + public override void Dispose() { } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs new file mode 100644 index 0000000..6ef7184 --- /dev/null +++ b/code/Infrastructure/Verticals/DownloadClient/DownloadServiceBase.cs @@ -0,0 +1,42 @@ +using Common.Configuration.QueueCleaner; +using Domain.Enums; +using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; +using Microsoft.Extensions.Logging; +using Microsoft.Extensions.Options; + +namespace Infrastructure.Verticals.DownloadClient; + +public abstract class DownloadServiceBase : IDownloadService +{ + protected readonly ILogger _logger; + protected readonly QueueCleanerConfig _queueCleanerConfig; + protected readonly FilenameEvaluator _filenameEvaluator; + protected readonly Striker _striker; + + protected DownloadServiceBase( + ILogger logger, + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) + { + _logger = logger; + _queueCleanerConfig = queueCleanerConfig.Value; + _filenameEvaluator = filenameEvaluator; + _striker = striker; + } + + public abstract void Dispose(); + + public abstract Task LoginAsync(); + + public abstract Task ShouldRemoveFromArrQueueAsync(string hash); + + public abstract Task BlockUnwantedFilesAsync(string hash); + + protected bool StrikeAndCheckLimit(string hash, string itemName) + { + return _striker.StrikeAndCheckLimit(hash, itemName, _queueCleanerConfig.StalledMaxStrikes, StrikeType.Stalled); + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs index 7a09f60..d6b0de5 100644 --- a/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/QBittorrent/QBitService.cs @@ -1,33 +1,32 @@ -using Common.Configuration; -using Common.Configuration.DownloadClient; +using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using QBittorrent.Client; namespace Infrastructure.Verticals.DownloadClient.QBittorrent; -public sealed class QBitService : IDownloadService +public sealed class QBitService : DownloadServiceBase { - private readonly ILogger _logger; private readonly QBitConfig _config; private readonly QBittorrentClient _client; - private readonly FilenameEvaluator _filenameEvaluator; public QBitService( ILogger logger, IOptions config, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; _config = config.Value; _config.Validate(); _client = new(_config.Url); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { if (string.IsNullOrEmpty(_config.Username) && string.IsNullOrEmpty(_config.Password)) { @@ -37,13 +36,14 @@ public sealed class QBitService : IDownloadService await _client.LoginAsync(_config.Username, _config.Password); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { TorrentInfo? torrent = (await _client.GetTorrentListAsync(new TorrentListQuery { Hashes = [hash] })) .FirstOrDefault(); if (torrent is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } @@ -55,17 +55,16 @@ public sealed class QBitService : IDownloadService IReadOnlyList? files = await _client.GetTorrentContentsAsync(hash); - // if no files found, torrent might be stuck in Downloading metadata - if (files?.Count is null or 0) + // if all files are marked as skip + if (files?.Count is > 0 && files.All(x => x.Priority is TorrentContentPriority.Skip)) { - return false; + return true; } - // if all files are marked as skip - return files.All(x => x.Priority is TorrentContentPriority.Skip); + return IsItemStuckAndShouldRemove(torrent); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { IReadOnlyList? files = await _client.GetTorrentContentsAsync(hash); @@ -91,8 +90,20 @@ public sealed class QBitService : IDownloadService } } - public void Dispose() + public override void Dispose() { _client.Dispose(); } + + private bool IsItemStuckAndShouldRemove(TorrentInfo torrent) + { + if (torrent.State is not TorrentState.StalledDownload and not TorrentState.FetchingMetadata + and not TorrentState.ForcedFetchingMetadata) + { + // ignore other states + return false; + } + + return StrikeAndCheckLimit(torrent.Hash, torrent.Name); + } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs index 764d4e1..801968e 100644 --- a/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs +++ b/code/Infrastructure/Verticals/DownloadClient/Transmission/TransmissionService.cs @@ -1,6 +1,7 @@ -using Common.Configuration; -using Common.Configuration.DownloadClient; +using Common.Configuration.DownloadClient; +using Common.Configuration.QueueCleaner; using Infrastructure.Verticals.ContentBlocker; +using Infrastructure.Verticals.ItemStriker; using Microsoft.Extensions.Logging; using Microsoft.Extensions.Options; using Transmission.API.RPC; @@ -9,21 +10,20 @@ using Transmission.API.RPC.Entity; namespace Infrastructure.Verticals.DownloadClient.Transmission; -public sealed class TransmissionService : IDownloadService +public sealed class TransmissionService : DownloadServiceBase { - private readonly ILogger _logger; private readonly TransmissionConfig _config; private readonly Client _client; - private readonly FilenameEvaluator _filenameEvaluator; private TorrentInfo[]? _torrentsCache; public TransmissionService( ILogger logger, IOptions config, - FilenameEvaluator filenameEvaluator - ) + IOptions queueCleanerConfig, + FilenameEvaluator filenameEvaluator, + Striker striker + ) : base(logger, queueCleanerConfig, filenameEvaluator, striker) { - _logger = logger; _config = config.Value; _config.Validate(); _client = new( @@ -31,44 +31,45 @@ public sealed class TransmissionService : IDownloadService login: _config.Username, password: _config.Password ); - _filenameEvaluator = filenameEvaluator; } - public async Task LoginAsync() + public override async Task LoginAsync() { await _client.GetSessionInformationAsync(); } - public async Task ShouldRemoveFromArrQueueAsync(string hash) + public override async Task ShouldRemoveFromArrQueueAsync(string hash) { TorrentInfo? torrent = await GetTorrentAsync(hash); - // if no files found, torrent might be stuck in Downloading metadata - if (torrent?.FileStats?.Length is null or 0) + if (torrent is null) { + _logger.LogDebug("failed to find torrent {hash} in the download client", hash); return false; } + + bool shouldRemove = torrent.FileStats?.Length > 0; foreach (TransmissionTorrentFileStats? stats in torrent.FileStats ?? []) { if (!stats.Wanted.HasValue) { // if any files stats are missing, do not remove - return false; + shouldRemove = false; } if (stats.Wanted.HasValue && stats.Wanted.Value) { // if any files are wanted, do not remove - return false; + shouldRemove = false; } } // remove if all files are unwanted - return true; + return shouldRemove || IsItemStuckAndShouldRemove(torrent); } - public async Task BlockUnwantedFilesAsync(string hash) + public override async Task BlockUnwantedFilesAsync(string hash) { TorrentInfo? torrent = await GetTorrentAsync(hash); @@ -108,10 +109,26 @@ public sealed class TransmissionService : IDownloadService FilesUnwanted = unwantedFiles.ToArray(), }); } - - public void Dispose() + + public override void Dispose() { } + + private bool IsItemStuckAndShouldRemove(TorrentInfo torrent) + { + if (torrent.Status is not 4) + { + // not in downloading state + return false; + } + + if (torrent.Eta > 0) + { + return false; + } + + return StrikeAndCheckLimit(torrent.HashString!, torrent.Name!); + } private async Task GetTorrentAsync(string hash) { @@ -120,7 +137,15 @@ public sealed class TransmissionService : IDownloadService if (_torrentsCache is null || torrent is null) { - string[] fields = [TorrentFields.FILES, TorrentFields.FILE_STATS, TorrentFields.HASH_STRING, TorrentFields.ID]; + string[] fields = [ + TorrentFields.FILES, + TorrentFields.FILE_STATS, + TorrentFields.HASH_STRING, + TorrentFields.ID, + TorrentFields.ETA, + TorrentFields.NAME, + TorrentFields.STATUS + ]; // refresh cache _torrentsCache = (await _client.TorrentGetAsync(fields)) diff --git a/code/Infrastructure/Verticals/ItemStriker/Striker.cs b/code/Infrastructure/Verticals/ItemStriker/Striker.cs new file mode 100644 index 0000000..d07a3ae --- /dev/null +++ b/code/Infrastructure/Verticals/ItemStriker/Striker.cs @@ -0,0 +1,57 @@ +using Domain.Enums; +using Microsoft.Extensions.Caching.Memory; +using Microsoft.Extensions.Logging; + +namespace Infrastructure.Verticals.ItemStriker; + +public class Striker +{ + private readonly ILogger _logger; + private readonly IMemoryCache _cache; + private readonly MemoryCacheEntryOptions _cacheOptions; + + public Striker(ILogger logger, IMemoryCache cache) + { + _logger = logger; + _cache = cache; + _cacheOptions = new MemoryCacheEntryOptions() + .SetSlidingExpiration(TimeSpan.FromHours(2)); + } + + public bool StrikeAndCheckLimit(string hash, string itemName, ushort maxStrikes, StrikeType strikeType) + { + if (maxStrikes is 0) + { + return false; + } + + string key = $"{strikeType.ToString()}_{hash}"; + + if (!_cache.TryGetValue(key, out int? strikeCount)) + { + strikeCount = 1; + } + else + { + ++strikeCount; + } + + _logger.LogDebug("item on strike number {strike} | reason {reason} | {name}", strikeCount, strikeType.ToString(), itemName); + _cache.Set(key, strikeCount, _cacheOptions); + + if (strikeCount < maxStrikes) + { + return false; + } + + if (strikeCount > maxStrikes) + { + _logger.LogWarning("blocked item keeps coming back | {name}", itemName); + _logger.LogWarning("be sure to enable \"Reject Blocklisted Torrent Hashes While Grabbing\" on your indexers to reject blocked items"); + } + + _logger.LogInformation("removing item with max strikes | reason {reason} | {name}", strikeType.ToString(), itemName); + + return true; + } +} \ No newline at end of file diff --git a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs index e089b0e..864abd0 100644 --- a/code/Infrastructure/Verticals/Jobs/GenericHandler.cs +++ b/code/Infrastructure/Verticals/Jobs/GenericHandler.cs @@ -1,10 +1,11 @@ using Common.Configuration.Arr; -using Domain.Arr.Queue; using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Microsoft.Extensions.Logging; +using Newtonsoft.Json; namespace Infrastructure.Verticals.Jobs; @@ -88,18 +89,27 @@ public abstract class GenericHandler : IDisposable _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; - protected SearchItem GetRecordSearchItem(InstanceType type, QueueRecord record) => - type switch + protected SearchItem GetRecordSearchItem(InstanceType type, QueueRecord record, bool isPack = false) + { + return type switch { - InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode => new SonarrSearchItem + InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode && !isPack => new SonarrSearchItem { Id = record.EpisodeId, - SeriesId = record.SeriesId + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Episode + }, + InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Episode && isPack => new SonarrSearchItem + { + Id = record.SeasonNumber, + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Season }, InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Season => new SonarrSearchItem { Id = record.SeasonNumber, - SeriesId = record.SeriesId + SeriesId = record.SeriesId, + SearchType = SonarrSearchType.Series }, InstanceType.Sonarr when _sonarrConfig.SearchType is SonarrSearchType.Series => new SonarrSearchItem { @@ -111,4 +121,5 @@ public abstract class GenericHandler : IDisposable }, _ => throw new NotImplementedException($"instance type {type} is not yet supported") }; + } } \ No newline at end of file diff --git a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs index e9e7ae1..ec7098e 100644 --- a/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs +++ b/code/Infrastructure/Verticals/QueueCleaner/QueueCleaner.cs @@ -1,8 +1,8 @@ -using Common.Configuration; using Common.Configuration.Arr; -using Domain.Arr.Queue; +using Common.Configuration.QueueCleaner; using Domain.Enums; using Domain.Models.Arr; +using Domain.Models.Arr.Queue; using Infrastructure.Verticals.Arr; using Infrastructure.Verticals.DownloadClient; using Infrastructure.Verticals.Jobs; @@ -29,34 +29,45 @@ public sealed class QueueCleaner : GenericHandler { HashSet itemsToBeRefreshed = []; ArrClient arrClient = GetClient(instanceType); + ArrConfig arrConfig = GetConfig(instanceType); await _arrArrQueueIterator.Iterate(arrClient, instance, async items => { - foreach (QueueRecord record in items) + var groups = items + .GroupBy(x => x.DownloadId) + .ToList(); + + foreach (var group in groups) { + if (group.Any(x => !arrClient.IsRecordValid(x))) + { + continue; + } + + QueueRecord record = group.First(); + if (record.Protocol is not "torrent") { continue; } - if (string.IsNullOrEmpty(record.DownloadId)) + if (!arrClient.IsRecordValid(record)) { - _logger.LogDebug("skip | download id is null for {title}", record.Title); continue; } - if (!await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId)) + if (!arrClient.ShouldRemoveFromQueue(record) && !await _downloadService.ShouldRemoveFromArrQueueAsync(record.DownloadId)) { _logger.LogInformation("skip | {title}", record.Title); continue; } - - itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record)); + + itemsToBeRefreshed.Add(GetRecordSearchItem(instanceType, record, group.Count() > 1)); await arrClient.DeleteQueueItemAsync(instance, record); } }); - await arrClient.RefreshItemsAsync(instance, GetConfig(instanceType), itemsToBeRefreshed); + await arrClient.RefreshItemsAsync(instance, arrConfig, itemsToBeRefreshed); } } \ No newline at end of file diff --git a/code/test/data/nginx/sonarr.xml b/code/test/data/nginx/sonarr.xml index 344d8fb..0ec28cd 100644 --- a/code/test/data/nginx/sonarr.xml +++ b/code/test/data/nginx/sonarr.xml @@ -44,6 +44,17 @@ Sat, 24 Sep 2022 22:02:13 -0300 + + Top.Gear.S23E03.720p.x265.HDTV.HEVC.-.YSTEAM + Test + 4138858110 + magnet:?xt=urn:btih:cf92cf859b110af0ad3d94b846e006828417b193&dn=TPG.2303.720p.x265.yourserie.com.mkv + + 174674a88c8947f6f5057ac3f81efde384ed216c2de43564ec450f2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + + Top.Gear.S23E01.720p.x265.HDTV.HEVC.-.YSTEAM Test @@ -65,5 +76,16 @@ Sat, 24 Sep 2022 22:02:13 -0300 + + + Sherlock.S01.1080p.BluRay.DD5.1.x264-DON + Test + 4138858110 + http://nginx/custom/sonarr_bad_pack.torrent + + 174674a88c8947f6f9057ac3f82efde384ed216cade43564ec45gf2cb4677554 + + Sat, 24 Sep 2022 22:02:13 -0300 + \ No newline at end of file diff --git a/code/test/data/nginx/sonarr_bad_pack.torrent b/code/test/data/nginx/sonarr_bad_pack.torrent new file mode 100644 index 0000000..c920381 --- /dev/null +++ b/code/test/data/nginx/sonarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent new file mode 100644 index 0000000..baae24e --- /dev/null +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/fa800a7d7c443a2c3561d1f8f393c089036dade1.torrent @@ -0,0 +1 @@ +d10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue index a528236..e5d0f16 100644 --- a/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/BT_backup/queue @@ -1,5 +1 @@ -2b2ec156461d77bc48b8fe4d62cede50dcdff8e0 -a4a1d1dd1db25763caa8f5e4d25ad72ef304094b -b72541215214be2a1d96ef6b29ca1305f5e5e1f6 -59ab2bc053430fe53e06a93e2eadb7acb6a6bf2c 11cece7f8721c484126b66f609d52738ff1bbf1e diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb b/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb index 5e65de0..5157c0f 100644 Binary files a/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb and b/code/test/data/qbittorrent-bad/config/qBittorrent/GeoDB/dbip-country-lite.mmdb differ diff --git a/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf b/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf index 66611c8..25fa858 100644 --- a/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf +++ b/code/test/data/qbittorrent-bad/config/qBittorrent/qBittorrent-data.conf @@ -1,2 +1,2 @@ [Stats] -AllStats=@Variant(\0\0\0\x1c\0\0\0\x2\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0\x44\0L\0\0\0\x4\0\0\0\0\0\x61\xc0\xdf\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0U\0L\0\0\0\x4\0\0\0\0\0\x9b\xf9\x8a) +AllStats=@Variant(\0\0\0\x1c\0\0\0\x2\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0U\0L\0\0\0\x4\0\0\0\0\0\x9dm\x4\0\0\0\x12\0\x41\0l\0l\0t\0i\0m\0\x65\0\x44\0L\0\0\0\x4\0\0\0\0\0\x62_.) diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 0000000..d6def41 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 0000000..d6def41 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv new file mode 100644 index 0000000..d6def41 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/Sherlock.S01.1080p.BluRay.DD5.1.x264-DON/Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkv @@ -0,0 +1 @@ +episode \ No newline at end of file diff --git a/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent b/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent new file mode 100644 index 0000000..c920381 --- /dev/null +++ b/code/test/data/qbittorrent-bad/downloads/sonarr_bad_pack.torrent @@ -0,0 +1 @@ +d8:announce28:http://tracker:6969/announce10:created by26:Enhanced-CTorrent/dnh3.3.213:creation datei1734129464e4:infod5:filesld6:lengthi7e4:pathl47:Sherlock.S01E01.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E02.1080p.BluRay.DD5.1.x264-DON.mkveed6:lengthi7e4:pathl47:Sherlock.S01E03.1080p.BluRay.DD5.1.x264-DON.mkveee4:name40:Sherlock.S01.1080p.BluRay.DD5.1.x264-DON12:piece lengthi262144e6:pieces20:/˜ŽrÎèçƒlY€„·°|¶7ee \ No newline at end of file diff --git a/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session b/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session new file mode 100644 index 0000000..0644f27 --- /dev/null +++ b/code/test/data/sonarr/config/Sentry/07ADDC43B5669C4F6DB64F2EF2B23B3FEEDFE865/.session @@ -0,0 +1 @@ +{"update":{"sid":"4ee000d424144e078e7f3ef208e30647","did":"1df9f2cc-17dc-4130-9753-9b694f82f1b5","init":true,"started":"2024-12-13T22:41:57.8197572+00:00","timestamp":"2024-12-13T22:41:57.8202577+00:00","seq":0,"duration":0,"errors":0,"attrs":{"release":"4.0.10.2544-main","environment":"main"}}} \ No newline at end of file diff --git a/code/test/data/sonarr/config/logs.db b/code/test/data/sonarr/config/logs.db index 6999d2e..1198ce5 100644 Binary files a/code/test/data/sonarr/config/logs.db and b/code/test/data/sonarr/config/logs.db differ diff --git a/code/test/data/sonarr/config/logs.db-shm b/code/test/data/sonarr/config/logs.db-shm index 9617b44..9558dac 100644 Binary files a/code/test/data/sonarr/config/logs.db-shm and b/code/test/data/sonarr/config/logs.db-shm differ diff --git a/code/test/data/sonarr/config/logs.db-wal b/code/test/data/sonarr/config/logs.db-wal index 25ae729..443f505 100644 Binary files a/code/test/data/sonarr/config/logs.db-wal and b/code/test/data/sonarr/config/logs.db-wal differ diff --git a/code/test/data/sonarr/config/sonarr.db b/code/test/data/sonarr/config/sonarr.db index 5db88d2..7802f0f 100644 Binary files a/code/test/data/sonarr/config/sonarr.db and b/code/test/data/sonarr/config/sonarr.db differ diff --git a/code/test/data/sonarr/config/sonarr.db-shm b/code/test/data/sonarr/config/sonarr.db-shm index be74b3a..f5c93f4 100644 Binary files a/code/test/data/sonarr/config/sonarr.db-shm and b/code/test/data/sonarr/config/sonarr.db-shm differ diff --git a/code/test/data/sonarr/config/sonarr.db-wal b/code/test/data/sonarr/config/sonarr.db-wal index 475ce06..ebd3086 100644 Binary files a/code/test/data/sonarr/config/sonarr.db-wal and b/code/test/data/sonarr/config/sonarr.db-wal differ diff --git a/code/test/data/sonarr/config/sonarr.pid b/code/test/data/sonarr/config/sonarr.pid index bc768da..70e1a64 100644 --- a/code/test/data/sonarr/config/sonarr.pid +++ b/code/test/data/sonarr/config/sonarr.pid @@ -1 +1 @@ -146 \ No newline at end of file +144 \ No newline at end of file diff --git a/code/test/docker-compose.yml b/code/test/docker-compose.yml index 910eab9..e7fc73f 100644 --- a/code/test/docker-compose.yml +++ b/code/test/docker-compose.yml @@ -172,7 +172,7 @@ services: container_name: cleanuperr environment: - LOGGING__LOGLEVEL=Debug - - LOGGING__FILE__ENABLED=false + - LOGGING__FILE__ENABLED=true - LOGGING__FILE__PATH=/var/logs - LOGGING__ENHANCED=true @@ -181,6 +181,8 @@ services: - QUEUECLEANER__ENABLED=true - QUEUECLEANER__RUNSEQUENTIALLY=true + - QUEUECLEANER__IMPORT_FAILED_MAX_STRIKES=5 + - QUEUECLEANER__STALLED_MAX_STRIKES=5 - CONTENTBLOCKER__ENABLED=true - CONTENTBLOCKER__BLACKLIST__ENABLED=true