[Release] GTRU FlexGet YAML Script 2.3 - Automatic RSS Downloader

[LoopyD]

If you're looking for automatic torrent management, I have a thing for your seedbox. Power Users only, please.

DISCLAIMER: The baseline guide is provided, and doesn't work out of the box. You will have to configure it. I assume no responsibility for data loss as a result of misconfiguration.

Baseline Installation instructions:

I have been using flexget here for just over 7 years, and really swear by it on a properly configured seedbox.

This flexget configuration is compatible with Deluge, bittorrent software: https://dev.deluge-torrent.org/wiki/Download

You will require of course, flexget which can be found here: https://flexget.com/InstallWizard/Linux , which does need python 3.6 or above, and the deluge-client module installed via pip.

By default, my install assigns labels to the torrents, so you'll need to enable the LabelPlus plugin for deluge: https://forum.deluge-torrent.org/viewtopic.php?f=9&t=42629 Installation instructions for all deluge plugins are here: https://dev.deluge-torrent.org/wiki/Plugins#Download .

If your LabelPlus egg does not load when you import it, its because you've selected the wrong python version. You may need to adjust your python version for best results. This is why using virtualenv (or anaconda) is smart instead of the systemwide plugin.

Install flexget into python virtualenv following the instructions on the flexget website.

Configure scripts:

Set host to your server IP
Set port to Deluge daemon port
Set username to Deluge WebUI username in your auth file in your Deluge config directory.
Set password to Deluge WebUI password in your auth file in your Deluge config directory.
Change paths as nessecary per your configuration.

Install with supervisord:

/etc/supervisor/conf.d/flexget-supervisor.conf

[program:flexget]
command=/mnt/shared/config/flexget/flexget-daemon.sh
directory=/mnt/shared/config/flexget
user=heavypaws
redirect_stderr=true
stdout_logfile=/mnt/shared/logs/flexget/flexget-stdout.log
numprocess=1
autostart=true
startretries=3
autorestart=true
stopasgroup=true
killasgroup=true
stdout_logfile_maxbytes=0

terminal

sudo supervisorctl update
sudo supervisorctl reload
sudo supervisorctl enable

Install your script:

/mnt/shared/config/flexget/flexget-daemon.sh

#!/bin/bash

my_root_dir='/mnt/shared/config/flexget'
my_log_dir='/mnt/shared/logs/flexget'
my_app_dir="${my_root_dir}/app"

pkill "${my_app_dir}/bin/flexget" >/dev/null 2>&1
. $my_app_dir/bin/activate
if [[ -f "${my_log_dir}/flexget.log" ]]; then
        rm -f "${my_log_dir}/flexget.log"
fi
if [[ -f "${my_log_dir}/flexget-stdout.log" ]]; then
        rm -f "${my_log_dir}/flexget-stdout.log"
fi
pushd "${my_root_dir}/app"
flexget -c ${my_root_dir}/app-config.yml --logfile ${my_log_dir}/flexget.log daemon start
pop

Make the script executable:

terminal

sudo chmod +x /mnt/shared/config/flexget/flexget-daemon.sh

Install the configured flexget script:

templates:

  deluge_config_out:
    disable: [retry_failed]
    deluge:
      host: [YOUR_HOSTNAME]
      port: [YOUR_PORT]
      username: [YOUR_DELUGE_USERNAME]
      password: [YOUR_DELUGE_PASSWORD]
      config_path: /mnt/shared/config/deluge/

  deluge_config_in:
    disable: [seen, seen_info_hash]
    from_deluge: 
      host: [YOUR_HOSTNAME]
      port: [YOUR_PORT]
      username: [YOUR_DELUGE_USERNAME]
      password: [YOUR_DELUGE_PASSWORD]
      config_path: /mnt/shared/config/deluge/

  gaytorrent_ru_header:
    headers:
      Cookie: 'uid=[YOUR_UID];pass=[YOUR_PASS]'
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=17'
      link:
        - link
        - magneturi
      other_fields: [description]
      ascii: yes
      escape: yes
      silent: yes
      all_entries: no
    
  gaytorrent_ru_bears:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=17&num=10'

  gaytorrent_ru_chubbies:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=9&num=10'
      
  gaytorrent_ru_grey_older:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=27&num=10'
      
  gaytorrent_ru_mature:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=36&num=10'
      
  gaytorrent_ru_hunks:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=12&num=10'
      
  gaytorrent_ru_bdsm:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=19&num=10'
      
  gaytorrent_ru_daddies_sons:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=5&num=10'
      
  gaytorrent_ru_comic_yaoi:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=48&num=10'
      
  gaytorrent_ru_military:
    rss:
      url: 'https://www.gaytorrent.ru/rss.php?feed=dl&cat=38&num=10'

  content_blocker:
    regexp:
      reject:
        - "young|children|child|twink|femboy|ageplay|abdl|diaper|freshmen|freshman|junior|senior|sophomore|high school|highschool|barely 18|barely eighteen|teen|schoolboy|school|barely legal"
        - "rape|noncon|non-con|smegma|snegma"
        - "smoking|smoker|drug|drugs|weed|420|pnp|party n play|partynplay|p n p|party and play|partyandplay"
        - "asian|jap|japanese|oriental|black|chinese|oriental|interracial|interacial|gangsta|gangster|gang"
        - "0 seeders|no seeders|dead torrent|no seeds|0 seeds"
      from:
        - title
        - description
    content_filter:
      reject:
        - '*password*.txt'
        - '*password*.lnk'
        - '*.exe'
        - '*.msi'
        - '*.cab'
        - '*.rar'
        - '*.7z'
        - '*.7zip'
        - '*.zip'
        - '*.sfx'
        - '*.stuff'
        - '*.stuffit'
        - '*.stf'
        - '*.dat'
        - '*.bat'
        - '*.cmd'
        - '*.ps1'
        - '*.psm'
        - '*.vbs'
        - '*.vbscript'
        - '*.jscript'
        - '*.js'
        - '*.jsl'
        - '*.ahk'
        - '*.dll'
        - '*.jar'
        - '*.pdb'
        - '*.sln'
        - '*.bin'
        - '*.dump'
        - '*.hex'
        - '*.key'
        - '*.crt'
        - '*.pub'
        - '*cmake*'
        - '*.sh'
        - '*.bash'
        - '*.zsh'
    content_size:
      min: 10
      max: 2500
      strict: yes
    limit_new: 1
    accept_all: yes
  
  add_task_base:
    template:
      - deluge_config_out
      - gaytorrent_ru_header
      - content_blocker
    deluge:
      queue_to_top: yes
      path: "/mnt/shared/AppData/deluge/downloaded"
      action: add
      auto_managed: true
      
tasks:

  add_bear_task:
    template:
      - add_task_base
      - gaytorrent_ru_bears
    deluge:
      label: Bears
      move_completed_path: "/mnt/shared/AppData/plex/videos/Bears/"

  add_chubby_task:
    template:
      - add_task_base
      - gaytorrent_ru_chubbies
    deluge:
      label: Chubs
      move_completed_path: "/mnt/shared/AppData/plex/videos/Chubs/"
      
  add_grey_older_task:
    template:
      - add_task_base
      - gaytorrent_ru_grey_older
    deluge:
      label: GreyAndOlder
      move_completed_path: "/mnt/shared/AppData/plex/videos/Grey_And_Older/"
      
  add_mature_task:
    template:
      - add_task_base
      - gaytorrent_ru_mature
    deluge:
      label: Mature
      move_completed_path: "/mnt/shared/AppData/plex/videos/Mature/"

  add_hunk_task:
    template:
      - add_task_base
      - gaytorrent_ru_hunks
    deluge:
      label: Hunks
      move_completed_path: "/mnt/shared/AppData/plex/videos/Hunks/"

  add_bsdm_task:
    template:
      - add_task_base
      - gaytorrent_ru_bdsm
    deluge:
      label: BDSM
      move_completed_path: "/mnt/shared/AppData/plex/videos/BDSM/"

  add_daddies_sons_task:
    template:
      - add_task_base
      - gaytorrent_ru_daddies_sons
    deluge:
      label: DaddiesAndSons
      move_completed_path: "/mnt/shared/AppData/plex/videos/Daddies_And_Sons/"

  add_comic_and_yaoi_task:
    template:
      - add_task_base
      - gaytorrent_ru_comic_yaoi
    deluge:
      label: ComicAndYiff
      move_completed_path: "/mnt/shared/AppData/plex/videos/Comic_And_Yiff/"

  add_military_task:
    template:
      - add_task_base
      - gaytorrent_ru_military
    deluge:
      label: Military
      move_completed_path: "/mnt/shared/AppData/plex/videos/Military/"
      
  clean_error_task:
    template:
      - deluge_config_in
      - deluge_config_out
    if:
      # torrents with filesystem or tracker issues
      - "'error' in deluge_state|lower": accept
      - "'unregistered torrent' in deluge_tracker_status|lower": accept
      - "'missing files' in deluge_tracker_status|lower": accept
      - "'not exist' in deluge_tracker_status|lower": accept
      - "not deluge_is_finished and not deluge_is_seed and deluge_progress <= 40.0 and torrent_seeds == 0 and not 'checking' in deluge_state|lower and not 'queued' in deluge_state|lower": accept
    deluge:
      action: purge
  
  clean_old_task:
    template:
      - deluge_config_in
      - deluge_config_out
    if:
      # 3 days and hasn't seeded to minimum ratio 0.1
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 0.3 and deluge_seeding_time >= 259200": accept
      # 5 days and hasn't seeded to minimum ratio 0.3
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 0.3 and deluge_seeding_time >= 432000": accept
      # 1 week and hasn't seeded to minimum ratio 0.5
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 0.5 and deluge_seeding_time >= 604800": accept
      # 2 weeks and hasn't seeded to minimum ratio 1.0
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 1.0 and deluge_seeding_time >= 1209600": accept
      # 3 and a half weeks and hasn't seeded to minimum ratio 2.0
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 2.0 and deluge_seeding_time >= 2116800": accept
      # 1 month and hasn't seeded to minimum ratio 3.0
      - "deluge_is_finished and deluge_is_seed and deluge_ratio <= 3.0 and deluge_seeding_time >= 3024000": accept
      # all seeding torrents greater than 3 months old
      - "deluge_is_finished and deluge_is_seed and deluge_ratio >= 3.0 and deluge_seeding_time >= 9072000": accept
      # stalled downloads (hit and runs)
    deluge:
      action: remove

schedules:
  - tasks: [add_bear_task, add_bsdm_task, add_chubby_task, add_comic_and_yaoi_task, add_daddies_sons_task, add_grey_older_task, add_hunk_task, add_mature_task, add_military_task]
    interval:
      minutes: 15
      jitter: 300
  - tasks: clean_error_task
    interval:
      minutes: 60
  - tasks: clean_old_task
    interval:
      days: 1

Don't forget to taylor all the scripts to your preferences, including your paths, and your cookies!

If you want more categories, you can add them in the YAML and then add them a task and add it to the list of tasks that execute in the scheduler. If you want to block more or block less, you can reconfigure content_blocker task.

You can start/stop it manually via:

terminal

sudo supervisorctl start flexget
sudo supervisorctl stop flexget

The main thing here is that the majority of work has been done here by an IT professional. Customize to your liking. The version if it I have shared here works real well with plex collections . So that you can watch porn in a private repository of your own.

Have fun. The Release is GPL 3.0 licensed.