Why does Laravel / Horizon create `queue:failed` processes that eat up memory?

We have recently encountered the problem that our laravel server creates many processes using the command php artisan queue:failed that take about 1.2 GB of RAM each and don't seem to resolve on their own.

htop looks like this:

It is entirely possible that this is some weird edge-case configuration that we inherited from the previous developers, but maybe somebody has encountered a similar problem.

The laravel version is 5.7, horizon is 3.7 on php version 7.2.34-23. The horizon config is (we assume the queue workers are started by horizon):

[
    'domain' => null,
    'path' => 'horizon',
    'use' => 'default',
    'prefix' => env('HORIZON_PREFIX', 'horizon:'),
    'middleware' => ['web', 'auth'],
    'waits' => [ 'redis:default' => 600 ],
    'trim' => [
        'recent' => 60,
        'recent_failed' => 4320,
        'failed' => 4320,
        'monitored' => 10080,
    ],
    'fast_termination' => false,
    'memory_limit' => 1024,
    'environments' => [
        'production' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['high', 'normal'],
                'balance' => 'auto',
                'minProcesses' => 1,
                'maxProcesses' => 20,
                'maxJobs' => 1000,
                'maxTime' => 3600,
                'tries' => 1,
                'timeout' => 90,
                'sleep' => 3,
            ],
            'supervisor-2' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'auto',
                'minProcesses' => 2,
                'maxProcesses' => 20,
                'maxJobs' => 1000,
                'maxTime' => 3600,
                'tries' => 1,
                'timeout' => 90,
                'sleep' => 3,
            ],
            'supervisor-3' => [
                'connection' => 'redis-long-running',
                'queue' => ['low', 'very low'],
                'balance' => 'simple',
                'minProcesses' => 1,
                'maxProcesses' => 10,
                'maxJobs' => 100,
                'maxTime' => 3600,
                'tries' => 1,
                'timeout' => 3600,
                'sleep' => 3,
            ],
        ],

        'local' => [
            'supervisor-1' => [
                'connection' => 'redis',
                'queue' => ['default'],
                'balance' => 'simple',
                'processes' => 3,
                'tries' => 1,
            ],
        ],
    ],
]

Update

Running php artisan horizon:terminate seems to kill the queue:failed processes. The problem seems to appear when running a ShouldQueue Job directly from php artisan tinker.

via Chebli Mohamed