There have been multiple accounts created with the sole purpose of posting advertisement posts or replies containing unsolicited advertising.

Accounts which solely post advertisements, or persistently post them may be terminated.

Data HDD with SSD catch drive

After upgrading my internet connection I immediatelly noticed that my HDD tops 40 MB/s and bottlnecking download speed in qbittorrent. Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on? If yes, does it make sense? Anyone using anything simmilar? Would 512 GB be enough or could I benefit from 2TB SSD?

HDD is just for jellyfin (movies/shows), not in raid, dont need backup for that drive, I can afford risking data if that matters at all

All suggestions are welcome, Thx in advance

ShortN0te ,

40MB/s is very very low even for a HDD. I would eventually debug why it’s that low.

Yes it’s possible. FS like zfs btrfs etc. support that.

acosmichippo , (edited )
@acosmichippo@lemmy.world avatar

agreed, I think there is something else going on here. test the write speed with another application, I doubt the drive actually maxes out at 40MB/s unless it’s severely fragmented or failing.

incidentally what OP wants is how most people set up Unraid servers. SSD cache takes incoming files for write speed, then at a later time the OS moves the files to the spinning disk array.

rambos OP ,

Its the cheapest drive I could find (refurbished seagate from amazon), I thought thats the reason for being slow, but wasnt aware its that low. Im also getting 25-40 MB/s (200-320 Mbps) when copying files from this drive over network. Streaming works great so its not too slow at all. Is there better way of debugging this? What speeds can I expect from good drive or best drive?

Ill research more about BTRFS and ZFS, thx

acosmichippo ,
@acosmichippo@lemmy.world avatar

can you copy files to it from another local disk?

rambos OP ,

Yeah, but need to figure out how to see transfer speed using ssh. Sorry noob here :)

not_fond_of_reddit ,

If you use scp (cp over ssh) you should see the transfer speed.

rambos OP ,

I have managed to copy with rsync and getting 180 MB/s. I guess my initial assumption was wrong, HDD is obviously not bottleneck here, it can get close to ISP speed. Thank you for pointing this out, Ill do more testing these days. Im kinda shocked because I never knew HDD can be that fast. Gonna reread all the comments as well

possiblylinux127 ,

Btrfs doesn’t support using a cache drive

catloaf ,

It’s probably a 5400rpm drive, and/or SMR. Both are going to make it slower.

johntash ,

Unraid has this with their cache pools. ZFS can also be configured to have a cache drive for writes.

You can also DIY with something like mergerfs and separate file systems.

rambos OP ,

Ive heard about all of these before, gonna do more research. Thank you

braindefragger ,

Yes. It’s part of the application and well documented. What did you try and not work?

rambos OP ,

Are you also talking about incomplete directory in qbit? Doesnt make it faster afaik, but I might be wrong. I havent tried anything yet, wanted to check is it something usual or not worth at all. Got zero experience with using SSD as catch drive, it just made sense to me

braindefragger ,

Yes, if the temporary directory where the files are being downloaded (incomplete folder) is on the SSD, then it will be faster, especially if you’ve identified a cheap HDD as your bottleneck.

Unless you are incorrect about the HDD being the bottleneck.

rambos OP ,

Yeah it will be faster, but its extra step before the files get available on HDD.

Even if my HDD is super fast and healthy it would still be a bottleneck for 2Gbps fiber? Ill deffo play with HDD more to find max speeds, wasnt paying attention before because it felt normal to me

braindefragger , (edited )

Of course it’s an additional step. But it will download faster. Which was what you asked for, specifically in your post above.

If you write directly to your HDD it will take longer to download. If you write to your ( faster?) SSD the download will be faster but yeah, processing has another step of copying.

I’m sorry, but I have no idea what you’re asking.

Best of luck.

rambos OP ,

Yeah feels like that lol. Thx anyway, have a nice day dude

acosmichippo ,
@acosmichippo@lemmy.world avatar

what OP wants is to download the file to a SSD, be able to use it on the SSD for a time, and then have the file moved to spinning disk later when they don’t need to wait for it.

this is just adding an extra step to the process before the file can be available to use. you’re just saving the copying to the HDD until the very end of the torrent.

braindefragger ,

Yeah, of course it is. Because that’s what OP asked for. I don’t see ( use it for a bit first and then automatically copy it over ).

I see:

Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on?

I assumed OP wanted Faster Download Speeds > Time to Access File

You know what. I don’t care. This whole post is ridiculous.

acosmichippo ,
@acosmichippo@lemmy.world avatar

what is the point of faster download if you just have to do another entire copy after that?

braindefragger ,

Ask OP

acosmichippo ,
@acosmichippo@lemmy.world avatar

or you could, you know, think about it for a second from their point of view. and they have already clarified this in other comments.

braindefragger , (edited )

Man you’re slow.

I don’t need to think about anything. I’ve built out millions upon millions of dollars worth of infrastructure over the past 20 years. I understand the options.

I answered the question asked. Others hinted at raid solutions. OP went out of his way to let us know how he’s a newb so good luck with that.

Long term solution is 100% setting up a raid.

acosmichippo ,
@acosmichippo@lemmy.world avatar

wtf does raid have to do with anything here? yeah, sure, I’m the slow one.

braindefragger ,

Wow. Good luck with life bro.

acosmichippo ,
@acosmichippo@lemmy.world avatar

the dude asks about SSD cash for torrents and your multimillion-dollar answer is “raid”. lol

as people have already pointed out multiple times, what OP wants is something like mergerfs or unraid which can handle files on SSD cash and then move to spinning disks later.

DaGeek247 ,
@DaGeek247@fedia.io avatar

Yeah, I use the incomplete folder location as a cache drive for my downloads as well. works quite nicely. It also keeps the incomplete ISOs out of jellyfin until they're actually ready to watch, so, bonus.

If it's not going faster for you there's probably something else that's broke.

rambos OP ,

It will download faster to SSD, but then I have to wait the files to be moved to HDD before getting them imported in media server. Im not after big numbers in qbit, I just want to start watching faster if possible. Sorry Im probably not explaining well and Im not sure if Im asking for something that even make sense

DaGeek247 ,
@DaGeek247@fedia.io avatar

qbittorrent moves the completed files to the assigned literally as soon as it is done.

rambos OP ,

Im doing more research, but will defo test this

acosmichippo ,
@acosmichippo@lemmy.world avatar

but if the disk is actually bottlenecking at 40MB/s it will still take time to copy from the SSD. That plus the initial download to SSD will just end up being more time than downloading to the spinning disk at 40MB/s in the first place.

theterrasque ,

I doubt the disk will bottleneck at 40mb/s when doing sequential write. Torrent downloads are usually heavy random writes, which is the worst you can do to a HDD.

DaGeek247 ,
@DaGeek247@fedia.io avatar

That's not how hard drives work, and doesn't take into account that OP might want to download more than one thing at a time.

Hard drives are fastest when they are moving large single files. SSDs are way better than hard drives at lots of small random reads/writes.Setting qbittorrent up so that all the random writes inherent to downloading a torrent go to a small ssd, and then moving that file over to the big hard drive with a single long writer operation is how you make both devices perform to their best.

slazer2au ,

You can and Qbittorrent has this functionality built in. You set your in progress download folder to be the SSD then set the move when completed to your HDD.

As for the size, that would depend on how much you are downloading.

rambos OP ,

But that would first download to SSD, then move to HDD and then become available (arr import) on jellyfin server, making it slower than not using SSD. Am I missing something?

BombOmOm ,
@BombOmOm@lemmy.world avatar

The biggest thing is you have changed a random write to a linear write, something HDDs are significantly better at. The torrent is downloading little pieces from all over the place, requiring the HDD to move it’s head all over the place to write them. But when simply copying off the ssd, it keeps the head in roughly one place and just writes lineally, utilizing it’s maximum write speed.

I would say try it out, see if it helps.

Also, if the HDD is having to do other tasks at the same time, that will slow it down as the head can only ever be in one place.

rambos OP ,

I might try that, thx

Maxy ,

qBittorrent has exactly the option you’re looking for, I believe it’s called “incomplete download path” in the settings, letting you store incomplete downloads at a temporary path and moving them to their regular location when the download finishes. Aside from the download speed improvement, this will also lead to less fragmentation on your HDD (which might be part of the reason why it is so slow when downloading directly to it). Pre-allocating space could have the same effect, but I would recommend only using one of these two solutions at once (pre-allocating space on your SSD would only waste space)

rambos OP ,

But that would first download to SSD, then move to HDD and then become available (arr import) on jellyfin server, making it slower than not using SSD. Am I missing something?

braindefragger ,

Is it possible to use SSD drive as a catch drive for 12 TB HDD so it uses SSD speeds when downloading and moves files to HDD later on?

Is that not what you asked for?

rambos OP ,

Well yes, but I was hoping files can be available (imported to media server) before they are moved to HDD. Import is not possible from incomplete directory if I understood that correctly (*arr stack)

catloaf ,

You would have to add both directories to your library.

Lost_My_Mind ,

Great that you have a catch drive. I assume the data drive manages everything. So I’m going to call that the manager drive.

Now you just need:

A 1st base drive.

A 2nd base drive.

A 3rd base drive.

A shortstop drive.

A left Field drive.

A center field drive.

A right field drive.

About 3-4 starting drives

A half dozen reliever drives.

A closer drive.

A hittch coach drive

And a couple of base running coach drives!

Got yourself a baseball team!

capital , (edited )

I do this with mergerfs.

I then periodically use their prewritten scripts to move things off the cache and to the backing drives.

I should say it’s not really caching but effectively works to take care of this issue. Bonus since all that storage isn’t just used for cache but also long term storage. For me, that’s a better value proposition.

rambos OP ,

Thanks, Ill check mergefs

fiddlesticks ,
@fiddlesticks@lemmy.dbzer0.com avatar

Depends on the file system, I know for a fact that ZFS supports ssd caches (in the form of l2arc and slog) and I believe that lvm does something similar (although I’ve never used it).

As for the size, it really depends how big the downloads are if you’re not downloading the biggest 4k movies in existence then you should be fine with something reasonably small like a 250 or 500gb ssd (although I’d always recommend higher because of durability and speed)

lemmylommy ,

l2arc is a read cache. Slog only is for synchronous writes.

fiddlesticks ,
@fiddlesticks@lemmy.dbzer0.com avatar

Welp, guess I should do my research next time. Thanks for the heads up.

rambos OP ,

Thx. I use ext4 right now. I might consider reformating, but so many new words to reasearch before deciding that. I heard about ZFS, but not sure is that right for me since I only have 16 GB of RAM.

Downloads are 100-200 GB max, but less than 40 GB most of the time. I have 512 GB in use and 2TB SSD not in use, can swap them if needed

possiblylinux127 ,

bcachefs will fill this role someday.

For now there is ZFS which as a cache drive option. Keep in mind it will absolutely destroy the cache drive by wearing out the flash

You also could look into ZFS special disks. However, if you are going that way already you might as well get a bunch of disks.

rambos OP ,

Ill look into ZFS, but in meantime I found out my HDD is probably not bottleneck. Still want to learn about this so thanks for your comment

lemmylommy ,

Any HDD should be able to get at least 100MB/s sequential write speed. Unfortunately torrent writes are usually very random, which just kills hdd performance. Multiple parallel downloads or concurrent playback from the same disk will only make it worse.

Using a SSD for temporary files will absolutely help. It should be big enough to hold all the files you are downloading at any one time.

You could also try to find a write cache setting that works for you. That way what would usually be many small writes can be combined to bigger chunks in memory before sending them to storage. Depending on how much ram is available I would start at 1GB or so and if it is still bottlenecking try in- or decreasing until it improves. Of course always stay in the range of free ram.

Back when I was torrenting (ages ago) write cache helped a lot. It should be somewhere in the settings menu.

rambos OP ,

Oh, you are talking about torrent client settings? I could spare 1-2 GB of RAM, but not more than that (got 16 GB in total). I see this might help a lot, but I would I still be limited with HDD max write speed? Using SSD for temporary files sounds great, but waiting files to be coppied to HDD would slow it down if I understood correctly

osaerisxero ,

My solution to this was to put the default download folder on an nvme and then move the torrent to a storage hdd after completion

  • All
  • Subscribed
  • Moderated
  • Favorites
  • •
  • [email protected]
  • random
  • lifeLocal
  • goranko
  • All magazines
    •