Ugly Hedgehog - Photography Forum
Home Active Topics Newest Pictures Search Login Register
Main Photography Discussion
When is an external backup disk "too full"?
Page <prev 2 of 3 next>
Jun 3, 2018 10:11:23   #
AndyH Loc: Massachusetts and New Hampshire
 
Gene51 wrote:
This is not true. There is no consensus on when a hard drive is too full, but common sense suggests that you need enough room to be able to defragment the drive if you write to and delete from that drive regularly. When I have less than 25% free space on a drive, defragging really takes a lot of time, when there is less than 10% it takes forever, so I would think that keeping 15% to 20% free is a reasonable goal. You will also see a big difference in performance when you start getting below 15%. So to your two points, yes, a drive can be too full, and yes, there are some seriously good reasons to leave some unused space on a drive. The only disagreement among IT people is on what constitutes enough free space.
This is not true. There is no consensus on when a ... (show quote)


Exactly. I defrag both main and external drive weekly (or as close to weekly as my schedule permits) and consider the drive full when I hit 80%, which is what my IT consultant recommended to me.

Andy

Reply
Jun 3, 2018 10:53:07   #
gessman Loc: Colorado
 
Rather than using enclosed external hard drives I suggest using something like https://www.amazon.com/Kingwin-Station-Compatible-Performance-Tool-Free/dp/B00JKM0KUE/ref=sr_1_2?ie=UTF8&qid=1528036486&sr=8-2&keywords=ez-dock

There are other brands available but the one above is offered as an example. It lets you swap
drives out with ease and allow "hot-swapping" of drives. They can also be turned off when not
actually needed, saving wear and tear on your drives, potentially extending the life of a drive.

I see, once again, the mention that some are using SSD for backup storage. I will, once again, without value judgement of validity or any pretense of expertise on the subject, refer anyone who wants to use SSD as a long-term backup storage to the following excerpt from Wikipedia, especially paragraph two:

"As of 2017, most SSDs use 3D TLC NAND-based flash memory, which is a type of non-volatile memory that retains data when power is lost. For applications requiring fast access but not necessarily data persistence after power loss, SSDs may be constructed from random-access memory (RAM). Such devices may employ batteries as integrated power sources to retain data for a certain amount of time after external power is lost.[4]

However, all SSDs still store data in electrical charges, which slowly leak over time if left without power. This causes worn out drives (that have exceeded their endurance rating) to start losing data typically after one (if stored at 30 °C) to two (at 25 °C) years in storage; for new drives it takes longer.[7] Therefore, SSDs are not suited for archival purposes."

Reply
Jun 3, 2018 12:22:15   #
gvarner Loc: Central Oregon Coast
 
gessman wrote:
Rather than using enclosed external hard drives I suggest using something like https://www.amazon.com/Kingwin-Station-Compatible-Performance-Tool-Free/dp/B00JKM0KUE/ref=sr_1_2?ie=UTF8&qid=1528036486&sr=8-2&keywords=ez-dock

There are other brands available but the one above is offered as an example. It lets you swap
drives out with ease and allow "hot-swapping" of drives. They can also be turned off when not
actually needed, saving wear and tear on your drives, potentially extending the life of a drive.

I see, once again, the mention that some are using SSD for backup storage. I will, once again, without value judgement of validity or any pretense of expertise on the subject, refer anyone who wants to use SSD as a long-term backup storage to the following excerpt from Wikipedia, especially paragraph two:

"As of 2017, most SSDs use 3D TLC NAND-based flash memory, which is a type of non-volatile memory that retains data when power is lost. For applications requiring fast access but not necessarily data persistence after power loss, SSDs may be constructed from random-access memory (RAM). Such devices may employ batteries as integrated power sources to retain data for a certain amount of time after external power is lost.[4]

However, all SSDs still store data in electrical charges, which slowly leak over time if left without power. This causes worn out drives (that have exceeded their endurance rating) to start losing data typically after one (if stored at 30 °C) to two (at 25 °C) years in storage; for new drives it takes longer.[7] Therefore, SSDs are not suited for archival purposes."
Rather than using enclosed external hard drives I ... (show quote)


That's good to know about SSD's. Thanks.

Reply
 
 
Jun 3, 2018 13:16:00   #
bpulv Loc: Buena Park, CA
 
jerryc41 wrote:
https://www.digitalcitizen.life/simple-questions-what-trim-ssds-why-it-useful
https://www.howtogeek.com/257196/how-to-check-if-trim-is-enabled-for-your-ssd-and-enable-it-if-it-isnt/


Is there a trim command for a Mac running OS10? If so, how is it enabled?

Reply
Jun 3, 2018 13:55:55   #
drklrd Loc: Cincinnati Ohio
 
You really need to copy all your files to a second drive instead of just backing them up. I keep the drive in computer for programs only and pull my picks from a 5 terra drive. Which has copy files from three other drives on it. I plan to get another 5 terra and copy all of this drive to it. I keep copies of my personal and professional on a 2 terra as well as the 5 terra. The other 3 drives sit on a shelf until needed. One terra has only music on it and its about half full. I have three copies of this file on three different drives. I label the drives so I know what they have on them. Backups seem to have the ability to be prone to virus or other problems trying to recall them from old backup routines. I just make multiple copies of data as i have time to make them.

Reply
Jun 3, 2018 14:03:17   #
CamB Loc: Juneau, Alaska
 
Something that concerns me. Is your editing program, lightroom, photoshop or whatever, on the same drive as your picture files? I wouldn't do that. Just one more chance for all to go wrong. My picture files, 240,000 or so, are on an external drive that I work from and are backed up to other drives and to the cloud. These are just raw files, no applications. Lightroom lives on one of the hard drives in my computer and is backed up separately the same way.
In answer to your question, I think about 85%.
...Cam

wrangler5 wrote:
My images reside on an external hard drive, from where they are accessed and manipulated by Lightroom. Every night the top-level Photo folder is backed up to another external hard drive - just in case. This backup drive is never accessed unless something goes wrong which (so far, knock on wood, etc.) it hasn't.

The backup drive is filling up, and I'm wondering when it makes sense to replace it with a larger one? I have a recollection that in the days when a gigabyte was a BIG disk (my original IBM PC has a 20 MEGAbyte hard disk, which was an expensive upgrade from the standard 10 Mb disk) there was a rule of thumb that you shouldn't let your hard drive get much more than 90% full. But in today's muti-terabyte drives, 10% of, say, a 2TB drive is still 200 GIGAbytes of space. Even with the largest Nikon or Canon sensors, 200GB will still hold a LOT of images.

I understand that with an "active" disk that gets written to and read from a lot, with constant changes to files, there needs to be disk space for the operating system to do its work properly. But for a backup disk, where basically each file gets written once and then is left untouched, unless it gets read off to replace one that got corrupted or lost, how much "spare" room does the disk really need to have for these single-write and occasional single-read operations to work properly? (I assume the same factors would apply to my external movie hard drive, to which I rip my DVDs so they can be viewed at will on various monitors around the house using Apple TVs - movies get ripped and written once, and then just read on demand.)

Thanks for any thoughts.
My images reside on an external hard drive, from w... (show quote)

Reply
Jun 3, 2018 14:51:02   #
bpulv Loc: Buena Park, CA
 
CamB wrote:
Something that concerns me. Is your editing program, lightroom, photoshop or whatever, on the same drive as your picture files? I wouldn't do that. Just one more chance for all to go wrong. My picture files, 240,000 or so, are on an external drive that I work from and are backed up to other drives and to the cloud. These are just raw files, no applications. Lightroom lives on one of the hard drives in my computer and is backed up separately the same way.
In answer to your question, I think about 85%.
...Cam
Something that concerns me. Is your editing progra... (show quote)


Since questions have been asked so many times on UHH about backup, that I have developed the following description of my backup system. Note that I agree that the backup drives should only be used for photo and data files and all programs should reside on the computer's internal drive for maximum speed and efficiency.

Computer security experts say you should always maintain a minimum of three (3) forms of backup for your photographs. At least one should be off site to protect against loss in the event of fire or natural disaster. The Cloud is not the best choice because your photos will be under the control of a second party that may have technical problems, financial failure or bankruptcy, sale to an unreliable party or may hold your data hostage to rising costs. Furthermore, upload speeds for large RAW files is way too slow and over time causes undue ware to hard drives. My system provides both on and off site redundant backup that is completely under my own control.

My system consists of the following:

1) All storage of photographs is external to the computer’s (iMac) internal drive.

2) Downloads from the camera, editing, final product and primary storage is on a 5Tb Western Digital My Book drive.
a) Each shoot is stored in a folder identified by date taken in the format "YYYY/MM/DD" followed by a title/description to allow sorting by date.
b) Within each folder are three sub-folders labeled “RAW”, “Edit” and “Final”. All uploads are made to the RAW folder and only copies of the raw files are copied to the Edit folder for editing. The final output that may include PSD, JEPG, TIFF, etc. files are transferred to the Final folder. After all editing is completed and the drive is backed up, the duplicate RAW files in the Edit folder are trashed leaving one complete set of RAW files in the RAW folder.

3) The complete primary storage drive is backed up to a Drobo 5-drive RAID (Redundant Array of Independent Disks). This allows multiple backup copies of the data along with offsite backup. In addition, the Drobo’s firmware automatically corrects any data transfer errors and insures that all backup drives have matched data. The RAID consists of five Western Digital Red Drives, which are designed for this type of service duty cycle and have an excellent reputation for reliability. Furthermore they are “hot swappable” which means they can be safely inserted and removed from the Drobo without removing power from the system.

a) The Drobo is only powered on when backups are being made. Otherwise, it is powered off. This limits the exposure to possible hacking.
c) Although the Drobo can hold up to five drives, only four are in the unit at any given time. Three of the drives are never removed and provide the basic redundancy.
d) The fourth drive position is used for the offsite backup. Two drives are used for that purpose. Once a week or as necessary, drive 4 is removed from the Drobo and taken to the bank. Drive 5 is removed from my safe deposit box and drive 4 takes its place in the box.

4) Drive 5 is inserted into the Drobo where the data from Drives 1, 2 and 3 are automatically copied to it.

Reply
 
 
Jun 3, 2018 14:55:51   #
wrangler5 Loc: Missouri
 
Let me clarify the setup that raises the original question. LR and PS Elements live on the 1.2TB fusion drive in my Mac Mini, which is just about half full. This drive holds the LR catalog, and serves as the scratch disk for the (rare) occasions when I do something in PSE (LR doesn't use a scratch disk.) The Mini has a quad core i7 processor and the maximum 16GB of RAM - it seems to run LR (and all the other stuff I have on it) just fine.

My image files (approaching 100,000, last I looked, mostly Nikon NEFs plus whatever jpegs I occasionally shoot or generate thru LR) are on an external hard drive which is also about half full. Call this drive PHOTOS.

I have another external hard drive that holds what am referring to as my "backup" photo files - every night, the backup software (Carbon Copy Cloner) copies over any new files it finds on the PHOTOS disk but does not delete anything. Call this drive BACKUP PHOTOS.

I have yet another external hard drive to which I rip my DVDs, for easy watching on several screens throughout the house using Apple TVs. Call this drive MOVIES.

The PHOTOS drive obviously gets lots of read and write attention, and I know that it will need empty space to allow that activity to run smoothly. But I view both the MOVIES drive and the BACKUP PHOTOS drive as essentially write-once drives. These are the ones I'm wondering about being "too full."

MOVIES is read from constantly, but once written, the mp4 files stored there are virtually never changed. BACKUP PHOTOS is rarely even read from, and nothing is deleted. I wouldn't think this kind of activity needs much disk headroom, but I'm uncertain enough that I asked the question.

Reply
Jun 3, 2018 15:05:22   #
RichLacey Loc: Atlanta
 
This is just to let you know that Sam’s has 4tb USB 3.0 external drives for just under a hundred bucks. I bought 4 of them and that should keep me in business for several years. No worries about them getting full.

Reply
Jun 3, 2018 16:29:14   #
TheShoe Loc: Lacey, WA
 
chrissybabe wrote:
If you are using Win 10 then typically your drives will be defragged or optimised on a weekly schedule (time can be changed or you can deal with it manually). So mostly not an issue if you are lowish on disk space. However you do need some scratch space to allow the process to proceed without too much hassle. When Windows optimises it doesn't necessarily defrag every file but does enough to improve the overall efficiency of drive access. If you want to have 100% defrag then you will need a 3rd party application. Or you can reformat the destination drive and copy over everything. This will give you a very tidy disk. SSD drives have their own commands to defrag but if you are at all interested then google "SSD trim".
Contrary to what I said earlier you don't get warnings at 90% BUT when remaining space is 200MB, then 80MB then 50MB. On a 4TB drive by the time you get the warning there isn't much space left. Space remaining gets checked every 10 minutes. You can with a registry edit remove the warning but since some of my wifes edits can be up to 1GB in size it does pay to keep a close eye on available space and to do something before you see the warning message. Saving unwanted edits is even more trouble than just saving every shot.
If you are using Win 10 then typically your drives... (show quote)

Two points: 1. SSD drives do not need to be defragged. There is no SEEK or rotational delay and because the time to read or write is the same for all sectors, they are always optimized. 2. Your spinning drive is kept pretty clean all of the time if you are using Win 10 at the current maintenance level. Windows 10 Defrag runs in the background. In truth, having the Defrag in the background will not be as squeaky clean as stopping all other activity and running nothing other than a disk optimizing program, but a squeaky clean disk will not stay that way for long.

Reply
Jun 3, 2018 16:44:54   #
rmorrison1116 Loc: Near Valley Forge, Pennsylvania
 
Gene51 wrote:
This is not true. There is no consensus on when a hard drive is too full, but common sense suggests that you need enough room to be able to defragment the drive if you write to and delete from that drive regularly. When I have less than 25% free space on a drive, defragging really takes a lot of time, when there is less than 10% it takes forever, so I would think that keeping 15% to 20% free is a reasonable goal. You will also see a big difference in performance when you start getting below 15%. So to your two points, yes, a drive can be too full, and yes, there are some seriously good reasons to leave some unused space on a drive. The only disagreement among IT people is on what constitutes enough free space.
This is not true. There is no consensus on when a ... (show quote)


There is a difference between how a partitioned drive with OS and paging is used vs an archival drive.

Reply
 
 
Jun 3, 2018 20:59:31   #
burkphoto Loc: High Point, NC
 
wrangler5 wrote:
My images reside on an external hard drive, from where they are accessed and manipulated by Lightroom. Every night the top-level Photo folder is backed up to another external hard drive - just in case. This backup drive is never accessed unless something goes wrong which (so far, knock on wood, etc.) it hasn't.

The backup drive is filling up, and I'm wondering when it makes sense to replace it with a larger one? I have a recollection that in the days when a gigabyte was a BIG disk (my original IBM PC has a 20 MEGAbyte hard disk, which was an expensive upgrade from the standard 10 Mb disk) there was a rule of thumb that you shouldn't let your hard drive get much more than 90% full. But in today's muti-terabyte drives, 10% of, say, a 2TB drive is still 200 GIGAbytes of space. Even with the largest Nikon or Canon sensors, 200GB will still hold a LOT of images.

I understand that with an "active" disk that gets written to and read from a lot, with constant changes to files, there needs to be disk space for the operating system to do its work properly. But for a backup disk, where basically each file gets written once and then is left untouched, unless it gets read off to replace one that got corrupted or lost, how much "spare" room does the disk really need to have for these single-write and occasional single-read operations to work properly? (I assume the same factors would apply to my external movie hard drive, to which I rip my DVDs so they can be viewed at will on various monitors around the house using Apple TVs - movies get ripped and written once, and then just read on demand.)

Thanks for any thoughts.
My images reside on an external hard drive, from w... (show quote)


Leave 10% to 20% “breathing room” for operating system functions, even on backup drives. Filling up a conventional hard drive slows it down. It becomes more and more fragmented. At first, files are written as single segments (continuous blocks of data). But over time...)

Reply
Jun 3, 2018 22:50:59   #
rfmaude41 Loc: Lancaster, Texas (DFW area)
 
bpulv wrote:
NEVER fill a disk to more then and preferably less then 90% of its rated capacity including the hidden data and "housekeeping" files that every disk system contains for data management.


A great many applications (Oracle, SQLServer, TeraData, etc. advise to not fill over 85%; these are "big users" that need exceptionally fast response. However, being a database admin for well over 30 years, I subscribe always to this "rule", no matter if I'm running RAID 5 or (what I always use) RAID 0+1 ( sometimes called INCORECTLY Raid 10).

Reply
Jun 3, 2018 23:20:20   #
Nikon_DonB Loc: Chicago
 
I'd say 90% is a workable limit.

Reply
Jun 4, 2018 00:47:17   #
11bravo
 
I worry more about temperature and health than "fullness". For windows, I use Hard Disk Sentinel for temperature monitoring (with alerts and auto shut down on custom set over-temps), and StableBit Scanner for automatic surface scans. Both paid programs, but well supported. Before I use a HDD, it is provisioned: a long (not quick) format, then a full chkdsk, then a StableBit scanner scan. Sure, a lot of duplication, but that's the point.

Don't know for Mac, but something similar must exist.

For "fullness", for my write once, infrequent read disks, they're packed (4tb with 500mb free). Never defrag. And if a backup drive slows down a bit as it has less and less free space, the backup runs while I'm sleeping, so I don't care, just as long as there is sufficient space. After all, I'm not accessing it all that much.

Reply
Page <prev 2 of 3 next>
If you want to reply, then register here. Registration is free and your account is created instantly, so you can post right away.
Main Photography Discussion
UglyHedgehog.com - Forum
Copyright 2011-2024 Ugly Hedgehog, Inc.