drizztguen77 Loc: Tualatin, OR

I've had this happen to me a couple times and am wondering if anyone else has had this problem and how to overcome it.

Basically, I backup my images to 4 different hard drives and also to the cloud. If we lose power or the computer somehow gets shut off while my external hard drives are attached, it apparently can corrupt files. I've had pictures that were fine originally (I edited them in Lightroom so know they were good) and then sometime down the road I found them with a 0 byte length and the picture was gone. The other problem was that I couldn't restore them because I didn't know the pictures files had gotten corrupted so it backed up the 0 byte files do my other hard drives and to the cloud. My cloud backup, IDrive, only saves versions for a month so if I don't notice within a month of when the files are corrupted then those old backups drop off and my files are gone forever.

Anyone have any ideas on how to prevent this from happening? I'm wondering if I could somehow generate some sort of checksum or something for each file and then find some software that would look at all the checksums of every file that has changed before backing it up. Or ?????

Thoughts or ideas???


Curtis

TriX Loc: Raleigh, NC

The way this is handled in computer storage is the concept of “snapshots”, which was popularized by NetApp NAS and incorporated into the WAFL OS and OneFS file system. The idea is that the file system periodically records a “picture” of the file system at a point in time. This can be hourly, weekly, monthly and is user-defined. Typically, 20 snapshots of varying granularity are stored. Since each new snapshot after the first contains only changes to the I-node structure and additions to the FS, the snapshots (after the first) are small. At any time, you can revert the file system to any of these points in time (before the corruption). That is one way to address the issue. Windows with NTFS has a similar capability (VSS or shadow copy) but not sure if it applies only to the OS or to the data/FS as well. I’ll have to research that a bit further...

rook2c4 Loc: Philadelphia, PA USA

Maybe don't connect all four external drives to the computer simultaneously, but only one at a time. If power unexpectedly cuts out, the contents on the other external drives not connected will at least remain safe.

drizztguen77 Loc: Tualatin, OR

I don't have them all connected at once. But the problem is that the files get corrupted on the one that is connected so when I go to backup the drive it copies the corrupted files over the good files on the other drives. I didn't know they were corrupted until a few months down the road.

blackest Loc: Ireland

A UPS could be a good investment, 'rsync' might be better for copying it does generate a checksum and only copies changed files, and 'find' can be used to generate a list of zero byte files.

e.g

find /path/to/dir -size 0 -type f -name "*.jpg"

would find all 0 byte jpeg files in a particular set of folders. This works as is in Linux and OSX but might be different in windows

drizztguen77 Loc: Tualatin, OR

I do run on windows but I can run the commands using cygwin. That's probably what I need to do. I didn't think about the find command. My new computer is a laptop so it will stay running for at least a couple hours if we were to lose power at night or when I'm not home.

blackest Loc: Ireland

if you can run cygwin then definitely look at rsync my nas runs linux but it only has 256MB of ram and sometimes it will run out of ram and drop a process if it stops partway through you just run the exact same command again and it figures out where it had got to and continues from there. With copy you never quite know where it was at when it failed so you end up restarting the job and it might take hours to run, especially with USB2.

You can do some pretty useful things with rsync such as copy just files of a particular type and remove the originals. I must get round to shifting all the files on one hdd to another because its formatted ntfs and the kernel sometimes drops the ntfs driver when its low on ram it wouldn't do that if it was ext4 and theres a lot of junk on that drive which i'd rather use for photos. I have another linuxbox (a converted android tv box actually) that has 2GB of Ram and a faster CPU it basically runs 2 cron jobs it logs into the other box with ssh and rsyncs my lightroom folders and my itunes folders every other hour. So after 2 hours every file i put on one Nas is duplicated on to the backup.

it's set up not to delete files that have been removed from the main nas but i also have a couple of scripts that i run manually which move files on the backup that have been deleted from the main system to live under a set of folders under one called deleted. If I get in a mess i can always restore
I also change the root folder under deleted so it might be deleted/lightroom-17-10-2018/.. the next time i run the newly deleted files go to /deleted/lightroom/... If space becomes a problem then i just delete the oldest 'deleted' backup.

Longshadow Loc: Audubon, PA, United States

A UPS will help greatly, you'll need to activate the auto-shutdown for power failure in case you are not at home though.

(BTW - never disconnect an external/pocket/USB (card) drive without ejecting it first, same thing can & will happen!)

bpulv Loc: Buena Park, CA

Buy a UPS (Uninterruptible Power Supply) for your computer and external drives. You can save money by buying one that has enough wattage capacity to support only the computer and drives. You don't need to backup non-essential devices such as printers or extra monitors. If there is a power failure, surge or spike, the UPS batteries will seamlessly kick in and keep your equipment going for some period of time. During that time you can do a normal shutdown of your equipment if you anticipate a long power outage. Some APC brand UPS systems come with PC (not Mac) software that will sense a power failure and automatically turn your equipment off for you in an orderly fashion. Not only does a UPS prevent data loss during a power interruption, but it will make your system much more bullet proof when there is a power surge or spike than a surge protector (MOV) power strip because the DC batteries act as a large filter for any power fluctuations.

blackest Loc: Ireland

Mac OS X contains built in, basic support for an APC UPS and if it's not enough, apcupsd is far more capable.

bpulv Loc: Buena Park, CA

How would I access that on my iMac?

gvarner Loc: Central Oregon Coast

So the result is that you backup a corrupted file? I put all my originals on an external drive before putting them on my PC. And I only connect that drive when I want to put more files on it. If so inclined, you could copy that to a second external for redundancy in case one fails.

blackest Loc: Ireland

It depends on the ups to be honest I have an iDialog and i'm connecting to one of my Nas boxes (runs linux) but there are a couple of 3rd party programs and the UPS manufacturer may have something else (powershield3 for my idialog).

JCam Loc: MD Eastern Shore

Curtis, Are you saying that all five backups are linked, done at the same time, likewise all got corrupted at the same time?? I don't have a clue as to how to restore the lost files, but some commercial recovery firm might be able to do it, but possibly at a significant cost. Wouldn't a simple solution, though at additional back up time, be to remove one of the four hard drives from of the string and back up to it separately?

Good luck. Jim

TriX Loc: Raleigh, NC

If you’re running a recent version of Windows, do you have Previous Versions implemented? If so, have you tried it for the effected file(s)/folders? If not implemented, why not?