soaro77 wrote:
I've had this happen to me a couple times and am wondering if anyone else has had this problem and how to overcome it.
Basically, I backup my images to 4 different hard drives and also to the cloud. If we lose power or the computer somehow gets shut off while my external hard drives are attached, it apparently can corrupt files. I've had pictures that were fine originally (I edited them in Lightroom so know they were good) and then sometime down the road I found them with a 0 byte length and the picture was gone. The other problem was that I couldn't restore them because I didn't know the pictures files had gotten corrupted so it backed up the 0 byte files do my other hard drives and to the cloud. My cloud backup, IDrive, only saves versions for a month so if I don't notice within a month of when the files are corrupted then those old backups drop off and my files are gone forever.
Anyone have any ideas on how to prevent this from happening? I'm wondering if I could somehow generate some sort of checksum or something for each file and then find some software that would look at all the checksums of every file that has changed before backing it up. Or ?????
Thoughts or ideas???
Curtis
I've had this happen to me a couple times and am w... (
show quote)
Hi, Curtis,
There are two questions here: how to detect a corrupted file, and how to prevent files from becomming corrupted.
You didn't mention what OS you are running, so I'll try to answer in general terms.
But before delving into that, I'd suggest you run file system checker (e.g, chkdsk on Windows or fsck on Linux)
on your boot drive and backup drive in order to verify the file systems and fix any problems. A corrupted file
system will create bizarre disk file behavior. (It's always tricky to run a file system check on the boot drive--
since it's always mounted. Usually it requires bringing the system down to a single-user run level. Windows
chkdsk seems to get around this--but I will still quiet the system as much as practical before running
chkdsk /f C:.
Now as to detecting corrputed files. Obviously, any copy with a different file size than the original is corrupted.
So lets suppose the copy is the same size as the original.
Checksums and message digests are probabilistic: they work most, but not all of the time. The only way to be
absolutely sure that the copy has not been corrupted is to compare it with the original, byte-by-byte. Windows
comp command (run from theWindows command line) will do this. However, it is slow.
Backups should be under the control of a backup manager software or a script. Either should allow you to
verify the backup in the background, without having to sit there and wait. So there is no reason not to verify
every file byte-by-byte against the original. It will slow down your system to a crawl, but if it happens at
night (or while you take a break) it wont matter.
Now, as to fixing the problem you need to diagnose what event is triggering the data corruption: power failure,
system lockup, program lockup, or something else. Any such system malfunction will need to be found and
fixed. That leaves the case where there is no obvious cause.
Unfortunately, disk writes are not simple. To improve efficiency, most programs used buffered write operations:
data is streamed into a buffered in RAM until the buffer is full, then it written to disk. So if the program is killed
or locks up for any reason, data will be lost.
In programming languages, there is often a statement that causes the write buffer to be flushed. For example,
in the C programming language, the statement "flush(stdout)" will flush the standad output stream. When
writing critical data, it's not a bad idea to use an unbuffered write or to call flush(), just in case the system
locks up All open files should be flushed by the OS when the program terminates, but who knows when that
will be.
Similarly, to improve efficiency, file systems buffer write operations at the OS level. File systems write both
user data (your image file) and meta data (directory entries, etc) to disk, so the file system can become corrrupted
if meta data does not get written. Journaling file systems keep a list of pending write operations, so after a power
failure they can be restarted without corrupting the file system (some user data may still be lost). Some journaling
file systems work better than others.
The Cloud drive adds additional layers: TCP/IP, encryption, and the remote server's OS and filesystem. I don't
know how you verify that all that is working. You could try asking a Russian hacker.
Sometimes the OS includes a command, usually "sync", to cause all buffered data to be written to disk. Sync
must always be called before the system is shut down. Usually it is incorporated into the systems shutdown
command.
If no cause can be found, then a possible work-around is to sync the filesystem after each backup file is written to
disk. How you do this depends on the OS. Windows and Mac OS/X try to hide all the details of how the system
works, which makes the system administrator's job very difficult. On Linux/UNIX, you just type "sync" at the
command prompt and it will flush alll the mounted file systems.
If you could figure out how to reproduce the problem on demand, then it will become much easier to diagnose
and fix it. Good luck!