drizztguen77 Loc: Tualatin, OR

No I have 1 master external hard drive that holds all my pictures. Then I connect the other hard drives and backup the pictures to those hard drives. Then I disconnect the other hard drives. But the problem happens when the pictures on the original hard drive become corrupted (0 byte) due to losing power and I don't know it so unknowingly copy the 0 byte files out to my other hard drives and lose all copies of the pictures.

drizztguen77 Loc: Tualatin, OR

Are you referring to previous versions of Windows where you can restore to a previous version? I'm not sure that would work because the pictures are stored on an external hard drive. I don't understand previous versions enough to be certain but I thought that just restored a previous version of Windows and not necessarily every file on the hard drive. Am I not understanding it correctly?

drizztguen77 Loc: Tualatin, OR

I am always careful about this. Sometimes Windows won't allow me to eject it so I have to reboot to get it to let go of them.

JCam Loc: MD Eastern Shore

I'm not a computer expert so only take this at face value; like most free advice it's worth what you pay for it

By "original hard drive" do you mean the one in your computer"? That may bring up another situation; I've always been told that hard drive failure is not a question of "if" but rather "when". How old is the drive? Does it seem to be working ok? If it is getting flaky,perhaps a backup occured at just the wrong time and the empty file got transferred to the other HD's, or the sector being used for your pictures has failed. If you are running a Windows system, doesn't the software have a trouble shooting program that could check the hard drive, or perhaps pull it out and have a local "computer fix it" company check it.

How large is your Pictures file? With four duplicate external drives, personally I'd feel better if one or even two were out of the chain and were backed up separately. Mine isn't that large, and I back it up the evening before we hit the sack so I don't really care how long it takes; it is always done by morning.

drizztguen77 Loc: Tualatin, OR

By "original hard drive" I mean my primary external hard drive. I have too many pictures to put them on my internal hard drive. It would be full. So I have four 4 gig external drives. I use one as my primary drive that I copy my pictures to and process them there with Lightroom and Photoshop. I keep the Lightroom catalog on my internal hard drive and copy it and all the preview directories to the main external drive after I'm done processing my pictures. Then I plug in my other hard drives and copy all the pictures to them. Then I tell it to back them all up to the cloud with IDrive.

I was using my D750 before so picture sizes weren't too large but not tiny either. Now I'm using my D850 and the files are pretty big.

Bipod

Hi, Curtis,

There are two questions here: how to detect a corrupted file, and how to prevent files from becomming corrupted.
You didn't mention what OS you are running, so I'll try to answer in general terms.

But before delving into that, I'd suggest you run file system checker (e.g, chkdsk on Windows or fsck on Linux)
on your boot drive and backup drive in order to verify the file systems and fix any problems. A corrupted file
system will create bizarre disk file behavior. (It's always tricky to run a file system check on the boot drive--
since it's always mounted. Usually it requires bringing the system down to a single-user run level. Windows
chkdsk seems to get around this--but I will still quiet the system as much as practical before running chkdsk /f C:.

Now as to detecting corrputed files. Obviously, any copy with a different file size than the original is corrupted.
So lets suppose the copy is the same size as the original.

Checksums and message digests are probabilistic: they work most, but not all of the time. The only way to be
absolutely sure that the copy has not been corrupted is to compare it with the original, byte-by-byte. Windows
comp command (run from theWindows command line) will do this. However, it is slow.

Backups should be under the control of a backup manager software or a script. Either should allow you to
verify the backup in the background, without having to sit there and wait. So there is no reason not to verify
every file byte-by-byte against the original. It will slow down your system to a crawl, but if it happens at
night (or while you take a break) it wont matter.

Now, as to fixing the problem you need to diagnose what event is triggering the data corruption: power failure,
system lockup, program lockup, or something else. Any such system malfunction will need to be found and
fixed. That leaves the case where there is no obvious cause.

Unfortunately, disk writes are not simple. To improve efficiency, most programs used buffered write operations:
data is streamed into a buffered in RAM until the buffer is full, then it written to disk. So if the program is killed
or locks up for any reason, data will be lost.

In programming languages, there is often a statement that causes the write buffer to be flushed. For example,
in the C programming language, the statement "flush(stdout)" will flush the standad output stream. When
writing critical data, it's not a bad idea to use an unbuffered write or to call flush(), just in case the system
locks up All open files should be flushed by the OS when the program terminates, but who knows when that
will be.

Similarly, to improve efficiency, file systems buffer write operations at the OS level. File systems write both
user data (your image file) and meta data (directory entries, etc) to disk, so the file system can become corrrupted
if meta data does not get written. Journaling file systems keep a list of pending write operations, so after a power
failure they can be restarted without corrupting the file system (some user data may still be lost). Some journaling
file systems work better than others.

The Cloud drive adds additional layers: TCP/IP, encryption, and the remote server's OS and filesystem. I don't
know how you verify that all that is working. You could try asking a Russian hacker.

Sometimes the OS includes a command, usually "sync", to cause all buffered data to be written to disk. Sync
must always be called before the system is shut down. Usually it is incorporated into the systems shutdown
command.

If no cause can be found, then a possible work-around is to sync the filesystem after each backup file is written to
disk. How you do this depends on the OS. Windows and Mac OS/X try to hide all the details of how the system
works, which makes the system administrator's job very difficult. On Linux/UNIX, you just type "sync" at the
command prompt and it will flush alll the mounted file systems.

If you could figure out how to reproduce the problem on demand, then it will become much easier to diagnose
and fix it. Good luck!

TriX Loc: Raleigh, NC

As I remember it (and I haven’t had to use it because my NAS supports snapshots), if you right click on a data file or folder (and you have previous versions implemented), you have the option of restoring that file to one of the previous versions. I will check it and see and amend my response if this is incorrect.

aellman Loc: Boston MA

One word: Carbonite.

drizztguen77 Loc: Tualatin, OR

They no longer sell it for personal use, only for business use.

aellman Loc: Boston MA

If you're talking about Carbonite, see the screen cap.

rck281 Loc: Overland Park, KS

You can always verify the size of a folder by right clicking the folder and selecting Properties. Record the size and check at an appropriate interval. Also, I agree with the others who suggested a UPS.

Bipod

Two words: private company.
All finanical data secret. Carbonite could file for bankruptcy tomorrow.
And then it would be up to the creditors what happens to your data.
Or the company could be acquired, in which case it would be up to the
new owner.

Of course, no .com ever got acquired or went out of business....

Two more words: unlimited backup.
This means small uses (such as you) are subsidizing huge corporate users.
But hey, those CEOs really appreciate you contributing to their bonuses!

One final word: contract.
Check yours, and see if it limits the amount of damages you can claim, and
excludes loss of business or goodwill, and other consequential damage, and
if you indemnify Carbonite for any losses sustained by third parties as a result
of Carbonite losing your data.

Chances are your sole remedy is limited to the market value of your image files--
which is??? And you would have to prove it in court: you'd need evidence
such as business papers showing actual sales of image files to an "arm's length"
buyer (not a friend, relative, etc) at a stated price. But you've got that, right?

Possibly you even agreed to binding arbitration, in which case it will never get
before a judge or a jury---a single arbitrator will decide. The National Arbitration
Forum used to conduct such arbitrations, but it was sued by the DoJ for being
biased towards businesses and signed a consent decree in 2009 promissing not
to do any more consumer arbitraition. The National Arbitration Association
and JAMS are still doing them.

You didn't negotiate this agreement--you just swallowed it whole. They have a
corporate counsel on staff, you don't. The lawyer who drafted the agreement
was working for Carbonite, not for you. He protected his employer, not you.

Chances are your data is protected by an "Oklahoma Guarantee": if Carbonite
accidentally deletes it, Carbonite promises to say "Ah, shucks".

But I'll admit, Carbonite has very good radio advertising. Who do you think
pays for that?

For all I know, Carbonite may be a very good company selling an excellent service.
But corporations ain't your friends or partners. Unless you are one of Carbonite's
biggest customers, if something goes wrong, they are likely to just blow you off--
what's the downside for them? Whatcha gonna do about it?

TriX Loc: Raleigh, NC

I think your comments concerning cloud storage run by (relatively small) non-public companies are spot on, but I wouldn’t let that dissuade potential cloud users from utilizing major cloud providers such as Amazon, Google, Microsoft, Apple, etc. All of these are public with solid financials, are extremely unlikely to go belly-up without notice and provide a robust DR (disaster revcovery) solution that you cannot possibly duplicate at home. None of them or ANY other data storage company will however, provide any guarantee against lost data or any consequential damages - the best you will get (at high cost) is an SLA that covers availability, not loss of data - that is and has always been the user’s responsibility, whether it’s a single HD or an enterprise-level SAN or NAS.

Your recourse as a data storage user is to design/implement a robust storage system with redundancy at every level, enterprise class components, environmental and power conditioning and professional level administration, but most importantly, multiple copies of your data with at least one off-site DR copy. If implementing such a system sounds daunting, and you have reasonable internet access, a major cloud provider can provide all that including multiple geographically seperate copies, at a very reasonable cost.

Bipod

I agree on the balance sheets of those large-cap companies. But as we found out about banks in the 2008 mortgage crisis,
balance sheets can be misleading. Each company is a separate case that needs to be analzyed

Most of Google's revenue comes from advertising. A single scandal could dry up that revenue overnight.
Moreover, Google's business model is not entirely clear, and it is spending vast amounts of cash on very shakey
business ventures: Google LOON for example. There are a lot of financial shoes that have yet to drop.

Apple used to be a computer company, but is now a consumer products company. Most consomer products
companies (Sony, Proctor and Gamble) have hundreds of products. But only six products account for most of
Apple's revenue. That make it a high risk business. The consumer tastes are notoriously fickle, and if
Joe Consumer suddenly decides he doesn't want to pay a huge premium for a white plastic case, Apple is in
trouble. It's happened before.

But the financial concerns pales to insignificance compared with the failure of thse companies to really commit
to being backup providers. Is Ford Motor Company in the car business? You bet it is! Is Apple in the Cloud
business this week, or have they spun it off? Let me check it's website and see.

If no "Cloud" provider will stand behind their service with a warranty that's a good reason to not use the Cloud.
Would you have your car engine rebuilt by a shop that would not provide a warranty?

There is no track record on any of these "Cloud" services, since none of them are more than a few years old, and most have
never been through a hurricane, tornado, flood, earthquake, wildfire, civil unrest, world war, or nuclear war. These things
do happen and are part of the reason why you need an offsite backup. But if your provider is under six feet of water,
ti doesn't do you any good.

Why should we accept any of thse companies on faith? What have they ever done to earn our trust? How is Amazon's
dirigible book warehouse and drone book delivery scheme working out? How is the Google barge doing these days?
How is the safety record of driverless cars?

The person who can best be counted on to pull your bacon out of the fire is: you. Not some big company for whom
you are a tiny and totally insignificant customer. Business relationships are most likely to work out when they are
between equals, and negotiated face-to-face by top management. That's the only way to find out whether there is
or isn't a commitment to the deal. You have to look into their eyes, and decide if you believe them.

I've set up an enterprise backup system. Anybody who wants to take the time and do the research can do it.

If you bought a nuclear reactor to heat your house, I guess you'd have to hire the Jiffy Reactor Company to run it for you.
Or maybe Apple (the iReactor).. "Perilous to us all are the devices of an art deeper than we possess ourselves."
― J.R.R. Tolkien, The Lord of the Rings

TriX Loc: Raleigh, NC

Well, let’s add some facts as opposed to speculation on cloud providers.

Every company I mentioned has been in business for decades and their financials have been solid, even through the 2008 crisis. For information, Amazon is by far the largest cloud provider - as large as the next five combined. I initially left off names like IBM and HP, both of which are at least 60 years old and both provide cloud services. Do you actually think Microsoft is in danger of going belly-up? If it does, then you’ll have lots more important things to worry about than your photo storage.

Two thirds of US business and almost all branches of the federal government use cloud services, including the DOD and the intelligence agencies (NSA, CIA...) - these use highly secure cloud services by major providers such as Amazon that are NIST compliant (Google NIST compliant or the classified cloud initiative for further information). All your critical data is already in the cloud including medical records and images, Social Security, credit card and banking, military records, IRS, etc.

The concept of cloud storage is not going away - it’s growing yearly, and the old-line data storage companies are terrified as more and more companies outsource data storage to the cloud. Why? Because it’s often more cost effective to purchase it as they do other services as opposed to purchasing and administering the storage in-house. Small and medium sized companies in particular benefit from not having to build, maintain and administer data storage facilities. That used to be the only option, but not any more. If you spend some time with CIOs of companies all across the spectrum, from small to major, that’s what they’ll tell you (I’ve been hearing it for almost a decade, straight from the people who are actually responsible for a company’s data). They’re outsourcing all aspects of computing. Are you familiar with terms like IAAS (infrastructure as a service) and SAAS (software as a service)? Well, if you’re in the computer business, you better be because that’s where the industry is headed, and that’s what both IT companies and CIOs are spending their time discussing. Have you noticed that IBM is morphing from a hardware company into a service company? Sorry, like it or not, that’s the way IT is headed.

Now let’s discuss reliability/availability and safety of your data. Fact: ONE small/medium cloud service has gone belly up, and in that case, customers were notified approximately a month before hand and had time to migrate the data - I know because I worked for a global file system provider that utilized their cloud at the time. I invite you to find examples of a medium-sized or major cloud provider going under without notice - just one. Note, that I’m NOT discussing or endorsing the use of cloud storage from your local mom and pop cloud storage or even mid sized storage providers like Carbonite, I’m talking about MAJOR cloud providers like the ones I just mentioned.

Let’s also dispel the idea that a flood, earthquake or other natural disaster will lose your data. Every MAJOR provider of cloud storage keeps 3-5 copies of your data at widely seperate data centers to mitigate just such a disaster, something you can’t possibly do if you keep your own DR. That hurricane, fire, flood or earthquake won’t take down all the multiple Amazon data centers holding your data, but it CAN take down your personal infrastructure (often permanently) and even that backup copy in the lockbox at your bank. No need to wonder where all the data that wasn’t In the cloud in structures on the Guld coast of Florida went - we already know the answer to that question.

Finally, let’s talk about one’s local home data storage, even for a computer professional. Do you have a hardened facility with redundant A/C, redundant power and redundant networking? No. Are you using HA enterprise class servers with failover, storage with redundant controllers, enterprise class drives? Not many. Do you have a nightly backup with off-site DR that’s more robust than a weekly trip to the bank with a consumer class drive that may or may not restart? I’m betting that most do not. And do you have the administrative skills AND the time to dedicate to manage it all? Judging from the questions and answers on this forum, less than 5% do (and that’s being very generous).

So what about the other 95+%? They spend thousand or tens of thousands of dollars on cameras and lenses to what purpose? To create data. What good is all that equipment and talent if we lose the end-product (the data)? So if you aren’t running your own data storage system that meets the above requirements, especially for those off-site disaster recovery copies of your data at multiple locations (to circumvent a natural disaster), what do you do? The answer is that if you have decent internet access, you keep the DR copy in a MAJOR provider’s cloud, and if you don’t like Amazon’s or Google’s commitment to data storage, then chose IBM, HP or Microsoft. I’ve been doing this since the 60s and have spent the last 25+ years specializing in data storage for companies like IBM, EMC, NetAPP and DDN who togather probably store 85% of the world’s data, and my DR copy is in Amazon S3 cloud. But hey, everyone is free to make their own decisions (and mistakes), and I don’t have any financial interest in any cloud storage company except as a customer, and I’m not preaching cloud storage, I’m preaching SAFETY of your data with a local backup and an off-site DR copy.