Ugly Hedgehog - Photography Forum
Home Active Topics Newest Pictures Search Login Register
General Chit-Chat (non-photography talk)
Having major file storage issues, help needed
Page <prev 2 of 2
Feb 23, 2020 11:03:25   #
SonyBug
 
TriX wrote:
Just curious - were you using the Windows OS to generate the RAID 1 (mirrored) set, or an external RAID controller?


The raid 1 was set up in the initial setup of the intel system. It was before the Win10 OS was activated. It had to be disconnected in the setup portion of the system. I think it is above what is called the kernel, but before the Win10 software.

The disadvantage of raid1 is that a corruption in the Windows system will be placed on both drives. Now, only one drive is corrupted if Win fails, and the data still will be secure. But the data is offsited every night to a 1tb external drive.

Reply
Feb 23, 2020 11:07:42   #
SonyBug
 
Sorry, double post.

Reply
Feb 23, 2020 11:24:23   #
TriX Loc: Raleigh, NC
 
SonyBug wrote:
The raid 1 was set up in the initial setup of the intel system. It was before the Win10 OS was activated. It had to be disconnected in the setup portion of the system. I think it is above what is called the kernel, but before the Win10 software.

The disadvantage of raid1 is that a corruption in the Windows system will be placed on both drives. Now, only one drive is corrupted if Win fails, and the data still will be secure. But the data is offsited every night to a 1tb external drive.


So Windows is managing the mirroring and the disks are set as dynamic?

You have hit on the disadvantage of mirroring (and backups for that matter) unless file versioning or snapshots are being used - corruption on the primary propagates to corruption on the mirror (or backup). In general, there are at least 3 solutions for this problem:

1) employ VSS (volume shadow copy service) in Windows/NTFS. That allows you to revert data files to a previous state. Combine that with system restore points which allows the OS to be reverted to a previous good state.

2) use a file system on your external devices that employs snapshots. A snapshot is a point in time picture of the file system. Each new snapshot contains any new files and changes since the last snapshots. You can revert to any previous snapshot before the file system or files became corrupted. WAFL is an example of a file system that incorporates snapshots, but there are others.

3) use a cloud storage backup/DR copy of your data that incorporates versioning so you can revert to a previous (uncorrupted) version of Ny file or the whole file system. Microsoft’s cloud storage has this feature as one example.

These suggestions don’t address your specific problem that I don’t think has been clearly defined yet, but may give you some ideas going forward as to how to backup AND protect against corruption being replicated.

Reply
 
 
Feb 23, 2020 11:59:59   #
CPR Loc: Nature Coast of Florida
 
Sounds like too many Chiefs and not enough Indians. All those different software's trying to work together is a challenge.

Reply
Feb 23, 2020 12:12:43   #
nadelewitz Loc: Ithaca NY
 
df61743 wrote:
If your budget allows it, the sentence above gives me reason to suggest taking a look at a Synology NAS RAID enclosure, and I would suggest populating it with WD Ultrastar Data Center drives.

I have a DS418 4-bay enclosure with a pair of 10TB drives and a pair of 12TB drives, each pair configured in RAID1 mode. With the Synology software, I can select and folders on my computer to instantly be mirrored on the 10TB drives. The "master" copies of my photo negatives, my music collection, and various other files are saved on the 12TB drives. The RAID configuration protects you from data loss upon a disk failure of any one drive of each pair. The DS418 is ethernet connected to my router, making the data stored on it available to any computer on my network.

You might ask your husbands advice about Network Attached Storage and RAID. You can check it out on Synology's web site or at B&H Photo.

Dick
If your budget allows it, the sentence above gives... (show quote)


Wow. Talk about straying far afield. If the poster can't save to ANY external media at all, there's a problem that needs to be isolated and solved. Suggesting a NAS, and RAID, is way too much "solution" for this user, retired tech husband notwithstanding.

Also, RAID 1 sounds like a great way to have backup drives. I first setup my NAS using it. Then I discovered, testing the system, that if I took a drive from the RAID array and connected it as an external drive to a computer to read its contents, it was not readable at all. Not in Windows or Linux or Mac. A little research showed me many sources saying that a RAID system formats drives that require being read through the exact same controller that created it. Every RAID controller sets up its drives in its own proprietary way. I have read peoples' comments that this is not true in their case, but those seem to be unusual exceptions.

I quickly dumped RAID on my NAS and set it up as the main network-shared drive with multiple USB backup drives formatted NTFS. The backup drives can all be read independently of the RAID array and can be synced to the main drive manually at will or scheduled for automatic syncing. My NAS OS is OpenMediaVault and I suppose any NAS OS will be the same.

Reply
Feb 23, 2020 12:42:20   #
TriX Loc: Raleigh, NC
 
nadelewitz wrote:
Wow. Talk about straying far afield. If the poster can't save to ANY external media at all, there's a problem that needs to be isolated and solved. Suggesting a NAS, and RAID, is way too much "solution" for this user, retired tech husband notwithstanding.

Also, RAID 1 sounds like a great way to have backup drives. I first setup my NAS using it. Then I discovered, testing the system, that if I took a drive from the RAID array and connected it as an external drive to a computer to read its contents, it was not readable at all. Not in Windows or Linux or Mac. A little research showed me many sources saying that a RAID drive formats drives that require being read through the exact same controller that created it. Every RAID controller sets up its drives in its own proprietary way. I have read peoples' comments that this is not true in their case, but those seem to be unusual exceptions.

I quickly dumped RAID on my NAS and set it up as the main network-shared drive with multiple USB backup drives formatted NTFS. The backup drives can all be read independently of the RAID array and can be synced to the main drive manually at will or scheduled for automatic syncing. My NAS OS is OpenMediaVault and I suppose any NAS OS will be the same.
Wow. Talk about straying far afield. If the poster... (show quote)


Yep. A NAS device by definition “owns” the file system, and when you buy a 3rd party NAS, you are typically buying a 3rd party file system as well, and that is the issue. The file system is the next most critical SW item after the OS - all your data depends on it, so would you trust your data to xyz file system, written and (maybe) supported by some small 3rd party company? Not me. If I were going to run a NAS, I would create a Linux server, run ZFS as a file system, and if necessary, run Samba to export CIFS shares, or else I’d buy an old used NetAPP “filer” with the appropriate licenses.

And if I were going to run a RAiD, I wouldn’t use a 3rd party external HW controller, or you could run into the exact issue Mr. Nadelewitz describes - I’d let Windows create the RAID group using the built-in RAID functionality in Windows.

Reply
Feb 23, 2020 13:38:12   #
chrissybabe Loc: New Zealand
 
I use bvckup2 to create my backups. Also use it to backup my wifes computer (from my PC). For a few months I always had a few corrupted files that were never the same files nor from any particular disks and could vary from 1 or 2 files to maybe 10 or more. Even the supplier of bvckup2 couldn't help. I explored and checked all sorts of things like network issues, faulty drives etc and was about to give up on bvckup2 when I let one of Windows interminable updates happen. After the update everything started working again. Never have I seen such a definite fix before or since. It is such a permanent fix that whenever I get an error using bvckup2 now it always means a faulty disk drive rather than something unknown.
This OP so far has an undefined file copy problem and suggesting alternatives like Raid etc are of little help until the original file copy error is fixed. Without more information than given I cannot suggest a fix but I might suggest something like StableBits as a starter.

Reply
 
 
Feb 23, 2020 14:12:24   #
rck281 Loc: Overland Park, KS
 
I have used an internal Windows RAID controller for many years (maybe 15 years) without a problem. The IRST software always told me if a drive failed or got flakey (twice).

Reply
Feb 23, 2020 14:28:33   #
TriX Loc: Raleigh, NC
 
chrissybabe wrote:
I use bvckup2 to create my backups. Also use it to backup my wifes computer (from my PC). For a few months I always had a few corrupted files that were never the same files nor from any particular disks and could vary from 1 or 2 files to maybe 10 or more. Even the supplier of bvckup2 couldn't help. I explored and checked all sorts of things like network issues, faulty drives etc and was about to give up on bvckup2 when I let one of Windows interminable updates happen. After the update everything started working again. Never have I seen such a definite fix before or since. It is such a permanent fix that whenever I get an error using bvckup2 now it always means a faulty disk drive rather than something unknown.
This OP so far has an undefined file copy problem and suggesting alternatives like Raid etc are of little help until the original file copy error is fixed. Without more information than given I cannot suggest a fix but I might suggest something like StableBits as a starter.
I use bvckup2 to create my backups. Also use it to... (show quote)


I agree - the problem has not been well defined yet. Waiting for further...

Reply
Feb 23, 2020 15:51:15   #
nadelewitz Loc: Ithaca NY
 
The whole issue about using RAID as a BACKUP system is the problem of the drives not being individually readable outside the RAID array they were created in.

If your desired intent is to be able to take one of the RAID drives and read it on another computer to retrieve what's on it should the NAS server fail, RAID is not advisable.

BTW, since we like to advertise the specific hardware we use....my NAS server is a Raspberry Pi running OpenMediaVault OS. $35.00 computer with an excellent free system. It's a Linux-based OS, and prefers to have the main drive formatted EXT4, the standard Linux format, so it is readable by any Linux machine. The USB-connected backups are formatted NTFS, readable on Windows, Linux and Mac.

Reply
Feb 23, 2020 16:26:34   #
moxie grrl Loc: Tennessee
 
Hi guys.. thanks so much for the input. The computer is bootable and is working well since we did a disk repair on the problem drive.
I went with Raid 1 because we have had several drives get corrupted and become un-recoverable. I should be able to just replace the bad drive with a good one and have it restore the volume. The big challenge is the volume of data to back up. I just added them up and it is 8.092 TB. I have an ASUSTOR NAS with proprietary software. At work I had access to windows Server 2012 to manage backups but those days are gone.I am open to a different software as long as it is not crazy expensive. We are backing up her main photo drive now, it has been in progress for aprox 30 hours now and is at 40pct. Raspberry PI computers are amazing, so much bang for the buck. I will probably find a different back up software for this. Marla is a heavy user of Photoshop/Lightroom/ON1 and affinity. She reads and writes a lot more than a normal user to the drives and it may just be the volume of data that wears the drive out faster, could be unstable electric as we live in Rural TN. I did put a UPS on her system recently so that may help.

Reply
 
 
Feb 23, 2020 16:41:52   #
TriX Loc: Raleigh, NC
 
moxie grrl wrote:
Hi guys.. thanks so much for the input. The computer is bootable and is working well since we did a disk repair on the problem drive.
I went with Raid 1 because we have had several drives get corrupted and become un-recoverable. I should be able to just replace the bad drive with a good one and have it restore the volume. The big challenge is the volume of data to back up. I just added them up and it is 8.092 TB. I have an ASUSTOR NAS with proprietary software. At work I had access to windows Server 2012 to manage backups but those days are gone.I am open to a different software as long as it is not crazy expensive. We are backing up her main photo drive now, it has been in progress for aprox 30 hours now and is at 40pct. Raspberry PI computers are amazing, so much bang for the buck. I will probably find a different back up software for this. Marla is a heavy user of Photoshop/Lightroom/ON1 and affinity. She reads and writes a lot more than a normal user to the drives and it may just be the volume of data that wears the drive out faster, could be unstable electric as we live in Rural TN. I did put a UPS on her system recently so that may help.
Hi guys.. thanks so much for the input. The comput... (show quote)


If I were losing that many drives, I would be using enterprise class drives and looking at the power. An UPS may help, but it has limits depending on how bad the power is. Most consumer UPSs are not conditioning the power at all times, they are merely a backup power system consisting of a battery and an inverter which kicks in when the main power fails. They also typically contain MOVs (a good one will also have transorbs) as surge suppressors, but their effectiveness is limited. I recently had an incoming power drop short phase to neutral putting 240V on one phase (which would normally be at 120V), destroying everything on that phase that was turned on. The APC UPS smoked and everything plugged into it was destroyed. An extreme case to be sure, but it points out the limitations of consumer grade UPSs. An internal examination showed no fuses (!) and MOVs that had exploded.

Reply
Feb 23, 2020 16:56:13   #
chrissybabe Loc: New Zealand
 
Raid 1 just mirrors the corruption. Plus drives can only be read by the same controller. Plus NAS's are usually connected via networks so are slower. I no longer use ANY sort of raid controller as their disadvantages outweigh their potential advantages. And after exposure to a few Raid errors you get sick of dealing with systems where TWO drives simultaneous fail.
I have found that instead of Raid 1 you are better having 2 individual drives containing the same data. Plenty of backup programs around that will keep two copies running.
And I have found that you need to use enterprise drives or the more expensive raid drives as the lower end drives tend to run only at 5400rpm which coupled with a slow Raid controller CPU and network connections tends to be a lot slower than a SATA or 1GB network connected drive (on another PC).
My wife, who tends to be the type of person who likes to keep EVERY photo she has ever taken, has been told to get a handle on it and delete stuff. She is allowed a 4TB drive and if that fills up she has to go back and delete stuff. As she gets older she is starting to realise that upon her death EVERYTHING will probably be deleted anyway so why bother now. I am now seeing a little progress with her hoarding desire.
I have the same problem with my movie collection and limit this to 4TB as well. Even then we still have 12TB each X 3. I refuse to start investing in 8TB drives because of the expense.
If I backup a complete 4TB drive from scratch over the network it still takes about 12 hours so 8TB taking 30 hours and only at 40% gives you an idea how slow your current system is.
I would suggest you work on her hoarding problem (not trying to be smart here but we all have a touch of this) and maybe work out some different backing up strategies.

Reply
Page <prev 2 of 2
If you want to reply, then register here. Registration is free and your account is created instantly, so you can post right away.
General Chit-Chat (non-photography talk)
UglyHedgehog.com - Forum
Copyright 2011-2024 Ugly Hedgehog, Inc.