Ugly Hedgehog - Photography Forum
Home Active Topics Newest Pictures Search Login Register
Main Photography Discussion
Raid Storage
Page <<first <prev 4 of 6 next> last>>
Feb 26, 2020 10:41:57   #
TriX Loc: Raleigh, NC
 
Just for background:

RAID 0 is striping for speed. Half of each file resides on each drive. Double the speed, but no redundancy - lose 1 drive, and you lose everything.

RAID 1 is simple mirroring between the drives. Lose one and you still have the other (redundancy). The downside is you only get 50% usable of the total capacity.

RAID 0+1 (sometimes called RAID 10) is striping plus mirroring - the favorite of DBAs. You get twice the speed of a single drive and some redundancy. If you lose 1 drive, you’re fine. If you lose 2, you MAY be fine depending on where the two failures are. If you lose both one one side of the mirror, you’re OK. If you lose the A stripe on one side and the B stripe on the other, you’re OK, but if you lose both A stripe drives or both B stripe drives, you lose all your data. Twice the speed of a single drive and some redundancy, but like RAID 1, you get 50% utilization.

No one uses RAID 2

RAID 3 Is byte striping for speed width a dedicated parity drive. The fastest and preferred configuration for large files like images. You get redundancy (from 1 failure), speed (x the number of data drives) and the usable capacity is high (total of all drives minus the parity drive).

RAID 4 is block striping plus a dedicated parity drive. A good compromise between large and small file performance and can stream multiple files at the same time. Same robustness and efficiency as RAID 3.

RAID 5 (most common for 3 drives or more) - block striping with rotating parity (the parity information is spread across all drives). Good small file performance, you can lose 1 drive, it can spool multiple files simultaneously, and the capacity efficiency depends on the number of drives (you get essentially the number of drives minus 1).

RAID 6 (not a defined RAID level) generally means RAID 5 with double parity. Can tolerate a double drive failure at the expenses of capacity efficiency (total of all drives minus 2).

The downside of any 3rd party external RAID is the HW controller. If it fails, you’re down until you can replace it and unless it’s just mirrored drives [RAID 1), the data on the drives may well not be readable by any other type/mfg RAID or directly attached. If I wanted to implement RAID, I’d strongly consider letting the OS/file system do the mirroring and either mount the drives internally or externally in a JBOD (just a bunch of drives) enclosure WITH dual power supplies. Not quite as fast as a HW stripped RAID, but simple and as reliable as the OS and file system. Windows (NTFS), MacOS, and Linux (EXT4 & ZFS) all support RAID natively.

Reply
Feb 26, 2020 10:52:00   #
TriX Loc: Raleigh, NC
 
SuperflyTNT wrote:
If you're worried about 2 drives failing at the same time you should be using RAID 6


I agree. Not as fast as 0+1 but will tolerate a dual drive failure with no loss of data. I would also add that with any RAID that will accommodate just a single drive failure, you are vulnerable to losing all your data if a second drive fails until the failed drive is replaced AND rebuilt, which with large capacity drives, can take DAYS. Double drive failures don’t happen often, but they DO happen - usually from a power or cooling issue or a lightning strike.

Reply
Feb 26, 2020 10:58:56   #
Saleavitt10 Loc: Maine
 
Thanks for all the reply’s. My use of the RAID will be for storage, not backup. My backup process is two-fold. Cloud along with external drives kept at a location separate from home.

Reply
 
 
Feb 26, 2020 11:36:42   #
wmontgomery Loc: Louisiana
 
dsmeltz wrote:
Related issue. I am also considering a RAID. Do any of you routinely swap out a drive and put it in a remote location? I am considering doing a RAID and periodically (monthly?) putting a copy in our safe deposit box or even just in a metal box in my car.

Yes.I have used a CRU ToughTech Duo for sometime. It is a handheld raid storage using 2 ½ inch SATA drives and you can hot swap the drives. I use 3 drives, 2 in the unit and the third I store offsite. When sufficient changes have occurred on the unit I pull one drive and replace it with the offsite drive, the unit automatically mirrors the drives. Unit just removed goes offsite.
I only store photos and video on this unit and I use SSD drives, less potential damage transporting.

Reply
Feb 26, 2020 12:46:28   #
Pumble
 
Saleavitt10 wrote:
Considering setting up a Raid system to store my growing photography collection. I’d like to get rid of the multiple external hard drives that currently hold the collection. I’d love to hear from those of you that use a Raid. Why you chose the one you have, etc.

Thanks in advance.


I use the WD MyCloud Ex2 Ultra NAS solution. Its a Raid 1 mirror. I bought it with the 4TB drives. It's been stable and just what I needed to start. You can get it with varied drive capacities, its available on the net and does auto mirrors. Its a great starter solution. https://www.bhphotovideo.com/c/product/1224323-REG/wd_wdbvbz0080jch_nesn_8tb_my_cloud_ex2.html

Reply
Feb 26, 2020 13:20:18   #
bpulv Loc: Buena Park, CA
 
Pumble wrote:
I use the WD MyCloud Ex2 Ultra NAS solution. Its a Raid 1 mirror. I bought it with the 4TB drives. It's been stable and just what I needed to start. You can get it with varied drive capacities, its available on the net and does auto mirrors. Its a great starter solution. https://www.bhphotovideo.com/c/product/1224323-REG/wd_wdbvbz0080jch_nesn_8tb_my_cloud_ex2.html


I have become very leery of WD drives since two of my WB My Book drives failed simultaneously three weeks ago while operating off the very stable 119 VAC output of my UPS system. Thank goodness that I had my photographs backed up on my Drobo.

Reply
Feb 26, 2020 13:28:20   #
jldodge
 
I am on my third NAS with a RAID6 configuration. I have always used Seagate hard drives in all three, although now I use their enterprise version instead of the lower cost entry level. Agree whole heartedly that the primary purpose is to have a fault tolerant system, NOT maximizing capacity. My first two NAS boxes where Thecus; my third is QNAP and I like their boxes and software much better. With RAID6, up to two drives can fail and you still retain your data (although I have never let it go that far). I have had to swap out a single drive on two separate occasions. It was very simple, pull out the bad drive and insert the new drive. The NAS will rebuild with the new drive and (depending on the size of the drive) return to normal operation. My remote/offsite backup is a copy of the NAS on USB drives using Syncback Pro.

This is the system I have used for quite a few years now and it works well for me. The ONLY time I have ever lost any files is when I have done something stupid and not because my system malfunctioned on me.

Reply
 
 
Feb 26, 2020 13:31:43   #
cbtsam Loc: Monkton, MD
 
abc1234 wrote:
RAID is certainly the way to go regarding redundancy of drives. But for backing up, the way to go is the cloud. Carbonite backs up my files in real time and I can download them instantly anywhere. Storing back up discs offsite does not do that.


So I'm a bit confused, as usual, and I'm trying to learn. If you have excellent backup with Carbonite, what is the function of redundant drives? It would seem that, if one of your drives fails, you could get a new one and restore the lost files from Carbonite. Are you worried Carbonite might get bombed by terrorists? Not trying to be sarcastic, just, as I said, confused and trying to learn.

Reply
Feb 26, 2020 14:03:07   #
TriX Loc: Raleigh, NC
 
cbtsam wrote:
So I'm a bit confused, as usual, and I'm trying to learn. If you have excellent backup with Carbonite, what is the function of redundant drives? It would seem that, if one of your drives fails, you could get a new one and restore the lost files from Carbonite. Are you worried Carbonite might get bombed by terrorists? Not trying to be sarcastic, just, as I said, confused and trying to learn.


A local backup is faster than a cloud restore (which is typically speed-limited by your internet download speed). The cloud is primarily for a disaster recovery (DR) copy. Three copies of your data is optimum. Primary (working) storage, backup, and an off-site DR copy.

Reply
Feb 26, 2020 14:04:37   #
bpulv Loc: Buena Park, CA
 
cbtsam wrote:
So I'm a bit confused, as usual, and I'm trying to learn. If you have excellent backup with Carbonite, what is the function of redundant drives? It would seem that, if one of your drives fails, you could get a new one and restore the lost files from Carbonite. Are you worried Carbonite might get bombed by terrorists? Not trying to be sarcastic, just, as I said, confused and trying to learn.


I do not and will not use cloud storage for a number of reason because you lose some control of your photographs/property. Some of the possible problems with iCloud storage are price increases, terms and conditions changes that adversely affect you, change in ownership of the storage company, bankruptcy (it happens to the biggest of them and your files could be out of your access and controls for years while the parties fight over ownership), hacking and a host of other bad possibilities that could leave you without control of your content.

As a result, I backup on multiple media including external hard drives, a Drobo and my bank safe deposit box where I frequently swap drives to and from my Drobo. Even if a bank files for bankruptcy and/or otherwise fails, the FDIC only places a short hold on access to safe deposit boxes and soon lets clients have access. That way, even if there is a fire or widespread desaster that comprises my photographs I will always have safe copies in reserve.

Reply
Feb 26, 2020 14:14:56   #
TriX Loc: Raleigh, NC
 
bpulv wrote:
I do not and will not use cloud storage for a number of reason because you lose some control of your photographs/property. Some of the possible problems with iCloud storage are price increases, terms and conditions changes that adversely affect you, change in ownership of the storage company, bankruptcy (it happens to the biggest of them and your files could be out of your access and controls for years while the parties fight over ownership), hacking and a host of other bad possibilities that could leave you without control of your content.

As a result, I backup on multiple media including external hard drives, a Drobo and my bank safe deposit box where I frequently swap drives to and from my Drobo. Even if a bank files for bankruptcy and/or otherwise fails, the FDIC only places a short hold on access to safe deposit boxes and soon lets clients have access. That way, even if there is a fire or widespread desaster that comprises my photographs I will always have safe copies in reserve.
I do not and will not use cloud storage for a numb... (show quote)


A MAJOR cloud provider such as Amazon, Google, Apple, IBM, etc) has NEVER gone belly up, gone into bankruptcy or had data hacked unless the user did something stupid. The REAL danger to your data is H.D. failure (especially cheap consumer class drives), no or untested backup, and poor computing practices. The danger of losing your data in the cloud is infinitesimal compared to these, but having an off-site disaster recovery copy (in the cloud) can save your ass as multiple people on UHH have posted. I’ve seen plenty of success stories, but never a story of data loss in the cloud, just lots of fear mongering.

Reply
 
 
Feb 26, 2020 15:25:19   #
1grumpybear
 
Been using Adaptec cards for over 25 years both at home and at my business. Started out with their scsi cards and migrated to the raid controller cards. My first raid controller cards were the 6805 cards and they are still in use as raid 6 controller in the backup computer at work. Upgraded to 71605 cards on other 3 other computers and all have ran great. With raid 6 you can loose any 2 of the 5 hard drives and not lose any data. Yes you need to replace the drives. Then December 2019 on my home computer I started having what I thought was drive problems and was replacing the drives. Adaptec could not explain what was going on, tried several thing but it look like I was going to lose 60 years of my photos. I was afraid to do any major changes for I had the fear of losing every thing. After all said and done I did not lose any data but had many sleepless nights. The only thing I can think of that caused this problem was WINDOWS 10 and their upgrades. Not sure what to think because I do not and did not have this problem at work. The system is working fine now and is the same as before the problem started. Also the drives I replaced are usable and work fine. If I had it to do over again I would go with mirrored drives and back up. Also with the RAID 6 you get about 5/8 of the total drives capacity. (if you use 5 10tb drives your storage capacity is about 26tb)

Reply
Feb 26, 2020 15:33:38   #
TriX Loc: Raleigh, NC
 
1grumpybear wrote:
Been using Adaptec cards for over 25 years both at home and at my business. Started out with their scsi cards and migrated to the raid controller cards. My first raid controller cards were the 6805 cards and they are still in use as raid 6 controller in the backup computer at work. Upgraded to 71605 cards on other 3 other computers and all have ran great. With raid 6 you can loose any 2 of the 5 hard drives and not lose any data. Yes you need to replace the drives. Then December 2019 on my home computer I started having what I thought was drive problems and was replacing the drives. Adaptec could not explain what was going on, tried several thing but it look like I was going to lose 60 years of my photos. I was afraid to do any major changes for I had the fear of losing every thing. After all said and done I did not lose any data but had many sleepless nights. The only thing I can think of that caused this problem was WINDOWS 10 and their upgrades. Not sure what to think because I do not and did not have this problem at work. The system is working fine now and is the same as before the problem started. Also the drives I replaced are usable and work fine. If I had it to do over again I would go with mirrored drives and back up. Also with the RAID 6 you get about 5/8 of the total drives capacity. (if you use 5 10tb drives your storage capacity is about 26tb)
Been using Adaptec cards for over 25 years both at... (show quote)


If I were going to use a HW RAID controller card, Adaptec is as good as any. I too have used them since the SCSI days. Drivers are always an issue, and that is one of the reasons I let Windows do RAID management if I want to implement a RAID, but not sure without checking if Windows can do RAID 6 dual parity.

Reply
Feb 26, 2020 15:38:31   #
nadelewitz Loc: Ithaca NY
 
[quote=TriX]
The downside of any 3rd party external RAID is the HW controller. If it fails, you’re down until you can replace it and unless it’s just mirrored drives [RAID 1), the data on the drives may well not be readable by any other type/mfg RAID or directly attached. If I wanted to implement RAID, I’d strongly consider letting the OS/file system do the mirroring and either mount the drives internally or externally in a JBOD (just a bunch of drives) enclosure WITH dual power supplies. Not quite as fast as a HW stripped RAID, but simple and as reliable as the OS and file system. Windows (NTFS), MacOS, and Linux (EXT4 & ZFS) all support RAID natively.[/quote]

Wrong. A drive out of a RAID 1 mirror array WILL ALSO BE UNREADABLE, as would a drive from any other RAID array type created by the same controller.

Reply
Feb 26, 2020 15:46:25   #
chrissybabe Loc: New Zealand
 
Please excuse not referencing the owners of these quotes. I don’t know how (even if it is possible) to select several quotes into a single reply.

“ RAID is a REDUNDANCY system (that's what the R stands for), not a backup system. It is intended to mirror or spread data across multiple drives, so if one of them fails, the replacement can be reconstructed by the RAID array and data is not lost. A BACKUP is media that can be read anytime, anywhere, on any standard computer. “

Raid is a bit of a hangup from the old days when drives were small and much less reliable. You wanted capacity ? go with Raid 5, you wanted reliability look at Raid 1 or 10, you wanted performance then go with Raid 0. Things have changed. A NAS is maybe okay but remember there are sacrifices here too. The main one being it never has the same speed as a directly connected drive.

" RAID 0+1 (sometimes called RAID 10) is striping plus mirroring - the favorite of DBAs. You get twice the speed of a single drive and some redundancy. If you lose 1 drive, you’re fine. If you lose 2, you MAY be fine depending on where the two failures are. If you lose both one one side of the mirror, you’re OK. If you lose the A stripe on one side and the B stripe on the other, you’re OK, but if you lose both A stripe drives or both B stripe drives, you lose all your data. Twice the speed of a single drive and some redundancy, but like RAID 1, you get 50% utilization. "

Slightly more fault tolerance than Raid 1 and can minimise the problem with 2 drives failing. Same issue with a failing controller.

“ The downside of any 3rd party external RAID is the HW controller. If it fails, you’re down until you can replace it and unless it’s just mirrored drives [RAID 1), the data on the drives may well not be readable by any other type/mfg RAID or directly attached. If I wanted to implement RAID, I’d strongly consider letting the OS/file system do the mirroring and either mount the drives internally or externally in a JBOD (just a bunch of drives) enclosure WITH dual power supplies. Not quite as fast as a HW stripped RAID, but simple and as reliable as the OS and file system. Windows (NTFS), MacOS, and Linux (EXT4 & ZFS) all support RAID “

If you have a Raid controller that has worked for a few years and then fails DON’T assume that buying the latest (assuming anything is still available anyway) and swapping the drives over will work because it is almost certainly at a later revision and won’t read the drives. The only cure is to buy two of them at the same time and keep one as a spare. Maybe swapping them over every 6 months to make sure they both still work.

“ I agree. Not as fast as 0+1 but will tolerate a dual drive failure with no loss of data. I would also add that with any RAID that will accommodate just a single drive failure, you are vulnerable to losing all your data if a second drive fails until the failed drive is replaced AND rebuilt, which with large capacity drives, can take DAYS. Double drive failures don’t happen often, but they DO happen - usually from a power or cooling issue or a lightning strike. “

I had a customer with a Raid 10 once. They had ignored all warnings about drives failing because everything still worked. When the 3rd drive became flaky and the system failed they were forced to restore from their tape backup (which fortunately had worked). Turned out the first drive had failed 18 months earlier. These were accountants.

“ I have become very leery of WD drives since two of my WB My Book drives failed simultaneously three weeks ago while operating off the very stable 119 VAC output of my UPS system. Thank goodness that I had my photographs backed up on my Drobo. “

Several issue here. You had two drives fail simultaneously ! It also happens in Raid systems. As an individual you may have had a system running great for 10 years so your impression of Raid systems is much higher. I have built and maintained hundreds of raid systems. I have seen about 20% of these fail with 2 drives (or more) and controllers fail. Without a completely independent backup you are screwed. And UPS’s need to have their batteries replaced every 3 years as a minimum. And tested every 6 months otherwise little point in having a UPS. The ratio of UPS’s operating correctly (with plenty of running time) is about 10%. The other 90% were the users finding their UPS was not operating, or had failed batteries when the power failed. And the chances of being in the 10% are much better if it is a decent UPS and not a cheapy. And finally the cheap portable backup drives are NOT the manufacturers best drives. They are built to a price not a quality hence they fail more often than an enclosure with enterprise drives in it. You can sort of guess this when the portable drive has a 1 year warranty but enterprise drives have a 5 year warranty. If the portable drive has a life time warranty you know that the manf either expects you not to bother bringing it back AND a certain failure rate is built into the price you pay initially. The manf does not care about any hassle this causes you.

“ So I'm a bit confused, as usual, and I'm trying to learn. If you have excellent backup with Carbonite, what is the function of redundant drives? It would seem that, if one of your drives fails, you could get a new one and restore the lost files from Carbonite. Are you worried Carbonite might get bombed by terrorists? Not trying to be sarcastic, just, as I said, confused and trying to learn. “

Other than the obvious things like the online going out of business, and it has happened, there are a lot of things between where the online storage is and your place. Any of those goes down and you can’t restore. I have seen this happen a lot.

Reply
Page <<first <prev 4 of 6 next> last>>
If you want to reply, then register here. Registration is free and your account is created instantly, so you can post right away.
Main Photography Discussion
UglyHedgehog.com - Forum
Copyright 2011-2024 Ugly Hedgehog, Inc.