[I'm Rookie] The nightmare of NAS:Silent Data Corruption - Q6 12/1 updated

本帖最後由 fatdog 於 2016-1-12 08:42 編輯

如果死幾個 file 你唔怕的話,請按此:link
如果死一個 file 都怕怕的話,一齊學野啦
---------------------------------------------------

YOUR NAS IS NAKED TO SDC!


Silent Data Corruption (SDC)
Some errors go unnoticed, without being detected by the disk firmware or the host operating system; these errors are known as silent data corruption.

https://en.wikipedia.org/wiki/Da ... ent_data_corruption

Data degradation (data rot)
also known as data decay or data rot, is a colloquial computing phrase for the gradual decay of storage media.

https://en.wikipedia.org/wiki/Data_degradation
---------------------------------------------------





Q1. Does Raid1 protect (detect & correct) the data from SDC automatically?
The answer is NO for detection & NO for correction.
http://superuser.com/questions/1 ... -against-corruption
%---%
Raid is aimed at disk failure(as redundancy), not data integrity.



Q2. Does Raid5 protect (detect & correct) the data from SDC automatically?
The answer is NO for detection & NO for correction.
Raid 5 might detect data corruption if you do disk scrubbing ON THAT TIME.
http://boink.superatomic.com/2009/04/25/the-raid-5-write-hole/
http://superuser.com/questions/4 ... s-other-raid-levels
%---%
Raid is aimed at disk failure(as redundancy), not data integrity.



Q3. If Raid is not that reliable on data integrity, what would be the better solutions for home user?
The answer is Raid with BtrFS or ZFS because these kind of solution included continuous integrity checking and automatic repair.
https://en.wikipedia.org/wiki/Btrfs
https://en.wikipedia.org/wiki/ZFS
%---%
BtrFS: only Raid1 is stable for production.
ZFS: RaidZ1, RaidZ2, RaidZ3 are all stable for production https://en.wikipedia.org/wiki/Non-standard_RAID_levels#RAID-Z



Q4. If we need to avoid damage from SDC, would it be costly to purchase or even build such Btrfs/ZFS NAS for home user?
The answer is NO, those server board & ecc ram are bearable for home user now.
~HK$2000 budget: 4 bay HP microserver gen8 & 4GB ECC ram (incl. shipping fee)
~HK$2000 budget: 6+ bay HP ML10 v2 & 4GB ECC ram (incl. shipping fee)
with extra HK$500: DDR3 1600MHz 8GB ECC RAM (max. 16gb for gen8 & 32gb for ml10 v2)
from here
you can even upgrade to Xeon CPU later.
%---% It's not expensive to get server grade hardware for building your Raid1 Btrfs or Raidz1-3 ZFS NAS!


Q5. For the case of ZFS system, how many hard disk is required for RaidZ1, RaidZ2 & RaidZ3?
The answer is at least 3 same size hard drives.
http://wintelguy.com/raidcalc.pl
http://biduleohm.free.fr/zfsraidsarc/
RaidZ1(single parity) - min. 3 HDD = 2 usable + 1 can be failed at the same time. (4:3+1, 5:4+1, 6:5+1 ... etc)
RaidZ2(double parity) - min. 4 HDD = 2 usable + 2 can be failed at the same  time. (5:3+2, 6:4+2, 7:5+2 ... etc)
RaidZ3(triple parity) - min. 5 HDD = 2 usable + 3 can be failed at the same  time. (6:3+3, 7:4+3, 8:5+3 ... etc)
More HDD more Storage in each raid.
http://jrs-s.net/2015/02/06/zfs- ... or-vdevs-not-raidz/
%---% For affordable home used server with general 6 bay:
better Reliability & Performance: For 6 HDD, Stripped RaidZ1 = only 2 usable + 1 can be failed at the same time in each Raid group (total 2)
More Storage: For 6 HDD, RaidZ2 = 4 usable + 2 can be failed at the same time in each Raid group.
https://icesquare.com/wordpress/ ... -raidz3-vs-striped/



Q6. I was a Qnap/ Synology/ Windows NAS user before, which OS is good for a Btrfs/ZFS NAS?
The answer is ... none of them.
Qnap is going to sell their 1st ZFS NAS which is target for high-end enterprise, NOT for home user.
http://www.anandtech.com/show/99 ... on-zfs-nas-and-more
Synology has DS716+ with BtrFS for home user BUT it's 2 bay mirrored only ... as danger as RAIDZ1
https://www.synology.com/zh-tw/products/DS716+#spec
Windows NAS with RoFS "What Is the Development Status of ReFS?"
http://www.makeuseof.com/tag/try-resilient-file-system-windows/
%---% NAS OS work with BtrFS or ZFS on home used & production level:
[ZFS] FreeNAS: www.freenas.org
[ZFS] NAS4Free: www.nas4free.org
[ZFS] ZFS Guru: www.zfsguru.com
[BtrFS] Rockstor: www.rockstor.com




Q7. Could those BtrFS & ZFS OS for NAS provided similar service like Synology & Qnap?
The answer is ... [updated later]


Any input is welcome

提示: 作者被禁止或刪除 內容自動屏蔽

TOP

提示: 作者被禁止或刪除 內容自動屏蔽

TOP

ECC RAM +ZFS/Btrfs + Regular Scrub

再應真啲可以做 Hash record
hkecp3 發表於 2016-1-5 19:03


會唔會太快到答案 ...

TOP

最驚是 loss of magnetic property!
啲 files 長時間無 read 過就有機會發生咗都不知! ...
toylet 發表於 2016-1-5 19:07


去到咁遠 ...
本來諗住講到 power supply ...

cHing,你提供下答案先:
Qn. What is loss of magnetic property?

TOP

提示: 作者被禁止或刪除 內容自動屏蔽

TOP

即兮儲住資料那個磁區的磁極改變咗, 啲資料即時錯咗或無咗!
不記得正式學名... spontaneous decay? ...
toylet 發表於 2016-1-5 19:25


我建議你都係自己唔好答,搵d reference support 就算

TOP

提示: 作者被禁止或刪除 內容自動屏蔽

TOP

提示: 作者被禁止或刪除 內容自動屏蔽

TOP

fatdog 師兄,我bookmark咗, 跟你學野....

TOP