{"id":291,"date":"2022-10-23T00:47:56","date_gmt":"2022-10-22T16:47:56","guid":{"rendered":"https:\/\/www.ddzheng.cc\/?p=291"},"modified":"2023-08-15T00:40:10","modified_gmt":"2023-08-14T16:40:10","slug":"building-a-nas-with-a-cursed-motherboard-%e7%94%a8%e7%81%b5%e8%bd%a6%e4%b8%bb%e6%9d%bf%e7%bb%84%e8%a3%85%e4%b8%80%e5%8f%b0nas","status":"publish","type":"post","link":"https:\/\/www.ddzheng.cc\/?p=291","title":{"rendered":"Building a NAS with a Cursed Motherboard \/ \u7528\u7075\u8f66\u4e3b\u677f\u7ec4\u88c5\u4e00\u53f0NAS"},"content":{"rendered":"\n

I have been wanting to build a NAS for a long time. My main goal is to provide some reliability improvement to the data safety issue I have been facing, where I’m constantly running out of storage space and a lot of my data does not have a local backup.<\/p>\n\n\n\n

For this build, I used TrueNAS Scale which is a fairly new addition to the TrueNAS portfolio. I choose this version over traditional BSD-based TrueNAS since I’m a lot more familiar with Debian.<\/p>\n\n\n\n

To safely use ZFS, ECC memory is required.<\/p>\n\n\n

The Motherboard<\/h1>\n\n\n

The motherboard is a JGINYUE(\u7cbe\u7ca4) X99I-SERVER, a new ITX motherboard with x99 chipset and dual 2.5GbE<\/strong>. I acquired this set from a friend, it cost me \uffe51400 or about $200, which includes the motherboard, a Xeon E5-2650v4 CPU and a pair of Kingston DDR4 2400 32G REG-ECC memory. <\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

The heatsink is a COOLSERVER(\u91d1\u94b1\u8c79) R16<\/a> with 6cm fan for LGA2011. One thing to remember for LGA2011 is that it has two types, square and narrow.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n

The Chassis<\/h1>\n\n\n

The chassis I used is a 4-bay NAS chassis, branded as “MineNAS”, which is a lower trim version of the TrueNAS Mini E<\/a> or Supermicro 5028D-TN4T<\/a>. I bought it second-hand on Xianyu (\u95f2\u9c7c) for \uffe5350 including the original 250W PSU. <\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n
\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n

Storage<\/h1>\n\n\n

For system disks 2 Toshiba MLC SATA SSD were used in a mirrored fashion.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

For actual data storage, I used 4 Western Digital (formerly Hitachi, HGST) 16TB HC550 datacenter hard drives.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n

Hardware Assembling<\/h1>\n\n\n

Assembling was mostly easy, but there were also quite a few annoying quirks. Firstly the chassis is missing the front optical drive panel, and the PCI-E slot cover, which I solved by 3D printing.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The chassis fan supplied by the former owner was complete garbage. I dug through my component bin and found a mighty Nidec TA450 fan which has 160 CFM, rated at 12V 1.4A.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The problem with this fan is that it was ripped out of a Dell Optiplex, and has Dell fan connector. Luckily, the pitch of that connector is the same as normal fan connector, so I only had to change the pin order.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The SATA backplane was also a lower-trim version which only had one power connector. Although one is absolutely fine, for some peace of mind I still decided to solder on another one.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n\n

The motherboard tested fine when used out of the chassis, but once I put it in the tray it no longer want to boot. Took some time before I found out the problem was the external clear-CMOS button was stuck pressed. The back panel was about 1mm higher than it should be and pinched the button. So I drilled the button hole to a much larger diameter.<\/p>\n\n\n\n

\n
\"\"<\/figure>\n\n\n\n
\"\"<\/figure>\n<\/figure>\n\n\n

System Installation<\/h1>\n\n\n

I just plugged the USB disk and followed the GUI to install TrueNAS Scale, and before long I was greeted with a functional WebUI.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

At this stage, I found that the system can boot, but cannot reboot. On reboot, it get stuck at<\/p>\n\n\n\n

freeing unused kernel image memory<\/code><\/pre>\n\n\n\n
Since it only happen at reboot, it is mostly related to ACPI or other power settings. After limiting processor C-state to C2, it now can boot and reboot just fine.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n
Software Configuration<\/h1>\n\n\n
I created a main pool called tank<\/code> and create a dataset called config<\/code> to act as home directories and store scripts.<\/p>\n\n\n
Fan Control Scripts<\/h2>\n\n\n
Warning: TrueNAS is an appliance, and is meant to be used not modified.<\/strong> <\/p>\n\n\n\n
The motherboard has a very poor fan regulation function, which starts out at about 70% PWM where all the fans are screaming. So I have to set the fans to fixed 100% in BIOS and do something on the software side.<\/p>\n\n\n\n
I first confirm that my SuperIO chip is supported by running<\/p>\n\n\n\n
sensors-detect<\/code><\/pre>\n\n\n\n
\"\"<\/figure>\n\n\n\n
At the end of the detection, sensors-detect can automatically insert needed kernel module into \/etc\/modules<\/code>. Note this may not persist across TrueNAS update.<\/p>\n\n\n\n
Next I have to find a way to control the fan speed. For this I need the fancontrol<\/code> package, which unfortunately didn’t come with install. TrueNAS also disabled apt<\/code> use by default. So I had to do the following to install the package. This will not persist across update as well.<\/p>\n\n\n\n
chmod +x \/bin\/apt*\napt-get install fancontrol<\/code><\/pre>\n\n\n\n
I would like to control the chassis fan based on the temperature of the disk which is hottest, and fancontrol<\/code> does not offer this function. I found some related stuff here<\/a>, and wrote some script based on the code there. Basically it reads drive temperature, found out the maximum value and store it as milli-degree-Celsius in a file for fancontrol<\/code> to read in as a sensor value. It also handles situation where return value is “” (when drive is in spin-down).<\/p>\n\n\n\n
For this to work I used the hddtemp<\/code> package:<\/p>\n\n\n\n
apt-get install hddtemp<\/code><\/pre>\n\n\n\n
Here is my script (drive string is removed), located at \/mnt\/tank\/config\/home\/admin\/gethddtemp.sh<\/code>. To use it one need to add their disks to the array. The strings can be found at \/dev\/disk\/by-id\/<\/code><\/p>\n\n\n\n
admin@TRUENAS:~$ cat gethddtemp.sh\ndeclare -a DiskList=(\n\"ata-WDC_WUH721816ALE6L4_2P111111\"\n\"ata-WDC_WUH721816ALE6L4_2P222222\"\n\"ata-WDC_WUH721816ALE6L4_2P333333\"\n\"ata-WDC_WUH721816ALE6L4_2P444444\"\n)\n\nFILE=\/tmp\/Hddtemp\nre='^[0-9]+$'\n\nwhile true\ndo\n  maxtemp=0\n  for disk in \"${DiskList[@]}\"; do\n    temperature=$(hddtemp -n \/dev\/disk\/by-id\/$disk 2>\/dev\/null)\n    # Set base temp if we got nothing in stdout, this mean we are asleep\n    if ! [[ \"$temperature\" =~ $re ]];\n      then\n        temperature=0;\n      else\n        if [ \"$temperature\" -ge \"$maxtemp\" ];\n          then maxtemp=\"$temperature\";\n        fi\n    fi\n  done\n\n\n  # Get a tempfile to write out temp\n  NEW=$(mktemp \/tmp\/Hddtemp.XXXX)\n  echo $(($maxtemp * 1000)) > $NEW\n  # Handle case of initial\/clean run\n  if [ -e $FILE ]; then\n    CLEAN=\"rm -f $(readlink -f $FILE)\"\n  fi\n  # Atomic symlink replacement\n  ln -s $NEW $FILE.lnk\n  mv -T $FILE.lnk $FILE\n  # Clean old link, if it existed\n  $CLEAN\n  sleep 30\n\ndone\n<\/code><\/pre>\n\n\n\n
I wrote a service file \/lib\/systemd\/system\/gethddtemp.service<\/code> for this script to be used as a systemd service.<\/p>\n\n\n\n
admin@TRUENAS:~$ cat \/lib\/systemd\/system\/gethddtemp.service\n[Service]\nExecStart=\/mnt\/tank\/config\/home\/admin\/gethddtemp.sh\nRestart=on-failure\nRestartSec=3s\n\n[Install]\nWantedBy=multi-user.target\n<\/code><\/pre>\n\n\n\n
Next step is to configure the fancontrol config file. Here’s my \/etc\/fancontrol<\/code><\/p>\n\n\n\n
admin@TRUENAS:~$ cat \/etc\/fancontrol\n# Configuration file generated by pwmconfig, changes will be lost\nINTERVAL=1\nDEVPATH=hwmon0=devices\/platform\/coretemp.0 hwmon1=devices\/platform\/nct6775.2592\nDEVNAME=hwmon0=coretemp hwmon1=nct6779\nFCTEMPS=hwmon1\/pwm2=hwmon0\/temp1_input hwmon1\/pwm1=\/tmp\/Hddtemp\nFCFANS=hwmon1\/pwm2=hwmon1\/fan2_input hwmon1\/pwm1=hwmon1\/fan1_input\nMINTEMP=hwmon1\/pwm2=55 hwmon1\/pwm1=30\nMAXTEMP=hwmon1\/pwm2=85 hwmon1\/pwm1=50\nMINSTART=hwmon1\/pwm2=150 hwmon1\/pwm1=150\nMINSTOP=hwmon1\/pwm2=0 hwmon1\/pwm1=0\nMAXPWM=hwmon1\/pwm2=255\n<\/code><\/pre>\n\n\n\n
This will update the fan PWM every second, and maximum temperature allowed for hard drives are 50 \u2103 beyond which the fan will spin at full speed.<\/p>\n\n\n\n
I also modified the fancontrol<\/code> systemd service file to depend on my service, and auto restart on failure.<\/p>\n\n\n\n
admin@TRUENAS:~$ cat \/lib\/systemd\/system\/fancontrol.service\n[Unit]\nDescription=fan speed regulator\n# Run pwmconfig to create this file.\nConditionFileNotEmpty=\/etc\/fancontrol\nAfter=lm-sensors.service\nAfter=gethddtemp.service<\/strong>\nDocumentation=man:fancontrol(8) man:pwmconfig(8)\n\n[Service]\nExecStartPre=\/usr\/sbin\/fancontrol --check\nExecStart=\/usr\/sbin\/fancontrol\nPIDFile=\/run\/fancontrol.pid\nRestart=on-failure\nRestartSec=3s<\/strong>\n\n[Install]\nWantedBy=multi-user.target<\/code><\/pre>\n\n\n\n
Note: if you need to used fancontrol<\/code> to control fan speed based on the temperature of one hard drive, you can use the drivetemp<\/code> kernel module, which parse drive temperature into ACPI sensor zones:<\/p>\n\n\n\n
admin@TRUENAS:~$ sudo modprobe drivetemp\nadmin@TRUENAS:~$ sensors\n\n...\ndrivetemp-scsi-4-0\nAdapter: SCSI adapter\ntemp1:        +30.0\u00b0C  (low  =  +5.0\u00b0C, high = +40.0\u00b0C)\n                       (crit low =  +0.0\u00b0C, crit = +80.0\u00b0C)\n                       (lowest = +27.0\u00b0C, highest = +33.0\u00b0C)\n\ndrivetemp-scsi-2-0\nAdapter: SCSI adapter\ntemp1:        +38.0\u00b0C  (low  =  +0.0\u00b0C, high = +60.0\u00b0C)\n                       (crit low = -40.0\u00b0C, crit = +70.0\u00b0C)\n                       (lowest = +30.0\u00b0C, highest = +39.0\u00b0C)\n\ndrivetemp-scsi-0-0\nAdapter: SCSI adapter\ntemp1:        +36.0\u00b0C  (low  =  +0.0\u00b0C, high = +60.0\u00b0C)\n                       (crit low = -40.0\u00b0C, crit = +70.0\u00b0C)\n                       (lowest = +29.0\u00b0C, highest = +37.0\u00b0C)\n\ndrivetemp-scsi-5-0\nAdapter: SCSI adapter\ntemp1:        +29.0\u00b0C  (low  =  +5.0\u00b0C, high = +40.0\u00b0C)\n                       (crit low =  +0.0\u00b0C, crit = +80.0\u00b0C)\n                       (lowest = +26.0\u00b0C, highest = +32.0\u00b0C)\n\ndrivetemp-scsi-3-0\nAdapter: SCSI adapter\ntemp1:        +36.0\u00b0C  (low  =  +0.0\u00b0C, high = +60.0\u00b0C)\n                       (crit low = -40.0\u00b0C, crit = +70.0\u00b0C)\n                       (lowest = +29.0\u00b0C, highest = +37.0\u00b0C)\n\ndrivetemp-scsi-1-0\nAdapter: SCSI adapter\ntemp1:        +37.0\u00b0C  (low  =  +0.0\u00b0C, high = +60.0\u00b0C)\n                       (crit low = -40.0\u00b0C, crit = +70.0\u00b0C)\n                       (lowest = +30.0\u00b0C, highest = +38.0\u00b0C)\n\n...<\/code><\/pre>\n\n\n\n
To find out which SCSI host is which drive, use lsblk -S<\/code><\/p>\n\n\n\n
admin@TRUENAS:~$ lsblk -S\nNAME HCTL       TYPE VENDOR   MODEL                  REV SERIAL       TRAN\nsda  0:0:0:0    disk ATA      WDC_WUH721816ALE6L4   W680 2P111111     sata\nsdb  1:0:0:0    disk ATA      WDC_WUH721816ALE6L4   W232 2P222222     sata\nsdc  2:0:0:0    disk ATA      WDC_WUH721816ALE6L4   W232 2P333333     sata\nsdd  3:0:0:0    disk ATA      WDC_WUH721816ALE6L4   W232 2P444444     sata\nsde  4:0:0:0    disk ATA      TOSHIBA_THNSNJ128GCSU 0101 XXXXXXXXTKJW sata\nsdf  5:0:0:0    disk ATA      TOSHIBA_THNSNJ128GCSU 0101 XXXXXXXXTKJW sata\n<\/code><\/pre>\n\n\n\n
This way you can use fancontrol<\/code> in the normal way without having to resort to custom scripts.<\/p>\n\n\n
Putting the NAS to Use<\/h1>\n\n\n
\"\"<\/figure>\n\n\n\n
For now I put the machine on the shelf in the living room. It’s October now, room temperature is about 22\u2103 and the fan noise is very low, hard drive temperatures are about 37\u2103. In the summer it may be louder and hotter.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
I don’t have multi-gig infrastructure at home so I’m using my GbE connection for now. In the future I may upgrade my network system to multi-gig or directly to 10G.<\/p>\n","protected":false},"excerpt":{"rendered":"
I built a NAS with a cursed motherboard and TrueNAS Scale<\/p>\n","protected":false},"author":1,"featured_media":315,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[4],"tags":[7,22,14,10],"class_list":["post-291","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-tech","tag-hw","tag-server","tag-software","tag-writeup"],"_links":{"self":[{"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/posts\/291","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=291"}],"version-history":[{"count":7,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/posts\/291\/revisions"}],"predecessor-version":[{"id":365,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/posts\/291\/revisions\/365"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=\/wp\/v2\/media\/315"}],"wp:attachment":[{"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=291"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=291"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ddzheng.cc\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=291"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}