Specs were:
IBM 486SLC-2/66 10MB RAM 512MB Samsung Hard Drive 10base2 NIC (ne2000-based) Boca 14.4 modem 4x Mitsumi CD-ROM Oak Video 256k VGA Card RedHat 3.0.3 (and later RedHat 4.2)
The specs are as follows:
Pentium 166 48 Mb RAM 768Mb Quantum Hard drive 10baseT Realtek 8029AS NIC US Robotics 56k Modem S3 Virge 4Mb PCI Video card 4x Mitsumi CD-ROM Windows 98SE
AMD K6/300 96Mb RAM 1.6Gb Western Digital HD 2x 100BaseT Realtek 8039 S3 Virge 4Mb PCI Video Card 4x Mitsumi CD-ROM RedHat 6.2
AMD K6/300 96Mb RAM 2.5Gb Western Digital HD 2x 100BaseT Realtek 8039 S3 Virge 4Mb PCI Video Card 4x Mitsumi CD-ROM RedHat 6.2 / EnGarde Secure Linux
AMD K6/300 96Mb RAM 2.5Gb Western Digital HD (/) 80Mb Seagate (/boot) 2x 100BaseT Realtek 8039 S3 Virge 4Mb PCI Video Card 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
Pentium 150 64Mb RAM 2.5Gb Western Digital HD (/) 80Mb Seagate (/boot) 2x 100BaseT Realtek 8039 Oak VGA 256k ISA video card 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
K6-3/450 196Mb RAM 10Gb 7200RPM Maxtor HD (/home) 2.5Gb Western Digital HD (/) 80Mb Seagate (/boot) 2x 100BaseT Realtek 8039 S3Virge 4Mg PCI Video 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
Pics and details can be found here regarding that system.
K6-3/450 196Mb RAM 2x Seagate Barracuda 18.2Gb (md0 -- /home) 2.5Gb Western Digital HD (/) 80Mb Seagate (/boot) 2x 100BaseT Realtek 8039 Adaptec AIC-7881U S3Virge 4Mb PCI Video 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
K6-3/450 196Mb RAM 2x Seagate Barracuda 18.2Gb (md0 -- /home) 2.5Gb Western Digital HD (/) 768Mb Maxtor (/boot) 2x 100BaseT Realtek 8039 Adaptec AIC-7881U S3Virge 4Mb PCI Video 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
Pentium 150 64Mb RAM 2x Seagate Barracuda 18.2Gb (md0 -- /home) 2.5Gb Western Digital HD (/) 768Mb Maxtor (/boot) 2x 100BaseT Realtek 8039 Adaptec AIC-7881U S3Virge 4Mb PCI Video 4x Mitsumi CD-ROM IBM UPS RedHat 6.2 / EnGarde Secure Linux
Brent hooked me up with 3 4.1Gb WD SCSI disks. I decided to use these. All IDE discs are removed from the system. I added 8 extra cooling fans to this case as well. Mounted all the drives in 5.25" bays with two sets of 3 cooling fans.
The final hardware specs are as follows:
K6-2/450 32Mb RAM (I will be increasing this soon) 2x Western Digital 4.1Gb SCSI discs (md1 -- /) 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) 2x 100BaseT Realtek 8139 (eth1 is dropping packets) Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video (overheats when you use it for games) 6x Sony CD-ROM drive AT Full-tower IBM UPS RedHat 6.2 / EnGarde Secure Linux
K6-2/450 128Mb RAM 2x Western Digital 4.1Gb SCSI discs (md1 -- /) 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) 2x 100BaseT Realtek 8139 (eth1 is dropping packets) Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video (overheats when you use it for games) 6x Sony CD-ROM drive AT Full-tower IBM UPS EnGarde Secure Professional 1.5.4
To this date (Feb 18 2004) the following hardware has failed:
md1 : active raid1 sdd1[1](F) sdc1[0] 3855488 blocks [2/1] [U_] I/O error: dev 08:31, sector 3708088 raid1: Disk failure on sdd1, disabling device. Operation continuing on 1 devices
K6-2/450 192Mb RAM 2x Western Digital 4.1Gb SCSI discs (md1 -- /) 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) 100BaseT Realtek 8139 Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video (overheats when you use it for games) 6x Sony CD-ROM drive AT Full-tower IBM UPS EnGarde Secure Professional 1.5.4
K6-2/450 192Mb RAM 2x Western Digital 4.1Gb SCSI discs (md1 -- /mnt/tmp) (both drives bad but the array is still "up") 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) Maxtor 8Gb IDE (/dev/hda1 /) 100BaseT Realtek 8139 Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video (overheats when you use it for games) 6x Sony CD-ROM drive AT Full-tower IBM UPS EnGarde Secure Professional 1.5. And the RAID arrays: Personalities : [raid1] read_ahead 1024 sectors md0 : active raid1 sdb1[1] sda1[0] 17775808 blocks [2/2] [UU] md1 : active raid1 sdd1[1] 3855488 blocks [2/1] [_U] and of course, the wonderful hard drive failure errors in dmesg scsi0: ERROR on channel 0, id 6, lun 0, CDB: 0x28 00 00 00 00 bf 00 00 08 00 Info fld=0xbf, Current sd08:31: sns = f0 3 ASC=11 ASCQ= 1 Raw sense data:0xf0 0x00 0x03 0x00 0x00 0x00 0xbf 0x18 0xff 0xff 0xff 0xff 0x11 0x01 0x00 0x80 0x00 0xac 0x00 0x00 0x07 0x03 0x00 0x4a 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 I/O error: dev 08:31, sector 128 raid1: sdd1: rescheduling block 128 raid1: sdd1: unrecoverable I/O read error for block 128 EXT2-fs error (device md(9,1)): ext2_read_inode: unable to read inode block - inode=416, block=16
Anticipated stats: Athlon Tbird 800 1.5GB RAM 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) Unknown '/' disks Intel Etherexpress 1Gb NIC Adaptec AIC-7881U ATI Rage 128 Pro 16Mb AGP 12/4/32x SCSI PlexWriter ATX Full-tower IBM UPS Unknown OS (Dunno if I will keep EnGarde or move to Debian perhaps, or even try a BSD out)
K6-2/450 192Mb RAM 2x Western Digital 4.1Gb SCSI discs (md1 -- /mnt/tmp) (both drives bad but the array is still "up") 2x Seagate Barracuda 18.2Gb discs (md0 -- /home) Maxtor 8Gb IDE (/dev/hda1 /) 100BaseT Realtek 8139 Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video (overheats when you use it for games) 6x Sony CD-ROM drive AT Full-tower IBM UPS EnGarde Secure Professional 1.5.
The server has finally been moved over to my house. I received a minor upgrade to one of my desktops so Demandred is getting some new hardware. Most notably upgrade to an 800Mhz Athlon (tbird) and 1.5GB of RAM. My ex-wife was pregnant with our first child and things have been getting busy. My desktop is running most of Demandred's services for the time and I haven't had time to rebuild demandred. I bought two more SCSI disks to replace the single IDE root partition. I figured it's time for a fresh install as the system was originally RedHat 6.2 that got merged by hand with EnGarde 1.0 which got manually upgraded to EnGarde Professional 1.5 and has been lingering in existence like this for the better part of 5 years. So I figured I would give Ubuntu Server (7.04) a shot.
The installation was rough. Quite a challenge getting the Software RAID arrays to play friendly. I had to do a number of software RAID things manually. Anyway, once I got it working and built one of the two new (refurbed) SCSI disks I bought died. So I started everything over from scratch with two new hard drives.
Everything is finally installed and the server is running off of a wireless connection and currently resides in my garage. I have since upgraded to Ubuntu 7.10 without issue. One last "issue" I am sure will come. The garage is nice and cold in the winter; however, during the summer months to come I am expecting to possibly see some hard drive failures. As of right now the machine is way too loud to keep in the house. We don't have a basement or any secluded areas. This is the first time since January 2001 that Demandred has not also served as a router. When I moved it from my parents' place they bought their first wireless router. I had purchased one several months earlier.
I also finally spell checked this file.
AMD Athlon 800MHz (tbird) 1.5GB RAM 2x Seagate Barracuda 18.2GB (md0 -- /home) 2x Seagate Barracuda 18.2GB (md1 -- /) 100BaseT Realtek 8139 Ra-Link RT2561/RT61 802.11g PCI Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video Mitsumi 24x/12x CD/RW ATX Full-tower Ubuntu Server 7.10
It's been a year since I've posted anything regarding Demandred and the good news is there's been none. Demandred is sitting in my garage just happily doing it's job. Everything is good so far, knock on wood.
I have a general hardware status page for demandred here.
I've had more headaches from my other computers, failed hard drives, bad memory, two dead laptops, etc. then I'd ever like to tackle, but demandred for once is headache free.
AMD Athlon 800MHz (tbird) 1.5GB RAM 2x Seagate Barracuda 18.2GB (md0 -- /home) 2x Seagate Barracuda 18.2GB (md1 -- /) 100BaseT Realtek 8139 Ra-Link RT2561/RT61 802.11g PCI Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video Mitsumi 24x/12x CD/RW ATX Full-tower Ubuntu Server 7.10
The server appeared to go down, so I connected to it via the serial connection and confirmed the network was down. The system was refusing to recognize the wireless card. After a reboot and the wireless NIC failed to appear. So I swapped the wireless card with another one and the server failed to boot. Being that I had only the serial connection I needed to hook up a monitor at this point.
After some troubleshooting I discovered that one of the SCSI disks in md1 is failing to read properly. It IDs itself to the controller but the controller can't access it beyond that. The same problem with the linux SCSI driver. The RAID array refused to come up with only one disk.
For now I changed 'root=/dev/md1' to 'root=/dev/sdd2' and all is running on the single disk. I have a cold spare SOMEWHERE. I bought it a LONG time ago, and it made several moves. I THINK it's at my parents' house. I'll have to check it out the next time I am there.
These are the hardware errors I am seeing on the bad disk:
# dmesg | grep -i sdb [ 120.340784] sd 0:0:2:0: [sdb] Spinning up disk........................not responding... [ 221.290137] sd 0:0:2:0: [sdb] READ CAPACITY failed [ 221.290147] sd 0:0:2:0: [sdb] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE,SUGGEST_OK [ 221.290159] sd 0:0:2:0: [sdb] Sense Key : Not Ready [current] [ 221.290168] sd 0:0:2:0: [sdb] Add. Sense: Logical unit not ready, cause not reportable [ 221.294715] sd 0:0:2:0: [sdb] Test WP failed, assume Write Enabled [ 221.296227] sd 0:0:2:0: [sdb] Asking for cache data failed [ 221.296285] sd 0:0:2:0: [sdb] Assuming drive cache: write through [ 221.296493] sd 0:0:2:0: [sdb] Attached SCSI diskI confirmed that all the connection were solid. The SCSI controller gives errors when trying to pull up simple data in the BIOS on the drive and the system BIOS does not recognize that disk as a 'bootable' disk at all. So I'll just have to replace it, assuming I can find my spare. I left the replacement wireless NIC in there.
AMD Athlon 800MHz (tbird) 1.5GB RAM 2x Seagate Barracuda 18.2GB (md0 -- /home) 2x Seagate Barracuda 18.2GB (md1 -- /) (Running on one disk) 100BaseT Realtek 8139 Ra-Link RT2500 802.11g PCI (rev 01) Adaptec AIC-7881U NVidia TNT2 16Mb AGP Video Mitsumi 24x/12x CD/RW ATX Full-tower Ubuntu Server 7.10
Increasing wireless problems and complex hardware configurations finally just pushed me to the point of tearing everything apart and starting from scratch. Sort of. I had a spare 1700+ system. Full system with an 80gb hard drive. I upgraded my main system to a new hard drive. It previously had a 160gb and a 200gb drive in it. The 200gb drive was a backup drive for the house.
So I took 20gb of the 80gb for '/' and then grabbed the remaining 60gb and 60gb from the 160gb a created a stripped RAID array for '/home'. The remaining space on the 160gb (100gb) was used for swap and for a network drive I used to run off of my desktop machine. The 200gb drive is remaining as a backup.
The new system is an Athlon 1700+ with 1.5GB of RAM, it's in a smaller case next to my desk and is WIRED to the network on a 1GB connection. Yay. The system was installed with Ubuntu 8.04LTS. Lets see how long this lasts...
So after 4+ years, the refurbished SCSI drives I bought are still running just fine. However, I am not using them simply for reduced complexity in my system. If I continue to have hard drive problems, I will definitely consider putting these back in. The big downside is they draw a lot of power, generate a lot of heat and are extremely noisy.
AMD Athlon 1700+ 1.5GB RAM 80GB ( /, /home (md0) ) Seagate 160GB ( /home (md0), network drive and swap ) Maxtor 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) ATI Radeon 9200 64Mb NEC 12x DVD Burner ATX Mid-Tower Ubuntu Server 8.04LTS
Memorial Day morning I got up, got Connor ready and made a cup of coffee. Moments after sitting down at my computer I hear a series of fizzles and pops. Then the hard drive light on demandred goes solid red. I switch over to demandred and it's dead.
After a couple reboot retries I open demandred up and discover 11 capacitors on the motherboard are blown. Oh well, I am not too surprised and not that upset. The motherboard was 8 years old. I have a spare P4 2ghz box but I only have 512MB I can stuff in it. That's just enough to squeeze by but I have some virtual machines running on Demandred now and what-not. So I looked around and scored a new Athlon 64 3200+ w/ motherboard for $36 + shipping. So I bought that.
In the interim I was going to put the P4 but I couldn't get it running. I didn't have time to mess with demandred much as we were getting everything ready for our annual Memorial Day BBQ.
When I mentioned this to Brent he brought a spare computer with him to the BBQ. Later that evening, after helping us clean up (thanks again Brent!) he tossed the motherboard/processor/memory in to demandred. After some fiddling around he got it all working. The new motherboard and processor should come by this weekend but I will be using my spare time to bottle my beer. So I'll have Brent's hardware running in here for 2-3 weeks. It has only 512MB of RAM for now but I can make do for a bit without my virtual boxen running. Thanks again Brent for all your help and the hardware loan :)
I also finally added some navigation at the top and this passed through the w3c validator, yay! :p
AMD Athlon 1800+ 512MB RAM 80GB ( /, /home (md0) ) Seagate 160GB ( /home (md0), network drive and swap ) Maxtor 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) ATI Radeon 8500 LE NEC 12x DVD Burner ATX Mid-Tower Ubuntu Server 8.04LTS
A new demandred is born. After receiving the new 3200+ motherboard and chip I realized I needed to upgrade the power supply. I bought a new PC Power & Cooling Silencer PPCS370X 370W power supply. I hope this lasts as long as the last one. This PS came in towards the end of the week. So Saturday morning I tossed it all together.
Everything went mostly smoothly except for the RAM. I have 1x2GB of DDR400 (pc3200) high density RAM. Unfortunately the motherboard is not compatible with this. Brent's DDR333 512MB stick works fine it however. So that is what it is running off of for now.
Everything appears to be running well and is quite fast except when swapping. I will be upgrading the RAM at some point in the not so distant future. For now, I am really happy to have Demanded stable again. Current system stats:
AMD Athlon 64 3200+ 512MB DDR333 RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower w/ upgraded power supply Ubuntu Server 8.04LTS
I was hoping this entry would be regarding something along the lines of a RAM upgrade. Unfortunately, it's not. Some cracker from Korea or China broke in to demandred and installed a rootkit. According to rkhunter I had SHV4 and SHV5 rootkits installed on my system. I detected a problem with the system late Saturday but dismissed it as something I possibly screwed up. I didn't look at it again until Monday morning and promptly took Demandred off the main network and started to investigate. I also checked the other 4 systems running Linux to see if there were any problems, there were none. After further investigation, I discovered it was Linux.RST.B-1 that installed on my server.
After investigating I saw a number of attacks from Korean and Chinese owned IP addresses. So I am concluding the attack came from there. I am uncertain how they got in exactly but I have to admit I was getting lax with security and leaving a lot of ports opened. So I have things locked down much better now.
So tonight I re-installed Ubuntu but upgraded to version 9.04 and to the 64bit version as well. The install went smoothly and I managed to bring a number of configuration files over from my backups and got 95% of everything working in less than an hour. So I ended up with a minor OS upgrade through all this. Hopefully my next entry will concern a memory upgrade :)
AMD Athlon 64 3200+ 512MB DDR333 RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower w/ upgraded power supply Ubuntu Server 9.04 (64bit)
A friend of mine managed to secure a really nice, APC 1200VA 780 Watt UPS for me to have, thanks Sean! I hooked this up to demandred as well as my desktop, router and DSL modem. I haven't hooked up demandred to interact with the UPS as of yet but I think I'll investigate using 'nut' to manage everything.
I also wanted to make a note. The power supply I bought a few months back works incredibly well. It's extremely quiet and generates very little heat. The old demandred set up was very hot over the rear power supply, especially with my switch, router and DSL modem sitting atop it. This new power supply barely sits above room temperature.
AMD Athlon 64 3200+ 512MB DDR333 RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 9.04 (64bit)
I hope I am not jinxing anything by posting this. Demandred broke 200 days consecutive uptime.
10:36:25 up 202 days, 11:16, 29 users, load average: 0.14, 0.10, 0.02
Since I have the UPS in and some stable hardware things have been humming along very smoothly. We've lost power a number of times at the house and the UPS kicked right in no problem and with mostly all new hardware in demandred things have been running smooth. That new power supply generates very little heat as well. The only issue I had lately was my Linksys router finally gave out after 5 years of use.
AMD Athlon 64 3200+ 512MB DDR333 RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 9.04 (64bit)
So after 258 days uptime, demandred was accidently rebooted today. So I decided to take this time to upgrade from Ubuntu 9.04 to 9.10 and upgrade the memory. I upgraded the OS because 9.04 is end of life in a couple of months. I want to go to the 10.04 LTS but have to go to 9.10 first to get there. I also am waiting until August 12th for 10.04.1 point release to come out before upgrading to 10.04. So they'll be one more reboot.
As far as memory goes, I came across a 1GB 400mhz stick of RAM so I dropped that in for a total of 1.5GB of RAM. I also found a 512MB stick of 400MHz RAM to replace the 333MHz stick in there now. No more swapping and more speed...
AMD Athlon 64 3200+ 1.5GB DDR 400MHz RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 9.10 (64bit)
I've been having problems with demandred randomly tossing out kernel panics. I think it's from the memory. I've had quite a few reboots. So I put the old memory back in. I also took this opportunity to perform the 10.04.1 LTS upgrade which went without a hitch. I'll have to see what I can do about the memory.
AMD Athlon 64 3200+ 512MB DDR 333MHz RAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 10.04.1 LTS (64bit)
Finally got some decent memory and tossed it in. Bumped demandred up to 1GB now. This will alleviate the heavy swapping. It was getting so bad it would take a good 15 seconds to get a prompt after log in.
On a side note, today is my son's 3rd birthday. Happy Birthday Connor!
AMD Athlon 64 3200+ 1GB 400Mhz DRAM Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 10.04.1 LTS (64bit)
Why has memory been such an issue??? Even with 1GB in demandred it was swapping like crazy. As it turns out, Brendan's old desktop died a few months back and he (through AJ) donated me his 2GB of memory. I finally got around to getting it in there. Two nice, 1GB Corsair sticks. Thanks again Brendan!
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Western Digital (WD800JB-00JJ) 80GB ( /, /home (md0) ) Seagate (ST3160023A) 160GB ( /home (md0), network drive and swap ) Maxtor (6L200P0) 200GB ( backup drive) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 10.04.1 LTS (64bit)
I finally broke down and got some real hard drive space. I removed all 3 of the old drives and migrated over to two 2TB Seagate Barracudas. One drive is for backups and the other runs the system. Hopefully these will last quite a while. I also tossed in a Rosewell USB card reader in place of my 3.5" floppy drive.
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( /, /home ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( /home/backup ) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 10.04.4 LTS (64bit)
Pretty much, to do the day when I upgraded demandred to brand new hard drives, one failed. Hard drives really are the most god awful things in the world. Second one this year, one of my desktop hard drives failed back in January. These were the errors I was receiving for the failed demandred drive:
[583225.359901] ata3.00: status: { DRDY ERR }
[583225.364428] ata3.00: error: { UNC }
[583225.368918] ata3.00: failed command: READ FPDMA QUEUED
[583225.373422] ata3.00: cmd 60/50:78:17:83:e2/00:00:00:00:00/40 tag 15 ncq 40960 in
[583225.373425] res 41/40:50:47:07:80/40:00:00:00:00/40 Emask 0x9 (media error)
[583225.382484] ata3.00: status: { DRDY ERR }
[583225.387005] ata3.00: error: { UNC }
[583225.391523] ata3: hard resetting link
[583225.391527] ata3: nv: skipping hardreset on occupied port
[583225.880051] ata3: SATA link up 3.0 Gbps (SStatus 123 SControl 300)
[583225.940433] ata3.00: configured for UDMA/133
[583225.940481] sd 2:0:0:0: [sda] Unhandled sense code
[583225.940486] sd 2:0:0:0: [sda] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
[583225.940492] sd 2:0:0:0: [sda] Sense Key : Medium Error [current] [descriptor]
[583225.940500] Descriptor sense data with sense descriptors (in hex):
[583225.940504] 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00
[583225.940519] 00 80 07 47
[583225.940526] sd 2:0:0:0: [sda] Add. Sense: Unrecovered read error - auto reallocate failed
[583225.940535] sd 2:0:0:0: [sda] CDB: Read(10): 28 00 00 80 01 bf 00 00 50 00
[583225.940549] end_request: I/O error, dev sda, sector 8389055
[583225.945154] ata3: EH complete
I made the mistake of not RAIDing my drives together so this turned in to quite a headache. What I ended up doing was yanking the bad drive, which housed '/' and '/home' and using the backup drive in place of it. I had a spare 3TB drive I made the new backup drive. I also changed things over to RAID1; however, with a missing drive. So the array is running in degraded mode. However, this will allow me to add the replacement 2TB drive in to the system with minimal hassle. Though, this motherboard only had 2 SATA slots, so I will need to order an additional SATA controller. As it stands now, this is my RAID array status:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md127 : active raid1 sda1[0]
30121728 blocks [2/1] [U_]
md126 : active raid1 sda2[0]
1922048640 blocks [2/1] [U_]
unused devices: <none>
I had many difficulties moving root to a bootable RAID array, so after much frustration I decided to just install Ubuntu 12.04.4 LTS. Especially considering the support for 10.04 LTS ends this time next year and I would have needed to upgrade at that time regardless. I'm now covered until April of 2017, but might move to Trusty before than.
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST3000DM001-1CH1) Barracuda 3TB ( /home/backup ) Intel 8254 Gigabit Ethernet (e1000) NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.4 LTS (64bit)
I received the replacement drive from Seagate and installed it. However, given I have 3 SATA devices in the system now there are only 2 SATA ports on the motherboard. So I had to purchase a PCIe SATA Card. I found the IO Crest 2 Port SATA III PCIe 1x card on Amazon Prime for a good price. The chipset is the Asmedia 1061 (ASM1061) which I read has very good support in Linux. So after installing this card and the new drive the system found both with no extra work on my part. I simply used sfdisk to copy the partition layout of the first drive and then via mdadm added the two partitions to the existing array. The current RAID status is as follows:
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 sda2[1] sdb2[0]
1922048640 blocks [2/2] [UU]
md127 : active raid1 sda1[1] sdb1[0]
30121728 blocks [2/2] [UU]
unused devices: <none>
So now everything looks really good in the system. I do have one issue, the case I am using is a case I've had since circa 2000 which became demandred's case back in November 2008 so it lacks available drive bays without stacking the drives directly on top of eachother, which I want to avoid because of heat. So I currently have one drive just resting at the bottom of the case. Either it's time to upgrade to a newer case or come up with a more solid solution. The updated system specs are below.
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST3000DM001-1CH1) Barracuda 3TB ( /home/backup ) Intel 8254 Gigabit Ethernet (e1000) ASMedia Technology Inc. ASM1062 Serial ATA Controller NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.4 LTS (64bit)
So the other 2GB drive in the array failed. Really can't stand this crap. I just sent the drive back for an RMA about a week ago and put the new one in the other night. Today I got it back in the raid array. Just for my own reference, this is the commands I used to bring the new drive in to the array:
# sfdisk -d /dev/sdf | sfdisk /dev/sda # mdadm --manage /dev/md126 --add /dev/sda2 # mdadm --manage /dev/md127 --add /dev/sda1
Really nothing different in the hardware specs as Seagate sent me an exact replacement drive:
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST3000DM001-1CH1) Barracuda 3TB ( /home/backup ) Intel 8254 Gigabit Ethernet (e1000) ASMedia Technology Inc. ASM1062 Serial ATA Controller NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.4 LTS (64bit)
While I am not certain of the date as I was starting a new job and had a lot going on, I lost two more hard drives. One of the 2TB drives in my RAID array was reporting failed and the 3TB backup drive went into read only mode. After running through the drives I determined that the 3TB drive was completely toasted but the 2TB drive only had some bad sectors. However, I was preparing a complete overhaul of Demandred so I didn't attend to this immedietely and left the RAID array in degraded mode. I hooked up an external 2TB drive to handle my backups.
AMD Athlon 64 3200+ 2GB 400Mhz DRAM (2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) *DEAD* Seagate (ST3000DM001-1CH1) Barracuda 3TB ( /home/backup ) *DEAD* Seagate (Verbatim chassis) 2TB USB drive Intel 8254 Gigabit Ethernet (e1000) ASMedia Technology Inc. ASM1062 Serial ATA Controller NEC (ND-3520A) 12x DVD Burner nVidia GeForce 6100 (onboard) ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.4 LTS (64bit)
This is the largest upgrade Demandred has ever received. For the first time ever I purchased a good deal of new hardware including a new case, DVD burner, and front panel controller. The system got an upgrade from the AMD single code 3200+ to a dual core 5000+ and a memory bump from 2GB to 6GB DDR2. To manage the failing drives I'm increasing airflow with extra 120mm fans controlled by a fan controller that installs in a 5.25" bay, which also comes with a USB 3 PCI-E controller as well.
The hard drives are finally settled as well. The failed 2TB drive had only bad sectors. I did a badblocks test on it and when I reformatted it I had it skip the bad sectors. This is used as the second disk in my RAID 1 array. A bought a new 2TB drive and made this my backup drive.
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md126 : active raid1 sdb2[0] sda2[1]
1922048640 blocks [2/2] [UU]
md127 : active raid1 sdb1[0] sda1[1]
30121728 blocks [2/2] [UU]
Also, for the first time, I have X running on demandred. Since my new job is working from home I use demandred to run all my irc windows, pidgin, skype, music (pithos and audacious) and a few other things.
And a few pictures of the new setup:
|
|
|
|
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) nVidia GeForce GT 630 LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.5 LTS (64bit)
375 days uptime. Coincidentally, also the number of days one of my hard drives took to reach failure.
[Thu Jun 23 14:19:25 2016] ata1.00: status: { DRDY ERR }
[Thu Jun 23 14:19:25 2016] ata1.00: error: { ABRT }
[Thu Jun 23 14:19:25 2016] ata1.00: configured for UDMA/133
[Thu Jun 23 14:19:25 2016] ata1: EH complete
[Thu Jun 23 14:19:25 2016] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0
[Thu Jun 23 14:19:25 2016] ata1.00: irq_stat 0x40000001
[Thu Jun 23 14:19:25 2016] ata1.00: failed command: READ DMA
[Thu Jun 23 14:19:25 2016] ata1.00: cmd c8/00:02:00:00:00/00:00:00:00:00/e0 tag 12 dma 1024 in
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( / (md127), /home (md126) ) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) nVidia GeForce GT 630 LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.5 LTS (64bit)
After replacing that last hard drive in June, I had the next one fail. At least I am becoming a pro at swapping drives and moving them around in their RAID arrays. Anyway, I'm starting to get done with these drive failures and am thinking of moving as much as possible off the server. However, I still need the server. So in the interm, I moved the root partition and the home directories, excluding the 2TB network share onto a pair of RAID 1 SSDs. The system is so much more responsive now. I left the old network share on the remaining good disk. I have this backed up in two additional places. So hopefully this will last until I save up some cash for my final push to move it off.
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) PNY CS1311 240GB x2 ( / (md126), /home (md127) ) Seagate (ST2000DM001-1CH1) Barracuda 2TB ( /home/share (md0) ) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) nVidia GeForce GT 630 LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 12.04.5 LTS (64bit)
Jarvis gave me his old NAS, a D-Link DNS-323. I ordered two WDC 4TB drives for it only to discover it won't support anything over 2TB. I quickly resolved this by flashing the firmware of the DNS-323 with Alt-F and have everything up and running. My main network share is now off demandred and on a separate NAS, finally.
I forget exactly when, but I also upgraded to Ubuntu 14.04LTS as 12.04LTS was EOL.
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) PNY CS1311 240GB x2 ( / (md126), /home (md127) ) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) nVidia GeForce GT 630 LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Ubuntu Server 14.04.5 LTS (64bit)
Wow, just shy of 2 years with nothing to report. Which was the case until recently. With Ubuntu 14.04 hitting EOL I decided to make some massive changes instead of simply upgrading to Ubuntu 16.04. I had just dealt with that exact migration at my job and decided I wanted to do this quite differently. I've moved demandred to a virtual machine. The original hardware is now running Debian Testing (10 buster) with a Xen hypervisor configured. The underlying hardware is mostly the same, with an addition to a dedicated ethernet card for the virtual machine. Additionally, the entire system is now provisioned from Ansible. This includes the Xen host and provisioning of demandred. Everything is now code. While the overall system is vastly more complicated at this point, it allows me to make changes and OS upgrades to demandred on my desktop and then simply rebuild the VM. This should make for much easier management in the future.
I started this process about 9 months ago, writing all the scripts and what-not, in anticipation of the EOL for 14.04. However, during that time I also moved into a new house and the demandred hardware now takes residence in my basement but as part of my 12U rack.
It's also worth noting that one of my SSDs was failed so I had to replace it. Luckily this things are dirt cheap for a 240GB and it seemlessly synced back up with the old drive. One side effect I had is I set it up to have the Xen host manage the software RAID arrays so now demandred just sees the xen virtual disks, even though I am passing the physical drives to the VM.
The specs for the new setup I guess look like this now:
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) (512MB for Xen, 5632MB for Demandred) PNY CS1311 240GB Kingston SA400S3 240GB (Raid replacement disk) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) Seagate (ST2000DM001-1CH1) 2TB ( /home/backup2, Xen root) nVidia GeForce GT 630 Intel 82540EM Gigabit Ethernet Controller (guest Connection) Realtek RTL8111 (Host connection) LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Debian Testing 10 (64bit) (host) Ubuntu Server 18.04.2 LTS (64bit) (guest)
I've now added an additional 4TB NAS drive to this set up, removing the DVD drive as I am out of SATA slots for it. I've decided to use this 4TB NAS drive as my primary network share. It syncs hourly to my 4TB RAID 1 NAS which itself syncs nightly back here, though only a subset of that is backed up. I did this mostly for increased speed. I'm passing the shared drive through to the demandred VM where it is shared over NFS and Samba.
I've also upgraded my desktop which now leaves me with a spare Athlon Phenom II 6X 1045t system loaded with 16GB of RAM. I'm not certain if I want to upgrade this system or pass that system along to my son. It may come here though.
The specs now look like:
AMD Athlon X2 64 5000+ 6GB DDR2 (2x2GiB + 2x1GiB) (512MB for Xen, 5632MB for Demandred) PNY CS1311 240GB Kingston SA400S3 240GB (Raid replacement disk) Seagate (ST32000644NS) Constellation ES 2TB ( /home/backup ) Seagate (ST2000DM001-1CH1) 2TB ( /home/backup2, Xen root) Western Digital Red (WD40EFRX-68N) 4TB (/home/netshare) nVidia GeForce GT 630 Intel 82540EM Gigabit Ethernet Controller (guest Connection) Realtek RTL8111 (Host connection) LineOn iHAS124 24x DVD Burner 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Debian Testing 10 (64bit) (host) Ubuntu Server 18.04.2 LTS (64bit) (guest)
So, in typical fashion for me the new 4TB drive failed in about 48 hours. Whee... After removing it my 2TB Seagate Conestllation drive started clicking. Which is the host's root drive (mislabeled above). Great, so the system is toasted. I took the opportunity to dump the Phenom II in there, replace the bad Constellation drive with a spare SSD I had laying around and I managed to get a quick turn around on the 4TB drive. I've rebuilt the entire system with those parts, ran my Ansible playbooks and almost perfect. The Xen host came up with my provisioning scripts, Ansible provisioned demandred about 90%. I had virtual host issues with Apache and my e-mail was completely hosed. I'm not certain why but I'll be updating my scripts. I should perhaps run these scripts automatically every few weeks with some testing in place.
The new system's specs:
AMD Phenom(tm) II X6 1045T 16GB DDR3 (2x8GiB) (378MB for Xen, 15GB for Demandred) PNY CS1311 240gb x2 (Host root and Demandred /home) Kingston SA400S3 240GB (RAID together with one PNY drive for Demandred's /home) Seagate (ST2000DM001-1CH1) 2TB (/home/backup) Western Digital (WDC WD40EFRX-68N) 4TB (/home/netshare) nVidia GeForce GT 630 Intel 82540EM Gigabit Ethernet Controller (guest Connection) Realtek RTL8111 (Host connection) 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Debian Buster 10 (64bit) (host) Ubuntu Server 18.04.4 LTS (64bit) (guest)
No updates in 3 years. Thank goodness there hasn't been much to report. I had a drive fail in my NAS but that's external to demandred these days. I did have one of my 4tb shared drives fail but I'm starting to think it's a controller issue. I've removed the drive and it tests well. I'm going to investigate later. I'm still running Ubuntu 18.04 which his EOL in a month. I'm debating what to do as my Ansible provisioning scripts hvave greatly drifted and don't deploy well on Ubuntu 22. I'd like to redeploy demandred at some point.
System Specs:
AMD Phenom(tm) II X6 1045T 16GB DDR3 (2x8GiB) (378MB for Xen, 15GB for Demandred) PNY CS1311 240gb x2 (Host root and Demandred /home) Kingston SA400S3 240GB (RAID together with one PNY drive for Demandred's /home) Seagate (ST2000DM001-1CH1) 2TB (/home/backup) Western Digital (WDC WD40EFRX-68N) 4TB (/home/netshare) (dead?) nVidia GeForce GT 630 Intel 82540EM Gigabit Ethernet Controller (guest Connection) Realtek RTL8111 (Host connection) 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Debian Buster 10 (64bit) (host) Ubuntu Server 18.04.4 LTS (64bit) (guest)
This has been nagging at me since th fall and that is Ubuntu 18.04LTS end-of-life has come and gone. So I needed to update, which I did and now I'm running 22.04LTS. I've updated my Ansible scripts and redeployed with a few hiccups. First, my underlying host, Debian Buster does not support jammy. I successfully upgraded to bullseye only to find it too doesn't support jammy. So I moved to bookworm (testing). A few minor issues but everything's running smoothly now.
System Specs:
AMD Phenom(tm) II X6 1045T 16GB DDR3 (2x8GiB) (378MB for Xen, 15GB for Demandred) PNY CS1311 240gb x2 (Host root and Demandred /home) Kingston SA400S3 240GB (RAID together with one PNY drive for Demandred's /home) Seagate (ST2000DM001-1CH1) 2TB (/home/backup) Western Digital (WDC WD40EFRX-68N) 4TB (/home/netshare) (dead?) nVidia GeForce GT 630 Intel 82540EM Gigabit Ethernet Controller (guest Connection) Realtek RTL8111 (Host connection) 5.25" USB 3.0 MultiMedia Front Panel PCI-E card reader and fan controller ATX Mid-Tower PC Power & Cooling Silencer PPCS370X 370w Power Supply APC Back-UPS 1200VA Debian 12 Bookworm (testing) (64bit) (host) Ubuntu Server 22.04.2 LTS (64bit) (guest)
Last Updated -- March 30th, 2023