Recover.IT – HP EX490 MediaSmart Server – Part 2

This is part 2 of the recovery of the HP EX490 MediaSmart Server – which is a Windows Home Server machine. The second drive on this server was seen to be offline, so I had shut down this server to investigate the problem.

Last Saturday, I ran some tests on the drive as previously mentioned. On Wednesday night, I decided to copy the disk to a new 3TB disk that I had just bought – a Toshiba 3TB drive with 3 years warranty for $127 each from a local computer shop. I thought that this was a good price.

Anyway, as you might have guessed – I connected this disk on a Linux machine. The Linux in this case was Ubuntu. I used the dd command (that I have previously mentioned) to copy raw data from the disk directly to the new disk.

dd if=/dev/sdb of=/dev/sdc conv=noerror,sync 2>&1 | tee -a ./logfile.txt

Now, of course – I did a smartctl -a /dev/sdb first to check the source disk, and then another one – smartctl -a /dev/sdc to confirm the destination disk. The source disk is a Seagate – correct, and the destination disk is a Toshiba – also correct, so I was good to go. It is good to check, and don’t assume that because the Seagate is connected to SATA0 and the Toshiba is connected to SATA1 that the disk designations will be in the right order.

Ok, so on Wednesday, the copy was started – then I went back to the machine some time later to check on its progress and I see these errors on the display.

dd: error reading ‘/dev/sdb’: Input/output error
6364552+0 records in
6364552+0 records out
3258650624 bytes (3.3 GB) copied, 346.982 s, 9.4 MB/s
dd: error reading ‘/dev/sdb’: Input/output error
6364552+1 records in
6364553+0 records out

6,364,552 sectors were read and copied before an error occurred. The noerror parameter means that it will continue, and sync means that the unreadable sector will be replaced on the destination with a blank sector. I stopped the copy at that time, since it is not a good idea to keep trying to read bad sectors in case the drive decides to quit permanently.

Then last night, I decided to copy from a point after this sector. This time I used this command line and let it run overnight after it seemed to start without throwing up any errors.

dd if=/dev/sdb of=/dev/sdc conv=noerror,sync bs=1M skip=4000 seek=4000 2>&1 | tee -a ./logfile.txt
1903728+1 records in
1903729+0 records out
1996204539904 bytes (2.0 TB) copied, 24148.3 s, 82.7 MB/s

For that command, I set a block size (bs) of 1MB, then used the skip and seek parameters to begin at a point 4000MB into the drive, on both the source and the destination. I checked this morning when I woke up, and found that it had completed successfully – the time taken for the copy works out to about 6.7 hours.

This evening, I also bought a Toshiba 2TB disk drive on my way home – I will talk about this later on. Ok, so I had copied about 3.3GB on Wednesday before it hit the bad sectors. Last night – I started the copy at 4GB or thereabouts onwards and it copied to the end. Now I did a few more copying commands – I won’t bore you with all of the details however the result was to copy the remaining good sectors, using the count parameter to specify how many blocks to copy.

Eventually, I had copied every sector that was able to be copied. It turns out that sectors 6,364,553 to 6,364,568 – 16 of them was unable to be read, not too bad. I also copied a couple of blocks before and after the bad sectors and had a look at the data – it seems to be file information, most likely parts of the Master File Table – which means that a few files are potentially lost.

Ok, this is where my new 2TB drive comes in. I put the faulty Seagate drive back into the EX490, and then added the new Toshiba drive into the top-most bay. After powering up the MediaSmart Server, and waiting – I was eventually shown two solid green lights – which means that the Seagate drive is now online together with the main WD drive, and one blinking green light which was the new Toshiba drive. I logged onto my Windows Home Server console and went into Server Storage and proceeded to add the new drive.

Screenshot 2016-08-05 19.45.01

The idea is to add the new Toshiba drive, so that WHS knows that it is available for storage, and then tell WHS that I want to remove the Seagate drive.

Screenshot 2016-08-05 19.45.51

You might ask, why am I doing this? The drive has bad sectors – it isn’t a good idea to keep using it. Also WHS allows me to remove this disk – by moving and redistributing the files on the disk to other available disks, like the new one that I just added.

Screenshot 2016-08-05 19.46.11

Great, it says that I have sufficient storage space to have this drive removed.

Screenshot 2016-08-05 21.42.40

Ok, I am not actually going to sit here and wait for it, but eventually it will (hopefully) tell me that the drive is ready to be removed. Depending on how full the disk drive was, it can definitely take many hours. Windows Home Server is actually really good, because most storage systems don’t allow you to remove disk drives once they had been used for storing data.

What about the 3TB drive, you are thinking? That is for insurance – in case the disk stops working during the removal, then I have a copy of it that I can use to copy files from. If this removal works successfully, then my 3TB drive can be retasked. By the way, Windows Home Server cannot use disk drives larger than 2TB without major surgery. The reason for this is that WHS uses partitioning based on the Master Boot Record. In order to use drives larger than 2TB, it is necessary to use GPT partitioning – but that is another story.

What about the 16 bad sectors on this Seagate drive? Once I take it out, I plan to do a factory erase on the Seagate drive – this should rewrite every sector on the disk, including the bad ones and I should end up with a disk drive without bad sectors. I can then use it it either for temporary storage of non-critical data or run lots of diagnostics on it to see if it is continuing to fail. If it holds up to the diagnostics, maybe it gets a second chance on life.

In the meantime, I am off to bed!

Advertisements

Retask.IT, Replace.IT – Cryptomining & VMware ESXi 5.5 Update 2 Host Server

So, what has cryptomining got to do with VMware?

Late last year, when the bitcoin was around the US$600 mark, I embarked into cryptocurrency mining.  This was where I used my desktop together with some software like cgminer and began scrypt number crunching using my video card. During a couple of months of trial, I was mining Anoncoin, then moved on to Novacoin, and dabbled briefly on Peercoin which really didn’t work out. There was enough justification to go into this in a bigger way, i.e. 5 mining computers instead of one. I bought a few video cards, actually not a few, 3x Radeon 7950 cards, 7x Radeon 7850 cards, and a Radeon 7870 card. I even pressed into service my older Radeon 5850 card when gave up the ghost after its fan failed one day, but I replaced the fan and heatsink with an after-market cooler and kept it workng.  I played around with a lot of other cryptocoins – that is until the returns from mining would not cover the cost of our expensive electricity.  In addition the room was getting quite hot and having to have the aircon running during summer was just not acceptable. Okay – basically everything was shutdown in June this year, so now I have this hardware sitting around essentially doing nothing.

Retask.it” – the mining hardware, of course. My VMware ESXi 4.0 Host Server was getting old, having run for several years and perhaps now was an opportunity to “Replace.it“. The current version of VMware ESXi is 5.5 Update 2. I put together some hardware to test this version – and had lots of issues installing it because some previously working hardware was no longer supported. There is another story there that I might tell another day. Anyway, after creating my customized installation cd that contains the Realtek 8168 network drivers, and updated adaptec array controller drivers – I was ready to install the production server.

The current configuration for my server contains the following parts:

Asrock 970 Extreme 4 AMD AM3+ motherboard with AMD Athlon X3 420e triple-core cpu and 8GB of ram.  The motherboard can handle up to 64GB of ram, so is sufficient for future expansion. There is no onboard video so I had to buy a single slot Gigabyte GV-R545 video card which houses a Radeon 5450 for $33. I don’t need a high performance card, just one that is a low power card.  The disk storage is an Adaptec 5805 Sata Array Controller (this was found for $300, normally $700+) – initially with 3x WD 3TB Nas Red drives, configured with Raid Level 1E.  I chose the Nas Red drives because they are designed for 24 hr operation – a little more expensive but hopefully are worth it, only time will tell.

SONY DSC

My server needs multiple network cards, one for onboard management, one for internet connection, one for general network and one for backbone network.  Backbone is where I plan to have multiple host servers communicating – not implemented as yet.  The motherboard only has two PCI slots, so I could only install two network cards.  I have a couple of PCI-e networks cards on order – one of those will be for the backbone network.  I found from experience that having a few drives running 24 hours a day has a bit of heat, which requires a bit of cooling.  To that end, I have reused the Antec 1100 case to house all of these items.  This case is a very good for gaming and has lots of cooling, apparently better than the Corsair 500R that was also available.

One more thing is missing, the power supply – I have two FSP Aurum Pro 1000W power supplies left over – one of these was pressed into service and should easily handle another half a dozen drives for future expansion.  Almost forgot – add a cd/dvd-rom drive – I need one in order to install from my customized cd.  To save power, I can always disconnect it after installation – a good idea as this will be running 24×7, since one of the virtual machines is a firewall that protects my internal network from the world wide web.

Current capacity is 4.1TB of which I have used just 80GB, so still another 4000GB to go. If I add five more 3TB drives in Raid 5, this will give me 12TB additional capacity.  In comparison, the old ESXi 4.0 server had 5x 1TB drives in a Raid 1 and a Raid 5 configuration giving me a total of 3TB.  I didn’t know at the time that if I had upgraded the firmware on the Adaptec 5405 Sata Array Controller, I could have achieved this capacity with only 4 drives in Raid 5.  The older firmware at that time only allowed a maximum array size of 2TB to be created.  This was one of the benefits that came out of my testing of ESXi 5.5 – to work out what can be improved.

Anyway, still more work to do. Need to sort out all of the virtual machines, work out which to keep and migrate those to the new server. Better get on with it, I guess.

[PS]  There is a very good reason for including the updated adaptec array controller drivers in my customized installation cd.  During testing and installing adaptec monitoring software, I found that the included adaptec drivers that are bundled with VMware ESXi 5.5 Update 2 did not allow array monitoring, so I had to install some updated drivers from Adaptec.  After doing this, each time I rebooted the server, the datastore went missing.  The datastore houses all of the virtual machines – if this was missing, no virtual machines can run.  It turns out that upgrading the driver caused the VMware to think that the datastore is now a snapshot.  We cannot run from a snapshot (which is like a copy or an image), so the only thing that could be done to fix this permanently is to resignature the storage, but that meant I will need to relink every virtual machine (like 20 of them) – what a headache, so it is best not to upgrade the drivers unless absolutely necessary which means – use the right drivers from the start.
The technical document is here

http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1011387