ColinJong.com Experienced the Longest Down Time

ColinJong.com has just woke up from a long sleep. It has been sleeping since 02 Jan 2010 around 9:00 pm (88 hours).  It has gone into half-die mode on 03 Jan 2010 morning for a short while, but completely unconscious until noon time today.

This is because the server where ColinJong.com has been staying in, had hard disk drives failure. (Primary and Secondary spoiled at the same time.)  The worst news is the web hosting company could only restore it back-dated to 20 Dec 2009 which means we have lost our most latest posts and comments of 2 weeks!!!  Luckily I have my database backup on 01 Jan 2010, but I still lost some comments from visitors.

Sorry if you posted your comments on 1st or 2nd Jan 2010 and your comments did not appear. Please post your comments again.

Let me show you the announcements from the web host, it is very dramatic:-

++++++++++++++++++++++++++++++

Sun 03/01/2010 10:58

Dear Customers,

It has been reported that the server has not been in a stable condition since 1st Jan 2010 (Friday)

We have successfully fix & boot the server by running a hard drive scan utility. Unfortunately this does not solve the problem after seeing multiple failure everyday.

Therefore, we believe the server has a bad hard drive sector; We will replace the server primary hard drive and restore all accounts:

Date: 3 Jan 2010 (Sunday)
Time: 11:20AM (+8GMT Malaysia)
Scope 1: Shutting Down Server & start latest backup Scope 2: Reinstalling OS on new hard drive & Restore backup Estimate Duration: 48 Hours

We believe the server will be back online to run as usual on Monday afternoon. The speed are depending on the data transfer rate.

It will be our best effort to ensure all accounts are restored successfully. While the above take place, the server will be offline.

We regret for any inconvenience caused.

++++++++++++++++++++++++++++++

Mon 04/01/2010 08:31

Dear Customers,

xxxxxxx.xxxxxxxx.COM - Hard Drive Failure (Update #2)

As per our first announcement early morning, the server are currently still running it\’s latest backup.

We are expecting the restoration work to proceed as follows:

Date: 4 Jan 2010 (Monday)
Time: Between 11:30AM to 4:00PM (+8GMT Malaysia)
Scope: Restoration of Backup

The speed are depending on the data transfer rate. A new update will be sent in the next few hour.

Thank you

++++++++++++++++++++++++++++++

Mon 04/01/2010 14:17

Dear Customers,

xxxxxx.xxxxxxxx.COM - Hard Drive Failure (Update #3)

Please be informed as at this moment, our engineers are trying our best to proceed with restoration of new hard drive replacement.

In that process, we are having minor difficulty with our DVD drive, therefore the process would be extended

Therefore, we will be expecting later restoration work to proceed as follows:

Date: 4 Jan 2010 (Monday)
Time: Between 4:00PM to 9:00PM (+8GMT Malaysia)
Scope: Restoration of Backup

We apologize for all the inconvenience caused.

++++++++++++++++++++++++++++++

Mon 04/01/2010 15:59

Dear Customers,

We are pleased to inform that our engineers have successfully booted the server proceed with restoration after replacement of new hard drive.

The backup files that shall be installed would be back dated to 31 Dec 2009 - 1 Jan 2010. The restoration process may take hours.

Date: 4 Jan 2010 (Monday)
Time: Between 5:00PM to 10:00PM (+8GMT Malaysia)
Scope: Restoration of Backup

We expect everything to be fully restored latest before this midnight and the server service will run as usual.

We apologize for the delays due to technical issues earlier.

++++++++++++++++++++++++++++++

Tue 05/01/2010 10:02

Dear Customers,

xxxxxx.xxxxxxxx.COM - Hard Drive Failure (Update #6)

We regret to inform that the initially planned restoration were not successful.

As of today\’s result, we find out that the ATA1 (Secondary Drive) were the one which is not functioning well.

Therefore our engineers are on it\’s way to plug back the primary hard drive and we are expecting the server to come back online in the next 30 minutes.

However, we are scheduling for account transfers for all accounts in xxxxxx.xxxxxxxx.COM to xxxxxxxxx.xxxxxxxx.com

Date: 5 Jan 2010 (Tuesday)
Time: After 1:00PM (+8GMT Malaysia)
Scope: Transfer of Accounts

A new announcement of name server changes will be sent by stages to each individual customers via e-mail. For quick queries, please call our office: 0xx-xxxx-xx

Again, we apologize for the delays due to unexpected technical issues which is slowing the whole process.

++++++++++++++++++++++++++++++

Tue 05/01/2010 14:16

Dear Customers,

xxxxxx.xxxxxxxx.COM - Hard Drive Failure (Update #7)

We have successfully restored the server services.

However the status are temporary due to unexpected long downtime as we have planned earlier, and possible file system failure due to corrupted sector.

Therefore we are setting up another server (xxxxxxxxx.xxxxxxxx.com) as a Replacement server in the next 24 hours.

The migration will be done in stages. Such method above ensure we have the Latest data, and allow us to do it in stages (minimize downtime).

Although it may have downtime possibility due to read-write issues on the hard drives which require forceful reboots, but this gives us minimum downtime as other options available may take extra hours.

++++++++++++++++++++++++++++++

Tue 05/01/2010 20:48

Dear Customers,

xxxxxx.xxxxxxxx.COM - Hard Drive Failure (Update #8)

After sleepless nights of hard efforts, we have failed to bring back the server online as expected twice with our Plan A & Plan B:

I would bring 2 news to all affected users:
Plan A: To install a new hard drive and restore backup on secondary drive (Fail due to failure of secondary drive) Plan B: To restore primary drive and migrate accounts to another server (Fail due to failure of primary drive)

The Bad news is: Both our hard drive has failed/spoiled. With very high bad sectors and failure to mount secondary drives which also contains all backup on 1st Jan 2010 the day it shows sign of failure. I would personally say, bad luck striking 2 hard drives at one-time.

The Good news is: We have a remote backup server which keeps all accounts backup, back-dated 20th Dec 2009 (Our Insurance!)

Therefore we have decided to call off Plan A & Plan B but to proceed with new plan which is to restore the above backup on xxxxxxxxx.xxxxxxxx.COM

The new name server would be:
nsxx.xxxxxxxx.com (xxx.xx.xx.xxx)
nsxx.xxxxxxxx.com (xxx.xx.xx.xxx)

Our engineer has started the installation work earlier and we will be expecting another full night downtime.

I personally assure every customer, we are doing what is best only for our clients; neither we wish this issue take longer hours.

We have been working long hours; Instead of wasting more time to rescue to hard drives, it is my best opinion we restore the accounts on our remote server to ensure all accounts are online by next morning.

++++++++++++++++++++++++++++++

Wed 06/01/2010 06:49

Dear Customers,

* Final Restoration *

As per our e-mail on yesterday night on our last option, we have successfully perform the restoration of backup from our remote backup server back-dated 20th Dec 2009 due to failure in both primary & secondary hard drive.

Our customer service staff will be assisting you to make the name server changes as follows:

The new name server would be:
nsxx.xxxxxxxx.com (xxx.xx.xx.xxx)
nsxx.xxxxxxxx.com (xxx.xx.xx.xxx)

The restoration list is still in progress from A to Z

And the good news is, all e-mails, web files and databases are working in good condition.

Should you experience any difficulty, please write us an e-mail to support@xxxx.my in order for us to investigate further or call us at: 0xx-xxxx-xx

++++++++++++++++++++++++++++++

Wed 06/01/2010 13:44

Hello.

Your account has fully restored.  

If you see further technical issues with your website after restoration and require technical assistance, please do not hesitate to inform us directly.

Thank you

++++++++++++++++++++++++++++++

I have learned a lesson:- Prepare for the unexpected, BACKUP IS EXTREMELY IMPORTANT, it saves ColinJong.com’s life.