Honesty Box has been, up until recently, hosted at a company called ThePlanet who’s incompetence is absolutely amazing. During the 6 months that we hosted our application there, I encountered some of the worst technical support I have ever seen. The last request thread went something like this – due to copyright restrictions, I need to paraphrase the actual conversations. This is just one of a dozen similar interactions with their support department. I would like to say thank you to all of the tech support folks who handled our requests, I understand the nonsense isn’t your fault, but instead expectations of IT departments everywhere – your spread too thin and asked to do too much. If this type of thing interests you read on – otherwise, this is mostly geek. To summarize, don’t host your website with them.

At about 10am I log into the server and realize something is REALLY wrong. The root filesystem is mounted in Read Only – essentially preserving all data, but making the machine completely useless. I try running a couple of disk checking utilities and back up our data saving it to my local system.

Then I start a new ticket with their support system. Keep in mind, this is the merging of three tickets that were ultimately used to track the issue and I had an IM session with support mixed in here as well.

Erik @1:20pm Tuesday:
There’s something wrong with this server. I can’t run any programs (including Apache) and I can’t even write a simple file:
[root@ ~]# touch file.txt
touch: cannot touch `file.txt’: Read-only file system

Erik @1:59pm Tuesday:
What’s the status of my request?

At this point I decided I would contact support more directly and opened an IM session with their support staff to see how things were progressing with the ticket. During the chat session they actually asked me, “Can you describe what’s wrong?” to which I replied, “please read the ticket.” The Tech then followed up with, “what exactly is wrong?” to which I replied, “If I knew what was wrong I would fix it.” This process took 30 minutes but put the message on deck for their support staff to actually look at it.

Tech @2:48pm Tuesday:
The server is set up correctly, but there’s a problem. We recommend you check the disks using /sbin/shutdown -rF and let us know if you continue to have issues.

I ran this within a few minutes and watched the machine go down using ping.

Tech @3:46pm Tuesday:
Our automated system detected a Host Down Critical. We are attempting to contact you per your instructions.

Erik @3:53pm Tuesday:
I started the /sbin/shutdown -rF at 2:50pm and my machine is still unreachable, can you verify it worked?

Tech @4:02pm Tuesday:
I am forwarding your issue on to get someone to reboot and get fsck runned [sic] on your server.

Tech @4:29pm Tuesday:
Assessing issue

Tech @5:59pm Tuesday:
We attempted fsck manually several times. The drive is failing to boot the kernel and will need to be replaced. We’re trying to resurrect the server so you can make backups.

Tech @6:13pm Tuesday:
Server is back online. Please make backups of your data because we need to replace the drive and re-install the OS.

Erik @7:19pm Tuesday:
This is RAID 1, why do we need an OS reload?

Tech @7:33pm Tuesday:
The server isn’t RAID 1.

At this point I think I had fallen asleep – it had been a long couple of days – so bright and early the next day…

Erik @8:45am Wednesday:
Data is backed up, please take down ASAP and install OS, the sooner the better.
RedHat 64bit

Tech @9:50am Wednesday:
Please submit an OS reload request for this server referencing this ticket number

Erik @1:02pm Wednesday:
Was the drive replaced?

Erik @1:04pm Wednesday:
OS reload submitted and all terms etc were agreed too.

Tech @1:22pm Wednesday:
We’re looking for a drive.

Tech @1:27pm Wednesday:
We found a drive.

Tech @2:00pm Wednesday:
We replaced the drive.

Tech @2:57pm Wednesday:
Your OS has been loaded. We’re passing this onto accounting.

Erik @3:01pm Wednesday:
We’re not paying for the OS reload on failed hardware are we?

Erik @3:05pm Wednesday:
This machine did have two hard drives, now there’s only one! Where’s the second drive?
[root@ ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 67357816 1950656 61930296 4% /
/dev/sda1 101086 16045 79822 17% /boot
tmpfs 1029776 0 1029776 0% /dev/shm

Tech @9:13pm Wednesday:
The disk wasn’t mounted. We’ve added it to the fstab and mounted the file point.

What, is this ordeal finished? Oh my! we have a server back – good thing this wasn’t the core of our operations. Cya TP!