There are a number of reasons why we ultimately didn’t choose them for HonestyBox when leaving ThePlanet. However, high up on our list were costs, responsiveness and recommendations against them from their existing customers (who were planning to leave themselves). When we decided to go with a different provider our salesman very abruptly wished us “good luck” and ceased all communication. A word to the wise – steer clear of them – what once was the most reliable and dependable hosting provider in this segment has become toxic.

Some folks will recall back in November I wrote how OpenSocial was irrelevant as a platform for social networking applications. I’ve been working with it for the last three days on MySpace and have to say, over the last few months the folks at MySpace have been working hard to bring OpenSocial to a usable reality. I’ve had some time this week to experiment with it and I must say – while certainly not “done” it’s come a VERY long way. Warning: This is going to get kinda lame for those who don’t care about Social Networks and application development.

Signed Requests / Security

OAuth which allows the providers proxies to sign requests using a pre-shared secret key – limited docs on the MySpace implementation are available on the developer site but your best served looking through the forums. This is so huge I don’t even know where to begin. Essentially, prior to this – anyone could make a request, watch it go over the wire and then manipulate the parameters at will. This type of untrusted injection was problematic for any application that needs to maintain integrity of their users data. Consider the following request:


// AJAX request

While 99% of users would never see this request go, the 1% of users who are more aware of what’s going on could see that by simply calling the file directly in their browser would let them write unlimited messages to anyone – virtually unchecked!


// False AJAX requests

Imagine if I wrote a script to automatically call that page incrementing the ‘t’ (target) value by one each time. I could eventually send a message to every user on that social network. In case your wondering the level of difficulty of this, it’s not hard. OAuth changes this by signing the request.


// Pseudocode to sign
secret_key="password"
oauth_signature=sha1("m=write&t=1&s=1&c=Hi+friend&oauth_secret="+secret_key)
params="m=write&t=1&s=1&c=Hi+friend&oauth_signature="+oauth_signature

Now on the server side, we can validate the source of this data to be trusted because we can recalculate the oauth_signature value by reconstructing the string using the same calculation using the shared secret on the server side. If the keys match, the request is valid! So why not just “unsha1″ the value? SHA1 is a hashing algorithm that is generally not reversible. SHA1 is also implemented in most modern web languages so it’s easy to integrate. Only a full public key infrastructure would be more secure! BTW: I know this signature is invalid – it’s for illustrative purposes.

Developers, wherever you are, keep those keys secure and hard to guess!!!

Where MySpace is Still Failing

I commend MySpace for adopting a standard for their implementation of OpenSocial, however, there’s a critical flaw in the execution. It’s OpenSocial based, not a true OpenSocial application. For some unknown reason, I have to upload 3 different versions of my code (1 per “surface”) that are then bascially cached on the MySpace server. While this is great for load and scalability for a startup, it leaves me little room for tweaking and makes testing VERY difficult unless I work in their very small textarea. Thank God for Apple’s Safari which allows for scaling of textarea elements in real time. This also means I need to create a new version of my application for MySpace. It could have accomplished the same means to an end by having their responsibility for the application end with the directory registration process. In that process we could establish a shared key (see OAuth above) and any particular differentiations from the standard OpenSocial format. Furthermore, the write once – run anywhere mantra of Java OpenSocial could be preserved!

Developers Define Space

The Data Portability working group has been formed and has gained some mass acceptance among social network providers triggered mostly by actions of blogger Robert Scoble who downloaded all of his Facebook contacts using a script and sent the entire Social Networking community into an uproar. While nothing has actually come to the users as a result of the group … I’m hopeful that Social Networking will allow us to select the tools we wish to use without alienating our friends.

App Directory

At least in the MySpace arena, there will be a directory of applications that users can choose from. Other social networks also host their own directories and I suspect that will be the case for some time to come. I could see a future state where an automatic discovery tool worked its way in (like RSS for apps) but it may be trivial if developers are continuing to craft their applications for each individual network.

LAMP, for those who don’t know, is a software architecture commonly deployed for high availability web applications. It’s entirely open source (meaning free to use) and is very inexpensive to get started with (some hosting plans offer LAMP for as little as $2-3/mo). I’ve worked with a number of websites running on LAMP and find as a developer I think very little about what’s going on under the hood. Recently I had a horrible experience (fortunately one that was reversible) deploying a site onto FreeBSD — afterall, there’s a reason this is called LAMP and not FAMP. Before I get into details and start an OS war, I’d like to declare that I find there are different tasks that are best suited for different environments. It’s not about selecting the tools that I use most of the time, it’s about using the tool that works best in a given task. I also understand that sometimes, those tools are not available and your forced to make due with what you have. While screwdrivers don’t generally make good hammers, if it’s all you have to hang a picture on the wall, it can get the job done… but I digress.

Recently I was working on an infrastructure move. It involved a few servers which distributed the LAMP stack into a typical database application server model, segmenting rolls out over equipment. Nothing super complex, but sufficiently large and dealing with a large enough volume that the move was done in stages to minimize downtime. Working with our very talented team of System administrators we planned out the entire move and experienced only 1 minute of downtime. The gotcha wasn’t code based, but OS based. The only difference between our two infrastructures was our OS layer (the ‘L’ in LAMP), we had opted for FreeBSD for a number of valid reasons (which are beyond the scope of this rant.)

We began by replicating the database using standard MySQL replication from our production master into the new infrastructure, this went without much difficulty and the system was able to handle the query load nicely. The database replicated and was very quick. We moved our codebase over and made necessary configuration changes to work within the new hardware environment (mostly path & security changes). After we did this, we were able to successfully test and use the application in the new environment – all seemed well. Queries performed faster than they had in our old setup – long and short we were pleased. We flipped the switch and basked in the glory that is a smooth transition. The site worked wonderfully, load was well within reason and we had solid performance from the application.

Two or three hours after the transition (as our load started to increase with traffic) we began experiencing issues with the latency of pages going up and up and up. Working with our ISP we were able to locate the problem. Threading in FreeBSD != threading in Linux. After two more hours working on the equipment, we decided to implement our rollback plan and revert to our old equipment (which was standing by for just such an occasion.) We were able to reapply the data changes to the old equipment and revert just about everything back in just a few minutes and bring the site back online. Unfortunately we learned the hard way, you shouldn’t use (at least as of right now) current stable releases of MySQL and FreeBSD together. While there are a number of hacks to make this work – we decided that we wouldn’t be the guinnea pig who proved these solutions in a production environment. While I have nothing against FreeBSD as an OS, this threading difference was a make or break for us. In summary, if your using FreeBSD and MySQL, and you are expecting (or at least hoping) to become a high volume site, you’ve been warned if things stop working. It’s called LAMP for a reason, move to CentOS, RedHat or another Linux distribution for your database hardware – it will save you many headaches.

Honesty Box has been, up until recently, hosted at a company called ThePlanet who’s incompetence is absolutely amazing. During the 6 months that we hosted our application there, I encountered some of the worst technical support I have ever seen. The last request thread went something like this – due to copyright restrictions, I need to paraphrase the actual conversations. This is just one of a dozen similar interactions with their support department. I would like to say thank you to all of the tech support folks who handled our requests, I understand the nonsense isn’t your fault, but instead expectations of IT departments everywhere – your spread too thin and asked to do too much. If this type of thing interests you read on – otherwise, this is mostly geek. To summarize, don’t host your website with them.

At about 10am I log into the server and realize something is REALLY wrong. The root filesystem is mounted in Read Only – essentially preserving all data, but making the machine completely useless. I try running a couple of disk checking utilities and back up our data saving it to my local system.

Then I start a new ticket with their support system. Keep in mind, this is the merging of three tickets that were ultimately used to track the issue and I had an IM session with support mixed in here as well.

Erik @1:20pm Tuesday:
There’s something wrong with this server. I can’t run any programs (including Apache) and I can’t even write a simple file:
[root@ ~]# touch file.txt
touch: cannot touch `file.txt’: Read-only file system

Erik @1:59pm Tuesday:
What’s the status of my request?

At this point I decided I would contact support more directly and opened an IM session with their support staff to see how things were progressing with the ticket. During the chat session they actually asked me, “Can you describe what’s wrong?” to which I replied, “please read the ticket.” The Tech then followed up with, “what exactly is wrong?” to which I replied, “If I knew what was wrong I would fix it.” This process took 30 minutes but put the message on deck for their support staff to actually look at it.

Tech @2:48pm Tuesday:
The server is set up correctly, but there’s a problem. We recommend you check the disks using /sbin/shutdown -rF and let us know if you continue to have issues.

I ran this within a few minutes and watched the machine go down using ping.

Tech @3:46pm Tuesday:
Our automated system detected a Host Down Critical. We are attempting to contact you per your instructions.

Erik @3:53pm Tuesday:
I started the /sbin/shutdown -rF at 2:50pm and my machine is still unreachable, can you verify it worked?

Tech @4:02pm Tuesday:
I am forwarding your issue on to get someone to reboot and get fsck runned [sic] on your server.

Tech @4:29pm Tuesday:
Assessing issue

Tech @5:59pm Tuesday:
We attempted fsck manually several times. The drive is failing to boot the kernel and will need to be replaced. We’re trying to resurrect the server so you can make backups.

Tech @6:13pm Tuesday:
Server is back online. Please make backups of your data because we need to replace the drive and re-install the OS.

Erik @7:19pm Tuesday:
This is RAID 1, why do we need an OS reload?

Tech @7:33pm Tuesday:
The server isn’t RAID 1.

At this point I think I had fallen asleep – it had been a long couple of days – so bright and early the next day…

Erik @8:45am Wednesday:
Data is backed up, please take down ASAP and install OS, the sooner the better.
RedHat 64bit

Tech @9:50am Wednesday:
Please submit an OS reload request for this server referencing this ticket number

Erik @1:02pm Wednesday:
Was the drive replaced?

Erik @1:04pm Wednesday:
OS reload submitted and all terms etc were agreed too.

Tech @1:22pm Wednesday:
We’re looking for a drive.

Tech @1:27pm Wednesday:
We found a drive.

Tech @2:00pm Wednesday:
We replaced the drive.

Tech @2:57pm Wednesday:
Your OS has been loaded. We’re passing this onto accounting.

Erik @3:01pm Wednesday:
We’re not paying for the OS reload on failed hardware are we?

Erik @3:05pm Wednesday:
This machine did have two hard drives, now there’s only one! Where’s the second drive?
[root@ ~]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda3 67357816 1950656 61930296 4% /
/dev/sda1 101086 16045 79822 17% /boot
tmpfs 1029776 0 1029776 0% /dev/shm

Tech @9:13pm Wednesday:
The disk wasn’t mounted. We’ve added it to the fstab and mounted the file point.

What, is this ordeal finished? Oh my! we have a server back – good thing this wasn’t the core of our operations. Cya TP!

Honesty Box has been an interesting project to work on because of it’s growth. Each day, we have traffic that rivals some smaller dot com sites and we only exist in a small ecosystem known as Facebook. This was all due to their opening up their developer Platform for everyone to build applications. If you were lost by my OpenSocial rant a few weeks ago, Facebook allows anyone who wants to, to create an application and let it live within their website. It requires minimal skills to get started – but you have to have a good idea. This is a huge shift in the way websites are built and how they aggregate user data. During the first few weeks of the application being live, over 1,000,000 users signed up to use Honesty Box. We now have over 4M users, with over 250K of them visiting the application each day.