Failure of a cloud service company; Easy come, easy go

Cloud computing, that is the answer to every problem facing companies today, or at least you would think this is a truth if you listen to the marketing hype, magazine ads, and even the major analysis companies. This week some people are shocked as we read about the demise of Code Spaces, which was according to their Facebook page providing “a source code hosting platform that enables development and collaboration for software teams.” I would argue this is not even the first and will not be the last cloud-hosting provider that is going to go under. So what happened and what lessons can we learn from this.

What happened?

If you go to their web site you will see the notice that explains in some painful technical details that the company was attacked by hackers. The attack took place leveraging something called a denial of service attack, where a site is flooded with traffic to prevent anyone from using it. It also turns out, that hackers had broken into the management systems for the site had access to the configuration tools for the systems. If you want the more technical version the site explains:

On Tuesday the 17th of June 2014 we received a well orchestrated DDOS against our servers, this happens quite often and we normally overcome them in a way that is transparent to the Code Spaces community. On this occasion however the DDOS was just the start.

An unauthorised person who at this point who is still unknown (All we can say is that we have no reason to think its anyone who is or was employed with Code Spaces) had gained access to our Amazon EC2 control panel and had left a number of messages for us to contact them using a hotmail address

Reaching out to the address started a chain of events that revolved arount the person trying to extort a large fee in order to resolve the DDOS.

Upon realisation that somebody had access to our control panel we started to investigate how access had been gained and what access that person had to the data in our systems, it became clear that so far no machine access had been achieved due to the intruder not having our Private Keys.

So what Code Spaces tries to do at this point is regain control, but in our advanced technical world, that sounds easier then it is. Some would think this is a story line for a cool movie but in reality this cat and mouse game happens every day within organizations.

At this point we took action to take control back of our panel by changing passwords, however the intruder had prepared for this and had already created a number of backup logins to the panel and upon seeing us make the attempted recovery of the account he proceeded to randomly delete artifacts from the panel. We finally managed to get our panel access back but not before he had removed all EBS snapshots, S3 buckets, all AMI's, some EBS instances and several machine instances.

For those that do not speak technical, basically the hacker deleted data, a lot of data. When your a company that runs on data, which more companies do today then they realize, and someone deletes your data, well put simply, your company crashes to the ground.

Code Spaces will not be able to operate beyond this point, the cost of resolving this issue to date and the expected cost of refunding customers who have been left without the service they paid for will put Code Spaces in a irreversible position both financially and in terms of ongoing credibility.

So what have we learned here?

Ransom: First, this is another example of the criminal organizations leveraging ransom to make money. There has been a clear trend over the last year of criminals encrypting or blocking access to data or systems, then demanding payment to go away. I have seen this on laptops, on servers, and with many web sites. This is the same as the gun to the head in a dark alley; don’t move and give me all your money.

Failed architecture: The ability for a company to recover from a data loss is a result of failing to careful define the architecture. Clearly Code Services was missing key pieces of the design. They did a poor job of understanding the risks to the business and creating and testing response processes to these threats.

Bad security controls: You know I am going to say it, "I told you so." That is what the security people at the company are saying, if they even existed. So many companies keep doing the same things wrong. Bad security control design, lack of control effectiveness monitoring, weak admin access management, poor architecture, and the lists go on and on. There are all types of security control frameworks that tell you what you need to do, there are products all over the place, and if you pay enough, you can get highly skilled professionals that know how to do this correctly. First, many companies do not use properly trained and experiences security professionals, second the security professionals do not have the correct funding level to handle today's threats, and finally they often lack the authority to enforce the correct controls. Same story, different company…

Layered Cloud Services: This one is an interesting point. Code Space, a cloud provider, was using Amazon Web Services, another cloud provider. For all we know Code Space might have been using many 3 party organizations and could have really just been a shell of multiple outsourced cloud services. When you place your data into these cloud providers there really is no way to know how many organizations you are dealing with. With so many cooks in the kitchen, it is easy for critical steps or functions to get missed. Which company was doing the backup, where was it hosted, who controlled admin accounts, how where admin accounts managed, and many other critical questions.

Financial Viability: I have been saying for while now that the Cloud Computing experiment is still running, and we will have to wait to see how and if it succeeds. I feel like almost every vendor out there is building their own cloud solutions, or building cloud solutions on top of other cloud solutions. We have customers out there that think somehow these cloud providers are going to magically provide them services at a fraction of the costs compared to doing it themselves. Well, I do not buy into the math. Sure there are some economies of scale, and lower barriers to entry for services with properly designed cloud solutions but in the end you still have to manage the system correctly, make investments in hardware and software, and properly fund the companies. Just like in the 90’s when I was building ISPs, many companies just had bad financial models. They were spending more then they were making and in the end they went bankrupt. When that bankruptcy happened, the equipment was turned off, so many customers came into the office with no telecommunication lines, no Internet lines, and sometimes no phone systems.

Code Spaces is the perfect example of this. They failed to invest correctly in a full design, they failed to have correctly funded and execution risk response processes, which in then end, highlighted the failure to price and manage the financial state of the company. As they state themselves, “… Code Spaces (is) in a irreversible position both financially and in terms of ongoing credibility”. Codes spaces failed to properly quantify their risks, and build the correct financial plans to recover from a disaster. So the owners of the company just look at each other and say, oh darn that did not work, then go to the back wall and pull the plug. Oh wait, missed a step, the owners say, “We are sorry”. Sure they might loose some money, but that is OK, they can go down the street re-brand it quickly and open a new company to start again. As people like to say easy come, easy go.

Buyer Beware: The biggest lesson here is buyer beware. As company owners and managers we need to really look hard at all our current and future cloud service providers. We need to make sure we fully understand what they are doing, and how they are doing it. We need to develop a program to actually confirm what these providers say they are doing. We need to make sure our company’s contingency plans include a response process for the failure of a cloud provider. Finally, we really need to find a way to make sure our data is backed up somewhere else, where we can if needed, rebuild our business process with another provider.