Yesterday DNSimple, my DNS provider and domain registrar of choice, suffered from a Denial of Service (DoS) attack and as a result their service went down. ?This affected my sites and the sites of many people I know.
DNSimple is a bit of a different provider of services for people on the Internet since they are an infrastructure provider. ?This means they provide a fundamental service which I and others rely to deliver services to our customers. ?This makes a company like DNSimple that much more responsible for being up moire than most. Their outage yesterday, and actually going back a some period of time before yesterday,
They could have handled the outage in several ways. ?The way they chose to handle this outage is an example of how any provider of services on the Internet should handle sometimes uncontrolled events. ?As a company, DNSimple has provided great service in the limited amount of support I needed from them. ?You can distinguish a great company by how they react when faced with a crisis.
Great Service
Respond to your customers in a calm fashion, letting them know how important your issue is to them. ?If support issues come in via email or Twitter, respond to them, even to let them know you received their email or tweet and will get back as soon as you have something to tell them. ?It only takes a second and it keeps your customer calm. ?How many times have you sent a support request and heard crickets for a day or two? ?Me too, it’s annoying.
Treat your customers just the way you want to be treated.
Transparency
Probably the #1 failure of a company under pressure is their lack of transparency. ?What is transparency? ?Simply being open about what’s going on, the issues and how you are dealing with them. ?Most customers are reasonable and probably have been in a crisis themselves and being open with them, communicating what you’re finding takes very little time and is a huge plus.
Be proactive, if you recognize a problem let your customers know before they find it themselves. ?Be genuine, let people know you understand how important this to them.
DNSimple did a great job in their Twitter stream, responding to customers as well as general updates on what was going on and chronicling the ups and downs.
June 1st at 12:05am being proactive:
We?re responding to a denial of service attack. It?s paused for the moment. Details to follow as we learn more.
? DNSimple (@dnsimple) June 1, 2013From first thing in the morning:
We are experiencing a significantly larger and sustained DDoS as of a few minutes ago. We’re working on mitigating.
? DNSimple (@dnsimple) June 3, 2013 ?
Setting expectations:
We are experiencing a significantly larger and sustained DDoS as of a few minutes ago. We’re working on mitigating.
? DNSimple (@dnsimple) June 3, 2013 ?
Keeping your word:
We’re still attempting to get our name servers back online. We will be temporarily disabling ALIAS support in the process.
? DNSimple (@dnsimple) June 3, 2013 ?
Thanking those who are most important:
Thank you to everyone for your patience during this. We’re doing our best to mitigate with multiple actions.
? DNSimple (@dnsimple) June 3, 2013 ?
We are in the processing of fixing this issue for good:
We’ve identified an issue that allowed a DDoS to cause an outage. We’ve stabilized the network while we develop and deploy a software fix.
? DNSimple (@dnsimple) June 3, 2013 ?
I feel good using these folks as a vendor. ?The openness and the continued updates makes me feel like I can count on them to be there and doing all they can to fix this problem and any other problem that might arise in the future.
Offering details to an audience that would otherwise be in the dark cost them very little in the way of time but wins them big in the eyes of the people who rely on this service.
Running a business where you have customers who rely on the product or service you provide is a very rewarding experience, but it doesn’t always go smoothly. ?Customers will remember how you reacted when things got tough. ?What image do you want them to have of you?
UPDATE (6/4/2013 @ 5:00pm EST): Anthony Eden of DNSimple posted a great overview of the problem and the actions taken to resolve and prevent future attacks. ?A really nice way for anyone not following the drama as it occurred.