We had a great question from one of our readers about what actually happened today, and whether it was just random, or if it was a directed attack, so I thought I'd put together a more detailed post.
Was it a random hit?
This would be random – happens every day, and has nothing to do with security. A lot of large, well known firms have this happen to them also. Nothing about us in particular made us a target, except that we are working on new web infrastructure (adding new servers) that caught the interest of some… let’s call them 'creative web programmers from China and Russia'.
What happened?
To help explain what happened in layman’s terms, let’s try an analogy. If you were in a deli trying to place an order to the order taker and another patron was calling out their order loudly to the same person, the order taker would probably hear your order correctly. But if thousands of people are in the deli calling out their orders at the same time you are, the order taker would not possibly be able to tell who to pay attention to, and whose order to take.
The technical term for our adventure today is a “denial of service” attack, which is what happens when a very large number of requests are sent to the router that manages some of a firm’s web traffic. Routers are somewhat like the “traffic cops” of the Internet, telling traffic where to go. At some (huge) level of traffic, the router hits a point where it can no longer keep up with all of the requests, so it stops responding.
While it would be cool if the cause of this was a new knowledge article or video we posted on our site generating huge demand, but that was not, unfortunately, the cause of the increased requests.
That’s what happened today. When the router received so many requests at once, it got overwhelmed and could not determine who to serve, so it stopped serving everyone.
Why does this happen?
So, why do people send so much bogus traffic to a router that a server can’t keep up with the requests that it shuts down?
Well, some people do it for fun. Others do it because they are using the server’s responses to damage other websites. In this case, the ... 'creative web programmers from China and Russia' were trying to trick one of our web servers into sending responses to the Yahoo ad network, trying to trick Yahoo into thinking that real users had clicked on ads. It is very doubtful that their attempts to generate fake ad clicks actually worked – but it sure did shut our web servers down for a couple of hours.
How was Trumpet impacted?
During this disruption, Trumpet was still able to operate normally. Our IT staff was scrambling, but normal Trumpet operations, support, etc… continued on without issue.
What has Trumpet learned?
As part of the recovery effort, we have adjusted our router design so we can shift traffic between different servers better. In the low probability event that one of our sites attracts the interest of '... creative web programmers from China and Russia' again in the future, we will be able to take just that one site down (instead of all of our sites), and shift resources to bring that site back up more quickly with less interruption. We have also changed our process for registering new web servers so they will not attract attention.
Probably the biggest thing that we've learned is that our network monitoring systems work (we knew about the problem early), and that our disaster recovery systems are effective (for example, having this blog hosted on a completely different area of the Internet allowed us to communicate about the issue, even though our primary web servers were down).
