The third week of September 2016 was a dark and stormy one for KrebsOnSecurity. Wave after wave of huge denial-of-service attacks flooded this site, forcing me to pull the plug on it until I could secure protection from further assault. The site resurfaced three days later under the aegis of Google’s Project Shield, an initiative which seeks to protect journalists and news sites from being censored by these crippling digital sieges.
Damian Menscher, a Google security engineer with whom I worked very closely on the migration to Project Shield, spoke this week about the unique challenges involved in protecting a small site like this one from very large, sustained and constantly morphing attacks.
Google Security Reliability Engineer Damian Menscher speaking at the Enigma conference this week. Photo: @mrisher
Addressing the Enigma 2017 security conference in Oakland, Calif., Menscher said his team only briefly considered whether it was such a good idea to invite a news site that takes frequent swings at the DDoS-for-hire industry.
“What happens if this botnet actually takes down google.com and we lose all of our revenue?” Menscher recalled. “But we considered [that] if the botnet can take us down, we’re probably already at risk anyway. There’s nothing stopping them from attacking us at any time. So we really had nothing to lose here.”
Ars Technica’s Dan Goodin was at the Engima conference and filed this report:
“It took only about an hour for Menscher’s team to arrive at the decision to help Krebs. A much more lengthy process involved actually admitting KrebsOnSecurity into Project Shield…A key requirement for admittance is that the person requesting service proves they have control over the site. Because KrebsOnSecurity was down at that moment, Krebs was unable to satisfy this requirement.
Making matters worse, the domain-name system settings KrebsOnSecurity used had been locked to thwart the attempted domain hijacking attacks that regularly targeted the site. That prevented Krebs from showing he had control of the site’s DNS settings.
Once Project Shield ultimately got KrebsOnSecurity back online, it took just 14 minutes for the attacks to resume.”
For more, check out Dan Goodin’s excellent piece, How Google Fought Back Against a Crippling IoT-Powered Botnet and Won. And a rolling thanks to Damian (a true mensch) and to Project Shield for deflecting the evil bits.
For more background on the botnet responsible for knocking this site offline, see Who is Anna-Senpai, the Mirai Worm Author?
Friends younger than me are still using wall calendars, traditional appointment books and pocket phone books, with everything handwritten inside. As for me, I’ve actually said out loud “If Google ever disappears, I am so *****.” I don’t know anyone’s phone number or address, and my appointments for the next five years are all stored with Google. At least now, I am using multiple cloud sources besides Google, including Apple and a backup service that my cell phone company provides. But this is a good reminder that nothing is guaranteed – some crazy hacker (or organized, government-supported terrorist group) could kill Google’s data core, taking all of our cat photos offline forever. It probably won’t happen, but I do think that it’s a good idea to have multiple places to back up your data.
Your take on it was why I’ve always been against the digitization of everything. The prime example is manuals for operating our power plants. If the grid goes down, we are as you say, “so ******” because everything we use to run our businesses, our government, our air traffic control systems, etc., requires that ubiquitous thing we take for granted, electricity.
That can end at any time, as happened many moons ago when a solar storm knocked out the power grid in certain States/Canada (maybe it WAS a UFO, who knows, but the effect would be the same). If the manuals to restart the power plant are in electronic form, we are in DEEP kemchi, to say the least… Data backup, unless in handwritten spreadsheets of ancient days, would be to say the least, absolutely useless…
References:
https://www.nasa.gov/topics/earth/features/sun_darkness.html
https://en.wikipedia.org/wiki/Northeast_blackout_of_1965
There was a freaky thing that occurred in 1998, when a satellite either turned in the wrong direction or suddenly had a signal blocked. If I recall, this went on for a few weeks – gas pumps randomly didn’t work on the West Coast, some ATMs were out of service, and some credit card processing machines also randomly didn’t work. It turned out that all of these services were satellite based. I remember this distinctly because at my favorite local restaurant, the manager was letting me pay for my meals with a personal check, and he was even letting me write the check for more than the cost of the meal so I could get some cash back. I recall commenting to him one day: “You know, if a terrorist ever wanted to bring our society to a crashing halt, this would be a great way to do it.” The randomness was what drove people crazy – especially with the gasoline pumps. Sometimes they would work and other times they wouldn’t.
http://articles.chicagotribune.com/1998-05-21/news/9805210138_1_galaxy-iv-satellite-outage-pager-customers
The satellite in question lost gyroscopic stability. This caused it to start wobbling in orbit, so the ability to use it faded in and out with its orientation in space (i.e. pointed in the right direction.)
> if a terrorist ever wanted to bring our society to a crashing halt, this would be a great way to do it
Except, as you describe, it already happened and society did not crash to a halt. Your restaurant took effective action to continue. If the outage had been long lasting, they would have continued on that way or found viable alternatives.
Doomsday won’t happen just because some satellites stop working.
Works great when you shop at mom-and-pop places that recognize you. Not so great when you’re in a big city or traveling outside your normal area.
I’m guessing most places also have the ability to insta-verify checks to make sure the funds are available. If that system was down, I doubt they’d access checks from unknowns.
That was nearly nine years ago when people and businesses still relied on a combination of paper and electronic methods to do business. That does not happen very often anymore.
If a similar event were to occur now, most businesses would be gone for the duration and many people would be going hungry as they would have no way to pay for food.
The Internet and cellular communications have become critical and we rely almost comepletely on them – to our detriment in a severe incident.
Most ‘reputable’ organizations have battery/generator backups in case of grid failures or other outages and are required by law to maintain and regularly test Business Continuity and Disaster Recovery Plans. Obviously these are ‘short-term’ solutions intended to allow for an orderly ‘shut down’ to protect from data loss until which time the grid is back up, or until a viable work-around is available. If the grid can not be restored for an extended period of time there’s likely much bigger issues facing us consumers and access to our data…
I have lived almost 40 years without the internet and am still capable to do so anytime. Nothing in my life would ever be able to convince me to backup my data in a cloud.
I dont use a smartphone, have no faceboot or twitcher account but I do post here and on 2 other websites sometimes.
And yes I have a huge wall calendar behind my office chair, huge enough to write half a tweet into each days square 🙂
It is nice of google to help this site (but they also love the image boost) Brian’s research is often very deep and he spends a lot of time on it.
Let us see if his research into the mirai kiddies brings some fruit and these clowns see the judge and their now home soon.
Others will fill the gap…
a true Mensch, that’s very witty Name play
Glad you took up Google’s offer and not CloudFlare’s.
Thanks for posting this article ! :–)
Glad there are Menschen like Damian working at Google. Breathtaking story the whole thing, I’m sure we’re gonna hear more about this doxed Jha guy once the FBI finished their investigations.
I clicked on this link that I found via Google.com:
https://projectshield.withgoogle.com/public/about
It seemed to be ok, but then after a couple of seconds I was redirected to this URL:
https://projectshield.withgoogle.com/public/404
Each time I try the first link I get redirected to the second one.
I was going to contact someone at the site to let them know, but of course, they don’t want to talk to anyone about anything…
Use https://projectshield.withgoogle.com/public/
I appreciate all the work Google is doing to keep websites like this one on line with Project Shield. But I hate to point out that after the switch to Google hosting, krebsonsecurity.com is no longer accessible in Syria (and I assume more countries with embargo in force by United States). In fact, none of the websites hosted by Project Shield are. It seems strange of Google to deny access to journalists and human rights activists’ websites from places like Syria and Iraq when the whole point of Project Shield is to increase availability of said websites.
How do you know Google is purposely blocking these places? How do you know it isn’t something happening because of the US federal government or even the Syrian government?
A lot of things have to happen before a user (anywhere in the world) can access any particular website. Many governments purposely block their people from websites of other countries.
I’m in Damascus. This might answer the rest of your questions:
https://i.imgur.com/67yO0ly.png
Notice the domain, the https with lock icon. I’m posting this through a private network tunnel.
The lock means https (essentially). This does not mean anything that has anything to do with Google specifically. One way or the other. It also does not mean there are no government blocks on place. It sounds like your talking about a VPN or something though. Where as using a VPN might make a difference to things, I’m still not sure what makes you think your being blocked by Google.
I’m sorry i wasn’t clear.
I’m saying that it’s not a government block causing this page to appear since the https (combined with the green lock icon) in the URL indicate a secure connection between me and Google and no MITM can be responsible.
I’m also not wondering if there’s a embargo or ban in place, i know there is. Half of Google services just don’t work here, Project Shield is just one in a dozen.
The secure connection I was referring to is the one I’m using to access this website and type this text you read right now, since the website is available otherwise. It’s not a VPN, it’s a SSH tunnel I run through a VPS of mine in Europe.
That’s a standard 403 error. It’s Google branded because you’re using Google’s Chromium browser not because it’s Google throwing the 403.
You’re respectfully wrong. Here’s a screen shot from firefox:
https://i.imgur.com/0Z9Y9cx.png
Why does everyone assume I’m an idiot? I’m at least reading one of the same security blogs as you are. Give me a break.
This is how a locally blocked website looks like:
https://i.imgur.com/nEN60mB.png
And a different site blocked by Google:
https://i.imgur.com/WZX8R9l.png
For fairness, it’s not only Google that enforces this. Here’s what the homepage of oracle.com redirects to:
https://i.imgur.com/wLFNp04.png
Adobe.com:
https://i.imgur.com/vEYPU1r.png
This error comes from your Chrome browser, not Google’s servers…
No, a 403 error is sent by a server, load balancer or a CDN. If the server has not designed a 403 error page, then the browser will display its builtin error page for 403.
More information from RFC 7231 https://tools.ietf.org/html/rfc7231#section-6.5.4
6. Response Status Codes
“The 404 (Not Found) status code indicates that the origin server did
not find a current representation for the target resource or is not
willing to disclose that one exists.”
Jesus christmas, this is a security blog. You’re sitting here telling some guy that an obviously google-generated 403 error is generated by his browser? Comeon people, step up your game a little bit before playing internet know-it-all.
Can I ask what drive you to the Google’s offer?
You can find that out by reading this story:
https://krebsonsecurity.com/2016/09/the-democratization-of-censorship/
Brian, your reporting is always spot on and full of anecdotal goodies. One of the best article series I have read about how to protect yourself online is hosted on your site. Might you consider revamping it with the potential Demise of EMET?
in my professional life I poiint to your website for technical writinbg written so the non technicall can understand it.
As a local I’ve been a victim ( twice ) to card skimming so your articles even helped bank persoanel I have pointed your way.
KEEP up the FANTASTIC work!
From Microsoft’s EMET page
https://technet.microsoft.com/en-us/security/jj653751
| End of Life Statement
|
| We have listened to customers’ feedback regarding the
| January 27, 2017 end of life date for EMET and we are pleased
| to announce that the end of life date is being extended 18
| months. The new end of life date is July 31, 2018. There are no
| plans to offer support or security patching for EMET after July
| 31, 2018. For improved security, we recommend that
| customers migrate to the latest version of Windows 10.
My experience — this is 100% anecdotal:
If you’re using a Windows 7, EMET v5.2 (which precedes the current version) was probably as up-to-date as you wanted to be. I found that EMET’s current version (v5.5) didn’t play well with W-7. I can’t speak to EMET behavior with W-8 / W-8.1
My understanding was / is that EMET v5.5 offers better compatability with W-10, but I’ve also read that W-10’s internal security features actually make EMET redundant.
As to Microsatan’s end of life statement, I was / am unaware of “security patching” for EMET other than the release of later versions.
There may be no future updates for EMET; not so for malware. And that said, it’s worth closing by point out: EMET was free; can’t complain about that.
Thanks for the update I appreciate knowing they extended it 🙂
Thanks for the update I appreciate it!
try https://projectshield.withgoogle.com/public/404
it worked ONCE for me our of 3 tries, and did not disappear.
I’m surprised nobody made any reference to the whole point we were waiting for which was to see Google shield take on a 1Tbps attack. Instead the attacks Mensch talks to are very small in comparison and probably in line with what they have been defending for years prior to Krebs. Not that you’d wish a large attack on anyone but after Google’s report we are none the wiser about how to defend against such attacks and who out there (if anyone) has the capacity and know-how to defend against them.
It’s saddening to see sites get recked by DDoS Attacks. I have spent countless hours testing and building new protections for my sites. It’s always possible though to get hit and the only thing you can do is keep adding new rulesets. Rate limiting and other restrictions are great but they’re not always enough to stop the attacks. It’s a constant battle between you and the attackers. Things can often get restless and because of that the cost of mitigating attacks can get quite high :/
Tick Tock, Paras. Only a matter of time before you meet your new cellmates. Let’s see how the attitude of yours plays out in prison for the next 15 years.
I likje whqt yyou guyts aare upp too. Thhis kihd oof clver work annd reporting!
Keeep upp thee very goood works guus I’ve added yoou
guys tto mmy owwn blogroll.
>>The satellite in question lost gyroscopic stability.
Are you sure the person aiming the receive antenna didn’t have that problem after a few beers?
Does anyone have a link to Menscher’s actual presentation and/or slides from the Enigma conference?
Delilah! Good heavens, you can always make a new appointment but god forbid we loose the cat photos. 😉
Good Luck, Brian, hope they leave you alone a good long while. If these hacker-crackers are hell bent on this behavior…why don’t they do some good and go after ISIS !?
For more information about Creighton University’s renewable energy project,
please visit If you reside in Omaha, Nebraska or the surrounding metropolitan area
and are contemplating or have completed a solar energy project, please
send me an email describing the initiative and your contact
information to solaromaha@yahoo. We have taken special care for security of our guests and our campus has been completely covered under CCTV cameras which will be monitored 24
x 7 by a special surveillance team. Seven lease sales that were scheduled to take place after Obama’s inauguration have been now canceled
entirely: one in the western Gulf, one off the
coast of Virginia, and five in Alaskan waters, all with proven reserves.
Hi Krebs,
Is there a way to know if my router has been hacked? A few weeks ago, I was not able to access my router administration page (192.168.1.1). And yes, I had already changed the default credentials.
So I did a hard-reset of my router, upgraded the firmware, changed the default credentials (again) and this time I was able to see the router administration page. But this was short-lived, after 1 hour my browser stopped loading the router admin page. I am able to access the internet though. I am using a very old router-cum-modem (about 6 years old). Does this look like my router has been hacked or am I just getting paranoid reading too much of your blogs?! :-p