How Did I Find Out There Was A Problem?

I check my Google rankings for the term “iPhone blog” on a daily basis. iPhone Download Blog (my iPhone blog) usually ranks #4. Last Monday though, it was nowhere to be found! Doing a search for “site:www.iPhonedownloadblog.com” showed me I had 27,000 pages indexed but again, the homepage was nowhere to be found. Hmm, that doesn’t smell good…

I then went and checked my Google Webmaster Tools account. It usually helps me figure out quickly if there is anything wrong with any of my sites. Usually, everything is fine. From time to time, I get a little error on one or two of my sites, which in most cases can be fixed simply by generating a new sitemap and resubmitting it to Google.

Last Monday, I completely freaked out when I realized that all my sites were showing errors! All of them! I thought it was very strange but I soon realized that my server had been down for a few minutes the day before and I figured that Google must have tried to crawl my sites at this time and was then enable to do it. I simply resubmitted all my sitemaps thinking it would do the trick.

What the Hell Happened?

As a measure of precaution, I checked Google Webmasters Tools again a few minutes later and saw that Google, once again, was giving me errors. The issue was obviously bigger than I thought. I decided to look into the problem more deeply and realized that Google was giving me a 403 error, which means:

The server is refusing the request. If you see that Googlebot received this status code when trying to crawl valid pages of your site (you can see this on the Web crawl page under Diagnostics in Google Webmaster Tools), it’s possible that your server or host is blocking Googlebot’s access.

It was clear now. DreamHost, my hosting company, had restricted Googlebots from crawling my server. I had had the same problem with DreamHost before so it was just half a surprise…

How the Hell Do You Fix This?

The first thing I did was sending a support request to DreamHost. I tried to be kind in my email but I was really furious. I think it’s very unprofessional of them to block access to my server to anyone without even notifying me. A few hours later, I didn’t have an answer yet… DreamHost is usually fast at replying, but I guess they were not that day…

So I sent them a second support request email, a little more spicy than the first one. A couple hours later I received this email:

Hello Sebastien,

I sincerely apologize for this! This was handled incorrectly, and NOT according to our own policy. It looks like a few days ago you wrote in regarding server performance. It appears part of that was actually googlebot slamming your sites. This was blocked to preserve server stability (which is normal troubleshooting), what didn’t happen, and SHOULD have, was that you were not notified and told how to deal with this. I’ve removed the block, but please see the following article:

http://wiki.dreamhost.com/Bots_spiders_and_crawlers

There is a way to slow down googlebot without killing your search results OR the server. This is the preferred approach. We obviously don’t want to hurt your sites.

Again, I apologize for this, and if it helps any, it looks like just an IP range was blocked, and whoever was troubleshooting the server wasn’t really *trying* to block google from your site…just an IP that was causing problems for everyone.

Please let me know if you have any other questions.

Thanks!
Jeff H

Alright, fair enough… What can I do anyway? So this part of the problem was handled but I needed to tell Google everything was back up so it can start crawling my sites asap.

I resubmitted all my sitemaps again. I also wrote a few new posts on my iPhone blog in order to “prove” Google that the site was still alive and doing well. Finally, I linked to my main sites from various other websites in the hope it would help speed up the crawling process.

Googlebots came back and crawled the sites. The next day (on Tuesday), iPhone Download Blog was back in Google’s search results but it was ranking at about #35 for my keyword. I realized that even though it was back in the index, Google didn’t have a cached image of it yet. I assumed that it would come back and crawl the site again and then generate a cached image.

On Wednesday, still no cache! It was ranking slightly better though, showing up on the second page. Still far from what I was looking for, but at least it was progressing a bit.

On Wednesday night, the homepage completely disappeared from the index again. While some may worry about this, I am now used to it and I know what it means. It means that Google has crawled your site again and it is generating a cached image of it. This usually happens when you create a new page. Google will crawl it, rank it, then it will disappear from the index for a few hours, and finally it will come back for good. It’s like Google is hurrying to crawl it, but then takes its time to analyze it and make sure it’s not crap.

I went to bed that  night confident that my site would be back on the next morning when I get up. Sure enough, it is ranking again, as good as ever, this morning. Yeahhh!!! All my other sites are back too.

What’s the Moral of the Story?

Well, the moral of the story is that you should check your sites often, if not daily. You should make it a habit to check your Google Webmasters account to make sure there is no problem. If there is a problem, identify it and hurry to fix it. Google Webmaster Tools is a great help for this and if you haven’t created an account yet, I suggest you do now.

This little adventure cost me about a 40% decrease in traffic on my iPhone blog. Yes, it hurts! I haven’t measured my other sites loss as they are not as important as my iPhone blog. My money-making sites rely on PPC to drive traffic so it wasn’t a big issue for those sites.

I’m still pissed at DreamHost but you get what you pay for… I considered moving my sites to a new hosting company but it would be a nightmare. I have over 20 sites, most of them being database driven. My technical skills are just not sharp enough to do all this by myself. I guess I’ll stick with DreamHost until something really major happens.

Everything was doing pretty well. I was averaging 1,600 visits per day (and growing fast!) on my iPhone blog. 90% of my traffic came for one reason: finding out where to get all these cracked iPhone apps. That was until 1.50pm today…

I checked by email inbox and saw an email from DreamHost, my hosting company. The subject line was: “URGENT: DMCA Takedown Notification”. That didn’t smell good…

Here is the body of the email I received:

Hello Sebastien,We have received a formal DMCA (Digital Millennium Copyright Act) notice regarding allegedly infringing content hosted on your site. The specific content in question is as follows:

http://www.iphonedownloadblog.com/the-ultimate-list-of-cracked-games-for-iphone/

The party making the complaint (Ian Ramage, O’Melveny & Myers LLP, Phone: 415.984.8783 ,e-mail: iramage@omm.com), claims under penalty of perjury to be or represent the copyright owner of this content. Pursuant to 17 U.S.C. § 512(c), we have removed access to the content in question.

http://www.loc.gov/copyright/title17/92chap5.html#512

If you believe that these works belong to you and that the copyright ownership claims of this party are false, you may file a DMCA counter-notification in the form described by the DMCA, asking that the content in question be reinstated. Unless we receive notice from the complaining party that a lawsuit has been filed to restrain you from posting the content, we will reinstate the content in question within 10-14 days after receiving your counter-notification (which will also be forwarded on to the party making the complaint).

In the meantime, we ask that you do not replace the content in question, or in any other way distribute it in conjunction with our services. Please also be advised that copyright violation is strictly against our Terms and Conditions, and such offenses risk resulting in immediate disablement of your account should you not cooperate (not to mention the legal risk to you if they are true).

http://www.dreamhost.com/tos.html

We also ask that if you are indeed infringing upon the copyright associated with these works that you delete them from your account immediately, and let us know once this has been done.

We also ask that you delete any other infringing works not listed in this takedown notification, if they exist. If you have any questions, please don’t hesitate to let us know.

Glen,

I was wondering who this “Ian Ramage, O’Melveny & Myers LLP” law firm was. It didn’t take me too long to find out it actually is Apple’s law firm!

At first, I emailed Ian Ramage, the Attorney that filed the DMCA notice to tell him I was not doing anything illegal, that I was just pointing to cracked applications, but I didn’t crack them or uploaded them to servers.

In the meanwhile, I did some research. I read part of the US copyright law and didn’t find anything relevant. All this legal jargon is Chinese to me.

Then I stumbled upon an article that states that linking to infringing content is probably illegal in the US.

Here is what this website says:

Summary: There have been 3 court cases in the US where sites were forced to take down links leading to infringing material on external sites. The sites were found guilty of ‘contributory infringement’ for simply linking.

Although unclear the law leans in favor of content owner so if you linked to copyrighted material and don’t take it down you could very well lose a legal battle and face huge legal fees. Even blogs are at risk!

Lock and ChainA large number of cases never reach the courts where sites are served with takedown notices and dare not risk ignoring them as they may otherwise face liability.

Are copyright laws out of control and infringing our own right to share information and discuss what is available on the web?

Is Linking to Infringing Content Illegal in the US? Like it or not the short answer is yes. I don’t agree with it or like it but its true. The law currently leans towards the concept that linking to infringing content is illegal.

At first, I was pissed. After thinking about it, I am really pissed! It is not like I am linking to kiddie porn or something.

DreamHost did an outstanding job at helping me out with this issue. Glen, the CSR at DreamHost, was on top of it, replying to my emails within minutes to advice me on what to do. The $140/year I pay them is very well spent. Although they have downtime every once in a while, their service is just first-class.

Oh well, lesson learned.