How Did I Find Out There Was A Problem?

I check my Google rankings for the term “iPhone blog” on a daily basis. iPhone Download Blog (my iPhone blog) usually ranks #4. Last Monday though, it was nowhere to be found! Doing a search for “site:www.iPhonedownloadblog.com” showed me I had 27,000 pages indexed but again, the homepage was nowhere to be found. Hmm, that doesn’t smell good…

I then went and checked my Google Webmaster Tools account. It usually helps me figure out quickly if there is anything wrong with any of my sites. Usually, everything is fine. From time to time, I get a little error on one or two of my sites, which in most cases can be fixed simply by generating a new sitemap and resubmitting it to Google.

Last Monday, I completely freaked out when I realized that all my sites were showing errors! All of them! I thought it was very strange but I soon realized that my server had been down for a few minutes the day before and I figured that Google must have tried to crawl my sites at this time and was then enable to do it. I simply resubmitted all my sitemaps thinking it would do the trick.

What the Hell Happened?

As a measure of precaution, I checked Google Webmasters Tools again a few minutes later and saw that Google, once again, was giving me errors. The issue was obviously bigger than I thought. I decided to look into the problem more deeply and realized that Google was giving me a 403 error, which means:

The server is refusing the request. If you see that Googlebot received this status code when trying to crawl valid pages of your site (you can see this on the Web crawl page under Diagnostics in Google Webmaster Tools), it’s possible that your server or host is blocking Googlebot’s access.

It was clear now. DreamHost, my hosting company, had restricted Googlebots from crawling my server. I had had the same problem with DreamHost before so it was just half a surprise…

How the Hell Do You Fix This?

The first thing I did was sending a support request to DreamHost. I tried to be kind in my email but I was really furious. I think it’s very unprofessional of them to block access to my server to anyone without even notifying me. A few hours later, I didn’t have an answer yet… DreamHost is usually fast at replying, but I guess they were not that day…

So I sent them a second support request email, a little more spicy than the first one. A couple hours later I received this email:

Hello Sebastien,

I sincerely apologize for this! This was handled incorrectly, and NOT according to our own policy. It looks like a few days ago you wrote in regarding server performance. It appears part of that was actually googlebot slamming your sites. This was blocked to preserve server stability (which is normal troubleshooting), what didn’t happen, and SHOULD have, was that you were not notified and told how to deal with this. I’ve removed the block, but please see the following article:

http://wiki.dreamhost.com/Bots_spiders_and_crawlers

There is a way to slow down googlebot without killing your search results OR the server. This is the preferred approach. We obviously don’t want to hurt your sites.

Again, I apologize for this, and if it helps any, it looks like just an IP range was blocked, and whoever was troubleshooting the server wasn’t really *trying* to block google from your site…just an IP that was causing problems for everyone.

Please let me know if you have any other questions.

Thanks!
Jeff H

Alright, fair enough… What can I do anyway? So this part of the problem was handled but I needed to tell Google everything was back up so it can start crawling my sites asap.

I resubmitted all my sitemaps again. I also wrote a few new posts on my iPhone blog in order to “prove” Google that the site was still alive and doing well. Finally, I linked to my main sites from various other websites in the hope it would help speed up the crawling process.

Googlebots came back and crawled the sites. The next day (on Tuesday), iPhone Download Blog was back in Google’s search results but it was ranking at about #35 for my keyword. I realized that even though it was back in the index, Google didn’t have a cached image of it yet. I assumed that it would come back and crawl the site again and then generate a cached image.

On Wednesday, still no cache! It was ranking slightly better though, showing up on the second page. Still far from what I was looking for, but at least it was progressing a bit.

On Wednesday night, the homepage completely disappeared from the index again. While some may worry about this, I am now used to it and I know what it means. It means that Google has crawled your site again and it is generating a cached image of it. This usually happens when you create a new page. Google will crawl it, rank it, then it will disappear from the index for a few hours, and finally it will come back for good. It’s like Google is hurrying to crawl it, but then takes its time to analyze it and make sure it’s not crap.

I went to bed that  night confident that my site would be back on the next morning when I get up. Sure enough, it is ranking again, as good as ever, this morning. Yeahhh!!! All my other sites are back too.

What’s the Moral of the Story?

Well, the moral of the story is that you should check your sites often, if not daily. You should make it a habit to check your Google Webmasters account to make sure there is no problem. If there is a problem, identify it and hurry to fix it. Google Webmaster Tools is a great help for this and if you haven’t created an account yet, I suggest you do now.

This little adventure cost me about a 40% decrease in traffic on my iPhone blog. Yes, it hurts! I haven’t measured my other sites loss as they are not as important as my iPhone blog. My money-making sites rely on PPC to drive traffic so it wasn’t a big issue for those sites.

I’m still pissed at DreamHost but you get what you pay for… I considered moving my sites to a new hosting company but it would be a nightmare. I have over 20 sites, most of them being database driven. My technical skills are just not sharp enough to do all this by myself. I guess I’ll stick with DreamHost until something really major happens.

First, let’s see what’s good about Craigslist. To me Craigslist has 4 major benefits: it’s free; it’s a well known website; it is a trusted site; and most importantly, it is one of the most visited websites in the US.

Many internet marketers use Craigslist to make a quick buck. If you’ve been on Craigslist before, you’ve probably seen all these ads such as “Make $12,000 in 3 days”, or “Earn 2K residual for life”. These are ads made up to advertise affiliate programs. From an affiliate marketer standpoint, these ads have very little value because they will most likely be removed in a matter of minutes.

The real deal is to create ads that will stick (which means not be removed) to generate traffic to your site, not from Craigslist visitors, but from Google visitors.

Let me explain to you something you probably know… Google loves Craigslist and it crawls the site many many times a day. When you post an ad on Craigslist, Google will find it in a matter of minutes, index it, and give it much more importance than it actually deserves, putting it on the first page of Google for a given keyword. If one of your keywords shows up on the first page of the SERP, you’re gold! You will get good qualified traffic from this.

Let’s cut the crap. Here is what I recommend to do in order to get your Craigslist ad on the first page of Google:

1. Your ad title should be your main keyword/keyphrase
2. Choose your ad title wisely. If your website sells warts removal cures, don’t just name your ad “warts removal”. There is too much competition for this keyword, so be more specific: “Natural Warts Removal” would be good.
3. Write a very short post description. Just a few words are enough. You’re not trying to sell the people right here; you’re trying to get them to come to your site! A good description for our warts removal site would be: “Our natural warts removal cure will help you get rid of warts naturally without drugs or cream”.
4. Make sure to include your main keyphrase in the description.
5. Include your URL and make it clickable. Not good: www.SebastienPage.com – Good: www.SebastienPage.com
6. That’s it!

Now here is what you shouldn’t do:

1. Do not repeat the same process over and over again with the same keywords. I don’t even recommend doing this over and over again with various titles as the Craigslist community will get back at you by flagging your posts. Just do it a couple times a week, switching ad titles and ad descriptions. Note that ads show up quickly on Google, but they don’t stay for more than 2-3 days.
2. Do not use broad titles!!! As I mentioned above, do not use competitive keyphrases but go after the long tail.

I use this trick on the 3 of my money making sites and I get great traffic for free. Since you’re targeting the long tail, you’re most likely not gonna get thousands of visits. But as we say in French, “10 grams of fine patisseries are better than 10 pounds of potatoes”.

Here is a screenie of my fake ad posted on Craigslist for “Natural Warts Removal” ranking #1 for this search! Note that I posted this ad on Craigslist on 1/16/09. This screenshot was taken on 1/17/09. So by the time you see this, the results will most likely have changed.

Click on the image for a larger view.

This post was inspired by a tweet from Lorelle, my favorite WordPress expert. Lorelle’s tweet was saying “Make WordPress’ search function suck Less” with a link to this post. I had WordPress search issues in the past and found a few solutions and I thought I would learn a little bit more from this post, but I didn’t.

Basically this post was about manually tweaking your search.php file, which improves the WordPress search feature but it still doesn’t make it a good search feature. The main problem with the WordPress search feature, besides the irrelevancy, is that if you have a large blog of a few hundreds posts, then there is not much you can do to speed up the search.

I don’t pretend to be an expert about WordPress (I’m actually far from that!) but I do think both solutions I will talk about in this post are much better at delivering very good search results for your WordPress blog. By “good search results”, I mean results that are delivered quickly, that are relevant to your search query, and sorted by relevance.

wpSearch Plugin

The wpSearch plugin is like having your own custom search engine on your blog. wpSearch is based on “Lucene“, a full-featured open source text search engine. wpSearch is fast, and relevant, which is pretty much everything I need from a search engine on my blog.

To set up wpSearch on your blog, simply download the plugin from here, then upload it, set permission to 777 (very important), and activate the plugin from your admin panel. After activating it, you will have to build the search index database, which may take a while (about 5 posts/second) if you have many posts on your blog. That’s all there is to it. Now check your search feature and you will see a huge difference in the results you will get. The relevancy will be so much better that like me, you’ll want to blog about it ;-)

Adsense For Search

Adsense for Search (AFS) is now my favorite search feature for my blogs for 2 reasons: for one it is the best search engine you can get; and two, you can actually make money from it. Even though I hate Google as a company, I can’t deny that their search algorithm delivers the best you can get from a seach engine thus far.

Now the main concern about Adsense For Search is that you have to rely on Google to crawl your website to index your new posts. Some people won’t use Adsense For Search for this one reason. Yes, you have to wait for Google to crawl your site and index your pages but how much does that really matter in your case? I mean, is it that important to you that your post must be foundable immediately after being published?

Besides, one benefit of Adsense For Search that I have been able to verify on all my blogs is that when AFS is implemented on your site, Google will come visit you quickly and more often. I haven’t done scientific tests on this, but I was able to notice it on many occasions. Most of the time, when I publish something on my blogs, the post is picked up by Google within hours, sometimes within minutes.

To set up, AFS, you will need a Google Adsense account.

A while back, when I started this blog, I installed the DoFollow plugin which removes the nofollow attribute of the URL of your commenters and any URL included in the comment itself. The reason I installed this plugin was to quickly get many backlinks to this blog.

Indeed, when your blog is “dofollow”, you get featured on many sites or blogs that give list of dofollow blogs. Doing this got me listed on a few hundred blogs in no time creating a large amount of backlinks, which is probably the reason why my blog has a PR4.

The downside of this as you can imagine is that you get quite a bit of spam comment. Akismet handles most of it but from time to time I had to manually delete comments. But most people commenting on my blog just to get a link back are actually very well behaved. Most didn’t abuse it and left genuinely interesting comments. In this case, I didn’t delete the comment.

Today I decided it is time to go back to the nofollow rule. My blog is clearly not intended to make money or anything like that so I really don’t care about traffic and Google PR, which in my opinion doesn’t even mean anything anyways. Besides, I will probably remain listed as a dofollow blog on many sites so I don’t think I’m gonna suffer any link juice dicrease.

For those of you who don’t know yet, I operate an iPhone blog that’s getting quite some attention lately. We average between 8,000 and 10,000 visits per day and I had to hire 2 co-bloggers (Alicia and Cody) to help me satisfy our readers’ hunger for iPhone news, tips, hacks and cracks.

I got in touch with the AllTop.com team to help them put together the iPhone page. I sent them a list of the best iPhone websites and resources out there. Obviously, they also included the iPhone Download Blog on their page.

It’s nice to be able to participate in the iPhone community and to make it better, little by little. If you go to the AllTop iPhone page, you will see that my blog is right at the top, in second position, which gives me PR5 links to 5 of my most recent posts as well as a link to the homepage. That’s some solid SEO juice!

Next step is to get my company’s franchising blog on AllTop but I’m going to wait a few weeks since we just launched the blog a few days ago and it doesn’t have much content.

I have 3 different blogs (1, 2, 3) that are powered by WordPress, which is without a doubt the best blogging platform out there. The good thing with WordPress is that it is open source, which means anyone can create plugins to improve WordPress’ backend to fit one’s needs.

There are 12 plugins that I find extremely useful and I would like to share them with you.

Admin

Akismet

Akismet used to be a plugin you had to install once WordPress was set up. It now automatically comes with WordPress. Unless you have an old version of WP (2.1 and older), this plugin is already installed.

Akismet is a smart spam filter. Every time someone comments on one of your posts, Akismet checks its database to see if the commentor might be spam. How does it know it’s spam? Simply because WP users make it smarter by telling Akismet what is spam and what is not. Let’s say I receive a comment from someone and I realize it is spam. I will click “spam” and Akismet will put it in its database of spam. Now next time this spammer comments on someone else’s WP blog, Akismet will check its database and block the comment automatically.

One thing I am not sure of is how many times a commentor has to be checked as “spam” before Akismet adds it to its database. If you know, please let me know.

Contact Form ][

One thing WP lacks is a contact form, should someone want to be in touch with you. This plugin is a simple way to add a contact form to any page or post by simply dropping [contact-form] anywhere in the post or page.

The form is highly customizable and usually integrates perfectly with your CSS. You can choose what email address you want to send the form to, customize the messages, and much more.

Related Posts

This plugin will display related posts at the bottom of each post to give more exposure to you other posts. For example, if someone is reading a post on your blog about the new iPhone 3G, you can set the plugin to show any number of related posts. The reader may then see another posts about the subject and go read it.

You can choose how many posts you want to display, add basic HTML code before and after the post title.

Different Posts Per Page

WP lets you choose how many posts you want to display per page. The problem is you can only choose a number that is reflected on all pages. In my case, I want certain pages to show 100 posts (ie. my archives pages that are just links to the posts, without any excerpt), and I want other to show only 10. DiffPostsPerPage will do just that in a very easy way.

Subscribe To Comments

This plugin adds a checkbox to your comment form so that reader can opt in receiving follow up comments. Easy set-up and provides additional incentives to read your blog to your visitors.

SEO

All In One SEO Pack

This is the one plugin you want to use to improve your blog’s SEO. WP is pretty flexible but doesn’t allow you to give unique meta descriptions for individual categories, pages, and posts. All In One SEO does all this and more.

Among others, it will let you choose your pages, categories, posts’ title, description, and keywords. It also autogenerates meta descriptions by using the first few words of your post or page. Lastly, you can have “noindex” and the like generated for archives, categories, and tag pages if you seem to have trouble with duplicate content.

XML-Sitemap

If you’re a little into SEO, you know that providing Google with a good and updated XML sitemap is very important. Problem is you can’t manually keep track of all these posts you’re creating and update your sitemap regularly.

First, XML-Sitemap will build the sitemap in a normal XML file and in a gzipped file should you want to. It will upload the sitemap to your server. If set properly, the plugin will also ping Yahoo, Google, and MSN automatically to let them know your sitemap has been updated.

XML-Sitemap lets you choose what pages you want to include in the sitemap, set your changes frequencies and pages priorities.

It is a powerful tool that will considerably help you. Thanks to this plugin, Google crawls my iPhone site several times a day and indexes pages sometimes within minutes of being posted!

DoFollow

By default, when someone leaves a comment on your blog and gives his URL, WP adds a “no-follow” tag to this URL. This means that when GoogleBot comes crawl your page, it’s going to see this link but it will not follow it, thus not giving SEO juice to the website.

Why am I using a “do-follow” tag? For one big reason… By being a “do-follow blog”, my blog is added on a bunch of websites and blogs. This brings many back links to my site. The downside is that every once in a while I have a spam comment. In most cases, those are caught by Akismet but I sometimes have to delete them manually. All in all, this plugin is very beneficial to my blog.

Mobile

iWPhone

This plugin creates a new theme for your blog that is specifically designed to fit the iPhone screen. Now every time someone accesses your blog from an iPhone, the plugin will switch your current theme to this iPhone theme, making it easy for your reader to navigate your blog without having to zoom in and out all the time.

WPhone

WP recently released its own native iPhone application but I find it very limited in what you can do with it. WPhone will create an iPhone-friendly version of your admin panel and offer you all the features of WP. It is a great tool to update, edit or delete posts on the go.

Social

Twitter Updater

The name says it all…. Twitter Updater will automatically update your Twitter status every time a post is created, edited, or published. You are able to customize what you want to send and how you want to send it. I set mine to only update Twitter when I add a new post and to Tweet the title of the post as well as a link to it. If too longs, links will automatically be shortened by tinyURL.

flickRSS

This plugin is great to display your Flickr images on your site. You can set it to display your full photostream, a group, a set, or tags.

You can display up to 20 images. Just add <?php get_flickrRSS(); ?>anywhere in your template to instantly display your Flick feed.

1. Google Keyword Tool
2. WordTracker Free Edition
3. Google Suggest
4. SEO Book Keyword Suggestion Tool
5. Good Keywords
6. Keyword Discovery

1. Do your keyword research
2. Stay in touch with the SEO world, read blogs, websites, etc…
3. Get the most of your code; use Cascading Style Sheets (CSS)
4. Make navigation easy so bots can easily crawl your site
5. Get links from trusted and relevant resources
6. Build a sitemap and submit it to major search engines
7. Make sure the backend of your site is SEO friendly (ie. your host, redirects, etc..)
8. Use web analytics to track your progress
9. Submit your content to search engines
10. Provide unique and relevant content

SEOmoz Whiteboard Friday-Blogging for Higher Rankings from Scott Willoughby on Vimeo.

• The fewer the parameters in your dynamic URL, the better. One or two parameters is much better than seven or eight. Avoid superfluous/nonessential parameters like tracking codes.
• A static looking URL (containing no ampersands, equals signs, or question marks) is more search optimal than a dynamic one.
• Having keywords in the URL is more optimal than no keywords.
• A keyword in the filename portion of the URL is more beneficial than in a directory/subdirectory name.
• Hyphens are the preferred word separator, although underscores are gaining acceptance over times past . So if you have multiple-word keyword phrases in your URLs, I’d recommend using dashes to separate them.
• Stuffing too many keywords in the URL looks spammy. Three, four, or five words in a URL looks perfectly normal. A little longer and it starts to look worse to Google, according to Matt Cutts.
• The domain name is not a good place for multiple hyphens, as it can make your URL look spammy. Although that said, sometimes a domain name should have a hyphen, as the domain faux pas ‘arsecommerce.com’ demonstrates (you may not get this joke if you don’t recognize Queen’s English!).

Source: Search Engine Land