ArJ.uNo

Scottish bazillionaire Brian Souter made headlines this week when he pointedly accused Google of censoring him because his site dropped in the SERPs. Said Sir Brian, “It’s not Google’s place to decide which sites we can see and those we can’t.”

These complaints happen every day, actually; stop by the Google Webmaster Forum and pick one of the dozens threads accusing Google of arbitrarily penalizing or deindexing a site. They surface daily. Most of these complaints don’t get this kind of exposure, though.

The common theme in the majority of the complaints seems to be that webmasters haven’t looked at their site objectively in an effort to determine what they might have done to cause the issue. Often, the problems are obvious and fixable.

Timing and Bite of the Evil Panda

Sir Brian fell back several pages in the SERPs on August 13. Souter claimed that his site had “mysteriously disappeared,” but he still had 46 pages indexed, so he hadn’t been removed by Google.

Is it any coincidence that Google officially launched Panda internationally on August 12, the day before Sir Souter’s ranking dropped? I think not.

Until yesterday, you could find an exact duplicate of Souter’s site at www.briansouter.com.14feb-youth.com/; it was removed sometime overnight.

Image Credit: Attach Media

Your Baby is Ugly

No one wants to hear it, I know, it’s cruel. But it is what it is. Souter’s site is clunky, old, and not user-friendly at all.

In the generation of me, what are you doing to attract me, your reader, to your site and make it easy for me to find what I’m looking for, Sir Brian?

You have all of this information about Stagecoach in different links within your Profile page, but I had no idea it was there until I actually visited the Profile page. A dropdown menu on the main navigation would give me some clue as to what your site contains and why I should want to stay and keep reading.

Few Backlinks & Social Presence Lacking

Your website isn’t a billboard; you can’t just stick it up and leave it for 10 years and hope for the best. Millions of others are making an effort to meet the needs of their audience and stay current.

The online landscape is constantly changing and webmasters have to make at least some semblance of an effort to keep up… even if they feel entitled because of who they are.

Rankings Dropped? What You Should Do First

If your ranking has suddenly dropped or you believe it might have been removed from Google’s index, do at least the following things before throwing on your tinfoil hat and assuming the Intrawebz are out to get you:

• Do a Google search for site:yourwebsite.com with your URL to see if your site is still indexed. If you get results, you haven’t been removed; your ranking has dropped for some reason. If you’ve been removed, you need to find out why and apply for re-inclusion at Webmaster Central once you’ve cleaned up your problem.
• Do a Google search for “yourwebsite.com” with your URL in quotations. You should be in the top 10 results. If your site appears back on page four or five, you have most likely been penalized by Google.
• Have you a bought or exchanged links?
• Have you hired or otherwise allowed someone else to make changes to your site, and do you understand what they did? If you were ranking really well for a time and suddenly dropped, you need to find out if they were using black hat tactics such as cloaking on your site.
• Look at your site objectively. Better yet, have someone else go over it for you. Are you giving Google the information it needs to crawl and rank your site for the terms you’re after? Is your site easy to navigate and user friendly? Be honest, the only person you’re hurting by wearing blinders is yourself.

Search Engine Manipulation?

Sir Brian Souter released a statement this morning asking webmasters to contact him if their sites have been blocked by Google, seemingly unaware that his own site, in fact, isn’t blocked at all. Is he ever in for a treat!

I suspect he’s about to get a flood of emails, 95 percent of which will be from webmasters whose sites were legitimately penalized or removed for things they could simply fix. It’s always easier to throw the problem in someone else’s lap than to admit that yes, your baby is, in fact, ugly.

How to not get caught in spam filters

Reliably sending email without getting caught in spam filters is a full-time job, for someone. Surely not for an end-user, but for every end-user email, there is an administrator somewhere who has to deal with daily occurrences of some user message not getting through because it got stuck in a spam filter on the other end.

At the enterprise level, this could easily be several people’s full-time jobs. Spam filtering is constantly evolving. This is partly due to new spam filtering initiatives that require administrators to configure something new, such as SPF or DKIM. A few years ago, SPF didn’t exist. Now, anyone who sends lots of email virtually has to implement it. It’s also partly due to other administrators; sometimes you just have to get on the phone with the recipient’s admin to figure out what’s going wrong.

This guide is not for those enterprise admins. It’s for the hapless developers pressed into Postfix config duty for a small start-up, or for the first time admin just getting into outbound mail. What follows is a quick and dirty guide to making sure 99% of your email is delivered.

Make sure you’re not on a DNS blacklist (aka RBL: Reverse Blacklist)

By far the most frequently used type of spam filter is the DNS blacklist. There are hundreds of free services out there that keep records of IP addresses they think send a lot of spam. Virtually every spam filtering product on the market comes pre-configured to look at a few of these every time they get a new connection. It’s fast due to extremely low over-head (DNS scales, baby), and relatively accurate.

You will need to know what IP you’re sending from. You can check many blacklists at once via various different sites.

If you are on a blacklist, you might be wondering how to get off it, and how you got on in the first place. Unfortunately, there is no single answer. Each blacklist has its own criteria for who it lists, and has its own process for removal. Indeed, many lists don’t allow removal at all. It’s the wild-west out there. If you find yourself unable to be removed from a popular blacklist, you may have no choice but to buy another IP address. Just make sure it’s clean first!

Some people think blacklists are the devil. If you have ever found yourself at the mercy of a popular, but totally non-responsive blacklist, you might agree. But in general, the problem is that some administrators outright block email that matches a single blacklist. If you’re an inbound admin, don’t do that! You want to weigh many factors, and multiple blacklists, before you decide to reject a message. Regardless, they are a reality of the modern Internet you need to just deal with.

Make sure you’re not an open relay

If you want to STAY off blacklists, you at the very least need to make sure you’re not an open relay. Basically, you should not accept and definitely not send out any mail that’s not destined for a domain you actually own. Testing can be done via telnet, or via a web-based tool.

Reverse DNS (aka PTR records)

Another very common check is whether your IP address is named, or unnamed. The idea here is that dynamic IPs, such as those given to home users by their ISP, generally don’t need to have names associated with them. A lot of spam these days comes from zombied home machines.

This is a simple DNS fix. You just need to create an PTR record for that IP address. You can check if your PTR is setup correctly with the following command.

MX Records, postmaster, root & abuse

While the standards RFCs don’t require you to receive mail just because you’re sending mail, in reality many anti-spam systems are biased against message from a domain that does not also accept mail. You don’t have to send and receive from the same server(s), but if you’re sending mail from @example.com, it’s a good idea to make sure some real human somewhere is getting any messages sent to [email protected], [email protected] and [email protected].

Postmaster IS strictly required by the RFC. Root is a legacy version of postmaster. Abuse is a relatively new “standard” that many administrators would try first to resolve a spam issue.

Inbound email is a whole other subject. But the basic gist is that you need an MX record for example.com, and it needs to point to a server that can accept mail for example.com. If you don’t have an existing inbound server, or don’t want to run your own, many hosted alternatives exists.

You should explicitly test postmaster, root & abuse manually via your regular email client to make sure they actually work.

HELO, I’m your mail server

Mail servers communicate via a protocol called SMTP. It’s actually a plain-text protocol, which you can easily emulate via telnet. The very first line of a SMTP handshake is the “HELO” command, where the sending server identifies itself. A typical example would be “HELO example.com”, meaning, “Hi, I’m the mail server for example.com”.

Many spammers set this to a bogus value, or try to use the recipient’s host name or IP address, which is nonsensical. In any case, the correct thing to do is for you to set it to your domain.

How you set this will vary by mailserver. In Postfix, it’s the myhostname parameter in /etc/postfix/main.cf. Checking it is easy; just send a message through the server, and look at the headers on the remote end. Your hostname will show up on the first “Received” header line:

SPF/DKIM

SPF and DKIM are newer standards that are slowly gaining popularity. The basic idea is that your DNS records can encode a list of rules about what IP addresses are allowed to send mail for your domain. It’s a whitelist, versus a blacklist. Typically, you can ignore these unless you’re sending a large volume of mail.

Monitoring

That just about covers anti-anti-spam 101. As mentioned, this will likely be an ongoing effort, and you need to keep on top of how it’s going. Ideally, there would be an administrator who would be alerted if emails are bouncing due to spam filters. For postfix, I would recommend pflogsumm.

Well, I seen many people going carzy as to weather choose Google+ or Facebook, so I found this interesting image that distinguishes both of them.