3 Essential SEO Lessons Hauled In from the ‘Deep Web’

(Disclaimer: These are all 100% white-hat ideas.)


If Google succeeds at becoming the ‘third half of your brain’, then it’s probably going to be pretty empty. You see, despite our utter dependence on the almighty search engines, Google and the rest of the gang merely scratch the surface of the Web — literally. Right now, search engine crawlers are only able to access and index less than 10% of the world’s half-a-trillion public web pages. The remaining 90% belong to a place called the Deep Web’.


As it turns out, the way pages and information are organized and made available on the Deep Web has a lot to teach us about optimizing our sites here on the surface. So, let’s take a deep dive and see what we can learn.


The Deep Web is the part of the internet hidden from conventional search engines and consequently inaccessible to most of us. Why? It’s because pages on the Deep Web are usually dynamically-generated or are protected by subscription-based firewalls and context filters. To access the hidden Web, you need to make use of some special browsing and search tools.


This is why, before the well-publicized raid on Silk Road hit the newswires, most people had only a vague idea of the invisible Web’s existence. Unfortunately, incidents like that have disproportionately given the hidden Web a nasty reputation. In reality, the Deep Web isn’t all bad, and it actually surpasses the visible Web as a knowledge resource.


In terms of content quantity and quality, the Deep Web is many orders of magnitude better than the surface Web. It’s estimated that the hidden Web contains about 400 to 500 times more publicly-available information than on the visible Web. The total quality content you’ll find on the invisible Web is 1,000 to 2,000 times greater than on the Web we’re familiar with.


So, what does any of that have to do with your SEO approach? Well, to begin with, monthly site traffic is 1.5 times higher on deep websites than their surface counterparts, and pages on the invisible web are more highly linked to despite being practically unheard-of. These are due to a lot of factors, but it all boils down to a few powerful principles that you can apply:


1. People aren’t searching; they’re completing a task.  This is an important lesson for both search engines and SEO practitioners to live by. Whatever they’re doing, people only search the web for things that can get them from step X to step Y. It’s this complete-a-task approach that makes the invisible Web vastly superior to the visible net when it comes to generating quality traffic. Forget keywords; remove clutter; stop trying to outsmart the next Google update. Focus instead on letting your site visitors finish a task.


2. The Long-tail is your shortest way home.  The tiny piece of real estate on SERP rankings is what all of us want to call home. But most of the prime locations have already been taken. So, we focus instead on other areas where we’d have better chances at landing the best spots. That’s why we optimize for long-tail keywords. It also happens that the Deep Web thrives on long-tail search queries. Just take a look at a typical Deep Web search engine — from the search forms all the way to the results pages, everything’s really meant to capture the long-tail.


3. Narrow is the new wide.  Compared to conventional websites, sites on the Deep Web tend to focus more on a narrower band of the audience spectrum, prioritizing depth over breadth. If you’ve ever tried exploring the invisible Web, you’ll find entire search tools dedicated solely to a specific niche. A whitepaper on harnessing the Deep Web’s value  estimates that more than half of content on the invisible Web resides in ‘topic-specific’ databases, making it ‘highly relevant’ to every information need, market, and domain. This really shows you a lot about the power of targeting and positioning when it comes to receiving search traffic.


Now, it’s time for us to go back to the surface. Hopefully, with what you’ve hauled in from the deepest parts of the Web, your SEO efforts will push your site further and further to the top. Remember: find your audience by narrowing it down; get found by focusing on long-tail queries; and, once found, help your visitors get something done.