Wednesday, July 22, 2015

The Deep Web you don't know about

From CNN:

What we commonly call the Web is really just the surface. Beneath that is a vast, mostly uncharted ocean called the Deep Web. By its very nature, the size of the Deep Web is difficult to calculate. But top university researchers say the Web you know -- Facebook (FB), Wikipedia, news -- makes up less than 1% of the entire World Wide Web.

When you surf the Web, you really are just floating at the surface. Dive below and there are tens of trillions of pages -- an unfathomable number -- that most people have never seen. They include everything from boring statistics to human body parts for sale (illegally).

Though the Deep Web is little understood, the concept is quite simple. Think about it in terms of search engines. To give you results, Google (GOOG), Yahoo (YHOO) and Microsoft's (MSFT) Bing constantly index pages. They do that by following the links between sites, crawling the Web's threads like a spider. But that only lets them gather static pages, like the one you're on right now.

What they don't capture are dynamic pages, like the ones that get generated when you ask an online database a question. Google and others also don't capture pages behind private networks or standalone pages that connect to nothing at all. These are all part of the Deep Web.

So, what's down there? It depends on where you look.

SEE Deep Web Search – A How-To Site.

No comments: