Deep Web

[Irina Shamaeva]

The Deep Web is "simply" all the pages that are not indexed by search engines.
The search engines like Google get from one page to another by following links. Any page that is not being linked to would belong to the Deep Web, unless someone submits it to a search engine. The phrase "deep web" sounds mysterious but in fact we all use it to an extent. If you are on a password protected site like Monster or Dice, you are using the deep web. If you run a database query somewhere on the web, and get a dynamically generated page, you are on the deep web.

Here are some categories of ways to access the Deep web. There are some sites that collect "favorite" links submitted by people. There are sites that provide the front end to all sorts of databases. Also, there are many human-built directories that are part of the deep web. Google and Yahoo have general purposes directories. There are specialized directories as well.

The deep web sounds exciting since its size is way, way bigger than the total size of indexed pages.
Have you used the Deep Web in your sourcing? Would you have some good sites / hints to share?

[Jan Kirchner]

Being a technical recruiter, I built myself a database of password-protected forums on technical subjects. Quite often these sites offer a "job wanted" corner where you can look for leads and a job board where you can post your openings. Since quite often only the parts of these forums are password-protected and therefore invisible, and rest is indexed by google| yahoo | live, you can locate them using Booleans and advanced operators. How to find the really invisible ones? Rely on your serendipity (unless someone else knows a good trick to locate these too?)

[Irina Shamaeva]

A good article can be found here: Exploring a ‘Deep Web’ That Google Can’t Grasp

[Martin Lee]

http://pipl.com/ is a great "deep web" search when knowing someone's name. It goes into most social networking sites and a few others, saves loads of time.

[gary cozin]

For corporations that have AIRS SourcePoint, it searches the deep web automatically along with other sites you have licensed. http://www.airsdirectory.com/mc/products_sourcepoint.guid

Not sure of costs, however would be for large enterprises I would assume.

Used it at my last company.

[Karen Antrim]

Can you use any deep web techniques to get a list a members from a specific association?
(one where you have to log on to be a member).

[Irina Shamaeva]

You can get lists if they are not protected. Trying the site: command on Google helps. Sometimes there are bugs that let you find a "hole" and get in.

Of course, you can't get information if it is well protected! There's a big field of computer security and most sites protect their data so that only authorized people can get it.

Karen Antrim said:

Can you use any deep web techniques to get a list a members from a specific association?
(one where you have to log on to be a member).

[gary cozin]

This was the string I submitted for Karen on one of the LI groups where I saw her post if anyone wants to modify or refine it or try it:

site:www.michcpa.org (inurl:list | inurl:~members | inurl:directory | intitle:list | intitle:~members | intitle:~directory | inurl:staff | inurl:association | inurl:board | inurl:committee | intitle:association | intitle:board | intitle:committee | intitle:staff)

It did bring back this that has member names:

http://www.michcpa.org/Content/18163.aspx

[Sanjeev Ammanna]

Hi Irina,

Can Bookmarklets help us in any way to explore the Deep Web?

-Sanjeev

[Amitai Givertz]

Gary, nice string. I modified it slightly to bring bag just spreadsheets, looking for lists in answer to Karen's question.

site:www.nsbe.org (inurl:list | inurl:~members | inurl:directory | intitle:list | intitle:~members | intitle:~directory | inurl:staff | inurl:association | inurl:board | inurl:committee | intitle:association | intitle:board | intitle:committee | intitle:staff) ext:xls

gary cozin said:

This was the string I submitted for Karen on one of the LI groups where I saw her post if anyone wants to modify or refine it or try it:

site:www.michcpa.org (inurl:list | inurl:~members | inurl:directory | intitle:list | intitle:~members | intitle:~directory | inurl:staff | inurl:association | inurl:board | inurl:committee | intitle:association | intitle:board | intitle:committee | intitle:staff)

It did bring back this that has member names:

http://www.michcpa.org/Content/18163.aspx

[gary cozin]

Thx Amitai. Also try filetype:xls at the end-

Amitai Givertz said:

Gary, nice string. I modified it slightly to bring bag just spreadsheets, looking for lists in answer to Karen's question.

site:www.nsbe.org (inurl:list | inurl:~members | inurl:directory | intitle:list | intitle:~members | intitle:~directory | inurl:staff | inurl:association | inurl:board | inurl:committee | intitle:association | intitle:board | intitle:committee | intitle:staff) ext:xls

gary cozin said:

This was the string I submitted for Karen on one of the LI groups where I saw her post if anyone wants to modify or refine it or try it:

site:www.michcpa.org (inurl:list | inurl:~members | inurl:directory | intitle:list | intitle:~members | intitle:~directory | inurl:staff | inurl:association | inurl:board | inurl:committee | intitle:association | intitle:board | intitle:committee | intitle:staff)

It did bring back this that has member names:

http://www.michcpa.org/Content/18163.aspx