Vertical Search Engines or Deep web search capabilities

The article "Vertical Search Engines Target New Niches " discusses the recent growing appearance of vertically focused – In a nutshell, search engines serve a specific target audience with specific information needs and they focus their , and content delivery efforts around a specific subject. Still, vertical search engines hide unique posture that can prove as a solid in their future with general purpose search engines.

A part of the article : "The generalized search engines like Yahoo and Google don't index the individual job pages," Paul Forster, chief executive of Indeed.com, said. "If you search for terms like 'finance job' they will bring up job sites but won't bring up the actual jobs themselves that match your criteria. We index our content in a way that enables people to bring up relevant matching job listings." grabbed my attention and cleared to me an underlying difference between vertically focused search engines and general purpose search engines (SE) and that is the usage of .

Deep web crawling is a designed to let search engines index and crawl the dynamic part of websites, the dynamic content that is usually generated on the fly from a back-end database based on users' web-queries (For example a search query on Gap's website for a specific shirt would yield local search results generated from the company's catalog database that would not appear on regular search engine results – At least not in the same level of indexing and presentation provided on the company's website).

A general purpose SE that wishes to integrate deep web indexing can encounter two new problems – dynamic content page relevancy and dynamic content link validity: The two problems occur due to the fact that dynamic content can not be "normally" referenced via common web links and it can be linked and accessed only via web queries. The first problem search engines encounter is the problem – Thoeretically a catalog item can be a very relevant information piece but still it won't have high ranking because there are no other pages on the web that can point to it. The second problem is the different results that will come up every time a dynamic content query is being submitted (Although the dynamic content's search results may seem as static web pages they are fully generated by the company's back-end database and applications), a fact that makes the SE life harder while trying to maintain a relevant index of web pages.

Still, it is tempting for search engines to index this un-touched bulk of information for the simple reason of increasing the SE's advertising inventory. In simple words it is difficult to integrate dynamic content of pages to a regular search engine due to their non static nature.

A vertically focused search engine presents a new type of search engines that use deep web searching technology to index RELEVANT dynamic content. Vertical search engines can do that successfully as opposed to general purpose search engines thanks to two main reasons: The audience of the vertical SE searches with inherently relevant queries and that is sufficient enough for generating relevant results based on the content of the dynamic pages using a regular term matching algorithm. The second reason is the low and controllable number of dynamic content sources per vertical that can be manually integrated and indexed by the SE (A travel site can index all the airlines' dynamic content and the viewers who query the SE with traveling relevant queries will receive effective search results).

From a different angel it is an expansion of the web/public information being indexed and that means a for search engines while currently only vertical search engines can enjoy it thanks to their focus.

UPDATE: See also "New search engines narrowing their focus Vertical sites cater to specific interests like shopping, job hunting, doing research"

Site Footer