IBM would like to see its WebFountain supercomputing project become the next big thing in Web search.
The Internet can be a treasure trove of business intelligence–but only if you can make sense of the data.
Along with competitors such as ClearForest, Fast Search and Transfer, and Mindfabric, Big Blue hopes to foster demand for new data-mining services that ferret out meaning and context, not just lists of more-or-less relevant links.
It’s a tall order, one that’s pushing the limits of supercomputing design and stretching expectations as to what raw processing power can accomplish when set to work on the world’s largest document library. Traditional search engines such as Google are already hard-pressed to match search terms to specific Web pages. Now WebFountain and other projects will take on a task that’s exponentially more complex.
“Search is trying to find the best page on a topic. WebFountain wants to find the trend,” said Dan Gruhl, chief architect of the project at IBM’s Almaden Research Center in South San Jose, Calif. Harnessing the Internet’s data to find meaning is a visionary ideal of Web search that has yet to be attained. As more companies manage their businesses on the Web, however, analysts predict they will be looking to extract value from its bits and bytes, and many software companies are now examining ways to bring that value to them.
IBM is hoping to cash in on the trend with the four-year-old WebFountain project, which is just now coming of age. It’s an ambitious research platform that relies on the Web’s structured and unstructured data, as well as on storage and computational capacity, and IBM’s computing expertise.
Whether WebFountain can deliver today, the problem it hopes to crack holds particular attractions for IBM. Big Blue has been pushing a new computing business model in which customers would rent processing power from a central provider rather than purchase their own hardware and software. WebFountain dovetails nicely with this utility computing model. IBM hopes to use the project to create a platform that would be used as a back end by other software developers interested in tapping data-mining capabilities.
In one of the first public applications of the technology, IBM on Tuesday teamed with software provider Semagix to offer an anti-money-laundering system for financial institutions, with Citibank as its first customer.
The two companies have quietly been working together for months to develop an application that helps banks flag suspects attempting to legitimize stolen funds. Those efforts are in accordance with the USA Patriot Act, signed into law two years ago to fight terrorism.
The WebFountain-Semagix system automates a process that has previously fallen onto the shoulders of compliance officers, who manually compare a person’s name against lists of known suspects.
“This is a classic IT solution,” WebFountain President Rob Carlson said. “It’s not replacing people, rather it organizes unstructured information from the Web to the point they can look at what’s important rather than sifting through a lot of data and manually trying to figure out who’s related to whom.”
In a sign of growing demand for money-laundering filters among banks, Fast Search and Transfer recently announced that financial institutions could build a similar application, and Cap Gemini is said to be a first customer, according to analysts.
A rapidly growing market
WebFountain traces its roots back to Stanford University and another groundbreaking research tool, Google. Its origins lie in a scholarly paper about text mining–authored jointly by researchers at IBM’s Almaden site and at Stanford–that discusses an idea known as hubs and authorities.
That theory suggests that the best way to find information on the Web is to look at the biggest and most popular sites and Web pages. Hubs, for example, are usually defined as Web portals and expert communities. Similarly, the concept of authorities rests on identifying the most important Web pages, including looking at the number and influence of other pages that link to them. The latter concept is mirrored in Google’s main algorithm, called PageRank.
IBM applied the same concepts in an early Web data-mining project called Clever, but shortcomings eventually led researchers to turn the theory of hubs and authorities on its head. In short, IBM found that it could excavate more interesting data from pages that the theory of hubs and authorities normally pushed to the bottom of the heap–unstructured pages like discussion boards, Web logs, newsgroups and other pages. With that insight, WebFountain was born.
“We’re looking at…the low-level grungy pages,” said Gruhl. Analysts said they expect to see increasing demand from corporations for services that mine so-called unstructured data on the Web. According to a study from researchers at the University of California at Berkeley, the static Web is an estimated 167 terabytes of data. In contrast, the deep Web is between 66,800 and 91,850 terabytes of data.
Providing services for unstructured-information management is an estimated $6.46 billion market this year and a $9.72 billion industry by 2006, according to research from IDC.
Any doubts about the scale of processing power required to tackle this task are quickly dispelled with a visit to WebFountain’s server farm, housed at IBM’s Almaden Research Center.
The company employs about 200 researchers in eight research labs around the world, including in India, New York and Beijing. But the heartbeat of the operation is here.
After clearing a gated security checkpoint, guests follow a long driveway to a low-slung, 1960s-era office building tucked away behind rolling foothills and parklands above Silicon Valley.
The steady whirr of fans signals the presence of something big down the hall.
A main cluster consists of 32 eight-server racks running dual 2.4GHz Intel Xeon processors, capable of writing 10GB of data per second to disk. Each rack has 5 terabytes of storage, for a total of 40 terabytes for the system.
The central cluster is supported by two adjacent 64 dual-processor clusters that handle auxiliary tasks. One bank crawls the Web–indexing about 250 million pages weekly–while the other handles queries.
The three clusters together currently run a total of 768 processors, and that number is growing fast.
The cluster and storage is migrating to blade servers this year, which will save space and provide a total of 896 processors for data mining and 256 for storage. In total, the system will add 1,152 processors, allowing it to collect and store as many as 8 billion Web pages within 24 hours.
Like Web search engines, WebFountain can be used to try to find a needle in a haystack, but unlike Web search, it’s designed to scope back and identify trends or answer unknowns like, “What is my corporate reputation?”
That goes well beyond the capabilities of Web search engines developed by companies such as Google, Inktomi and Fast Search and Transfer. These products typically scour the Web to find the documents that best match a given query, typically analyzing links to important Web pages or matching similar chunks of text. With these and other methods, search lets people browse, locate or relocate information, and get background information on a topic.
By contrast, IBM’s WebFountain wants to help find meaning in the glut of online data. It’s based on text mining, or what’s called natural language processing (NLP). While it indexes Web pages, it tags all the words on a page, examines their inherent structure, and analyzes their relationship to one another. The process is much like diagramming a sentence in fifth grade, but on a massive scale. Text mining extracts blocks of data, nouns-verb-nouns, and analyzes them to show causal relationships.
WebFountain promises to combine its intelligence with visualization tools to chart industry trends or identify a set of emerging rivals to a particular company. The platform could be used to analyze financial information over a five-year span to see if the economy is growing, for example. Or it could be used to look at job listings to pinpoint emerging trends in employment.
“The Web has become just a huge bulletin board, and if you can look at that over time and see how things have changed, it answers the question, ‘Tell me what’s going on?'” said Sue Feldman, analyst at market research firm IDC. “This looks for the predicable structure in text, and uses that just the way people do, to do some analysis, categorize information and to understand it.”
To be sure, some critics say WebFountain and other projects still have a long way to go in proving they can deliver on their ambitious promises.
“IBM is trying to unleash this cannon of 20 years of research–it’s a nice big gun, but it may be ill-suited to the task in some cases,” said Jim Pitkow, president of search company Moreover, which has a deal with IBM rival Microsoft. He argued that companies may not need to have 3 billion pages crawled in order to do an analysis of their corporate reputation or marketing effectiveness online, because many pages don’t address the topic.
“Automatically detecting sentiment is a tricky thing,” Pitkow said. IBM says the WebFountain service has already yielded some promising results in early test runs, pointing to 2002 market research done on behalf of oil conglomerate British Petroleum as one telling example.
BP already knew that gas prices and car washes are customers’ chief concerns while at the pump. But by unearthing news of a tiny Chicago-area gas station that created “cop-landing” areas for police officers, WebFountain called attention to another consumer worry: crime. Now BP is exploring plans to improve safety at its stations, giving away coffee, doughnuts and Internet connections to attract police officers.
Other WebFountain developments include an application expected to make its debut this summer from Factiva, an information retrieval company owned by Dow Jones and Reuters. Factiva licensed WebFountain in September and has been building software to sit on top of the platform and gauge corporate reputation.
In an era of corporate scandals and fierce competition, measuring public perception could become a key focus of many companies. Already, at least one company that has tested WebFountain has named a corporate reputation officer, according to Gruhl.
“The problem has always been the difficulty of doing systematic mining of such a large amount of data, and distinguishing the important from the trivial,” said Charles Frombrun, executive director of the Reputation Institute.
“If the venture works out,” Frombrun said, “there should be a great deal to learn from combining retrospective data from print sources with emerging data from Web analyses.”
Thanks to Monica ofAdvertorial.org for the tip.
