Managing metadata for distributed information servers.

Abstract

The amount of available information on the World Wide Web (WWW) and the access frequencies have grown rapidly with network size, but methods to locate and access desired data remain rudimentary. As a result, the information consumer must sift inefficiently through large amounts of data. Users are increasingly interested in accessing structured data sets such as phone books which contain names, addresses and phone numbers, or product catalogs which typically contain product names, features and prices. This pattern resembles distributed database access more than traditional file or document system access. The problem of finding data of interest, from a large set of distributed sites, has been called <italic>information discovery</italic>. This study explores information discovery issues associated with the rapid growth of catalog information available at Internet Web sites. We present the design and implementation of the Domain Specific Metadata Service (DSMS), a general distributed index system that improves metadata discovery on large networks. The primary feature of this architecture is its novel use of hierarchical metadata to describe Web site contents. We have found that using hierarchical metadata to organize and distribute indices can facilitate the search of metadata. We present experimental results demonstrating search response times and scalability improvements. Finally, we present the design and testing of a cache consistency manager to mitigate the effects of communication overhead in our distributed system. Simulation results show that hierarchical grouping of replica peers can provide efficient update dissemination as well as fault tolerance.