by Mohamamdreza Khelghati, Maurice van Keulen, and Djoerd Hiemstra
To make deep web data accessible, harvesters have a crucial role. Targeting different domains and websites requires the need of a general-purpose harvester which can be applied to different settings and situations. To develop such a harvester, a large number of issues should be addressed. To have all influential elements in one big picture, a new concept, called harvestability factor (HF), is introduced in this paper. The HF is defined as an attribute of a website (HFW) or a harvester (HFH) representing the extent to which the website can be harvested or the harvester can harvest. The comprising elements of these factors are different websites' or harvesters' features. These elements are gathered from literature or introduced through the authors' experiments. In addition to enabling designers of evaluating where they products stand from the harvesting perspective, the HF can act as a framework for designing harvesters. Designers can define the list of features and prioritize their implementations. To validate the effectiveness of HF in practice, it is shown how the HFs' elements can be applied in categorizing deep websites and how this is useful in designing a harvester. To validate the HFH as an evaluation metric, it is shown how it can be calculated for the harvester implemented by the authors. The results show that the developed harvester works pretty well for the targeted test set by a score of 14.8 of 15.
To be presented at Riva del Garda, Trentino, Italy at the Workshop on Surfacing the Deep and the Social Web (SDSW 2014), a workshop co-located with The 13th International Semantic Web Conference