PFIRE - A Parallel Full Information Retrieval Engine

Description

Objective: The application of massive parallelism to information retrieval with large document collections
Approach: The vector space model of the sequential FIRE is implemented on the massive parallel platform.
Results: Almost optimal (linear) speed up achieved for document collections of appropriate size. For document collections of up to 2 GB, can be on-line searched.
Remarkable code flexibility: The same executable can be started on processor partitions with 1,2,4,8,..., and 512 processors and supports virtual tree topologies of any degree (2,3, etc.).

Publications/Technical Reports

Pavlos S. Efraimidis and Christos Glymidakis (Diplom work - or equivalently our Master's Thesis -  Dept. Computer Engineering and Informatics, School of Engineering, University of Patras, Hellas, 1994)

Pavlos S. Efraimidis, Christos Glymidakis, Basilis Mamalis, Paul G. Spirakis, and Basil Tampakas. Parallel Text Retrieval on a High Performance Super Computer Using the Vector Space Model. SIGIR 1995: 58-66. (Available electronically at ACM's digital library).

Abstract
This paper discusses the efficiency of a parallel text retrieval system that is based on the Vector Space Model. Specifically, we describe a general parallel retrieval algorithm for use with this model, the application of the algorithm in the FIRE system [1], and its implementation on the high performance GCe131512 Parsytec parallel machine [2]. The use of this machine’s two-dimensional grid of processors provides an efficient basis for the virtual tree that lies at the heart of our retrieval algorithm. Analytical and experimental evidence is presented to demonstrate the efficiency of the algorithm performance.

Implementation

PFIRE has been implemented on the massive parallel supercomputer GCel 3/512 of Parsytec, with 512 Transputer nodes connected on a 2-dimensional grid topology. The source code (Parallel C for Parix 1.2) freely available here.