Proc. IEEE ADL -- Advances in Digital Libraries 1998

Towards an Archival Intermemory

Andrew V. Goldberg and Peter N. Yianilos

Abstract: We propose a self-organizing archival Intermemory. That is, a noncommercial subscriber-provided distributed information storage service built on the existing Internet. Given an assumption of continued growth in the memory's total size, a subscriber's participation for only a finite time can nevertheless ensure archival preservation of the subscriber's data. Information disperses through the network over time and memories become more difficult to erase as they age. The probability of losing an old memory given random node failures is vanishingly small -- and an adversary would have to corrupt hundreds of thousands of nodes to destroy a very old memory.

This paper presents a framework for the design of an Intermemory, and considers certain aspects of the design in greater detail. In particular, the aspects of addressing, space efficiency, and redundant coding are discussed.

Keywords: Archival Storage, Distributed Redundant Databases, Electronic Publishing, Distributed Algorithms, Error Correcting Codes, Erasure-Resilient Codes, Information Dispersal Algorithm, Internet.


The ideas reported in this paper emerged over the period 1994-1997. We acknowledge informal discussions with several people including Sam Buss, Joe Kilian, and Scott Stornetta. This is the first manuscript describing the concept and was first released in October 1997.