Comparing Costs for Cloud-based Data Archives

Abstract

Research data management is an expensive enterprise. Computing infrastructure for storing, retrieving, and preserving data is one area of expenses, and computing infrastructure costs grow as the size and number of datasets and demands for their retrieval grow. This paper compares the costs and performance of two database infrastructures, PostgreSQL and Elasticsearch, for digital data archives. We used benchmarking experiments and data from social media to estimate the costs of loading, indexing, and querying data from these two databases. The results show that traditional relational open-source databases can be effective for large social science data and run on relatively low-cost computing infrastructure, where PostgreSQL queries can be faster and less expensive than Elasticsearch. PostgreSQL required higher up front costs and time, and adding computing resources did not improve Elasticsearch’s query performance. These findings are useful for digital archives evaluating back-end storage systems.