Suffix-sorting via Shannon-Fano-Elias codes (Q1662548)

Summary: Given a sequence \(T=t_0t_1\dots t_{n-1}\) of size \(n=| T|\), with symbols from a fixed alphabet \(\Sigma\), \((|\Sigma|\leq n)\), the suffix array provides a listing of all the suffixes of \(T\) in a lexicographic order. Given \(T\), the suffix sorting problem is to construct its suffix array. The direct suffix sorting problem is to construct the suffix array of \(T\) directly without using the suffix tree data structure. While algorithims for linear time, linear space direct suffix sorting have been proposed, the actual constant in the linear space is still a major concern, given that the applications of suffix trees and suffix arrays (such as in whole-genome analysis) often involve huge data sets. In this work, we reduce the gap between current results and the minimal space requirement. We introduce an algorithm for the direct suffix sorting problem with worst case time complexity in \(O(n)\), requiring only \((1\frac{2}{3}n\log n-n\log|\Sigma|+O(1))\) bits in memory space. This implies \(5\frac{2}{3}n+O(1)\) bytes for total space requirment, (including space for both the output suffix array and the input sequence \(T\)) assuming \(n\leq 2^{32}\), \(|\Sigma|\leq 256\), and 4 bytes per integer. The basis of our algorithm is an extension of Shannon-Fano-Elias codes used in source coding and information theory. This is the first time information-theoretic methods have been used as the basis for solving the suffix sorting problem.