Exploiting Intel Optane Persistent Memory for Full Text Search
In our information-driven societies, full-text search is ubiquitous. Search is memory-intensive. Quickly searching massive corpora requires building indices, which consumes big volatile heaps. Search is storage I/O-intensive. Limited main memory necessitates writing large partial indices on non-volatile storage, where they finally live in merged form. These indices reside in memory, in full or in part, during query evaluation. Memory and I/O intensity make it hard to index and search content rapidly and efficiently. On the hardware side, the recently introduced Intel Optane DC persistent memory (PM) offers byte-addressability, high capacity, and non-volatility. This paper evaluates and exploits Optane PM for text indexing and search on multicore platforms.
We identify essential structures in inverted indices (hash table, merge tree, and key-value store), where they reside (memory or storage), and key operations over them (sort, flush, and merge). We allocate these structures in DRAM, Optane PM, and block storage by modifying an existing search engine. We then evaluate a myriad of hybrid memory and storage configurations. Our findings include: (1) careful placement of index structures across DRAM, Optane PM, and SSD, speeds up indexing with a single core compared to a high-performance baseline but does not scale to many cores, (2) crash-consistent indexing with Optane PM is feasible without incurring a high overhead, and (3) the tail latency of the longest multi-term conjunctive queries is lower with a PM-backed index than an SSD-backed one. This paper opens up persistent memory to a practical role in full-text search.
presentation slides (ismm-2021-search.pdf) | 6.39MiB |
Shoaib Akram is an assistant professor at the Australian National University. He holds a Ph.D. in Computer Science and Engineering from Ghent University in Belgium and an M.S. in Electrical and Computer Engineering from the University of Illinois at Urbana-Champaign. His current research focuses on the evaluation, optimization, and applications of emerging memory and storage hardware.
Tue 22 JunDisplayed time zone: Eastern Time (US & Canada) change
18:00 - 21:00 | Session 4: Compacting/Indexing/Transactioning & ClosingISMM 2021 at ISMM Chair(s): Timothy M. Jones University of Cambridge, UK | ||
18:00 30mTalk | Exploiting Intel Optane Persistent Memory for Full Text Search ISMM 2021 Shoaib Akram Australian National University Pre-print File Attached | ||
18:30 30mTalk | Understanding and Utilizing Hardware Transactional Memory Capacity ISMM 2021 Zixian Cai Australian National University, Steve Blackburn Australian National University, Michael D. Bond Ohio State University, USA Link to publication DOI Media Attached | ||
19:00 30mTalk | Fusuma: Double-ended Threaded Compaction ISMM 2021 Hiro Onozawa The University of Electro-Communications, Tomoharu Ugawa University of Tokyo, Hideya Iwasaki University of Electro-Communications, Japan | ||
19:30 15mDay closing | Closing Remarks ISMM 2021 |