KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language

KVDK

KVDK (Key-Value Development Kit) is a key-value store library implemented in C++ language. It is designed for persistent memory and provides unified APIs for both volatile and persistent scenarios. It also demonstrates several optimization methods for high performance with persistent memory. Besides providing the basic APIs of key-value store, it offers several advanced features, like transaction, snapshot as well.

Features

  • The basic get/set/update/delete opertions on unsorted keys.
  • The basic get/set/update/delete/iterate operations on sorted keys.
  • Multiple changes on unsorted keys can be made in one atomic batch.
  • User can create multiple collections of sorted keys.
  • Support read-committed transaction. (TBD)
  • Support snapshot to get a consistent view of data. (TBD)

Limitations

  • Maximum supported key-value size are 64KB-64MB.
  • The maximum write thread number can't be dynamicly changed after start-up.
  • No support of key-value compression.
  • Persistent memory space can't be expanded on the fly.

Getting the Source

git clone --recurse-submodules https://github.com/pmem/kvdk.git

Building

mkdir -p build && cd build
cmake .. -DCMAKE_BUILD_TYPE=Release && make -j

Before each commit, please check its coding style with below instructions

cmake .. -DCHECK_CPP_STYLE=ON && make -j

Benchmarks

Here are the examples on how to benchmark the performance of KVDK on your own systems.

Documentations

User Guide

Please reference to User guide for API introductions of KVDK.

Architecture

Owner
Persistent Memory Programming
Libraries and Examples for Persistent Memory Programming
Persistent Memory Programming
Comments
  • Unexpected InvalidDataSize

    Unexpected InvalidDataSize

    Bug Report

    KVDK version

    14f8778dea099225cd2bb1d9c8b406c7b53b8133

    System configuration

    Reproduce steps

        kvdk::Configs engine_configs;
        {
    
          engine_configs.pmem_file_size = 40 * 1024UL * 1024UL * 1024UL;
          engine_configs.pmem_segment_blocks = (1ull << 8);
          engine_configs.hash_bucket_num = (1ull << 10);
          engine_configs.log_level = kvdk::LogLevel::Debug;
        }
    
        std::string engine_path{kvpmem_data_file_path};
    
        // Purge old KVDK instance
        system(std::string{"rm -rf " + engine_path + "\n"}.c_str());
    
        status = kvdk::Engine::Open(engine_path, &engine, engine_configs, stdout);
        assert(status == kvdk::Status::Ok);
        DBUG_PRINT("KVDK",
                   ("Successfully created KVDK engine %s", kvpmem_data_file_path));
    
        status = engine->SSet("hi", "yo", "peeep");
        ASSERT(status == kvdk::Status::Ok, (int)status);
      }
    

    Expect behavior

    Should put value in the ordered set.

    Current behavior

    Assert fails with InvalidDataSize. If I print the size of the string view inside the engine I get following output [ERROR] time 40009 ms: SSET size collection: 1147430597 userkey: 1147430594 value: 1147430588.

    I think there is an issue with the string view but I don't understand why since the usage is very similar to the usage guide. I also tried to pass a static string. Any ideas?

  • Why not use libpmemobj ? [NEW]

    Why not use libpmemobj ? [NEW]

    Bug Report

    Why not use libpmemobj ? I do not think kvdk can support atomic operation only using APIs in libpmem. batchWrite can write multiple KV pairs one by one, but could not roll back if failed. DLinked_list could not update prev/next pointer in one transaction. But it shows "Provide APIs to write multiple key-value pairs in an atomic batch. ", Is there something wrong ?

  • Why is the reading and writing speed of kvdk not as fast as leveldb on ssd?

    Why is the reading and writing speed of kvdk not as fast as leveldb on ssd?

    Why is the reading and writing speed of kvdk not as fast as leveldb? I think this is very strange. Is it the problem of my use or the problem of kvdb itself? If it is the problem of kvdb itself, do you have a good recommendation for kv database, it is best to use B+ tree as the engine, thank you.

  • Reorg PMEM FreeList: global free entry pool + thread local entry cache

    Reorg PMEM FreeList: global free entry pool + thread local entry cache

    In this patch:

    1. Move pmem allocator related codes to directory engine/pmem_allocator
    2. Reorg free list:
    • Use a global free space entry pool to store free space
    • Each write thread has a cache structure, which consists of a active free space list and a backup free space list. Each threads store just freed space to and use free space from own active list. To balance free space entries among threads, if too many entries cached by a thread, newly freed entries will be stored in backup list, then move the whole backup list to entry pool which shared by all threads
    • If active list is empty, then swap backup list as active list, or fetch a free list from entry pool
  • Cannot compile kvrocks

    Cannot compile kvrocks

    Bug Report

    I cannot compile the kvrocks correctly by following the steps mentioned in the example/kvrocks/README I found that the README is updated 9 months ago. However, the patch is updated three days ago.

    Current behavior

    When I compile with running ‘make’, there's an error:

    incubator-kvrocks/external/kvdk/include/types.hpp:7:10: fatal error: libpmemobj++/string_view.hpp: No such file or directory
    

    This error could be fixed by copying libpmemobj++/ directory from kvdk to the kvrocks.

    But there is another error:

    In file included from redis_bitmap_string.cc:5:
    redis_string.h:9:10: fatal error: namespace.hpp: No such file or directory
    #include "namespace.hpp"
    

    I found that you are developing kvdk and there is no namespace.h file in the latest version. I would be grateful if you fixing the issue and update the README.

  • Allow empty string as key

    Allow empty string as key

    The empty string as key acts normally in KVDK. No need to check for zero-sized keys. Added a file debug.cpp under tests directory and is compiled to dbdebug. This file is almost a copy of tutorial.cpp but for debugging purpose, because it's very simple and easy to change.

  • bench tool:  Set error happened when using sorted-type and latency enable

    bench tool: Set error happened when using sorted-type and latency enable

    "Set error" happened when using sorted-type and latency enable, option type inclues "Fill, update, Insert",for example:

    Insert new sorted-type kv Write latency overflow: 12191390 us Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error Set error sh: line 1: 71030 Segmentation fault (core dumped) numactl --cpunodebind=0 --membind=0 ./bench -latency=1 -populate=1 -value_size=128 -threads=96 -time=10 -path=/mnt/pmem0/kvdk -num=789516047 -space=536870912000 -max_write_threads=96 -fill=0 -type=sorted -read_ratio =0 -existing_keys_ratio=0 > ./results/sorted_vs128_insert_thread96

  • Implement batch write for sorted collections

    Implement batch write for sorted collections

    What is changed and how it works?

    Implement batch write for sorted collections

    Check List

    Tests

    • Unit test
    • Integration test

    Performance (120Byte value size)

    Batch insert (100 per batch): 6.32 MOPS Normal insert: 6.38 MOPS

  • Fix test cases with sync points

    Fix test cases with sync points

    Signed-off-by: ZiyanShi [email protected]

    What problem does this PR solve?

    Fix test throw an exception if use gtest_repeat option. This patch disable and reset sync points for each test case on setup and teardown so that sync points will not live through test cases.

    Tests

    • Unit test
  • fix warning and background block problem

    fix warning and background block problem

    Signed-off-by: zhichenj [email protected]

    What problem does this PR solve?

    1. Fix warning about no return type.
    2. Workaround to solve block problem for background cleaner.

    Tests

    • Unit test
    • Integration test
    • No code

    Side effects No measure

  • Will kvdk support other types of Key or value such as int64,selfdefined structure? [NEW]

    Will kvdk support other types of Key or value such as int64,selfdefined structure? [NEW]

    As a kv system, it may save any type of data except for std::basic_string_view, will kvdk support other types? Following codes only support c-like string as input data, why not support any data with size ?

    /**

    • Constructor initialized with data and its size.
    • @param[in] data pointer to the C-like string to initialize with,
    • it can contain null characters.
    • @param[in] size length of the given data. */ template <typename CharT, typename Traits> constexpr inline basic_string_view<CharT, Traits>::basic_string_view( const CharT *data, size_type size) : data_(data), size_(size) { }
  • Fix List and Hash Iterator Behavior

    Fix List and Hash Iterator Behavior

    Signed-off-by: ZiyanShi [email protected]

    What problem does this PR solve?

    Calling Prev() and Next() on a Iterator where Valid() == false will do nothing. This prevents misuse of an invalid iterator when user forgets to check Valid() right after a call of Seek().

    Tests

    • Unit test
  • [Pending] Clean cached outdated records in foreground threads

    [Pending] Clean cached outdated records in foreground threads

    What is changed and how it works?

    What's Changed:

    • Clean outdated records in foreground threads to improve performance
    • Background thread do global cache clean before every purge for correctness

    Check List

    Tests

    • Unit test
    • Integration test

    Performance:

    • Updated performance improved 50%
  • Use std::exit instead of exception to implement crash point

    Use std::exit instead of exception to implement crash point

    What problem does this PR solve?

    Use std::exit instead of exception to implement crash point so that we can kill all threads within the process.

    Tests

    • Unit test failed
Related tags
Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.
Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Jun 23, 2022
Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.
Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Kvrocks is a distributed key value NoSQL database based on RocksDB and compatible with Redis protocol.

Jun 24, 2022
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.
🥑 ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

?? ArangoDB is a native multi-model database with flexible data models for documents, graphs, and key-values. Build high performance applications using a convenient SQL-like query language or JavaScript extensions.

Jun 24, 2022
Kreon is a key-value store library optimized for flash-based storage
Kreon is a key-value store library optimized for flash-based storage

Kreon is a key-value store library optimized for flash-based storage, where CPU overhead and I/O amplification are more significant bottlenecks compared to I/O randomness.

Jun 10, 2022
RocksDB: A Persistent Key-Value Store for Flash and RAM Storage

RocksDB is developed and maintained by Facebook Database Engineering Team. It is built on earlier work on LevelDB

Jun 20, 2022
BerylDB is a data structure data manager that can be used to store data as key-value entries.
BerylDB is a data structure data manager that can be used to store data as key-value entries.

BerylDB is a data structure data manager that can be used to store data as key-value entries. The server allows channel subscription and is optimized to be used as a cache repository. Supported structures include lists, sets, and keys.

Jun 24, 2022
FoundationDB - the open source, distributed, transactional key-value store
FoundationDB - the open source, distributed, transactional key-value store

FoundationDB is a distributed database designed to handle large volumes of structured data across clusters of commodity servers. It organizes data as

Jun 17, 2022
A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store
A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store

SimDB A high performance, shared memory, lock free, cross platform, single file, no dependencies, C++11 key-value store. SimDB is part of LAVA (Live A

Jun 8, 2022
Kit: a magical, high performance programming language, designed for game development
Kit: a magical, high performance programming language, designed for game development

Kit: a magical, high performance programming language, designed for game development

May 26, 2022
cdk is a minimal cross-platform c language development kit.

Overview cdk is a minimal cross-platform c language development kit. Requirement Based on c11 standard. Compile create a build directory under the cdk

Jun 10, 2022
Jun 16, 2022
Data Plane Development Kit

DPDK is a set of libraries and drivers for fast packet processing. It supports many processor architectures and both FreeBSD and Linux. The DPDK uses

Jun 17, 2022
Internal Software Development Kit for Battlefield 2042

battlefield-2042-internal-sdk Internal Software Development Kit for Battlefield 2042 SDK Includes the following: Entity Classes Player Classes Vehicle

Mar 17, 2022
🎮 Cross platform development kit for Z80 and SM83 based consoles.

cdk ?? Cross platform development kit for Z80 and SM83 based consoles. Platform We planned to support the following consoles: Nintendo Game Boy Ninten

Jan 10, 2022
bl_mcu_sdk is MCU software development kit provided by Bouffalo Lab Team for BL602/BL604, BL702/BL704/BL706 and other series of RISC-V based chips in the future.
bl_mcu_sdk is MCU software development kit provided by Bouffalo Lab Team for BL602/BL604, BL702/BL704/BL706 and other series of  RISC-V based chips in the future.

bl mcu sdk is an MCU software development kit provided by the Bouffalo Lab Team for BL602/BL604, BL702/BL704/BL706 and other series of chips in the future

Jun 18, 2022
Simple constant key/value storage library, for read-heavy systems with infrequent large bulk inserts.

Sparkey is a simple constant key/value storage library. It is mostly suited for read heavy systems with infrequent large bulk inserts. It includes bot

Jun 19, 2022
LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values.

LevelDB is a fast key-value storage library written at Google that provides an ordered mapping from string keys to string values. Authors: Sanjay Ghem

Jun 22, 2022
Modern transactional key-value/row storage library.
Modern transactional key-value/row storage library.

Sophia is advanced transactional MVCC key-value/row storage library. How does it differ from other storages? Sophia is RAM-Disk hybrid storage. It is

Jun 25, 2022
This project implemented the Mean Value Coordinates in 3D algorithm in c++

Mean Value Coordinates in 3D [c++] | Paper link on Sciencedirect | Pdf version link | This project implemented the Mean Value Coordinates in 3D algori

Nov 18, 2021
Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.
Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Kvrocks is a key-value NoSQL database based on RocksDB and compatible with Redis protocol.

Jun 23, 2022