STXXL: Standard Template Library for Extra Large Data Sets

DESCRIPTION

STXXL is an implementation of the C++ standard template library STL for
external memory (out-of-core) computations, i. e. STXXL implements containers
and algorithms that can process huge volumes of data that only fit on disks.
While the closeness to the STL supports ease of use and compatibility with
existing applications, another design priority is high performance.


DOCUMENTATION

See the Doxygen documentation for installation manual and programmer
documentation: http://stxxl.sourceforge.net


LICENSE TERMS

STXXL is distributed under the Boost Software License, Version 1.0.
(See accompanying file LICENSE_1_0.txt or copy at
http://www.boost.org/LICENSE_1_0.txt)


BUGS AND QUESTIONS

If you find any bugs or have any questions, please visit the forums at

http://sourceforge.net/projects/stxxl/forums
Owner
STXXL
STXXL: Standard Template Library for Extra Large Data Sets
STXXL
Comments
  • No Mingw support?

    No Mingw support?

    I tried to compile stxxl under mingw-64 (from mingw-builds). Had one error in lib/io/wfs_file_base.cpp , added naive fix

    @@ -73,12 +73,11 @@ static HANDLE open_file_impl(const std::string& filename, int mode)
             dwFlagsAndAttributes |= FILE_FLAG_NO_BUFFERING;
             // TODO: try also FILE_FLAG_WRITE_THROUGH option ?
     #else
             if (mode & file::REQUIRE_DIRECT) {
                 STXXL_ERRMSG("Error: open()ing " << filename << " with DIRECT mode required, but the system does not support it.");
    -            file_des = INVALID_HANDLE_VALUE;
    -            return;
    +            return INVALID_HANDLE_VALUE;
             }
             else {
                 STXXL_MSG("Warning: open()ing " << filename << " without DIRECT mode, as the system does not support it.");
             }
     #endif
    

    Then it compiled successfully but runs with multiple problems: does not support DIRECT mode (MSVC does):

    [STXXL-MSG] Warning: open()ing C:\Users\ak\AppData\Local\Temp\stxxl.tmp without DIRECT mode, as the system does not support it.
    

    and often fails with mutex

    [extractor] Confirming/Writing used nodes     ... terminate called after throwing an instance of 'stxxl::resource_error'
      what():  Error in stxxl::mutex::~mutex() : pthread_mutex_destroy(&m_mutex) : Resource device
    This application has requested the Runtime to terminate it in an unusual way.
    Please contact the application's support team for more information.
    

    (tried to compile and run https://github.com/DennisOSRM/Project-OSRM software)

    Just wanted to know, is there any plans to support mingw compiler?

  • Support newer Mingw

    Support newer Mingw

    It seems I have accdentally fixed MINGW support :)

    It would be nice to have it "out of the box". The change tells cmake to use STD threads and DIRECT io by default on MINGW.

    Tested on http://sourceforge.net/projects/mingwbuilds/files/host-windows/releases/4.7.3/64-bit/threads-posix/sjlj/

  • Does stxxl's containers support embed

    Does stxxl's containers support embed "std::vector" into user-defined type?

    Hi all,

    Greetings from me!

    I write a simple program using stxxl's vector:

    #include <stxxl/vector>
    #include <vector>
    #include <iostream>
    
    class Foo
    {
    public:
        std::vector<double> vec;
        Foo()
        {
        }
    };
    
    int main()
    {
        typedef stxxl::VECTOR_GENERATOR<Foo>::result vector;
        vector my_vector;
        std::cout.precision(10);
    
        for (int i = 0; i < 1024 * 1024 * 1024; i++)
        {
            my_vector.push_back(Foo());
            my_vector[i].vec.push_back(i);
        }
    
    
        std::cout << my_vector[99].vec[0] << std::endl;
    
        return 0;
    }
    

    You can see the user-defined type Foo contains a std::vector. I expect the running result is 99, but in fact the output is 1073737827 . I guess the stxxl's vector did "resize" operation, but not sure.

    So does stxxl's containers support embed std::vector into user-defined type? Thanks very much in advance!

    Best Regards Nan Xiao

  • stxxl 1.3.1: crash after unsuccessful resize of stxxl::vector

    stxxl 1.3.1: crash after unsuccessful resize of stxxl::vector

    If stxxl::vector::resize(n) throws an exception due to insufficient disk space the vector object will crash in it's destructor. Looking at the crash in a debugger I see the stack like this: block_manager::delete_blocks(begin, end) -> block_manager::delete_block(bid) -> bid.is_managed() // here and the reason for the crash is that bid.storage is null. The problem is gone after I changed https://github.com/stxxl/stxxl/blob/1.3.1/include/stxxl/bits/mng/mng.h#L208 line 208 from if (!bid.is_managed()) to if (!bid.valid() || !bid.is_managed()) Would be great to know if this problem is fixed in 1.4.x?

  • stxxl::vector does not work if used as dynamic array

    stxxl::vector does not work if used as dynamic array

    Hi,

    I plan to use your library within my research projects, but I found a bug (or misusage) that prevents me from using it.

    The bug concerns the fact that vector cannot be used unless the size is specified beforehand (which is impossible in my case).

    To reproduce the bug, I wrote the following program:

    #include <iostream>
    #include <cstdlib>
    #include <stxxl/vector>
    
    typedef struct Triple {
    long t1, t2, t3;
    } Triple;
    
    typedef stxxl::VECTOR_GENERATOR<Triple>::result vector_type;
    
    int main(int argc, const char** argv) {
    vector_type vector;
    vector_type::bufwriter_type writer(vector);
    for(int i = 0; i < 1000000000; ++i) {
        std::cout << "Inserting element " << i << std::endl;
        Triple t;
        t.t1 = rand();
        t.t2 = rand();
        t.t3 = rand();
        writer << t;
    }
    writer.finish();
    }
    

    This program is supposed to create a very large array of Triple objects (three longs). Now, if I launch it the following happens:

    Inserting element 0 Inserting element 1 ... Inserting element 174761 Inserting element 174762 Segmentation fault: 11

    I'm currently using macos 10.9, stxxl v1.4.0 (prerelease/Release) and g++ --version is:

    Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/usr/include/c++/4.2.1 Apple LLVM version 5.0 (clang-500.2.79) (based on LLVM 3.3svn) Target: x86_64-apple-darwin13.0.0 Thread model: posix

    Thanks very much!

  • Syscall param write(buf) points to uninitialised byte(s)

    Syscall param write(buf) points to uninitialised byte(s)

    Hello,

    i have spent hours of trying to debug one class i am using stxxl::vector in. If i substitute with STL vector everything works. All variables are initialized, i dont know what is memcheck complaining about. Thank you for your help.

    //these are properties of EdgeIndex class
    stxxl::vector<WayInfo> way_info;
     sparse_hash_map<uint64_t, uint32_t> way_info_index; //OSM way id -> index to way_info
    
    //i am trying to do 
    void DiskSerializer::insertWayToIndex(const osmium::Way &way)
    {
    	WayInfo way_info;
    	//add way nodes
    	for (const osmium::NodeRef& nr : way.nodes())
    	{
    		OSMNodeID id = nr.ref();
    		way_info.nodes.push_back(id);
    	}
    	OSMWayID way_id = way.id();
    	edge_index->insertWayInfo(way_id, way_info);
    };
    void DiskEdgeIndex::insertWayInfo(OSMWayID &way_id, WayInfo &way_inf)
    {
        pair<uint64_t,uint32_t> way_id_index = make_pair(way_id,way_info.size());
        way_info.push_back(way_inf);    
        way_info_index.insert(way_id_index);
    };
    

    The stack trace

    STXXL-MSG] Disk '/home/doma/Desktop/test_files/monaco-latest.osm.pbf.stxxl' is allocated, space: 10240 MiB, I/O implementation: syscall direct=on queue=0 devid=0 unlink_on_open
    [STXXL-ERRMSG] stxxl::aligned_alloc: disabling realloc()
    ==2627== Thread 7:
    ==2627== Syscall param write(buf) points to uninitialised byte(s)
    ==2627==    at 0x52A068D: ??? (syscall-template.S:84)
    ==2627==    by 0x3AF83A: stxxl::syscall_file::serve(void*, unsigned long long, unsigned long long, stxxl::request_interface::request_type) (syscall_file.cpp:92)
    ==2627==    by 0x3C5007: stxxl::serving_request::serve() (serving_request.cpp:57)
    ==2627==    by 0x3C1FA9: stxxl::request_queue_impl_qwqr::worker(void*) (request_queue_impl_qwqr.cpp:162)
    ==2627==    by 0x5297709: start_thread (pthread_create.c:333)
    ==2627==    by 0x63470AE: clone (clone.S:105)
    ==2627==  Address 0xfe0eff8 is 2,101,176 bytes inside a block of size 67,112,976 alloc'd
    ==2627==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==2627==    by 0x2F14CC: void* stxxl::aligned_alloc<4096ul>(unsigned long, unsigned long) (aligned_alloc.h:58)
    ==2627==    by 0x2EDC13: stxxl::typed_block<2097152u, WayInfo, 0u, void>::operator new[](unsigned long) (typed_block.h:322)
    ==2627==    by 0x2E8BAC: stxxl::simple_vector<stxxl::typed_block<2097152u, WayInfo, 0u, void> >::simple_vector(unsigned long) (simple_vector.h:70)
    ==2627==    by 0x2E443F: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::allocate_page_cache() const (vector.h:1011)
    ==2627==    by 0x2E15FF: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::vector(unsigned long long, unsigned long long) (vector.h:966)
    ==2627==    by 0x2DCC13: DiskEdgeIndex::DiskEdgeIndex() (DiskEdgeIndex.cpp:4)
    ==2627==    by 0x30BA2D: main (parser.cpp:106)
    ==2627==  Uninitialised value was created by a heap allocation
    ==2627==    at 0x4C2CB3F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
    ==2627==    by 0x2F14CC: void* stxxl::aligned_alloc<4096ul>(unsigned long, unsigned long) (aligned_alloc.h:58)
    ==2627==    by 0x2EDC13: stxxl::typed_block<2097152u, WayInfo, 0u, void>::operator new[](unsigned long) (typed_block.h:322)
    ==2627==    by 0x2E8BAC: stxxl::simple_vector<stxxl::typed_block<2097152u, WayInfo, 0u, void> >::simple_vector(unsigned long) (simple_vector.h:70)
    ==2627==    by 0x2E443F: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::allocate_page_cache() const (vector.h:1011)
    ==2627==    by 0x2E15FF: stxxl::vector<WayInfo, 4u, stxxl::lru_pager<8u>, 2097152u, stxxl::RC, unsigned long long>::vector(unsigned long long, unsigned long long) (vector.h:966)
    ==2627==    by 0x2DCC13: DiskEdgeIndex::DiskEdgeIndex() (DiskEdgeIndex.cpp:4)
    ==2627==    by 0x30BA2D: main (parser.cpp:106)
    ==2627== 
    ==2627== 
    ==2627== HEAP SUMMARY:
    ==2627==     in use at exit: 271,491 bytes in 2,803 blocks
    ==2627==   total heap usage: 104,163 allocs, 101,360 frees, 552,918,871 bytes allocated
    ==2627== 
    ==2627== LEAK SUMMARY:
    ==2627==    definitely lost: 3,040 bytes in 25 blocks
    ==2627==    indirectly lost: 1,943 bytes in 34 blocks
    ==2627==      possibly lost: 912 bytes in 3 blocks
    ==2627==    still reachable: 265,596 bytes in 2,741 blocks
    ==2627==                       of which reachable via heuristic:
    ==2627==                         newarray           : 1,560 bytes in 3 blocks
    ==2627==         suppressed: 0 bytes in 0 blocks
    ==2627== Rerun with --leak-check=full to see details of leaked memory
    ==2627== 
    ==2627== For counts of detected and suppressed errors, rerun with: -v
    ==2627== ERROR SUMMARY: 1 errors from 1 contexts (suppressed: 0 from 0)
    
  • Next release?

    Next release?

    I notice that v1.4.1 is the latest release, but there have been over 350 commits to master since: https://github.com/stxxl/stxxl/releases/tag/1.4.1.

    Are there any plans for a new official release or an anticipated date?

  • spin_lock misses the default constructor on newer MSVC

    spin_lock misses the default constructor on newer MSVC

    When compiling under new Visual Studio 2015 RC I am getting the error about missing default constructor in spin_lock class. It is really turned off for MSC_VER 1800 and 1900 . Should it be returned in #else block?

    https://github.com/stxxl/stxxl/blob/master/include/stxxl/bits/common/mutex.h#L167

  • Deprecated hash_map header

    Deprecated hash_map header

    Newer Visual Studio 2015 RC stopped compilig deprecated hash_map header

    C:\Program Files (x86)\Microsoft Visual Studio 14.0\VC\include\hash_map(17): error C2338: <hash_map> is deprecated and will be REMOVED. Please use <unordered_map>. You can define _SILENCE_STDEXT_HASH_DEPRECATION_WARNINGS to acknowledge that you have received this warning. [D:\winbuilds\stxxl\build\tools\stxxl_tool.vcxproj]
    

    I guess that <unordered_map> should be activated in https://github.com/stxxl/stxxl/blob/master/include/stxxl/bits/compat/hash_map.h#L24 for _MSC_VER >= 1900 . Not sure about the best solution, so there is no PR.

  • Fix for DllMain() initialization deadlock

    Fix for DllMain() initialization deadlock

    [Only affects Windows - Visual Studio configuration]

    In the scenario when stxxl is being initialized in an DllMain() the application may or may mot deadlock, this patch fixes that.

  • can the container be loaded back from file created by stxxl

    can the container be loaded back from file created by stxxl

    Write the index stxxl::config * cfg = stxxl::config::get_instance(); stxxl::disk_config disk1("disk=/tmp/lsh.index, 10 GiB, syscall unlink"); disk1.direct = stxxl::disk_config::DIRECT_ON; // force O_DIRECT disk1.delete_on_exit = false; cfg->add_disk(disk1);

    What could be the possible way to read the file "/tmp/lsh.index" ?

  • Guard definition of log2 on versions of Visual C++ that provide it.

    Guard definition of log2 on versions of Visual C++ that provide it.

    Resolves:

    C:\PROGRA~1\MICROS~1\2022\ENTERP~1\VC\Tools\MSVC\1432~1.313\bin\Hostx64\arm64\cl.exe   /TP -D_CRT_NONSTDC_NO_DEPRECATE -D_CRT_SECURE_NO_WARNINGS -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGE_FILES -D_SCL_SECURE_NO_WARNINGS -ID:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\include -ID:\buildtrees\stxxl\arm64-windows-rel\include /nologo /DWIN32 /D_WINDOWS /W4 /utf-8 /GR /EHsc /MP  /wd4127 /wd4290 /wd4250 /wd4512 /wd4355 -openmp /MD /O2 /Oi /Gy /DNDEBUG /Z7 /showIncludes /Fotools\CMakeFiles\stxxl_tool.dir\benchmark_sort.cpp.obj /Fdtools\CMakeFiles\stxxl_tool.dir\ /FS -c D:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\tools\benchmark_sort.cpp
    D:\buildtrees\stxxl\src\53f2b75236-bdc19ed4a0.clean\include\stxxl/bits/msvc_compatibility.h(23): error C2169: 'log2': intrinsic function, cannot be defined
    ninja: build stopped: subcommand failed.
    
  • Is STXXL still maintained?

    Is STXXL still maintained?

    Hi, We are using stxxl as a submodule in our project QLEVER. stxxl currently only builds with warnings in C++17 mode, and does not build at all in C++20 mode (using G++ 10). I am able to locally fix the issues and would like to contribute them here as a PR, however I am not sure if this project is still active.

    • It's been almost 3 years since the last activity
    • @bingmann who seems to be the main contributor to this system maintains his own fork of stxxl , which has diverged quite a bit from this repository, but also there the last commit is 2 years old.

    Is stxxl still being actively maintained, and if yes, on which repository and by whom?

  • [Build] Build failure with boost-system

    [Build] Build failure with boost-system

    Hi guys, today I got a regression when building stxxl:

    D:\installed\x86-windows\include\boost/system/detail/system_category_message_win32.hpp(53): error C2039: '_snprintf': is not a member of 'std'
    C:\Program Files (x86)\Microsoft Visual Studio\2019\Enterprise\VC\Tools\MSVC\14.28.29333\include\mutex(30): note: see declaration of 'std'
    

    When I dig into this issue, I found that even if I set USE_BOOST to OFF, if HAVE_STD_THREAD or HAVE_STD_MUTEX is OFF, boost will be forced to be enabled [1]. Before using boost, stxxl defined snprintf as _snprintf [2], which caused the std::snprintf search statement in boost-system to fail [3].

    So I think that's a source bug.

    See boost official reply https://github.com/boostorg/system/issues/32.

    [1]. https://github.com/stxxl/stxxl/blob/b9e44f0ecba7d7111fbb33f3330c3e53f2b75236/CMakeLists.txt#L347-L352 [2]. https://github.com/stxxl/stxxl/blob/b9e44f0ecba7d7111fbb33f3330c3e53f2b75236/include/stxxl/bits/msvc_compatibility.h#L28 [3]. https://github.com/boostorg/system/blob/689e09e86ff7f4943a1c8236d316ba4608ec3142/include/boost/system/detail/system_category_message_win32.hpp#L53

  • Build fixes for C++17 (due to errors in Visual Studio)

    Build fixes for C++17 (due to errors in Visual Studio)

    There are several std classes/functions that are removed in C++17 and causes build errors. This PR removes and replaces them as necessary. In addition there are a few build warnings (in particular recommending enum class) that are fixed as well.

    Please check that this causes no problems/changes on Linux

  • Fixed C++17 compilation.

    Fixed C++17 compilation.

    I've update the code for handling with success C++17 standard. I've removed references to std::binary_function, std::random_shuffle and std::bind2nd. I've also fixed a __cplusplus issue with Visual Studio 2019 and now it creates unique_ptr.

FSD-Template - A template UE4.25 project for BP modding.

FSD-Template Project generated by Archengius' UE4 Template Generator. Reflected C++ classes generated by CheatingMuppet & Archengius' UE4SS UHT Genera

Jun 18, 2022
Libft is an individual project at 42 that requires us to re-create some standard C library functions including some additional ones that can be used later to build a library of useful functions for the rest of the program.

?? Index What is Libft? List of Functions Technologies ✨ What is Libft? Libft is an individual project at 42 that requires us to re-create some standa

Jan 17, 2022
Thrust is a C++ parallel programming library which resembles the C++ Standard Library.

Thrust: Code at the speed of light Thrust is a C++ parallel programming library which resembles the C++ Standard Library. Thrust's high-level interfac

Jun 22, 2022
jkds is a modern header-only C++20 library that complements the standard library.

jkds is a modern header-only C++20 library that complements the standard library. It provides generic atypical data structures, ergonomic functional programming abstractions, and then some.

May 24, 2022
Bionic BSD-3-ClauseBionic - Google's standard library, developed for Android. BSD-3-Clause

bionic bionic is Android's C library, math library, and dynamic linker. Using bionic as an app developer See the user documentation. Working on bionic

Jun 13, 2022
CloudABI's standard C library

NOTE: This project is unmaintained CloudABI is no longer being maintained. It was an awesome experiment, but it never got enough traction to be sustai

Jun 7, 2022
libcu++: The C++ Standard Library for Your Entire System

libcu++, the NVIDIA C++ Standard Library, is the C++ Standard Library for your entire system. It provides a heterogeneous implementation of the C++ Standard Library that can be used in and between CPU and GPU code.

Jun 16, 2022
An open source standard C library that includes useful functions && (Reimplementation of libc functions + own functions).
An open source standard C library that includes useful functions && (Reimplementation of libc functions + own functions).

?? LIBFT-42 : Artistic view of LIBC: ?? HOW DOES IT FEEL HAVING YOUR OWN LIB: SUBJECT : ENGLISH PDF ℹ️ What is LIBFT : This project aims to code a C l

Mar 24, 2022
Your standard library for metaprogramming

Boost.Hana Your standard library for metaprogramming Overview #include <boost/hana.hpp> #include <cassert> #include <string> namespace hana = boost::h

Jun 12, 2022
Reimplementation of some of the Standard C Library functions.
Reimplementation of some of the Standard C Library functions.

42-libft Reimplementation of some of the Standard C Library functions. This repository contains some of the standard library C functions. List of avai

Jan 26, 2022
MSVC's implementation of the C++ Standard Library.

Microsoft's C++ Standard Library This is the official repository for Microsoft's implementation of the C++ Standard Library (also known as the STL), w

Jun 17, 2022
mlibc is a C standard library

mlibc is a C standard library Official Discord server: https://discord.gg/7WB6Ur3 Design of the library Directory Purpose options/ (More or less) OS-i

Jun 14, 2022
A standard conforming C++20 implementation of std::optional.

A standard conforming C++20 implementation of std::optional.

Feb 11, 2022
RTL - Reactive Template Library for C++

RTL - Reactive Template Library It solves all my C++ animation problems -- me Using RTL Lazy, caching, reactive variables

Dec 22, 2021
Modern C++ generic header-only template library.

nytl A lightweight and generic header-only template library for C++17. Includes various utility of all kind that i needed across multiple projects: Ex

Feb 10, 2022
A template C project using CMAKE, logging library and basic memory handling.

C Project template Aim of this Repository is to create a template repository for C executable projects with following properties: Cmake project Loggin

May 23, 2022
this lib with 26 template container and 10 kinds of algorithm, it is a good lib for study and usage

simple stl this lib simplify the achievement detail of common container, but add the container variety, the whole code partily follow Google Style. Em

Mar 10, 2022
This is the template for peripheral projects using nRF5 SDK

NRF5_SDK template project About This is the template for peripheral projects using nRF5 SDK Getting Started To get a local copy up and running follow

Nov 27, 2021
Code::Blocks template for custom launcher executable.

Launcher Code::Blocks template for custom launcher executables. This is a basic Code::Blocks project for creating authentic Windows executables. Inclu

Feb 5, 2022