mpiFileUtils - File utilities designed for scalability and performance.

mpiFileUtils

mpiFileUtils provides both a library called libmfu and a suite of MPI-based tools to manage large datasets, which may vary from large directory trees to large files. High-performance computing users often generate large datasets with parallel applications that run with many processes (millions in some cases). However those users are then stuck with single-process tools like cp and rm to manage their datasets. This suite provides MPI-based tools to handle typical jobs like copy, remove, and compare for such datasets, providing speedups of up to 20-30x. It also provides a library that simplifies the creation of new tools or can be used in applications.

Documentation is available on ReadTheDocs.

DAOS Support

mpiFileUtils supports a DAOS backend for dcp, dsync, and dcmp. Custom serialization and deserialization for DAOS containers to and from a POSIX filesystem is provided with daos-serialize and daos-deserialize. Details and usage examples are provided in DAOS Support.

Contributors

We welcome contributions to the project. For details on how to help, see our Contributor Guide

Copyrights

Copyright (c) 2013-2015, Lawrence Livermore National Security, LLC. Produced at the Lawrence Livermore National Laboratory CODE-673838

Copyright (c) 2006-2007,2011-2015, Los Alamos National Security, LLC. (LA-CC-06-077, LA-CC-10-066, LA-CC-14-046)

Copyright (2013-2015) UT-Battelle, LLC under Contract No. DE-AC05-00OR22725 with the Department of Energy.

Copyright (c) 2015, DataDirect Networks, Inc.

All rights reserved.

Build Status

The current status of the mpiFileUtils master branch is Build Status.

Owner
High-Performance Computing
Computing that is highly performant
High-Performance Computing
Comments
  • dcp: support DAOS

    dcp: support DAOS

    - Use mfu_file_t struct to wrap fd vs obj
    - create DAOS specific io functions
    - Create mfu_file_* functions that call DAOS backend
    - Add DAOS I/O functions to common library
    - call daos io backend in dcp
    - Add FindCART.cmake and FindDAOS.cmake
    - Add ifdef to DAOS specific code
    - create mfu_file in all tools for walk
    

    Signed-off-by: Danielle Sikich [email protected]

  • Dcmp improve

    Dcmp improve

    Here are some minor improvements for dcmp. "dcmp: improve the usage() dcmp: print the matched count of each conjunctions group in summary" These two are used to improve the usage and summary log, in order to make it more friendly to users.

    "dcmp: record the rawtime directly rather than converting from wtime" This patch uses localtime() to record the time rather than MPI_Wtime (with additional conversion), to avoid some unexpected condition, e.g. time overflow.

    "dcmp: skip the size differ check for dir" This commit remove the size differ check for dir, as we are focusing the size check of files, not dirs, and on the other hand, size of dir may be different even contained files are the same, depends on the fs.

    Please refer to the commits for details.

  • dcp and dsync: add --dereference and --no-dereference

    dcp and dsync: add --dereference and --no-dereference

    Support DAOS --preserve

    Added new IO functions:

    • llistxattr
      • mfu_file_llistxattr
      • mfu_llistxattr
      • daos_llistxattr
    • lgetxattr
      • mfu_file_lgetxattr
      • mfu_lgetxattr
      • daos_lgetxattr
    • lsetxattr
      • mfu_file_lsetxattr
      • mfu_lsetxattr
      • daos_lsetxattr
    • lchown
      • mfu_file_lchown
      • mfu_lchown
      • daos_lchown
        • DFS has owner and group permissions only at the container level so this is just a placeholder to avoid errors.
    • utimensat
      • mfu_file_utimensat
      • mfu_utimensat
      • daos_utimensat

    #closes #404

    Implement --dereference and --no-dereference

    mfu_walk_opts

    • Added dereference option
      • default 0

    mfu_copy_opts

    • Added no-dereference option
      • default 0

    mfu_flist_walk.c -> walk_stat_process

    • Added DEREFERENCE global, similar to existing globals
    • If DEREFERENCE call stat
      • Effectively follows symlinks to files/dirs pointed to
    • Otherwise call lstat
    • Added error message for failed stat/lstat

    mfu_io

    • Altered daos_hash_lookup to set errno on error
      • Adjusted callers to account for this
    • Fixed missing parent lookup in daos_access
    • Added mfu_file_faccessat, mfu_faccessat, daos_faccessat
      • Parameters allow for not following symlinks
    • Renamed daos_stat to daos_lstat to be more accurate
    • Added mfu_file_stat, mfu_stat, daos_stat
      • Similar to lstat, but follows symlinks
      • daos_stat is a WIP, pending DFS library changes

    mfu_util.h

    • Explicitly initialize some DAOS structs to silence erroneous valgrind errors.

    mfu_param_path.c -> mfu_param_path_check_copy

    • Added no_dereference parameter
    • If no_dereference, call faccessat(..., AT_SYMLINK_NOFOLLOW)
      • Essentially allows broken links to be copied
    • Otherwise, call access

    dcp.c

    • Added --dereference
      • "-L"
    • Added --no-dereference
      • "-l" (short name hidden)

    dsync.c

    • Added --dereference
      • "-L"
    • Added --no-dereference
      • "-x" (short name hidden)

    dwalk.c

    • Added --dereference

    #closes #29

    Signed-off-by: Dalton Bohning [email protected]

  • daos: add support for copying daos containers at obj level

    daos: add support for copying daos containers at obj level

    This adds support for copying any type of daos container in parallel.

    * add obj_id_lo and obj_id_hi to elem_t struct
    * add total_oids field to flist
    * add a pack and unpack call for obj_id_lo and obj_id_hi
    * create mfu_flist_global_oid_size to return total oids
    * add boolean is_posix_copy for non-posix copies
    * adds daos_obj_copy function and helper functions
    

    This copy starts by retrieving all obj ids on rank 0, then it uses the mfu_flist_spread function to distribute all obj ids evenly across ranks. After the obj ids are distributed, the obj ids are copied using the local flist.

    Also, daos will split the anchor used to retrive all obj ids and enumerate the dkeys, akeys, and records. There is work that still needs to be done from the daos level to split the anchor for the object iterator table used to retrieve obj ids. In the case that a large record extent needs to be copied, daos already chunks the record extents using their index ranges. Each record is of a fixed size, where multiple records is a record extent.

    Signed-off-by: Danielle Sikich [email protected]

  • dwalk: add default file histogram

    dwalk: add default file histogram

    When file_histogram option is used this creates a file histogram of all walked files. This option does not require any range inputs. It will give the user a count of files in each bin. The bins are created based on the max file size. So if the max file size is between 2^20 - 2^30, then this is the last bin. The bins start with the first bin for 0 byte files, then goes up from 2^10, 2^20, 2^30, etc. The bins are created dynamically based on the max file size.

  • daos: add daos-serialize and daos-deserialize tools

    daos: add daos-serialize and daos-deserialize tools

    * add optional HDF5_SUPPORT to build
    * add helper functions to mfu_daos
    * daos-serialize serializes daos containers into hdf5
      files, for long term storage on a POSIX filesystem.
    * daos-deserialize deserializes an hdf5 file by
      restoring it back into a new daos container
    

    daos-deserialize can only be used when daos-serialize was the tool used to create the hdf5 file. The serialize tool evenly distributes the containers object ids among ranks, and each rank then writes to its own HDF5 file. The deserialize tool evenly distributes the HDF5 files among ranks.

    examples: mpirun -np 2 daos-serialize -v /pool/cont mpirun -np 2 daos-deserialize -v --pool h5file h5file ...

    TODO: add obj level statistics after dsync obj level support PR is merged TODO: investigate using libcircle for distribution of files among ranks in daos-deserialize

    Signed-off-by: Danielle Sikich [email protected]

  • feeding binary cache file from dfind or dwalk to drm causes seg fault

    feeding binary cache file from dfind or dwalk to drm causes seg fault

    Tried with both 0.10 and current master. Same behavior. Host is Centos 7

    1. gcc/8.2.0 2) openmpi/4.0.3 3) cmake/3.13.2

    Trying to build lists of files for scratch purge and I'm having trouble passing data between tools like dfind and dwalk. I notice if I strings the cache file it's storing much more data than the single entry my test data should. Though not knowing the format not sure but I'm seeing file paths outside the path i'm running in. If I use text output or print to screen the results are what I would expect.

    mpirun -np 4 ~/mpifileutils/build/src/dwalk/dwalk --output /tmp/output -p .

     mpirun -np 4 ~/mpifileutils/build/src/drm/drm --input /tmp/output --dryrun
    [2020-05-24T20:14:01] Reading from input file: /tmp/output
    [gl-login1:58972:0:58972] Caught signal 11 (Segmentation fault: address not mapped to object at address (nil))
    ==== backtrace (tid:  58972) ====
     0 0x0000000000010e52 mfu_flist_compute_depth()  ???:0
     1 0x0000000000021efd list_elem_decode()  mfu_flist_io.c:0
     2 0x0000000000022325 list_insert_decode()  mfu_flist_io.c:0
     3 0x0000000000022838 read_cache_variable()  mfu_flist_io.c:0
     4 0x0000000000023ea7 mfu_flist_read_cache()  ???:0
     5 0x0000000000401d18 main()  ???:0
     6 0x0000000000022505 __libc_start_main()  ???:0
     7 0x00000000004013c9 _start()  ???:0
    =================================
    
  • drm: add traceless mode support

    drm: add traceless mode support

    Add traceless mode support to drm (-T/--traceless), if the option is enabled, unlink file/dir/link will not change its parent dir's timestamp (atiem/mtime), it is useful if some guy want to remove the extra files under dst dir after a dcp, but not change the timestamp of the dst filesystem tree, in order to keep the same as src.

    Signed-off-by: Gu Zheng [email protected]

  • Reporting job progress during DCP transfer

    Reporting job progress during DCP transfer

    During our data migration from ORNL Spider 1 file system to Spider 2, it is not uncommon to have a transfer job that last more than a day. One issue user/Ops guys wonders is: how much has been transferred so far, and how much is left? Is there a solution to this problem? Thanks.

  • Make chunksize/blocksize runtime option

    Make chunksize/blocksize runtime option

    This changeset makes both chunksize and blocksize to be a runtime option instead of a compile time option. It provides two command line options:

    -k chunksize (MB unit, default is 1MB as it is) -b blockxize (MB unit, default is 1MB as it is)

    In my test, with 500GB dataset with 5 100GB file (large file case), using default setting, I am getting 392 MB/s average transfer speed in 3 test runs using runtime option -k 16 -b 4, I got 709 MB/s average transfer speed in 3 test runs.

    So the improvement seems to be tangible, and changes to the system to be minimal. ORNL is putting these changes into further testing.

    Please let me know if this is something of interests to merge.

    Best,

    Feiyi

  • Inclusion of parallel compression and decompression based on bzip2

    Inclusion of parallel compression and decompression based on bzip2

    I am working with Dr. Feiyi Wang and am in the process of developing parallel compression and decompression using libcircle and bzip2. Could I have permission to merge my changes? There are also changes to enable an user to use dtar with compression and/ or decompression. This makes some changes to dtar, but will not affect simple dtar or untar. It will use an additional -j flag to use compression.

  • [Request] Add exclude option to dtar

    [Request] Add exclude option to dtar

    I would like to be able to exclude files following a pattern, like tar does: mpirun -np 128 dtar --verbose --exclude="n*" --create --file /tmp/toto.tar * This is like feature request https://github.com/hpc/mpifileutils/issues/304 for dtar

  • Feature request : dtar append mode

    Feature request : dtar append mode

    When one runs dtar inside Slurm for instance, and when the allocated time is not sufficient, the tar file is not complete. I would be happy to have some append mode, where repeating the same command exactly would complete the tar.

  • dcp: User ID and Group ID

    dcp: User ID and Group ID

    As part of Near Node Flash (NNF) Data Movement implementation, there is the need to specify the User ID and Group ID when running dcp (explained here).

    For dcp, this work seems pretty straightforward to add with -U [uid] and -G [gid].

    For example dcp -U 1001 -G 1002 file.in file.out

    Does this work for everyone? Any other comments before I begin?

  • ddup hashing and openssl 3.x

    ddup hashing and openssl 3.x

    Was looking through the code to see if there was a less computationally expensive way to handle duplicate files (SHA256 is unnecessary IMO, but that's a separate issue/PR) and as it turns out openssl 3.x drops support for the low level interface to hash functions.

    Considering openssl 1.1.1 is EOL in under a year, this may be worth looking at. From what I'm reading, the correct functions are in evp.h per https://stackoverflow.com/questions/69806220/advice-needed-for-migration-of-low-level-openssl-api-to-high-level-openssl-apis

  • Add O_NOATIME to the flags for file open so the source file's atime i…

    Add O_NOATIME to the flags for file open so the source file's atime i…

    …s preserved (if the platform supports it). Signed-off-by: Doug Johnson [email protected].

    Lightly tested dsync on Linux (Red Hat 7.9). Not sure if this covers the bases for what's needed to reliably not change the source file's atime. I went off a comment src/common/mfu_flist.h:29 where the _GNU_SOURCE define appears to have been intended to be used for this purpose. I checked the open and closed issues and pull requests to see if this was ever discussed, surprisingly absent. This feature would be useful for those migrating data and that use atime for purging or migration purposes: the source file's atime needs to be preserved in the interregnum between when the copy is initiated, subsequent invocations of dsync, and the cutover to the new location of the data is performed.

Utilities and common code for use with raylib

Utilities and shared components for use with raylib

Dec 1, 2022
MacFlim flim player source code and utilities

MacFlim Video player source code Please do not barf on code quality. It was not in releasable state, but people wanted to use it. You may even be one

Jan 1, 2023
Isaac ROS common utilities and scripts for use in conjunction with the Isaac ROS suite of packages.

Isaac ROS Common Isaac ROS common utilities and scripts for use in conjunction with the Isaac ROS suite of packages. Docker Scripts run_dev.sh creates

Jan 8, 2023
Panda - is a set of utilities used to research how PsExec encrypts its traffic.

Panda Panda - is a set of utilities used to research how PsExec encrypts its traffic. Shared library used to inject into lsass.exe process to log NTLM

Jul 17, 2022
Dead by Daylight utilities created while researching

DeadByDaylight Research material and PoC for bugs found during the reversal of the game Dead by Daylight. All information provided is for educational

Dec 26, 2022
Utilities to extract secrets from 1Password

1PasswordSuite Blog https://posts.specterops.io/1password-secret-retrieval-methodology-and-implementation-6a9db3f3c709 1PasswordExtract This .NET appl

Dec 7, 2022
cavi is an open-source library that aims to provide performant utilities for closed hierarchies (i.e. all class types of the hierarchy are known at compile time).

cavi cavi is an open-source library that aims to provide performant utilities for closed hierarchies (i.e. all class types of the hierarchy are known

Mar 9, 2022
personal organization utilities

orgutils: Personal Organization Utilities orgutils are a set of utilities for personal and project organization. Each program has

Dec 8, 2021
A combined suite of utilities for manipulating binary data files.

BinaryTools A combined suite of utilities for manipulating binary data files. It was developed for use on Windows but might compile on other systems.

Oct 1, 2022
Utilities for use in a DPP based discord bot

DPPUtils NOTE: This repo is in development, use these utilities at your own risk Numerous utilities for use in your DPP bot. List of Utilities Youtube

Nov 5, 2022
Header-only lock-free synchronization utilities (one writer, many readers).

stupid Header-only lock-free synchronization utilities (one writer, many readers). No queues Base functionality The base functionality of this library

Nov 28, 2022
provide SFML Time utilities in pure C++20, no dependencies

SFML-Time-utilities-without-SFML provide SFML Time utilities in pure C++20, no dependencies Example int main() { Clock clock; Sleep(1000);

Apr 28, 2022
convert elf file to single c/c++ header file

elf-to-c-header Split ELF to single C/C++ header file

Nov 4, 2021
A ring buffer designed to work with embedded devices, does not use heap allocations.
A ring buffer designed to work with embedded devices, does not use heap allocations.

Embedded Ring Buffer This is a header only ring buffer that is designed to work on embedded devices, it is able to handle non-blocking ISR spooling

Dec 2, 2022
My CS:GO cheat, written with performance in mind.

sinclair_csgo My CS:GO cheat, written with performance in mind. Currently in development, and I plan to keep this as such!

Apr 9, 2022
Cheap: customized heaps for improved application performance.

Cheap: a malloc/new optimizer by Emery Berger About Cheap Cheap is a system that makes it easy to improve the performance of memory-intensive C/

Dec 6, 2022
CacheLib is a C++ library providing in-process high performance caching mechanism.
CacheLib is a C++ library providing in-process high performance caching mechanism.

Pluggable in-process caching engine to build and scale high performance services

Jan 5, 2023
Simple and lightweight pathname parser for C. This module helps to parse dirname, basename, filename and file extension .
Simple and lightweight pathname parser for C. This module helps to parse dirname, basename, filename and file extension .

Path Module For C File name and extension parsing functionality are removed because it's difficult to distinguish between a hidden dir (ex: .git) and

Feb 25, 2022
Locate the current executable and the current module/library on the file system

Where Am I? A drop-in two files library to locate the current executable and the current module on the file system. Supported platforms: Windows Linux

Dec 27, 2022