Ntfs-3g - NTFS-3G Safe Read/Write NTFS Driver

INTRODUCTION
============

The NTFS-3G driver is an open source, freely available read/write NTFS driver 
for Linux, FreeBSD, macOS, NetBSD, OpenIndiana, QNX and Haiku. It provides 
safe and fast handling of the Windows XP, Windows Server 2003, Windows 2000, 
Windows Vista, Windows Server 2008, Windows 7, Windows 8, Windows Server 2012,
Windows Server 2016, Windows 10 and Windows Server 2019 NTFS file systems.

The purpose of the project is to develop, quality assurance and support a 
trustable, featureful and high performance solution for hardware platforms 
and operating systems whose users need to reliably interoperate with NTFS. 
Besides this practical goal, the project also aims to explore the limits 
of the hybrid, kernel/user space filesystem driver approach, performance, 
reliability and feature richness per invested effort wise.

Besides the common file system features, NTFS-3G has support for file 
ownership and permissions, POSIX ACLs, junction points, extended attributes 
and creating internally compressed files (parameter files in the directory
.NTFS-3G may be required to enable them). The new compressed file formats
available in Windows 10 can also be read through a plugin. 

News, support answers, problem submission instructions, support and discussion 
forums, and other information are available on the project web site at

	https://github.com/tuxera/ntfs-3g/wiki

The project has been funded, supported and maintained since 2008 by Tuxera:

	https://tuxera.com


LICENSES
========

All the NTFS related components: the file system drivers, the ntfsprogs
utilities and the shared library libntfs-3g are distributed under the terms
of the GNU General Public License as published by the Free Software
Foundation; either version 2 of the License, or (at your option) any later
version. See the included file COPYING.

The fuse-lite library is distributed under the terms of the GNU LGPLv2.
See the included file COPYING.LIB.


QUICK INSTALLATION
==================

Most distributions have an up-to-date NTFS-3G package ready for use, and
the recommended way is to install it.

If you need some specific customization, you can compile and install from
the released source code. Make sure you have the basic development tools
and the kernel includes the FUSE kernel module. Then unpack the source
tarball and type:  

	./configure
	make
	make install      # or 'sudo make install' if you aren't root.

Please note that NTFS-3G doesn't require the FUSE user space package any
more.

The list of options for building specific configurations is displayed by
typing :

	./configure --help

Below are a few specific options to ./configure :
	--disable-ntfsprogs : do not build the ntfsprogs tools,
	--enable-extras : build more ntfsprogs tools,
	--disable-plugins : disable support for plugins
	--enable-posix-acls : enable support for Posix ACLs
	--enable-xattr-mappings : enable system extended attributes mappings
	--with-fuse=external : use external fuse (overriding Linux default)

There are also a few make targets for building parts :
	make libntfs : only build the libntfs-3g library
	make libs : only build libntfs-3g (and libfuse-lite, if relevant)
	make drivers : only build drivers and libraries, without ntfsprogs
	make ntfsprogs : only build ntfsprogs and libntfs-3g, without drivers


USAGE
=====

If there was no error during installation then the NTFS volume can be
read-write mounted for everybody the following way as the root user 
(unmount the volume if it was already mounted, and replace /dev/sda1 
and /mnt/windows, if needed):

	mount -t ntfs-3g /dev/sda1 /mnt/windows
or
	ntfs-3g /dev/sda1 /mnt/windows

Please see the ntfs-3g manual page for more options and examples.

You can also make NTFS to be mounted during boot by putting the below 
line at the END(!) of the /etc/fstab file:

	/dev/sda1 /mnt/windows ntfs-3g defaults 0 0


TESTING WITHOUT INSTALLING
=========================

Newer versions of ntfs-3g can be tested without installing anything and
without disturbing an existing installation. Just configure and make as
shown previously. This will create the scripts ntfs-3g and lowntfs-3g
in the src directory, which you may activate for testing:

	./configure
	make

then, as root:
	src/ntfs-3g [-o mount-options] /dev/sda1 /mnt/windows

And, to end the test, unmount the usual way:
	umount /dev/sda1


NTFS UTILITIES
==============

The ntfsprogs directory includes utilities for doing all required tasks to
NTFS partitions. In general, just run a utility without any command line
options to display the version number and usage syntax.

The following utilities are so far implemented:

ntfsfix - Attempt to fix an NTFS partition and force Windows to check NTFS.

mkntfs - Format a partition with the NTFS filesystem.  See man 8 mkntfs for
command line options.

ntfslabel - Display/change the label of an NTFS partition.  See man 8 ntfslabel
for details.

ntfsundelete - Recover deleted files from an NTFS volume.  See man 8
ntfsundelete for more details.

ntfsresize - Resize NTFS volumes.  See man 8 ntfsresize for details.

ntfsclone - Efficiently create/restore an image of an NTFS partition.  See
man 8 ntfsclone for details.

ntfscluster - Locate the owner of any given sector or cluster on an NTFS
partition.  See man 8 ntfscluster for details.

ntfsinfo - Show some information about an NTFS partition or one of the files
or directories within it.  See man 8 ntfsinfo for details.

ntfsrecover - Recover updates committed by Windows but interrupted before
being synced.

ntfsls - List information about files in a directory residing on an NTFS
partition.  See man 8 ntfsls for details.

ntfscat - Concatenate files and print their contents on the standard output.

ntfscp - Overwrite files on an NTFS partition.

ntfssecaudit - Audit the security metadata.

ntfsusermap - Assistance for building a user mapping file.
Comments
  • Fuse not needed, but building fails without it on Monterey

    Fuse not needed, but building fails without it on Monterey

    configure: error: FUSE >= 2.6.0 was not found. Either older FUSE is still present, or FUSE is not fully installed (e.g. fuse, libfuse, libfuse2, libfuse-dev, etc packages). Source code: http://fuse.sf.net
    
  • Minor autoconf improvements for portability

    Minor autoconf improvements for portability

    These two commits improve the portability of ntfs-3g on systems that do not use the standard Linux FHS (e.g. NixOS, Gobo, GuixSD, etc.).

    • The first commit is a bugfix: ntfs-3g already uses rootsbindir/sbindir in most places but not everywhere.
    • The second commit is a set of new configuration options for specifying the paths to /sbin helpers used at runtime by the ntfs-3g programs.
  • Copy NTFS files preserving all their attributes ?

    Copy NTFS files preserving all their attributes ?

    Hi,

    I've been looking for a way to copy NTFS directories preserving all attributes, ACLs, permissions, reparse points etc (surprisingly hard it turns out, short of cloning the whole partition).

    I came across the wiki page on Extended Attributes which gives a bit of hope :
    Looks like the ntfscp.c and ntfscp.sh additional tools do just that (!)

    What's the status on these ? Why are they not part of ntfs-3g ? Looks like that would be useful to a lot of people.

    (As a side-note it was really confusing intially that it has the same name as ntfs-3g's ntfscp which seem something else entirely)

  • Migration from Sourceforge is not complete

    Migration from Sourceforge is not complete

    Dear @tuxera team, @jpandre,

    It will be nice to remove old parts like:

    • https://sourceforge.net/p/ntfs-3g/cvs/
    • https://sourceforge.net/p/ntfs-3g/_list/git
    • https://sourceforge.net/p/ntfs-3g/ntfs-3g/ci/edge/tree/
    • https://sourceforge.net/p/ntfs-3g/old_ntfs-3g_ntfsprogs/ci/PERMISSION_HANDLING_BRANCH/tree/
    • https://sourceforge.net/p/ntfs-3g/pjd-fstest/ci/master/tree/

    Can you add a "move" box like https://sourceforge.net/projects/handbrake for example.

    • https://sourceforge.net/projects/ntfs-3g/

    To disable/delete tabs, go here: https://sourceforge.net/projects/ntfs-3g/ To add a new tab, go here: https://sourceforge.net/projects/ntfs-3g/ To change informations, go here: https://sourceforge.net/p/ntfs-3g/admin/overview To export, go here: https://sourceforge.net/p/ntfs-3g/admin/export

    The new place can be added in:

    • "Moved Project to", here: https://sourceforge.net/p/ntfs-3g/admin/overview
    • "Preferred Support Page (for users of your project)", here: https://sourceforge.net/p/ntfs-3g/admin/overview
    • A new tab, go here: https://sourceforge.net/projects/ntfs-3g/

    The website link is not good in Sourceforge options (better to change with github.com too):

    • http://www.tuxera.com/community/ntfs-3g-download/

    Thanks in advance.

  • OneDrive: unsupported reparse tag 0x9000701a

    OneDrive: unsupported reparse tag 0x9000701a

    I have a drive with an NTFS partition that OneDrive on Windows 10 is configured to sync to (I've tagged the whole onedrive folder with "keep offline" so everything is downloaded locally). However, when I use the latest version of ntfs-3g (2022.5.17) the folder shows up as a broken symlink, with the target listed as

    unsupported reparse tag 0x9000701a

    I also tried mounting the drive with the old version of ntfs-3g included with the distro (2017.3.23AR.3-3ubuntu1.2) and while that allowed me to see the contents of the folder, all files were broken links with

    unsupported reparse point

    I tried running ntfsinfo and got this debug information:

    > sudo ntfsprogs/ntfsinfo --file "folder/OneDrive (synced)/OneDrive - [my organization]" /dev/sdb1
    Dumping Inode 242195 (0x3b213)
    Upd. Seq. Array Off.:	 48 (0x30)
    Upd. Seq. Array Count:	 3 (0x3)
    Upd. Seq. Number:	 43 (0x2b)
    LogFile Seq. Number:	 0x4117129b
    MFT Record Seq. Numb.:	 5 (0x5)
    Number of Hard Links:	 2 (0x2)
    Attribute Offset:	 56 (0x38)
    MFT Record Flags:	 IN_USE DIRECTORY 
    Bytes Used:		 712 (0x2c8) bytes
    Bytes Allocated:	 1024 (0x400) bytes
    Next Attribute Instance: 28 (0x1c)
    MFT Padding:	00 00 
    Dumping attribute $STANDARD_INFORMATION (0x10) from mft record 242195 (0x3b213)
    	Resident: 		 Yes
    	Attribute flags:	 0x0000
    	Attribute instance:	 0 (0x0)
    	Data size:		 72 (0x48)
    	Resident flags:		 0x00
    	File Creation Time:	 Tue Jun 28 00:19:56 2022 UTC
    	File Altered Time:	 Wed Jun 29 04:17:41 2022 UTC
    	MFT Changed Time:	 Wed Jun 29 04:17:41 2022 UTC
    	Last Accessed Time:	 Wed Jun 29 07:01:07 2022 UTC
    	File attributes:	 READONLY ARCHIVE REPARSE_POINT UNKNOWN: 0x00080000 (0x00080421)
    	Maximum versions:	 0 
    	Version number:		 0 
    	Class ID:		 0 
    	User ID:		 0 (0x0)
    	Security ID:		 282 (0x11a)
    	Quota charged:		 0 (0x0)
    	Update Sequence Number:	 394371712 (0x1781a280)
    Dumping attribute $ATTRIBUTE_LIST (0x20) from mft record 242195 (0x3b213)
    	Resident: 		 No
    	Attribute flags:	 0x0000
    	Attribute instance:	 11 (0xb)
    	Compression unit:	 0 (0x0)
    	Data size:		 392 (0x188)
    	Allocated size:		 4096 (0x1000)
    	Initialized size:	 392 (0x188)
    Dumping attribute $FILE_NAME (0x30) from mft record 242195 (0x3b213)
    	Resident: 		 Yes
    	Attribute flags:	 0x0000
    	Attribute instance:	 3 (0x3)
    	Data size:		 82 (0x52)
    	Resident flags:		 0x01
    	Parent directory:	 239101 (0x3a5fd)
    	File Creation Time:	 Tue Jun 28 00:19:56 2022 UTC
    	File Altered Time:	 Tue Jun 28 00:19:56 2022 UTC
    	MFT Changed Time:	 Tue Jun 28 00:19:56 2022 UTC
    	Last Accessed Time:	 Tue Jun 28 00:19:56 2022 UTC
    	Allocated Size:		 0 (0x0)
    	Data Size:		 0 (0x0)
    	Filename Length:	 8 (0x8)
    	File attributes:	 I30_INDEX (0x10000000)
    	Namespace:		 DOS
    	Filename:		 'ONEDRI~1'
    Dumping attribute $FILE_NAME (0x30) from mft record 242204 (0x3b21c)
    	Resident: 		 Yes
    	Attribute flags:	 0x0000
    	Attribute instance:	 0 (0x0)
    	Data size:		 136 (0x88)
    	Resident flags:		 0x01
    	Parent directory:	 239101 (0x3a5fd)
    	File Creation Time:	 Tue Jun 28 00:19:56 2022 UTC
    	File Altered Time:	 Tue Jun 28 00:19:56 2022 UTC
    	MFT Changed Time:	 Tue Jun 28 00:19:56 2022 UTC
    	Last Accessed Time:	 Tue Jun 28 00:19:56 2022 UTC
    	Allocated Size:		 0 (0x0)
    	Data Size:		 0 (0x0)
    	Filename Length:	 35 (0x23)
    	File attributes:	 I30_INDEX (0x10000000)
    	Namespace:		 Win32
    	Filename:		 'OneDrive - [my organization]'
    Dumping attribute $DATA (0x80) from mft record 242204 (0x3b21c)
    	Resident: 		 No
    	Attribute name:		 '${3D0CE612-FDEE-43f7-8ACA-957BEC0CCBA0}.SyncRootIdentity'
    	Attribute flags:	 0x0000
    	Attribute instance:	 1 (0x1)
    	Compression unit:	 0 (0x0)
    	Data size:		 272 (0x110)
    	Allocated size:		 4096 (0x1000)
    	Initialized size:	 272 (0x110)
    Dumping attribute $INDEX_ROOT (0x90) from mft record 242195 (0x3b213)
    	Resident: 		 Yes
    	Attribute name:		 '$I30'
    	Attribute flags:	 0x0000
    	Attribute instance:	 13 (0xd)
    	Data size:		 56 (0x38)
    	Resident flags:		 0x00
    	Indexed Attr Type:	 DIRECTORY_I30
    	Collation Rule:		 1 (0x1)
    	Index Block Size:	 4096 (0x1000)
    	Clusters Per Block:	 1 (0x1)
    	Entries Offset:		 16 (0x10)
    	Index Size:		 40 (0x28)
    	Allocated Size:		 40 (0x28)
    	Index header flags:	 0x01
    	Index entries total:	 1
    Dumping attribute $INDEX_ALLOCATION (0xa0) from mft record 242195 (0x3b213)
    	Resident: 		 No
    	Attribute name:		 '$I30'
    	Attribute flags:	 0x0000
    	Attribute instance:	 9 (0x9)
    	Compression unit:	 0 (0x0)
    	Data size:		 32768 (0x8000)
    	Allocated size:		 32768 (0x8000)
    	Initialized size:	 32768 (0x8000)
    	Index entries total:	 171
    	INDX blocks total:	 8
    Dumping attribute $BITMAP (0xb0) from mft record 242195 (0x3b213)
    	Resident: 		 Yes
    	Attribute name:		 '$I30'
    	Attribute flags:	 0x0000
    	Attribute instance:	 10 (0xa)
    	Data size:		 8 (0x8)
    	Resident flags:		 0x00
    Dumping attribute $REPARSE_POINT (0xc0) from mft record 242195 (0x3b213)
    	Resident: 		 Yes
    	Attribute flags:	 0x0000
    	Attribute instance:	 27 (0x1b)
    	Data size:		 116 (0x74)
    	Resident flags:		 0x00
    	Reparse tag:		 0x9000701a (Cloud)
    	Data length:		 108 (0x6c)
    	Data:			 0x01006c0046655270ed47d01a6800000002000a0007000100600000000a000400...
    End of inode reached
    
  • NTFS allows files ending with blank/period when mounted as root but not when mounted as user

    NTFS allows files ending with blank/period when mounted as root but not when mounted as user

    Looking into the issue https://github.com/rdiff-backup/rdiff-backup/issues/579 I noticed a discrepancy in behaviour depending if a NTFS file system is mounted as root or non-root (we're talking about the same file system freshly created with sudo mkfs.ntfs /dev/sda1 on a USB stick):

    $ udisksctl mount -b /dev/sda1
    $ mount | grep sda1
    /dev/sda1 on /run/media/myuser/5291712E0E79C514 type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096,uhelper=udisks2
    $ cd /run/media/myuser/5291712E0E79C514
    $ touch a
    $ touch 'a '
    touch: setting times of 'a ': No such file or directory
    $ touch 'a.'
    touch: setting times of 'a.': No such file or directory
    $ ll
    total 0
    -rwxrwxrwx. 1 myuser myuser 0 May 26 07:44 a
    

    vs.

    $ sudo mount -t ntfs -o nosuid,nodev,uid=1000,gid=1000 /dev/sda1 /mnt  # to make sure the mount options are the same
    $ mount | grep sda1
    /dev/sda1 on /mnt type fuseblk (rw,nosuid,nodev,relatime,user_id=0,group_id=0,default_permissions,allow_other,blksize=4096)
    $ touch a
    $ touch 'a '
    $ touch 'a.'
    $ ll
    total 0
    -rwxrwxrwx. 1 myuser myuser 0 May 26 07:47  a
    -rwxrwxrwx. 1 myuser myuser 0 May 26 07:47 'a '
    -rwxrwxrwx. 1 myuser myuser 0 May 26 07:47  a.
    

    And I can't find any reason for the difference in behaviour.

  • configure.ac: fix bashism in fuse check

    configure.ac: fix bashism in fuse check

    configure scripts need to be runnable with a POSIX-compliant /bin/sh.

    On many (but not all!) systems, /bin/sh is provided by Bash, so errors like this aren't spotted. Notably Debian defaults to /bin/sh provided by dash which doesn't tolerate such bashisms as '=='.

    This retains compatibility with bash.

    Fixes configure warnings/errors like:

    checking Windows OS... no
    ./configure: 13360: test: xinternal: unexpected operator
    checking for pthread_create in -lpthread... yes
    checking Solaris OS... no
    

    Signed-off-by: Sam James [email protected]

  • how to support the other character set, such as GBK

    how to support the other character set, such as GBK

    the system has support "zh_CN.gbk", and i mounted the usb disk by "mount -t ntfs-3g -o locale=zh_CN.gbk /dev/xxx /mount_path". But I still can not create gbk filename. eg: [[email protected]_RV1109:/]# mount -t ntfs-3g -o locale=zh_CN.gbk /dev/sda1 /tmp/udisk/ [[email protected]_RV1109:/]# [[email protected]_RV1109:/]# cd /tmp/udisk/ [[email protected]_RV1109:/tmp/udisk]# touch 你好 touch: cannot touch ''$'\304\343\272\303': Invalid or incomplete multibyte or wide character

  • Autogen warning message.

    Autogen warning message.

    Autogen brings this error message (although it completes successfully and I can't see any negative effects):

    libtoolize: Remember to add 'LT_INIT' to configure.ac.

    The entire console output (including the above message) is here:

    >./autogen.sh
    Running autoreconf --verbose --install --force
    autoreconf: Entering directory `.'
    autoreconf: configure.ac: not using Gettext
    autoreconf: running: aclocal --force -I m4
    autoreconf: configure.ac: tracing
    autoreconf: running: libtoolize --copy --force
    libtoolize: putting auxiliary files in '.'.
    libtoolize: copying file './ltmain.sh'
    libtoolize: putting macros in AC_CONFIG_MACRO_DIRS, 'm4'.
    libtoolize: copying file 'm4/libtool.m4'
    libtoolize: copying file 'm4/ltoptions.m4'
    libtoolize: copying file 'm4/ltsugar.m4'
    libtoolize: copying file 'm4/ltversion.m4'
    libtoolize: copying file 'm4/lt~obsolete.m4'
    libtoolize: Remember to add 'LT_INIT' to configure.ac.
    autoreconf: running: /usr/bin/autoconf --force
    autoreconf: running: /usr/bin/autoheader --force
    autoreconf: running: automake --add-missing --copy --force-missing
    configure.ac:189: installing './compile'
    configure.ac:32: installing './config.guess'
    configure.ac:32: installing './config.sub'
    configure.ac:36: installing './install-sh'
    configure.ac:36: installing './missing'
    Makefile.am: installing './INSTALL'
    libfuse-lite/Makefile.am: installing './depcomp'
    autoreconf: Leaving directory `.'
    >
    

    I compiled NTFS-3G, at my 2 systems here: PCLinux and Devuan.

  • It should be possible to cancel mkfs.ntfs

    It should be possible to cancel mkfs.ntfs

    Example:

    # mkfs.ntfs /dev/sdc1 
    Cluster size has been automatically set to 4096 bytes.
    Initializing device with zeroes:   4%^C^C^C^C^C^C^C^C^C^C^C^C^C^C
    

    And no matter how many Ctrl+C I press, it keeps creating the filesystem on the disk. Shouldn't mkfs.ntfs stop after a Ctrl+C? Isn't it possible to capture SIGINT and stops whatever mkfs.ntfs is doing, please?

    ntfs-3g is Debian's 2021.8.22-3 Linux kernel is 5.15.15

    Thanks!

  • Heap overflow in ntfsck

    Heap overflow in ntfsck

    Hello.

    I have found a vulnerability in the NTFS-3G driver, specifically in the ntfsck tool (see: ntfsprogs/ntfsck.c).

    In the check_file_record function, the update sequence array is applied, but no proper boundary checks are implemented, so the function can write bytes from the update sequence array beyond the buffer being checked.

    The vulnerable code is here:

    usa_ofs = le16_to_cpu(mft_rec->usa_ofs);
    usa_count = le16_to_cpu(mft_rec->usa_count);
    
    [...]
    
    // Remove update seq & check it.
    usa = *(u16*)(buffer+usa_ofs); // The value that should be at the end of every sector.
    assert_u32_equal(usa_count-1, buflen/NTFS_BLOCK_SIZE, "USA length");
    for (i=1;i<usa_count;i++) {
        u16 *fixup = (u16*)(buffer+NTFS_BLOCK_SIZE*i-2); // the value at the end of the sector.
        u16 saved_val = *(u16*)(buffer+usa_ofs+2*i); // the actual data value that was saved in the us array.
    
        assert_u32_equal(*fixup, usa, "fixup");
        *fixup = saved_val; // remove it.
    }
    

    If buflen is 1024, but the update sequence array contains 4 entries (including the first one, which you call usa), the loop will replace bytes 3 times, at the following offsets: buffer+512*1−2 (within the buffer), buffer+512*2−2 (within the buffer), buffer+512*3−2 (beyond the allocated buffer size). (The offset of the first attribute should be set to make room for additional entries in the update sequence array, so the usa_ofs+usa_count <= attrs_offset check is passed.)

    Thus, bytes beyond the allocated buffer can be replaced, this is a heap overflow.

    It should be noted that the assert_u32_equal function just reports the errors, it doesn’t terminate the execution flow.

    Since the ntfsck tool is used in some GNU/Linux distributions (it’s fsck.ntfs in Fedora), I strongly suggest implementing a fix.


    Report date (to info at tuxera dot com): 2021-09-24. No reply. Ping (to info at tuxera dot com): 2021-09-29. No reply.

  • What are the optimal ntfs-3g or kernel parameters/tunables for use with SMR media?

    What are the optimal ntfs-3g or kernel parameters/tunables for use with SMR media?

    You can see from the title that this is NOT a bug. Please flag it as discussion. Maybe, if we reach some result, you can put it in the Wiki.

    Description

    I have searched low and high, but could practically find nothing about what ntfs-3g parameters and/or kernel I/O scheduler tunables to use when dealing with SMR (Shingled Magnetic Recording) media. In my situation, the SMR media are 2.5" external USB3 HDDs (not SSDs), encrypted with Truecrypt or Veracrypt. When I try to initially fill such a disk with backups, basically consisting of thousands of small (500K-2M) files, with, say, rsync, the write throughput slows down after the first 10-20 Gigabytes to a crawling 1MB/sec (one _Mega_byte per second)!

    System

    Gentoo Linux, 5.4.196 kernel, lots of RAM, ntfs3g-2022.10.3.

    Suggestions

    Searching around, I found (I should say stumbled, as it was rather by chance - I was searching for "How to increase file cache with ntfs-3g", rather than "Which I/O scheduler tunables to use with ntfs-3g and SMR media", which brings up nothing of interest in my search engine of choice), the following suggestion in Can I configure my Linux system for more aggressive file system caching?:

    
    #!/bin/bash
    modprobe bfq
    for d in /sys/block/sd?; do
      # HDD (tuned for Seagate SMR drive)
      echo bfq >"$d/queue/scheduler"
      echo 4 >"$d/queue/nr_requests"
      echo 32000 >"$d/queue/iosched/back_seek_max"
      echo 3 >"$d/queue/iosched/back_seek_penalty"
      echo 80 >"$d/queue/iosched/fifo_expire_sync"
      echo 1000 >"$d/queue/iosched/fifo_expire_async"
      echo 5300 >"$d/queue/iosched/slice_idle_us"
      echo 1 >"$d/queue/iosched/low_latency"
      echo 200 >"$d/queue/iosched/timeout_sync"
      echo 0 >"$d/queue/iosched/max_budget"
      echo 1 >"$d/queue/iosched/strict_guarantees"
    done
    

    However, this is not directly applicable in my case: it is for the bfq scheduler, while I have the mq-deadline scheduler.

    What I tried

    For me the tunables in question were the tunables of mq-deadline:

    for l in c d; do  echo "+++"; echo "sd$l"; echo "+++"; \
    echo ''; d=/sys/block/sd$l/queue; echo -n "Scheduler: "; \
    cat "$d/scheduler"; echo '-----------------------------'; \
    echo -n "nr_requests: "; cat $d/nr_requests; echo '';\
     for v in $(ls $d/iosched/); do echo -n "$v:";\
     cat $d/iosched/$v; echo ''; done; echo ''; echo ''; done
    
    +++
    sdc
    +++
    
    Scheduler: [mq-deadline] none
    -----------------------------
    nr_requests: 2
    
    fifo_batch:16
    
    front_merges:1
    
    read_expire:500
    
    write_expire:5000
    
    writes_starved:2
    
    
    
    +++
    sdd
    +++
    
    Scheduler: [mq-deadline] none
    -----------------------------
    nr_requests: 2
    
    fifo_batch:16
    
    front_merges:1
    
    read_expire:500
    
    write_expire:5000
    
    writes_starved:2
    

    I have tried to set them to higher values:

    fifo_batch=65536
    nr_requests=32
    write_expire=30000
    

    I also set the kernel virtual memory management tunables:

    dirty_background_ratio=30
    dirty_ratio=50
    dirty_expire_centisecs=72000
    
    sysctl vm.dirty_background_ratio=$dirty_background_ratio
    sysctl vm.dirty_ratio=$dirty_ratio
    sysctl vm.dirty_expire_centisecs=$dirty_expire_centisecs
    

    The guiding idea behind all these settings was: increase file cache, keep the copied files in the cache as long as possible, increase the write batch - in general: write as many bytes as you can in one batch to the drive. The hope was that the more data I would write at once, the more consecutive I/O writes I would send to the drive, increasing the chance that the "zones" (pieces of 256MB consecutive data) would be written sequentially. Remember, the drives may be encrypted, which increases the entropy (data that were neighbors in the source filesystem, will be scattered around randomly). Since the data looks random to the drive, I hoped to gather as many consecutive ones as possible by gathering them in the file cache or a "write batch".

    In line with the above considerations I also

    • mounted the drive(s) with the big_writes mount option of ntfs-3g
    • tried to increase the commit interval, but found no option analogous to the commit mount option of ext4, or sysctl fs.xfs.xfssyncd_centisecs=XXX for XFS
    • enabled write-back:
    write_cache='write back'
    echo "$write_cache" > /sys/block/$dev/queue/write_cache
    

    Nothing worked. While big_writes did write large amounts of data at once, after the first 10, 20 GB, throughput grinded to a halt, with the drive being busy, while rsync stopped sending data, waiting for the drive's O.K. Average write throughput: ~1MB/sec.

    At this point I am out of ideas. What would you suggest? Is anything that can be done to increase write performance of drive-managed SMR disks with ntfs-3g?

  • ntfsfix can not correct MFT errors

    ntfsfix can not correct MFT errors

    After plug out sdcard and plug in again, though ntfsfix print ok and mount normally, but from logs MFT errors occurred:

    ntfs3: sdg1: Inode r=1d is not in use! ntfs3: sdg1: MFT: r=3c, expect seq=1 instead of 4! ntfs3: sdg1: MFT: r=3c, expect seq=2 instead of 4! ntfs3: sdg1: MFT: r=3c, expect seq=3 instead of 4! ...

    and the sdcard can not use normally, can not create or remove files. are there any way ntfsfix can correct these errors, how i can fix these errors from linux?

  • Memory access error with ntfssecaudit on Ubuntu

    Memory access error with ntfssecaudit on Ubuntu

    I get this error on Ubuntu 22.04:

    $ LC_ALL=C ntfssecaudit /media/pitpat/System/Dokumente\ und\ Einstellungen/Administrator/Desktop/Suse\ Euro-Bilanz.xls
    ntfssecaudit 1.5.0 : NTFS security data auditing
    File /media/pitpat/System/Dokumente und Einstellungen/Administrator/Desktop/Suse Euro-Bilanz.xls
    Windows attrib : 0x4020
    ** Could not find the user mapping file
    Windows owner S-1-5-21-746137067-2049760794-682003330-500
    Windows group S-1-5-21-746137067-2049760794-682003330-513
    Interpreted Unix owner 0, group 0, mode 0700
    Speicherzugriffsfehler (Speicherabzug geschrieben)
    

    My worry is about the last line in the output (unfortunately LC_ALL=C didn't translate it): German: Speicherzugriffsfehler (Speicherabzug geschrieben) English: Memory access error (Memory extract written)

    Additionally I can not open the file for read or copy it to somewhere (permission denied (13) ). I suspect, it is a consequence of the memory access error. There are many files on that volume I want to backup with that error, but 90 % of the files copy without problems. So additionally I'm asking for a quick workaround to backup the files. As the original permissions don't matter, may be there is a way just to delete the existing ACLs somehow?

    ntfsfix or chkdsk /F with Windows didn't help.

    Here the verbose output with sudo (this time without "Memory extract written"):

    $ LC_ALL=C sudo ntfssecaudit -vv /media/pitpat/System/Dokumente\ und\ Einstellungen/Administrator/Desktop/Suse\ Euro-Bilanz.xls 
    ntfssecaudit 1.5.0 : NTFS security data auditing
    File /media/pitpat/System/Dokumente und Einstellungen/Administrator/Desktop/Suse Euro-Bilanz.xls
            000000  01000484 90000000 ac000000 00000000
            000010  14000000 02007c00 04000000 00101800
            000020  ff011f00 01020000 00000005 20000000
            000030  20020000 00102400 ff011f00 01050000
            000040  00000005 15000000 eb25792c 1ade2c7a
            000050  828ba628 f4010000 00102400 ff011f00
            000060  01050000 00000005 15000000 a43b9221
            000070  b7e88c6d 8af61665 ec030000 00101400
            000080  ff011f00 01010000 00000005 12000000
            000090  01050000 00000005 15000000 eb25792c
            0000a0  1ade2c7a 828ba628 f4010000 01050000
            0000b0  00000005 15000000 eb25792c 1ade2c7a
            0000c0  828ba628 01020000
    Computed hash : 0x81348146
    Windows attrib : 0x4020
    ** Could not find the user mapping file
    Global header
        revision 1
        flags    0x8404
            DACL present
            DACL was inherited automatically
            self relative descriptor
        Off USID 0x90
        Off GSID 0xac
        Off SACL 0x0
        Off DACL 0x14
    Owner SID
        Administrator SID
        O:hex S-1-5-15-2c7925eb-7a2cde1a-28a68b82-1f4
        O:dec S-1-5-21-746137067-2049760794-682003330-500
    Group SID
        Domain Users SID
        G:hex S-1-5-15-2c7925eb-7a2cde1a-28a68b82-201
        G:dec S-1-5-21-746137067-2049760794-682003330-513
    DACL
        revision 2
        ACL size 124
        ACE cnt  4
        ACE 1 at 0x1c
            type     0
                Access allowed
            flags    0x10
                ACE was inherited
            Size     0x18
            Acc rgts 0x1f01ff
                Obj specific acc rgts 0x1ff
                    Read data
                    Write data
                    Append data
                    Read EA
                    Write EA
                    Execute
                    Read attributes
                    Write attributes
                standard acc rgts 0x1f
                    Delete
                    Read control
                    Write DAC
                    Write owner
                    Synchronize
            SID at 0x24
                Administrators SID
                hex S-1-5-20-220
                dec S-1-5-32-544
            Summary : grant rwx applied
        ACE 2 at 0x34
            type     0
                Access allowed
            flags    0x10
                ACE was inherited
            Size     0x24
            Acc rgts 0x1f01ff
                Obj specific acc rgts 0x1ff
                    Read data
                    Write data
                    Append data
                    Read EA
                    Write EA
                    Execute
                    Read attributes
                    Write attributes
                standard acc rgts 0x1f
                    Delete
                    Read control
                    Write DAC
                    Write owner
                    Synchronize
            SID at 0x3c
                Administrator SID
                hex S-1-5-15-2c7925eb-7a2cde1a-28a68b82-1f4
                dec S-1-5-21-746137067-2049760794-682003330-500
            Summary : grant rwx applied to owner
        ACE 3 at 0x58
            type     0
                Access allowed
            flags    0x10
                ACE was inherited
            Size     0x24
            Acc rgts 0x1f01ff
                Obj specific acc rgts 0x1ff
                    Read data
                    Write data
                    Append data
                    Read EA
                    Write EA
                    Execute
                    Read attributes
                    Write attributes
                standard acc rgts 0x1f
                    Delete
                    Read control
                    Write DAC
                    Write owner
                    Synchronize
            SID at 0x60
                Local user-1004 SID
                hex S-1-5-15-21923ba4-6d8ce8b7-6516f68a-3ec
                dec S-1-5-21-563231652-1837951159-1696003722-1004
            Summary : grant rwx applied
        ACE 4 at 0x7c
            type     0
                Access allowed
            flags    0x10
                ACE was inherited
            Size     0x14
            Acc rgts 0x1f01ff
                Obj specific acc rgts 0x1ff
                    Read data
                    Write data
                    Append data
                    Read EA
                    Write EA
                    Execute
                    Read attributes
                    Write attributes
                standard acc rgts 0x1f
                    Delete
                    Read control
                    Write DAC
                    Write owner
                    Synchronize
            SID at 0x84
                Local System SID
                hex S-1-5-12
                dec S-1-5-18
            Summary : grant rwx applied
    No SACL
    Windows owner S-1-5-21-746137067-2049760794-682003330-500
    Windows group S-1-5-21-746137067-2049760794-682003330-513
    Interpreted Unix owner 0, group 0, mode 0700
    Speicherzugriffsfehler
    
  • Mounting takes a long time with big UserMapping file

    Mounting takes a long time with big UserMapping file

    Hi,

    I try to get correct user mappings for a NTFS partition (inside VHDX backup) but so far failed. First I tried running the gauntlet with ntfsusermap but out of 5 folders I tested afterwarts only one had correct ownership. I love that this tool gives you no option to correct mistakes without running to the end. Then I tried to get the user mapping with LDAPsearch from our AD and had build a UserMapping with around 3300 entries. The mount command now hangs on me since 30 minutes. Don't know yet if it will succeed in the end. Also I'm pretty confused why I can't find a way to make NTFS-3G look up SIDs through Samba/Winbind which is configured and working fine on the workstation in question.

    Issues

    1. I think I overwelmed the thing with my UserMapping which I consider a bug.
    2. NTFS-3G doesn't seem to cooperate at all with Samba. Winbind already has all the information which NTFS-3G needs for user mapping. What can I do about that?

    Sorry for mixing up multiple problems here.

    Br, Thomas

    EDIT: Gave up waiting for mount because of fear of kernel mode memory corruptions.

  • List xattrs in system namespace

    List xattrs in system namespace

    Is there a reason that ntfs_fuse_listxattr_common does not list the virtual xattrs (eg. system.ntfs_attrib)? Its confusing that, for instance, system.ntfs_attrib can have a value as returned by getxattr, but it will not show up in listxattr. If its undesirable for the xattrs to be listed in some cases, perhaps there can be an option when mounting to enable/disable the listing of these xattrs.

RemixDB: A read- and write-optimized concurrent KV store. Fast point and range queries. Extremely low write-amplification.

REMIX and RemixDB The REMIX data structure was introduced in paper "REMIX: Efficient Range Query for LSM-trees", FAST'21. This repository maintains a

Dec 3, 2022
"SaferCPlusPlus" is essentially a collection of safe data types intended to facilitate memory and data race safe C++ programming

A collection of safe data types that are compatible with, and can substitute for, common unsafe native c++ types.

Nov 24, 2022
Haxe native extension to read and write windows clipboard.

Haxe Clipboard This is a native library to read and write clipboard data from Haxe. It uses Ammer to generate bindings. Note: This is a Windows only l

Nov 11, 2022
This software brings you the possibility to Read and Write the internal Flash of the Nordic nRF52 series with an ESP32
This software brings you the possibility to Read and Write the internal Flash of the Nordic nRF52 series with an ESP32

ESP32 nRF52 SWD flasher This software brings you the possibility to Read and Write the internal Flash of the Nordic nRF52 series with an ESP32 using t

Dec 31, 2022
This is a set of utilities that allow you to read, write or erase SPI flash chips using a Raspberry Pi Pico (RP2040) chip.

Pico SPI Utilities This is a set of utilities that allow you to read, write or erase SPI flash chips using a Raspberry Pi Pico (RP2040) chip. While th

Aug 7, 2022
Android PoC to read/write Huawei's NVME image

hisi-nve Android PoC to read/write Huawei's NVME image Disclaimers Use this tool at your own risk and always backup NVME. This tool was made for educa

Dec 15, 2022
Loads a signed kernel driver which allows you to map any driver to kernel mode without any traces of the signed / mapped driver.
Loads a signed kernel driver which allows you to map any driver to kernel mode without any traces of the signed / mapped driver.

CosMapper Loads a signed kernel driver (signed with leaked cert) which allows you to map any driver to kernel mode without any traces of the signed /

Jan 2, 2023
DLL Exports Extraction BOF with optional NTFS transactions.
DLL Exports Extraction BOF with optional NTFS transactions.

DLL Exports Extraction BOF What is this? This is a Cobalt Strike BOF file, meant to use two or three arguments (path to DLL, and/or a third argument [

Nov 9, 2022
x64 Windows kernel driver mapper, inject unsigned driver using anycall
x64 Windows kernel driver mapper, inject unsigned driver using anycall

anymapper x64 Windows kernel driver mapper, inject unsigned driver using anycall This project is WIP. Todo Fix: Can't make API calls from IAT nor func

Dec 26, 2022
Driver leap - Self-sustainable fork of SteamVR driver for Leap Motion controller with updated vendor libraries
Driver leap - Self-sustainable fork of SteamVR driver for Leap Motion controller with updated vendor libraries

Driver Leap Self-sustainable fork of SteamVR driver for Leap Motion controller with updated vendor libraries Installation (for users) Install Ultralea

Jan 5, 2023
Hygieia, a vulnerable driver traces scanner written in C++ as an x64 Windows kernel driver.

Hygieia The Greek goddess of health, her name is the source for the word "hygiene". Hygieia is a windows driver that works similarly to how pagewalkr

Dec 4, 2022
SinMapper - usermode driver mapper that forcefully loads any signed kernel driver
SinMapper - usermode driver mapper that forcefully loads any signed kernel driver

usermode driver mapper that forcefully loads any signed kernel driver (legit cert) with a big enough section (example: .data, .rdata) to map your driver over. the main focus of this project is to prevent modern anti-cheats (BattlEye, EAC) from finding your driver and having the power to hook anything due to being inside of legit memory (signed legit driver).

Dec 29, 2022
A simple thread-safe implementation of runtime obfuscation for Win32 applications.
A simple thread-safe implementation of runtime obfuscation for Win32 applications.

Thread-Safe Win32 Runtime Obfuscation A simple thread-safe implementation of runtime obfuscation for Win32 applications. The main use case for this is

Mar 6, 2022
Online multi-agent trajectory planner using linear safe corridor (LSC)

lsc_planner This package presents an efficient multi-agent trajectory planning algorithm which generates safe trajectories in obstacle-dense environme

Dec 27, 2022
EntityX - A fast, type-safe C++ Entity-Component system

EntityX - A fast, type-safe C++ Entity Component System NOTE: The current stable release 1.0.0 breaks backward compatibility with < 1.0.0. See the cha

Dec 29, 2022
Type-safe Printf in C

Type-Safe Printf For C This uses macro magic, compound literals, and _Generic to take printf() to the next level: type-safe printing, printing into co

Dec 18, 2022
KeePassXC is a cross-platform community-driven port of the Windows application “Keepass Password Safe”.

KeePassXC KeePassXC is a modern, secure, and open-source password manager that stores and manages your most sensitive information. You can run KeePass

Jan 2, 2023
Type safe - Zero overhead utilities for preventing bugs at compile time

type_safe type_safe provides zero overhead abstractions that use the C++ type system to prevent bugs. Zero overhead abstractions here and in following

Jan 8, 2023
Thread-safe cross-platform event loop library in C++

Dasynq Version 1.2.2 Dasynq is an event loop library similar to libevent, libev and libuv. Like other such libraries, it is crossplatform / portable.

Jan 9, 2023