Text - A spicy text library for C++ that has the explicit goal of enabling the entire ecosystem to share in proper forward progress towards a bright Unicode future.

Last update: Aug 30, 2022

Comments: 16

`ztd.text`

Because if text works well in two of the most popular systems programming languages, the entire world over can start to benefit properly.

This repository is an implementation of an up and coming proposal percolating through SG16, P1629 - Standard Text Encoding. It will also include implementations of some downstream ideas covered in Previous Work in this area, including Zach Laine's Boost.Text (proposed), rmf's libogonek, and Tom Honermann's text_view.

Current Plans

The current roadmap is to obtain funding to make text encoding viable in time for standardization for C++23. See the roadmap for details.

Please considering sponsoring the work via any of the available means listed near the bottom of this page or from the "Sponsor" button on this website.

Documentation

The documentation can be found in full on https://ztdtext.rtfd.io/!

License & Contributions

By contributing to this repository, you agree that your contributions will be released to the copyright of JeanHeyd "ThePhD" Meneide and Shepherd's Oasis, LLC under the Apache 2.0 License or the relevant Commercial License. See the LICENSE file and the CONTRIBUTING file for details.

Owner

Shepherd's Oasis

Software Services and Consulting.

https://github.com/soasis/text https://ztdtext.readthedocs.io/en/latest/

Comments

$Naming discussion: count_{en,de,trans}codable$
Naming discussion: count_{en,de,trans}codable
First off, thanks a lot for the awesome work you're doing! 🙃

I have been loosely following @ThePhD's stream (flood?) of papers & blog posts, and was wondering about a naming discussion they had posed on twitter a while ago.

Naming is hard, though worth it - the right name substantially reduces cognitive overhead, especially when accumulated over a long period of time. Since I was intrigued by the riddle of squaring that circle, I had written to @ThePhD with a suggestion, and they suggested to open an issue here.

One of the key points about the count_* functions that doesn't exactly jump out from the documentation (at least to me) is that the actual work of encoding/... (especially around memory allocation etc., AFAIU) is not being done. Quoth @ThePhD (in private communication):

I'm trying to avoid the participle because count_decoded sounds like "count after the work is already done", and that work expressly isn't being done. That's why I took the adjective-y version of decodable (and friends). Or, at least, that was the reasoning....! Not sure if it's good reasoning 😅

Since I wasn't warming up to count_encodable et. al (it's not intrinsically obvious to from the name whether things are being counted before or after the operation. The "-able" suffix makes it seem like it could be the former - "these code points are encodable [into code units], now count them"), I wanted to come up with something that more intrinsically reflects the absence of real work™.

My suggestion boils down to changing the verb from count to infer, freeing up the participle to rejoin the party unburdened of assumed labour. IOW: infer_encoded_{size,length,count}

count could even make a comeback as a noun, but whether the choice is size, length or count is (from my POV) not that important.

I have no beef with the validate_* functions (whose expressiveness I applaud), so, to sum up my proposed answer to the initial riddle from the tweet (using my preferred noun size, if only due to its lack thereof, typographically speaking):

encode decode transcode validate_encodable_as validate_decodable_as validate_transcodable_as infer_encoded_size infer_decoded_size infer_transcoded_size

Thanks for reading!
Request: Add port for vcpkg

First I'd like to thank you and all the contributors of the library for all your work done so far.

Going further, I'd like to propose making the library available through vcpkg (and other package managers) to ease up the process of integrating it on different projects, let them be existing or new ones. The library is header-only so should be (relatively) easy to integrate and sol2's "model" could be as well adapted for the library (not sure if there is any problem related to this).
🔨 Fixes for GCC 11
Tests fail under GCC 11 due to wider constexpr support. These are the fixes necessary to make tests pass

Fix wide_execution_cwchar copy-paste bug

Bump transcode_iterator tags to prevent type ambiguity

Make word_iterator::_M_base_range constexpr
decode_view iterator post-increment operator is broken

It posts but it does not increment. 3/5 :star:'s, room for improvement

More things are also probably broken but I only wrote the test for the decode_view, see https://github.com/soasis/text/pull/27

Cannot run tests with gcc 11 (error: invalid use of ‘this’ at top level)

Configured with:

CC=gcc CXX=g++ cmake ..  -DZTD_TEXT_TESTS=ON -DZTD_TEXT_EXAMPLES=ON -DZTD_TEXT_GENERATE_SINGLE=ON -GNinja

g++ (GCC) 11.1.1 20210531 (Red Hat 11.1.1-3)
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

CMake Output

-- The C compiler identification is GNU 11.1.1
-- The CXX compiler identification is GNU 11.1.1
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/gcc - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/g++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Performing Test CXX_CHECK_FLAG_DISABLE_PERMISSIVE
-- Performing Test CXX_CHECK_FLAG_DISABLE_PERMISSIVE - Success
-- Performing Test C_CHECK_FLAG_DISABLE_PERMISSIVE
-- Performing Test C_CHECK_FLAG_DISABLE_PERMISSIVE - Success
-- Performing Test CXX_CHECK_FLAG_WARN_PEDANTIC
-- Performing Test CXX_CHECK_FLAG_WARN_PEDANTIC - Success
-- Performing Test C_CHECK_FLAG_WARN_PEDANTIC
-- Performing Test C_CHECK_FLAG_WARN_PEDANTIC - Success
-- Performing Test CXX_CHECK_FLAG_WARN_ALL
-- Performing Test CXX_CHECK_FLAG_WARN_ALL - Success
-- Performing Test C_CHECK_FLAG_WARN_ALL
-- Performing Test C_CHECK_FLAG_WARN_ALL - Success
-- Performing Test CXX_CHECK_FLAG_WARN_ERRORS
-- Performing Test CXX_CHECK_FLAG_WARN_ERRORS - Success
-- Performing Test C_CHECK_FLAG_WARN_ERRORS
-- Performing Test C_CHECK_FLAG_WARN_ERRORS - Success
-- Performing Test CXX_CHECK_FLAG_UTF8_LITERAL_ENCODING
-- Performing Test CXX_CHECK_FLAG_UTF8_LITERAL_ENCODING - Success
-- Performing Test C_CHECK_FLAG_UTF8_LITERAL_ENCODING
-- Performing Test C_CHECK_FLAG_UTF8_LITERAL_ENCODING - Success
-- Performing Test CXX_CHECK_FLAG_UTF8_SOURCE_ENCODING
-- Performing Test CXX_CHECK_FLAG_UTF8_SOURCE_ENCODING - Success
-- Performing Test C_CHECK_FLAG_UTF8_SOURCE_ENCODING
-- Performing Test C_CHECK_FLAG_UTF8_SOURCE_ENCODING - Success
-- Performing Test CXX_CHECK_FLAG_EXTRA_CONSTEXPR_DEPTH
-- Performing Test CXX_CHECK_FLAG_EXTRA_CONSTEXPR_DEPTH - Success
-- Performing Test C_CHECK_FLAG_EXTRA_CONSTEXPR_DEPTH
-- Performing Test C_CHECK_FLAG_EXTRA_CONSTEXPR_DEPTH - Failed
-- Performing Test CXX_CHECK_FLAG_EXTRA_CONSTEXPR_STEPS
-- Performing Test CXX_CHECK_FLAG_EXTRA_CONSTEXPR_STEPS - Success
-- Performing Test C_CHECK_FLAG_EXTRA_CONSTEXPR_STEPS
-- Performing Test C_CHECK_FLAG_EXTRA_CONSTEXPR_STEPS - Failed
-- Performing Test CXX_CHECK_FLAG_TEMPLATE_DEBUGGING_MODE
-- Performing Test CXX_CHECK_FLAG_TEMPLATE_DEBUGGING_MODE - Success
-- Performing Test C_CHECK_FLAG_TEMPLATE_DEBUGGING_MODE
-- Performing Test C_CHECK_FLAG_TEMPLATE_DEBUGGING_MODE - Failed
-- Found PythonInterp: /usr/bin/python3 (found suitable version "3.9.6", minimum required is "3")
-- Configuring done
-- Generating done
-- Build files have been written to: /tmp/text/build

[3/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o -c ../tests/basic_run_time/source/detail/encoding_name.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/is_ignorable_error_handler.hpp:38,
                 from ../include/ztd/text/utf8.hpp:41,
                 from ../include/ztd/text/detail/encoding_name.hpp:38,
                 from ../tests/basic_run_time/source/detail/encoding_name.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~

Full build output

[0/2] Re-checking globbed directories...
[2/162] Generating include/ztd/text.hpp, include/ztd/text/forward.hpp
Current version: v0.0.1 (revision 811ecc2)
Project path: /tmp/text/single/../include

Creating single header for ztd.text
processing /tmp/text/include/ztd/text.hpp
processing /tmp/text/include/ztd/text/version.hpp
processing /tmp/text/include/ztd/text/forward.hpp
processing /tmp/text/include/ztd/text/encoding.hpp
processing /tmp/text/include/ztd/text/execution.hpp
processing /tmp/text/include/ztd/text/no_encoding.hpp
processing /tmp/text/include/ztd/text/code_point.hpp
processing /tmp/text/include/ztd/text/unicode_code_point.hpp
processing /tmp/text/include/ztd/text/type_traits.hpp
processing /tmp/text/include/ztd/text/detail/assert.hpp
processing /tmp/text/include/ztd/text/detail/unicode.hpp
processing /tmp/text/include/ztd/text/code_unit.hpp
processing /tmp/text/include/ztd/text/encode_result.hpp
processing /tmp/text/include/ztd/text/encoding_error.hpp
processing /tmp/text/include/ztd/text/detail/to_underlying.hpp
processing /tmp/text/include/ztd/text/detail/encoding_range.hpp
processing /tmp/text/include/ztd/text/state.hpp
processing /tmp/text/include/ztd/text/text_tag.hpp
processing /tmp/text/include/ztd/text/decode_result.hpp
processing /tmp/text/include/ztd/text/error_handler.hpp
processing /tmp/text/include/ztd/text/c_string_view.hpp
processing /tmp/text/include/ztd/text/basic_c_string_view.hpp
processing /tmp/text/include/ztd/text/is_code_points_replaceable.hpp
processing /tmp/text/include/ztd/text/is_code_units_replaceable.hpp
processing /tmp/text/include/ztd/text/is_unicode_code_point.hpp
processing /tmp/text/include/ztd/text/detail/pass_through_handler.hpp
processing /tmp/text/include/ztd/text/detail/forwarding_handler.hpp
processing /tmp/text/include/ztd/text/detail/transcode_one.hpp
processing /tmp/text/include/ztd/text/transcode_result.hpp
processing /tmp/text/include/ztd/text/validate_result.hpp
processing /tmp/text/include/ztd/text/count_result.hpp
processing /tmp/text/include/ztd/text/detail/is_lossless.hpp
processing /tmp/text/include/ztd/text/is_full_range_representable.hpp
processing /tmp/text/include/ztd/text/is_transcoding_compatible.hpp
processing /tmp/text/include/ztd/text/unicode_scalar_value.hpp
processing /tmp/text/include/ztd/text/detail/encoding_iterator_storage.hpp
processing /tmp/text/include/ztd/text/detail/span_or_reconstruct.hpp
processing /tmp/text/include/ztd/text/is_ignorable_error_handler.hpp
processing /tmp/text/include/ztd/text/detail/empty_state.hpp
processing /tmp/text/include/ztd/text/detail/execution_cuchar.hpp
processing /tmp/text/include/ztd/text/utf8.hpp
processing /tmp/text/include/ztd/text/detail/cast.hpp
processing /tmp/text/include/ztd/text/utf16.hpp
processing /tmp/text/include/ztd/text/detail/progress_handler.hpp
processing /tmp/text/include/ztd/text/detail/windows.hpp
processing /tmp/text/include/ztd/text/detail/posix.hpp
processing /tmp/text/include/ztd/text/detail/encoding_name.hpp
processing /tmp/text/include/ztd/text/utf32.hpp
processing /tmp/text/include/ztd/text/encoding_scheme.hpp
processing /tmp/text/include/ztd/text/is_bidirectional_encoding.hpp
processing /tmp/text/include/ztd/text/is_unicode_encoding.hpp
processing /tmp/text/include/ztd/text/ascii.hpp
processing /tmp/text/include/ztd/text/detail/replacement.hpp
processing /tmp/text/include/ztd/text/detail/execution_mac_os.hpp
processing /tmp/text/include/ztd/text/detail/execution_iconv.hpp
processing /tmp/text/include/ztd/text/wide_execution.hpp
processing /tmp/text/include/ztd/text/detail/wide_execution_windows.hpp
processing /tmp/text/include/ztd/text/detail/wide_execution_iso10646.hpp
processing /tmp/text/include/ztd/text/detail/wide_execution_iconv.hpp
processing /tmp/text/include/ztd/text/detail/wide_execution_cwchar.hpp
processing /tmp/text/include/ztd/text/literal.hpp
processing /tmp/text/include/ztd/text/wide_literal.hpp
processing /tmp/text/include/ztd/text/any_encoding.hpp
processing /tmp/text/include/ztd/text/default_encoding.hpp
processing /tmp/text/include/ztd/text/encode.hpp
processing /tmp/text/include/ztd/text/decode.hpp
processing /tmp/text/include/ztd/text/validate_decodable_as.hpp
processing /tmp/text/include/ztd/text/validate_encodable_as.hpp
processing /tmp/text/include/ztd/text/count_encodable.hpp
processing /tmp/text/include/ztd/text/count_decodable.hpp
processing /tmp/text/include/ztd/text/transcode.hpp
processing /tmp/text/include/ztd/text/detail/transcode_extension_points.hpp
processing /tmp/text/include/ztd/text/encode_view.hpp
processing /tmp/text/include/ztd/text/encode_iterator.hpp
processing /tmp/text/include/ztd/text/detail/encoding_iterator.hpp
processing /tmp/text/include/ztd/text/decode_view.hpp
processing /tmp/text/include/ztd/text/decode_iterator.hpp
processing /tmp/text/include/ztd/text/transcode_view.hpp
processing /tmp/text/include/ztd/text/transcode_iterator.hpp
processing /tmp/text/include/ztd/text/text_view.hpp
processing /tmp/text/include/ztd/text/basic_text_view.hpp
processing /tmp/text/include/ztd/text/normalization.hpp
processing /tmp/text/include/ztd/text/text.hpp
processing /tmp/text/include/ztd/text/basic_text.hpp
finished creating single header for ztd.text

Creating single forward declaration header for ztd.text
processing /tmp/text/include/ztd/text/forward.hpp
processing /tmp/text/include/ztd/text/version.hpp
finished creating single forward declaration header for ztd.text

writing /tmp/text/build/single/include/ztd/text.hpp...
writing /tmp/text/build/single/include/ztd/forward.hpp...
[3/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/detail/encoding_name.cpp.o -c ../tests/basic_run_time/source/detail/encoding_name.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/is_ignorable_error_handler.hpp:38,
                 from ../include/ztd/text/utf8.hpp:41,
                 from ../include/ztd/text/detail/encoding_name.hpp:38,
                 from ../tests/basic_run_time/source/detail/encoding_name.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[4/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode.cpp.o -c ../tests/basic_run_time/source/decode.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/no_encoding.hpp:42,
                 from ../include/ztd/text/execution.hpp:38,
                 from ../include/ztd/text/default_encoding.hpp:38,
                 from ../include/ztd/text/decode.hpp:39,
                 from ../tests/basic_run_time/source/decode.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[5/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode_view.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode_view.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode_view.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode_view.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/decode_view.cpp.o -c ../tests/basic_run_time/source/decode_view.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/decode_iterator.hpp:38,
                 from ../include/ztd/text/decode_view.hpp:38,
                 from ../tests/basic_run_time/source/decode_view.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[6/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/encode.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/encode.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/encode.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/encode.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/encode.cpp.o -c ../tests/basic_run_time/source/encode.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/encode.hpp:40,
                 from ../tests/basic_run_time/source/encode.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[7/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_encodable.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_encodable.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_encodable.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_encodable.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_encodable.cpp.o -c ../tests/basic_run_time/source/count_encodable.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/no_encoding.hpp:42,
                 from ../include/ztd/text/execution.hpp:38,
                 from ../include/ztd/text/default_encoding.hpp:38,
                 from ../include/ztd/text/count_encodable.hpp:39,
                 from ../tests/basic_run_time/source/count_encodable.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[8/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_decodable.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_decodable.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_decodable.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_decodable.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_decodable.cpp.o -c ../tests/basic_run_time/source/count_decodable.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/no_encoding.hpp:42,
                 from ../include/ztd/text/execution.hpp:38,
                 from ../include/ztd/text/default_encoding.hpp:38,
                 from ../include/ztd/text/count_decodable.hpp:39,
                 from ../tests/basic_run_time/source/count_decodable.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[9/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.encode.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.encode.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.encode.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.encode.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.encode.cpp.o -c ../tests/basic_run_time/source/any_encoding.encode.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/is_ignorable_error_handler.hpp:38,
                 from ../include/ztd/text/encoding_scheme.hpp:43,
                 from ../tests/basic_run_time/source/any_encoding.encode.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[10/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_transcodable.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_transcodable.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_transcodable.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_transcodable.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/count_transcodable.cpp.o -c ../tests/basic_run_time/source/count_transcodable.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/no_encoding.hpp:42,
                 from ../include/ztd/text/execution.hpp:38,
                 from ../include/ztd/text/default_encoding.hpp:38,
                 from ../include/ztd/text/count_transcodable.hpp:39,
                 from ../tests/basic_run_time/source/count_transcodable.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[11/162] Building CXX object tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.decode.cpp.o
FAILED: tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.decode.cpp.o
/usr/bin/g++ -DZTD_TEXT_COMPILE_TIME_ENCODING_NAME=\"UTF-8\" -I../tests/basic_run_time/../shared/include -I../include -Isingle/include -I_deps/ztd.idk-src/include -I_deps/catch2-src/single_include -fexec-charset=utf-8 -finput-charset=utf-8 -pedantic -Wall -Werror -std=gnu++17 -MD -MT tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.decode.cpp.o -MF tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.decode.cpp.o.d -o tests/basic_run_time/CMakeFiles/ztd.text.tests.basic_run_time.dir/source/any_encoding.decode.cpp.o -c ../tests/basic_run_time/source/any_encoding.decode.cpp
In file included from ../include/ztd/text/detail/transcode_one.hpp:56,
                 from ../include/ztd/text/error_handler.hpp:52,
                 from ../include/ztd/text/is_ignorable_error_handler.hpp:38,
                 from ../include/ztd/text/encoding_scheme.hpp:43,
                 from ../tests/basic_run_time/source/any_encoding.decode.cpp:31:
../include/ztd/ranges/unbounded.hpp:273:50: error: invalid use of ‘this’ at top level
  273 |                         noexcept(ranges::advance(this->_M_it, __diff))) {
      |                                                  ^~~~
../include/ztd/ranges/unbounded.hpp:287:49: error: invalid use of ‘this’ at top level
  287 |                         noexcept(ranges::recede(this->_M_it, __diff))) {
      |                                                 ^~~~
[12/162] Building CXX object _deps/catch2-build/CMakeFiles/Catch2WithMain.dir/src/catch_with_main.cpp.o
ninja: build stopped: subcommand failed.

CI is using GCC 9, which is why this hasn't come up. It's probably also worth testing clang there also, although it's Worked On My Machine (TM)

__iterator aliases assume std::iterator_traits has members

Most of the __iterator_* aliases depend on std::iterator_traits, but this doesn't necessarily mesh with C++20 iterators, since not all C++20 iterators necessarily have a populated std::iterator_traits (e.g. stdr::iterator_t<stdr::basic_istream_view<char, char, std::char_traits<char>>>).

Most of these either have a C++20 counterpart traits type (e.g. value_type) or can probably be worked out (e.g. pointer). One exception to this is iterator_concept, which might only exist as a member alias (e.g. basic_istream_view).

Compilation issue with CMAKE_BUILD_TYPE=RelWithDebInfo

Building with -DCMAKE_BUILD_TYPE=RelWithDebInfo produces errors.

/home/sdowney/src/text/text/include/ztd/text/utf32.hpp: In function ‘constexpr auto ztd::text::__v0::validate_code_points(_Input&&, _Encoding&&, _EncodeState&, _DecodeState&) [with _Input = std::span<const char32_t, 18446744073709551615>; _Encoding = const ztd::text::__v0::encoding_scheme<ztd::text::__v0::basic_utf32<char32_t, char32_t>, ztd::text::__v0::endian::little, std::byte>&; _EncodeState = ztd::text::__v0::__detail::__empty_state; _DecodeState = ztd::text::__v0::__detail::__empty_state]’:
/home/sdowney/src/text/text/include/ztd/text/utf32.hpp:189:38: error: ‘__code_unit_buf’ may be used uninitialized in this function [-Werror=maybe-uninitialized]
  189 |     _

Compile differences seem to be adding -O2 -g -DNDEBUG to the compile. First error is in any_encoding.encode.cpp

MAINT: renormalize line endings
Like #34, as windows line-endings are creeping in again. Quoting from a comment in that other PR:

I'd recommend that you ensure you have the equivalent of

git config --global core.autocrlf true

to make sure the files don't get messed up again.

If there are particular files that shouldn't be crlf-mangled by git, add them (directly or with a wildcard) to .gitattributes as binary, or specify the eol directly, such as:

*.patch binary build.sh text eol=lf build.bat text eol=crlf

Less likely but not impossible, it could be that you have an IDE setting that's overriding this somehow.
R-values? More like Revenge-values!!
The following code breaks:

std::stringstream message_stream; /* blah blah manipulate message stream */ std::wstring winMessage = ztd::text::transcode(message_stream.str(), ztd::text::compat_utf8, ztd::text::wide_utf16, ztd::text::replacement_handler); MessageBoxW(NULL, winMessage.c_str(), L"GlueGDLoader Error", MB_OK);

The error happens because passing a temporary into these functions (the message_stream.str()) breaks everything. This is one of the fundamental problems with std::ranges and ranges in general: r-values are not handled by the lower-levels.

But honestly, there's nothing wrong with this code in particular.

An owning-container (and not a temporary container) is handed to the end-user by the plain and _to versions of these functions. We should detect r-values in the plain functions and hold onto them normally, rather than forward them, and let the result be returned back to the end user. This would result in a more pleasant user experience.

We should also add an explicit top-level r-value screamer for the Input on the _into functions.

(Thanks for catching this one, @s5bug.)
Design: do we need state in the replacement functions?

It could be conceivable that, if an error handler is called and replacement is happening, replacement characters may be inserted into the stream which may need additional state information. Does this mean encode_state and decode_state need to be (required) parameters to replacement_code_units, replacement_code_points, maybe_replacement_code_units, and maybe_replacement_code_points?

Are we missing other places where the state information is necessary?

I didn't initially think of this because I assumed most people would override the entire error handler to take care of this case, but now I'm not sure. The parameter will be ignored in 99% of cases, but it's probably no issue to just make sure it takes it anyways even if we turn out to be wrong...
[BUG] Discrepency when determining result type's types
The below sample will fail to compile.

#include <ztd/text.hpp> void repro() { std::string str; ztd::text::utf8 enc; auto enc_state = ztd::text::make_encode_state(enc); auto dec_state = ztd::text::make_decode_state(enc); ztd::text::transcode_into(str, enc, str, enc, ztd::text::pass_handler{}, ztd::text::pass_handler{}, dec_state, enc_state); }

basic_transcode_into uses using _WorkingOutput = __recontsruct_t<Output> and declares a variable _WorkingOutput __working_output. Later, in a deeper call to __super_basic_transcode_one(), the output range type for the transcode_result is determined as the return type of __basic_encode_one(). __basic_encode_one uses a series of if constexpr checks to determines its return type. The type of transcode_result::output can differ from __working_output. basic_transcode_into then attempts to assign __transcode_result.output into __working_output, which will fail if they are not the same (or at least convertible).

In this case, the __basic_encode_one() is returning an unbounded_view<std::string::iterator>, whereas the __working_output is of type subrange<std::string::iterator>. I cannot understand the if constexpr checks enough to tell why the branch returning an unbounded_view is selected.
MAINT: renormalize line endings

If VSCode messes up the line endings relative to the .gitattributes settings discussed previously, perhaps a solution could be to do git add --renormalize . before creating the next mega commit. 🙃

PS. Advise not to squash, otherwise 17ecb4b48555deae3fa7c2b8584e15785a75d131 won't take effect.
fixed/static/inline_vector for decode_one/encode_one/transcode_one

These *_one functions return heap-allocating containers right now. This is not necessary, as the maximum output size is known ahead-of-time.

We should write a fixed/static/inline_vector<T, max_size>, alongside a fixed/static/inline_string<T, max_size, [char_traits<T> ??]>.
Don't Cursed Open Inside
This is a running list of all the (mildly to extremely) cursed encodings, and whether or not we should implement them. More can be suggested on Twitter here Here goes:

[ ] UNIVAC encoding

[ ] UTF-EBCDIC

[ ] UTF-7

[ ] UTF-7-IMAP

[ ] UTF-1

[ ] SHIFT-JIS (already implemented in example code)

[ ] ISO/IEC 2022 Encodings (https://en.wikipedia.org/wiki/ISO/IEC_2022)

[ ] ISO/IEC 646 Encodings (https://en.wikipedia.org/wiki/ISO/IEC_646)

Some that might not be possible within the framework of this library:

Early Canjie input method translation: this is moreso a system of input that is then converted to characters, rather than a character set itself. It also seems to have a (potentially?) unbounded set of inputs that can produce an equally wild amount of outputs, making the encode_one/decode_one limitations potentially useless? Needs more research
Missing encodings: Tatar
Two encodings that are missing from the list of available encodings are the Tatar encodings as specified by Resolution of the Cabinet of Ministers of the Republic of Tatarstan from the 9th of December, 1996 No. 1026 "On the standards of the encoding symbols of Tatar alphabet for computer applications"

The document specifies an ‘ASCII’ version (for DOS and console applications; compatible with cp866) and ‘ANSI’ version (for Windows applications; compatible with cp1251).

I don't think there are established codes for these encodings. I would suggest something along the lines of cp1251-tatar and cp866-tatar.

‘ANSI’ version (variant of cp1251)

The scans of the actual resolution are hard to read, but the ASCII version is described in Russian Wikipedia's article on cp1251, under the subheading «Татарский вариант». Here are the rows that are different from cp1251:

Basically, the letters that differ from cp1251 are:

0x80 = Ә (U+04D8) 0x8A = Ө (U+04E8) 0x8C = Ү (U+04AE) 0x8D = Җ (U+0496) 0x8E = Ң (U+04A2) 0x8F = Һ (U+04BA) 0x90 = ә (U+04D9) 0x9A = ө (U+04E9) 0x9C = ү (U+04AF) 0x9D = җ (U+0497) 0x9e = ң (U+04A3) 0x9F = һ (U+04BB)

This encoding can still be encountered in the internet. For example, search for хђзер (for хәзер 'now'), кирђк (for кирәк 'needed') or мљмкин (for мөмкин 'possible') in Google to find some example usage. Some fonts for this encoding can be found here: https://kashapovnail.ucoz.ru/load/1-1-0-1 (they would usually replace Southern Slavic letters with Tatar letters, so they're not really Unicode-compatible; it's a hard because old Windows version didn't support other way to input this encoding).

‘ASCII’ version (variant of cp866)

I don’t know how widely this encoding is used. But the document provides information about it, so it can be implemented:

It's basically cp866 with the following changes (0xF0 and 0xF1 might not be the change depending on what you consider the 'basic variant' of cp866):

0xF0 = Ё (U+0401) 0xF1 = ё (U+0451) 0xF2 = Ә (U+04D8) 0xF3 = Ө (U+04E8) 0xF4 = Ү (U+04AE) 0xF5 = Җ (U+0496) 0xF6 = Ң (U+04A2) 0xF7 = Һ (U+04BA) 0xF8 = ә (U+04D9) 0xF9 = ө (U+04E9) 0xFA = ү (U+04AF) 0xFB = җ (U+0497) 0xFC = ң (U+04A3) 0xFD = һ (U+04BB)
Improve CMake integration
At the moment, the CMake build files are messy.

There is no clear separation between consumer and developer code paths.
This makes it impossible to build/install/package the library without heavily patching build files, which is completely avoidable. It's not nice to force consumers to pass 10 flags just so they can dodge everything but the library itself.

The build files make network requests during configure.
If you wish this library to be reach a wide audience, this is an absolute no go. For dependencies like Catch2, you can use vcpkg, even for those you grab from GitHub, like the other ztd libraries.

The library does not declare its standard requirement.
I had to find out from #13 that this library is targetting C++17. A simple target_compile_features(ztd_text INTERFACE cxx_std_17) would be helpful. Hardcoding CMAKE_CXX_STANDARD is also wrong, because it's purpose is to be used from the CLI to control the build.

Globbing sources. This is mainly for scripting. Again, if you wish this library to be usable by a wide audience, this is an absolute no go. Mainly because it doesn't even work reliably for most build systems. Please read the big note from the CMake developers and heed their words.

Install rules are wrong.

Consumer side of lists files should do nothing more than describe the build and usage requirements of a library and the CMake build scripts should be as minimal as possible with no hardcoded details. It is noone's benefit to force people trying to build/package/use your library to patch things.

I recommend taking a look at cmake-init, which is designed to solve all of these issues. Since you have dependencies from GitHub as well, I recommend taking a look at the vcpkg example repository.
A better solution would be however, if instead of introducing potentially exponential complexity to building the library with optional dependencies, just make a separate library that depends on both this one and that optional dependency.

Text - A spicy text library for C++ that has the explicit goal of enabling the entire ecosystem to share in proper forward progress towards a bright Unicode future.

ztd.text

Current Plans

Documentation

License & Contributions

Owner

Shepherd's Oasis

Comments

Naming discussion: count_{en,de,trans}codable

Request: Add port for vcpkg

🔨 Fixes for GCC 11

decode_view iterator post-increment operator is broken

Cannot run tests with gcc 11 (error: invalid use of ‘this’ at top level)

__iterator aliases assume std::iterator_traits has members

Compilation issue with CMAKE_BUILD_TYPE=RelWithDebInfo

MAINT: renormalize line endings

R-values? More like Revenge-values!!

Design: do we need state in the replacement functions?

[BUG] Discrepency when determining result type's types

MAINT: renormalize line endings

fixed/static/inline_vector for decode_one/encode_one/transcode_one

Don't Cursed Open Inside

Missing encodings: Tatar

‘ANSI’ version (variant of cp1251)

‘ASCII’ version (variant of cp866)

Improve CMake integration

Related tags

A bright opening, a clear sight, a clean slate.

Adds proper Apple Pencil support to Unity's InputSystem.

A modern port of Turbo Vision 2.0, the classical framework for text-based user interfaces. Now cross-platform and with Unicode support.

Building game development ecosystem for @ziglang!

This is the repo for the microcontroller part of project Cleanurge - a scalable IoT powered waste management system, our solution towards a greener world.

FlexOS: Towards Flexible OS Isolation (ASPLOS'22) Artifact Evaluation Repository

Fastest tiktok share botter. Over 100k shares a second.

Header-only library providing unicode aware string support for C++

Libnaomi - Toolchain and system library enabling hobby/homebrew software to be developed for the Sega Naomi platform.

Path Tracking PID offers a tuneable PID control loop, decouling steering and forward velocity

Swift Framework for Apple Health Integration with Terra Enabling Developers LTD.

Kexts enabling native support for Intel Bluetooth chipsets in macOS.

libu8ident - Follow unicode security guidelines for identifiers

This is the laplight software for enabling flashlight support on a laptop/netbook. For the specification, see: https://github.com/LapLight/

Simdutf - Unicode routines (UTF8, UTF16): billions of characters per second.

Neo - Simulates the digital rain from "The Matrix" (cmatrix clone with 32-bit color and Unicode support)

CppUTest For QP/C++ implements a CppUTest port of the QP Framework, with supporting utilities, enabling easy host based unit testing of active objects.

Pandex is a light but FAST programming language written in C . Pandex goal is that be hard & it's good for eductional goals

Kernel source for j7y17lte - the goal is to make it as closest to linux-stable sources as possible without breaking OneUI compatibility.

`ztd.text`