Perl Incompatible Regular Expressions library

If a string passed to DoRun is less than sizeof(size_t), then its content can't be read as if its type is size_t*. DoRun splits the string to 3 parts: pre-aligned, aligned and post-aligned. If there is no aligned part of a string, then whole string is treated as pre- and post-aligned.

The error was caused by casting pre- and post-aligned parts to size_t*. Now they are processed by code operating on char* (function SafeRunChunk). (Actually pre-aligned part is allowed to be casted to size_t*, if aligned part exists, ie. length of the string > size(size_t).)

Target strings:

• length <= 6 - error
• length >= 7 - no error

String of length 7 fits in size_t (+1 byte for terminating '\0').

Valgrind output before this patch applied:

==26448== Invalid read of size 8                                                                      
==26448==    at 0x60439F5: void Pire::Impl::DoRun<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> >, Pire::Impl::RunPred<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > > >(Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > const&, Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> >::State&, char const*, char const*, Pire::Impl::RunPred<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > >) (run.h:129)                                                                                          
==26448==    by 0x603C906: void Pire::Run<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > >(Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > const&, Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> >::State&, char const*, char const*) (run.h:266)
==26448==    by 0x60438C6: Pire::RunHelper<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > >::Run(char const*, char const*) (run.h:341)                                           
==26448==    by 0x603C899: Pire::RunHelper<Pire::Impl::Scanner<Pire::Impl::Relocatable, Pire::Impl::ExitMasks<2ul> > >::Run(char const*, unsigned long) (run.h:342) 

Applying this patch the valgrind error was fixed.

I have a very simple regexp

std::string r;
r  = "(post|get|put|delete).*http/1\\.(1|0)\r\n.*\r\n\r\n";

The test data is:

GET / HTTP/1.1 User-Agent: chrome Host: ya.ru Accept: / Proxy-Connection: Keep-Alive

Which has a proper format i.e. it contains \r\n after each line and extra \r\n after the header. Creation of the scanner is done with the following function:

Pire::Scanner flow::scannerFor( std::string regexp ){
    if ( !regexp.size() ) {
        throw common::error( _ERR_EMPTY_REGEXP );
    }

    std::vector<Pire::wchar32> pattern;
    Pire::Scanner s;

    try {
        Pire::Encodings::Utf8().FromLocal( regexp.c_str(), regexp.c_str() + regexp.size(), std::back_inserter(pattern) );

        s = Pire::Lexer( pattern.begin(), pattern.end() )
            .SetEncoding( Pire::Encodings::Utf8() )
            .AddFeature( Pire::Features::CaseInsensitive() )
            .Parse()
            .Surround()
            .Compile<Pire::Scanner>();

    } catch ( ... ) {
        throw common::error( _ERR_REGEXP_COMPILE
            .arg( regexp ) 
        );
    }

    return s;
}

When I use scanner which is compiled for my regexp with the test data it fails to match. If I comment the line

.SetEncoding( Pire::Encodings::Utf8() )

in the scanner's creation function, scanner starts to match. Could you comment this situation?

On FreeBSD-9.1, with automake-1.14 autoconf-2.69, in generated pire/Makefile:in:

...
pire_hdr_HEADERS = \
        align.h \
        any.h \
        defs.h \
        determine.h \
        easy.h \
        encoding.h \
        extra.h \
        fsm.h \
        fwd.h \
        glue.h \
        partition.h \
        pire.h \
        re_lexer.h \
        re_parser.h \
        run.h \
        static_assert.h \
        platform.h \
        vbitset.h
...
BUILT_SOURCES = re_parser.h re_parser.cpp
CLEANFILES = re_parser.h re_parser.cpp
...
re_parser.hpp: re_parser.cpp
        @if test ! -f [email protected]; then rm -f re_parser.cpp; else :; fi
        @if test ! -f [email protected]; then $(MAKE) $(AM_MAKEFLAGS) re_parser.cpp; else :; fi
...

e.g. a rule for generating re_parser.hpp is generated, but re_parser.h is expected.

To keep header file naming uniform, either

• all headers should be renamed to .hpp as well
• maybe all sources may be renamed to .cc instead of this patch (haven't tested, just a wild guess)
• there may be a way to tell autotools explicitly that .h should be generated

There are 3 main differences, see unit tests.

1. AdvancedCountingScanner never enters a dead state, see test Count("[a-z\320\260-\321\217]+", " +", " \320\260\320\260\320\220 abc def \320\260 cd")
2. AdvancedCountingScanner returns 1 for Count("a", "b", "aaa"), while CountingScanner returns 0 there.
3. AdvancedCountingScanner better handles complex regular expressions, see test CountGreedy.

[email protected]:~/test/pire$ uname -a Linux ubuntu 2.6.28-11-server #42-Ubuntu SMP Fri Apr 17 02:45:36 UTC 2009 x86_64 GNU/Linux [email protected]:~/test/pire$ g++ -v Using built-in specs. Target: x86_64-linux-gnu Configured with: ../src/configure -v --with-pkgversion='Ubuntu 4.3.3-5ubuntu4' --with-bugurl=file:///usr/share/doc/gcc-4.3/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --enable-shared --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --enable-nls --with-gxx-include-dir=/usr/include/c++/4.3 --program-suffix=-4.3 --enable-clocale=gnu --enable-libstdcxx-debug --enable-objc-gc --enable-mpfr --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu Thread model: posix gcc version 4.3.3 (Ubuntu 4.3.3-5ubuntu4) [email protected]:~/test/pire$ make make all-recursive make[1]: Entering directory /home/yaoweibin/test/pire' Making all in pire make[2]: Entering directory/home/yaoweibin/test/pire/pire' make all-am make[3]: Entering directory /home/yaoweibin/test/pire/pire' /bin/bash ../ylwrap inline.lpp .c inline.cpp -- : make[3]: *** [inline.cpp] Error 1 make[3]: Leaving directory/home/yaoweibin/test/pire/pire' make[2]: *** [all] Error 2 make[2]: Leaving directory /home/yaoweibin/test/pire/pire' make[1]: *** [all-recursive] Error 1 make[1]: Leaving directory/home/yaoweibin/test/pire' make: *** [all] Error 2

Hi!

Thank you for the such cool library! I'm interested in packaging the library into some dependecy managers. It will be much easier if you make a release on GitHub, so in a package recipe I will rely on some "stable" version instead of specific commit. E.g. it much easier to create a package for Conan with release.

I found that from the last release (in 2013 - 7 years ago) there are a lot of changes. Will be fine if they will be released.

Thank you!

All Any elements took place on bison stack. Normally they are deleting during parsing process but if error happens they will remain on stack. Bison provides destructor operator for cleaning stack. We can use it we but should assure that all stack elements are valid for delete operator.

Added new class NoGlueLimitCountingScanner. It works exactly like AdvancedCountingScanner, but does not have limit on the number of glued regular expressions. New class is added to existing unit tests.

Added unit test for gluing many regular expressions. Current problem: classes CountingScanner and AdvancedCountingScanner don't check whether the limit on number of expressions is exceeded. Glueing too many expressions just leads to incorrect count results. Fix made: now Glue() function returns empty scanner when limit is exceeded. This corresponds to return value when glueing fails due to limit on number of states.

Some minor fixes and improvements: added move constructors, fixed compiler warnings, etc.

Add the new version of CapturingScanner based on NFA, which is more correct than the previous one. Add new operators of 'non-greedy' repetitions, for example, '+?', '??', '*?', which give a chance for users to tell, that the text, captured by those repetitions, should be the shortest one Added c++11 flag to configure.ac

IMHO the wording "the higher the priority" is somewhat misleading, as if it naturally means "the greater the priority", the statement is wrong, and if it means "the less the priority", it is not obvious.

Also fixed a typo in "eariler".

This is a cython-based python binding of PIRE. Mako template engine is used heavily to address repetitiveness of similar interfaces.

What is not wrapped:

• It is impossible yet to subclass Features and Scanners in python and pass the extension back to C++;
• Most low-level operations with Fsm;
• mmap- and Action-related methods and functions;
• Run for pair of scanners;
• Feature and Encoding classes.

Interface of the binding is similar to the original one. Differences:

• All C++-space global template functions are wrapped as python instance methods.
• Fsm::operator * () is wrapped as Fsm.Iterated().
• All scanners' states are represented as classes similar to Pire::RunHelper.
• Encoding, Feature and Option abstractions are replaced with single Options abstraction, which is used to tweak Lexer behavior. Options can be either parsed from string such as "aiyu" or composed of predefined constants such as I and UTF8.
• Instead of lexer::AddFeature(Capture(42)) you use lexer.AddCapturing(42).
• An OverflowError is raised on unsuccessful glue operation instead of returning empty scanner.

Hello, Do you know about Conan? Conan is modern dependency manager for C++. And will be great if your library will be available via package manager for other developers.

Here you can find example, how you can create package for the library.

If you have any questions, just ask :-)

Fix the following warning:

'*((void*)& s +72)' may be used uninitialized in this function
/multi.h:243:11: note: '*((void*)& s +72)' was declared here
Scanner s;

Other scanner types have this method. SlowScanner can implement it easily. I write template code parametrized with scanner type, in which I need LettersCount.