Experimental mutation testing tool for Swift and XCTest powered by mull

mull-xctest

Experimental mutation testing tool for Swift and XCTest powered by mull.

⚠️ This tool is still experimental and under development.

Installation

This tool is distributed through Homebrew. Run the following command to install:

brew install kateinoigakukun/tap/mull-xctest

Usage

SwiftPM

TODO: Add more docs

# Compile with bitcode and link by mull-ld in mull-tools directory.
$ swift build --build-tests \
    -Xswiftc -embed-bitcode \
    -Xswiftc -tools-directory -Xswiftc /usr/local/opt/mull-xctest/bin/mull-tools

# Run mutation tests and measure mutation score.
$ mull-xctest .build/debug/StubKitPackageTests.xctest

Xcode

TODO: Add more docs

$ xcodebuild build-for-testing \
  -project StubKit.xcodeproj -scheme StubKitTests \
  -destination "platform=iOS Simulator,name=iPhone 8" \
  OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" \
  OTHER_CFLAGS="\$(inherited) -fembed-bitcode" \
  LD="$(which mull-ld)" \
  OTHER_LDFLAGS="\$(inherited) \
    -Xmull --linker=clang \
    -Xmull --linker-flavor=clang"

$ mull-xctestrun \
  -Xxcodebuild -destination \
  -Xxcodebuild "platform=iOS Simulator,name=iPhone 8" \
  --test-target StubKitTests \
  path/to/Build/Products/StubKitTests_iphonesimulator14.4-x86_64.xctestrun

Similar Resources

A YAML parser and emitter in C++

yaml-cpp yaml-cpp is a YAML parser and emitter in C++ matching the YAML 1.2 spec. To get a feel for how it can be used, see the Tutorial or How to Emi

Dec 2, 2022

Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications

Zmeya Zmeya is a header-only C++11 binary serialization library designed for games and performance-critical applications. Zmeya is not even a serializ

Nov 2, 2022

Use to copy a file from an NTFS partitioned volume by reading the raw volume and parsing the NTFS structures.

ntfsDump Use to copy a file from an NTFS partitioned volume by reading the raw volume and parsing the NTFS structures. Similar to https://github.com/P

Nov 28, 2022

A C++11 ASN.1 BER Encoding and Decoding Library

fast_ber A performant ASN.1 BER encoding and decoding library written in C++11 Introduction fast_ber is a small, lightweight library for BER encoding

Dec 1, 2022

A high performance C++14 library for effortlessly reading and writing UBJSON

UbjsonCpp A high performance C++14 library for effortlessly reading and writing UBJSON This library implements UBJSON Draft 12 and Value semmantics Ab

Aug 2, 2022

High-performance specialized replacements for PHP's pack() and unpack() functions

ext-encoding High-performance specialized replacements for PHP's pack() and unpack() functions Under a profiler, it becomes obvious that PHP's pack()

Sep 17, 2022

SimpleBaseLib4CPP is a simple C++11 Base Encoding library that provides at the moment support for encoding and decoding various bases such as Base16, Base32 (various variants), Base58 (various variants), Base64 (various variants).

SimpleBaseLib4CPP SimpleBaseLib4CPP is a simple C++11 Base Encoding library that provides at the moment support for encoding and decoding various base

Nov 29, 2021

International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.

International obfuscated contest: Small C program to minify HTML sources and generate a minified HTML output.

HTML Minifier C International obfuscated contest: Just a small C program to minify HTML sources and generate a minified HTML output. Using $ gcc html-

Oct 27, 2022

Decode and Print Kadena Chainweb Block Headers

decode-header - Decode and Print Kadena Chainweb BlockHeaders The tool reads Kadena Chainweb Headers in various formats from stdin and prints a JSON r

Nov 5, 2021
Comments
  • `No mutants found. Mutation score: infinitely high` error

    `No mutants found. Mutation score: infinitely high` error

    I have a swift packet with UIKit import. Unfortunately there is no way to build it using swift build. That's why tried 3 different ways to use mull-xctests.

    1. I generate a project from a SPM with swift package generate-xcodeproj. Then I build it with xcodebuild build-for-testing -project PSBCore.xcodeproj -scheme PSBCore-Package -destination "platform=iOS Simulator,name=iPhone 11 Pro" OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" OTHER_CFLAGS="\$(inherited) -fembed-bitcode" LD="$(which mull-ld)" OTHER_LDFLAGS="\$(inherited) -Xmull --linker=clang -Xmull --linker-flavor=clang". It succeed. Then when I run mull-xctestrun -Xxcodebuild -destination -Xxcodebuild "platform=iOS Simulator,name=iPhone 11 Pro" --test-target PSBCoreTests /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/PSBCore-Package_iphonesimulator15.0-x86_64.xctestrun and get [info] No mutants found. Mutation score: infinitely high
    2. I even tried to test the project if the command similar to above, but if test instead of building-for-testing. After that I tried mull-xctest /path/to/Library/Developer/Xcode/DerivedData/PSBCore-btiytljcqmsisacjtlzuldouejsz/Build/Products/Debug-iphonesimulator/PSBCoreTests.xctest. It also gives me following:
    [info] Warm up run (threads: 1)
           [################################] 1/1. Finished in 45ms
    [info] Baseline run (threads: 1)
           [################################] 1/1. Finished in 41ms
    [info] No mutants found. Mutation score: infinitely high
    
    1. I've also tried to build a SPM with the command from the 1st, but without generating a Project file. This gives me an error:
    x86_64-apple-ios11.0-simulator: No such file or directory
    Command Ld failed with a nonzero exit code
    

    why is there x86_64-apple-ios11.0-simulator? I'm using Xcode 13.0, which doesn't seem to have ios11 simulator.

    What I'm doing wrong? Pls, help me.

  • Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib

    Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib

    Getting error on mull-ld launch and during xcodebuild setup:

    dyld: Library not loaded: /usr/local/opt/llvm/lib/libLLVM.dylib
      Referenced from: /usr/local/bin/mull-ld
      Reason: image not found
    Command Ld failed with a nonzero exit code
    
  • Final Release date?

    Final Release date?

    We are looking for a tool to implement Mutation Tests on iOS side in huge project. Do you guys have any ETA about releasing this library? Is this already usable?

  • Different result depending on number of workers

    Different result depending on number of workers

    I build your test project StubKit with command

    xcodebuild build-for-testing \
      -project StubKit.xcodeproj -scheme StubKit-Package \
      -destination "platform=iOS Simulator,name=iPhone 8" \
      OTHER_SWIFT_FLAGS="\$(inherited) -embed-bitcode" \
      OTHER_CFLAGS="\$(inherited) -fembed-bitcode" \
      LD="$(which mull-ld)" \
      OTHER_LDFLAGS="\$(inherited) \
        -Xmull --linker=clang \
        -Xmull --linker-flavor=clang"
    

    Then I try to find mutation with the following different --workers= values: 1, 2,4, 8, 9, 10. and finally, no parameter at all. Like follows:

    mull-xctestrun \              
      -Xxcodebuild -destination \
      -Xxcodebuild "platform=iOS Simulator,name=iPhone 8" \
      --test-target StubKitTests /Path/to/Library/Developer/Xcode/DerivedData/StubKit-errbmrafikpnlmgukqsogexfeeyk/Build/Products/StubKit-Package_iphonesimulator14.1-x86_64.xctestrun 
     --workers=1
    

    all of them produces different Mutation scores corresponding: 0%, 55%, 66%, 77%, 44%, 77%.

    It seems like workers don't continue to get the scheduled jobs. And the correct result is only available, when the number of workers are equal to number of mutants found. That become an issue for projects having more than 16 mutants ^.^

    It seems like there is a bug. Could you fix it or explain, how I could fix it by myself?

Header-only TOML config file parser and serializer for C++17 (and later!).
Header-only TOML config file parser and serializer for C++17 (and later!).

toml++ homepage ✨ This README is fine, but the toml++ homepage is better. ✨ Library features Header-only Supports the latest TOML release (v1.0.0), pl

Dec 4, 2022
A C++11 or library for parsing and serializing JSON to and from a DOM container in memory.
A C++11 or library for parsing and serializing JSON to and from a DOM container in memory.

Branch master develop Azure Docs Drone Matrix Fuzzing --- Appveyor codecov.io Boost.JSON Overview Boost.JSON is a portable C++ library which provides

Dec 1, 2022
libcluon is a small and efficient, single-file and header-only library written in modern C++ to power microservices.

libcluon Linux & OSX Build (TravisCI) Win64 Build (AppVeyor) Test Coverage Coverity Analysis CII Best Practices libcluon is a small single-file, heade

Nov 19, 2022
Nov 28, 2022
Cap'n Proto serialization/RPC system - core tools and C++ library
Cap'n Proto serialization/RPC system - core tools and C++ library

Cap'n Proto is an insanely fast data interchange format and capability-based RPC system. Think JSON, except binary. Or think Protocol Buffers, except

Dec 1, 2022
Header-only C++11 library to encode/decode base64, base64url, base32, base32hex and hex (a.k.a. base16) as specified in RFC 4648, plus Crockford's base32. MIT licensed with consistent, flexible API.

cppcodec Header-only C++11 library to encode/decode base64, base64url, base32, base32hex and hex (a.k.a. base16) as specified in RFC 4648, plus Crockf

Nov 17, 2022
MessagePack implementation for C and C++ / msgpack.org[C/C++]

msgpack for C/C++ It's like JSON but smaller and faster. Overview MessagePack is an efficient binary serialization format, which lets you exchange dat

Nov 24, 2022
FlatBuffers Compiler and Library in C for C

OS-X & Ubuntu: Windows: The JSON parser may change the interface for parsing union vectors in a future release which requires code generation to match

Nov 26, 2022
Utility to convert any binary file into C source that can be compiled and linked to the executable.

bin2c Utility to convert any binary file into C source that can be compiled and linked to the executable. bin2o Utility to convert any binary file int

Jul 14, 2021