An area to test reading in ATLAS xAOD format and writing out to Parquet

xaod_to_parquet

An area to test reading in ATLAS xAOD format and writing out to Parquet

Getting the Code

Clone the repository with the --recursive flag to pick up parquet-writer:

git clone --recursive https://github.com/dantrim/xaod_to_parquet.git

Building

We must have an environment with both ATLAS AnalysisBase and Apache Arrow & Parquet libraries. The image dantrim/analysisbase-pq is built and tagged for several AnalysisBase releases. It simply adds the Apache Arrow & Parquet libraries on top of the already existing ATLAS AnalysisBase images. For example:

docker pull dantrim/analysisbase-pq:21.2.192

Once you have pulled the image:

cd /path/to/xaod_to_parquet
docker run --rm -ti -v ${PWD}:/workdir dantrim/analysisbase-pq:21.2.192 /bin/bash
source /release_setup.sh
cd src/
mkdir build
cd build
cmake -DCMAKE_MODULE_PATH=$(find /usr/lib64 -type d -name arrow) ..
make
source x86*/setup.sh

Toying Around

Currently a dummy executable dump-parquet gets built, which simply writes a Parquet file provided the test layout file test_layout.json.

After successfully building you can do:

cd /path/to/build
./x86_64-centos7-gcc8-opt/bin/dump-parquet --layout ../xaod_to_parquet/layouts/test_layout.json

After which you will have a Parquet file my_dataset.parquet:

parquet-tools show my_dataset.parquet
+-------+-------+------------------+
|   foo |   bar | baz              |
|-------+-------+------------------|
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
|    42 |    42 | [42.  42.1 42.2] |
+-------+-------+------------------+
Owner
Daniel Antrim
consider again that dot
Daniel Antrim
Similar Resources

Project basicly find the test that fails some code.

Stress Test project Overview many and many times in competitve programming when you have a solution got WA(wrong answer) and you do not know why and c

Feb 15, 2022

C++ Testing using spies and fakes for isolation and simulation

C++ Testing using spies and fakes for isolation and simulation

ELFspy is a library for linux for writing tests using fakes and spies in C++. For very large call graphs, the testing of the higher nodes can be diffi

Jun 12, 2022

Native ApprovalTests for C++ on Linux, Mac and Windows

Approval Tests for C++ ⬇️ Download the latest version (v.10.8.0) of the single header file here. 📖 Read the Docs Contents What are Approval Tests? Re

May 27, 2022

Googletest - Google Testing and Mocking Framework

GoogleTest OSS Builds Status Announcements Release 1.10.x Release 1.10.x is now available. Coming Soon Post 1.10.x googletest will follow Abseil Live

Jun 23, 2022

Practical mutation testing tool for C and C++

Mull Mull is a tool for Mutation Testing based on LLVM/Clang with a strong focus on C and C++ languages. For installation and usage please refer to th

Jun 16, 2022

A modern, C++11-native, single-file header-only, tiny framework for unit-tests, TDD and BDD (includes C++98 variant)

lest – lest errors escape testing This tiny C++11 test framework is based on ideas and examples by Kevlin Henney [1,2] and on ideas found in the CATCH

Jun 15, 2022

Modern c++17 unit testing framework on Microsoft Windows, Apple macOS, Linux, iOS and android.

Modern c++17 unit testing framework on Microsoft Windows, Apple macOS, Linux, iOS and android.

tunit Modern c++17 unit testing framework on Windows, macOS, Linux, iOS and android. Continuous Integration build status Operating system Status Windo

Apr 5, 2022

testing joystick under Linux environment, support monitoring disconnection state and auto recovery mode

qjoystick This qjoystick class is rewritten based on the library: https://github.com/drewnoakes/joystick. Please look at this library if you want to s

Oct 30, 2021

Xerus - A general purpose library for numerical calculations with higher order tensors, Tensor-Train Decompositions / Matrix Product States and other Tensor Networks

About The xerus library is a general purpose library for numerical calculations with higher order tensors, Tensor-Train Decompositions / Matrix Produc

Apr 20, 2021
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)
A modern, C++-native, header-only, test framework for unit-tests, TDD and BDD - using C++11, C++14, C++17 and later (or C++03 on the Catch1.x branch)

Catch2 v3 is being developed! You are on the devel branch, where the next major version, v3, of Catch2 is being developed. As it is a significant rewo

Jun 23, 2022
A dynamic mock tool for C/C++ unit test on Linux&MacOS X86_64

lmock 接口 替换一个函数,修改机器指令,用新函数替换旧函数,支持全局函数(包括第三方和系统函数)、成员函数(包括静态和虚函数)

Jun 13, 2022
PlatformIO + BL602 Bouffalo Arduino Core Test
 PlatformIO + BL602 Bouffalo Arduino Core Test

PlatformIO + BL602 Bouffalo Arduino Core Test Description Uses A custom extension of the PlatformIO SiFive Platform (https://github.com/maxgerhardt/pl

May 31, 2022
A test showing a flipped bit in a file encrypted on two different machines

ChaCha ASM Test I have observed that the ChaCha cipher may have very rarely divergent code paths for AVX vs. SSE. I have seen this in earlier CryptoPP

Dec 6, 2021
A tool to test if a shared library is dlopen'ble

A tool to test if a shared library is dlopen'ble

Oct 17, 2021
End to end test framework designed for Juce applications

JUCE End to End test framework What is it? This package provides a mechanism to end-to-end test a JUCE application Prerequisites CMake. Must be 3.18 o

May 29, 2022
Various Framework to do Unit Test in C++
Various Framework to do Unit Test in C++

Unit Test in C++ There are many frameworks to performs unit test in C++, we will present the most popular ones and show how to use them. The testing f

Nov 18, 2021
Handy C++ test framework

C++ Voyager Test Framework Voyager is a simple and handy C++ Unit Test framework. It is designed to be beautiful and expressive both. Try it to feel i

Mar 16, 2022
Network utility tool which enables to prototype or test network things.

netsck netsck is a network utility tool which is developed to prototype or test network things. It provides a shell inside which runs javascript engin

May 29, 2022
Upp11 - C++11 lightweight single header unit test framework

upp11 Lightweight C++11 single header unit test framework To use framework: Copy upp11.h in you project dir. Create unit test source files or modify e

Apr 4, 2019