This is a study on how to do create a queue via IPC (inter-process communication)

IPC queue

This is a study on how to do create a queue via IPC (inter-process communication).

Please take a look at the examples of producer and consumer

Design

There are two main components that can be used independently:

SharedMem

exposes an interface to share memory among processes, it uses MMAP under the hood, and pre-allocates space that will be used.


SharedQueue

It uses the shared memory to create a ring buffer in the available space. Each message has a sequence number, that is prepended to the front on the message like a header, so we can keep track when reading.

The messages for this implementation have a fixed size, this way we can have O(1) access to memory by calculating the memory offset. This has some downsides if your domain has variable-length messages, but this could be easily implemented by modifying the queue and adding the message size into the header.

Due to each process being responsible for managing its own sequence number we end up having a lock-free, single-producer, multi-consumer model. This is heavily inspired by LMAX's Disruptor.

Be aware that due to the ring buffer reuse of space some messages can be lost if the consumer is lagging (not ingesting messages fast enough). This has to be measured for your use case and can be solved by increasing the buffer size of the shared memory.


Warning: those components can act as writer/reader (or producer/consumer) but never both simultaneously.

How to run

Just run make and then start the ./producer and ./consumer in different terminals.

Improvements

I'm not a C expert, and I'm sure this code can be improved a lot

This is just for demonstration purposes; thus, there are lots of cases that are not covered, some examples:

  • trying to write to a read-only resources and so on
  • try to change initiate more than once (writer becomes a reader)

Feel free to contribute.


Shout-out to @apuxbt for sharing some snippets that inspired me to move to mmap

Owner
Tarcísio Zotelli Ferraz
Tarcísio Zotelli Ferraz
Similar Resources

A library OS for Linux multi-process applications, with Intel SGX support

Graphene Library OS with Intel SGX Support A Linux-compatible Library OS for Multi-Process Applications NOTE: We are in the middle of transitioning ou

Jan 4, 2023

Inter-process communication library to enable allocation between processes/threads and send/receive of allocated regions between producers/consumer processes or threads using this ipc buffer.

This is a relatively simple IPC buffer that allows multiple processes and threads to share a dynamic heap allocator, designate "channels" between processes, and share that memory between producer/consumer pairs on those channels.

Aug 20, 2022

Implementation of System V shared memory (a type of inter process communication) in xv6 operating system.

NOTE: we have stopped maintaining the x86 version of xv6, and switched our efforts to the RISC-V version (https://github.com/mit-pdos/xv6-riscv.git)

Feb 21, 2022

iceoryx - true zero-copy inter-process-communication

 iceoryx - true zero-copy inter-process-communication

iceoryx is an inter-process-communication (IPC) middleware for various operating systems (currently we support Linux, MacOS and QNX). It has its origins in the automotive industry, where large amounts of data have to be transferred between different processes when it comes to driver assistance or automated driving systems.

Jan 6, 2023

The Efficient Study Planner (ESP) is a CLI app that gives an optimized plan to study for an upcoming exam.

Welcome to Efficient Study Planner 👋 Optimize your study plan with ESP! Table of Contents About The Project Tech Stack Prerequisites How To Use? Lice

Aug 3, 2022

C++11 thread safe, multi-producer, multi-consumer blocking queue, stack & priority queue class

BlockingCollection BlockingCollection is a C++11 thread safe collection class that provides the following features: Modeled after .NET BlockingCollect

Nov 23, 2022

CredBandit - Proof of concept Beacon Object File (BOF) that uses static x64 syscalls to perform a complete in memory dump of a process and send that back through your already existing Beacon communication channel

CredBandit CredBandit is a proof of concept Beacon Object File (BOF) that uses static x64 syscalls to perform a complete in memory dump of a process a

Dec 25, 2022

Cross-platform shared memory stream/buffer, header-only library for IPC in C/C++.

Cross-platform shared memory stream/buffer, header-only library for IPC in C/C++.

libsharedmemory libsharedmemory is a small C++11 header-only library for using shared memory on Windows, Linux and macOS. libsharedmemory makes it eas

Dec 4, 2022

Source Codes for Codimensional Incremental Potential Contact (C-IPC)

Source Codes for Codimensional Incremental Potential Contact (C-IPC) Reference This repository provides source code for: Minchen Li, Danny M. Kaufman,

Dec 29, 2022

Source Codes for Injective Deformation Processing (IDP) with Incremental Potential Contact (IPC)

Source Codes for Injective Deformation Processing Reference This repository provides source code for: Yu Fang*, Minchen Li* (equal contribution), Chen

Dec 6, 2022

ipcbuf - test/report the size of an IPC kernel buffer

ipcbuf - test/report the size of an IPC kernel buffer Different forms of IPC utilize different in-kernel buffers, depending on a variety of possibly s

Sep 7, 2022

Fast C++ IPC using shared memory

Fast C++ IPC using shared memory

Dec 14, 2022

A Solaris doors IPC implementation for FreeBSD.

General This repository contains a completely new, Illumos-compatible implementation of the Solaris Doors IPC mechanism for FreeBSD. The patch is comp

May 31, 2022

Project 1945 is a top down scroller game made with SDL2, CMake and programmed in C as a project of study for the Italian Videogames Academy for the 3rd year course of Videogames programming.

Project 1945 is a top down scroller game made with SDL2, CMake and programmed in C as a project of study for the Italian Videogames Academy for the 3rd year course of Videogames programming.

Project 1945 is a top down scroller game made with SDL2, CMake and programmed in C as a project of study for the Italian Videogames Academy for the 3rd year course of Videogames programming. The game is based on the old videogame 1945: The Final Front of the 2002.

Dec 30, 2022

Mecha allows you to debug, study, and modify the League of Legends client

Mecha allows you to debug, study, and modify the League of Legends client

Mecha allows you to debug, study, and modify the League of Legends client Usage To use Mecha, you must compile it first. See this for more information

Dec 29, 2022

TIP (translate it, please) is a plugin for VLC media player that helps you to study languages by watching videos.

vlc-tip-plugin TIP (translate it, please) is a plugin for VLC media player that helps you to study languages by watching videos. Features The plugin a

Oct 11, 2022

this lib with 26 template container and 10 kinds of algorithm, it is a good lib for study and usage

simple stl this lib simplify the achievement detail of common container, but add the container variety, the whole code partily follow Google Style. Em

Mar 10, 2022

A PE parser written as an exercise to study the PE file structure.

Description A PE parser written as an exercise to study the PE file structure. It parses the following parts of PE32 and PE32+ files: DOS Header Rich

Nov 18, 2022

Reverse engineering Genshin Impact anticheat to study how anticheats work on the Windows operating system.

Reverse engineering Genshin Impact anticheat to study how anticheats work on the Windows operating system.

mhyprot2 - Reverse engineering Genshin Impact anticheat for study 1. Demonstration video 2. Screenshots 2.1 Panel 3. Future updates and questions As I

Nov 16, 2022
A fast multi-producer, multi-consumer lock-free concurrent queue for C++11

moodycamel::ConcurrentQueue An industrial-strength lock-free queue for C++. Note: If all you need is a single-producer, single-consumer queue, I have

Jan 3, 2023
A fast single-producer, single-consumer lock-free queue for C++

A single-producer, single-consumer lock-free queue for C++ This mini-repository has my very own implementation of a lock-free queue (that I designed f

Jan 5, 2023
Code from https://queue.acm.org/detail.cfm?id=3448307 unzipped

Copyright (C) 2020-2021 Terence Kelly. All rights reserved. Author contact: [email protected], [email protected], [email protected] Adde

May 30, 2021
A bounded multi-producer multi-consumer concurrent queue written in C++11
A bounded multi-producer multi-consumer concurrent queue written in C++11

MPMCQueue.h A bounded multi-producer multi-consumer concurrent queue written in C++11. It's battle hardened and used daily in production: In the Frost

Dec 25, 2022
A bounded single-producer single-consumer wait-free and lock-free queue written in C++11
A bounded single-producer single-consumer wait-free and lock-free queue written in C++11

SPSCQueue.h A single producer single consumer wait-free and lock-free fixed size queue written in C++11. Example SPSCQueue<int> q(2); auto t = std::th

Dec 27, 2022
Smart queue that executes tasks in threadpool-like manner

execq execq is kind of task-based approach of processing data using threadpool idea with extended features. It supports different task sources and mai

Dec 22, 2022
Thread pool - Thread pool using std::* primitives from C++17, with optional priority queue/greenthreading for POSIX.

thread_pool Thread pool using std::* primitives from C++11. Also includes a class for a priority thread pool. Requires concepts and C++17, including c

Dec 30, 2022
Optimized primitives for collective multi-GPU communication

NCCL Optimized primitives for inter-GPU communication. Introduction NCCL (pronounced "Nickel") is a stand-alone library of standard communication rout

Dec 30, 2022
Coroutine - C++11 single .h asymmetric coroutine implementation via ucontext / fiber

C++11 single .h asymmetric coroutine implementation API in namespace coroutine: routine_t create(std::function<void()> f); void destroy(routine_t id);

Dec 20, 2022
The RaftLib C++ library, streaming/dataflow concurrency via C++ iostream-like operators

RaftLib is a C++ Library for enabling stream/data-flow parallel computation. Using simple right shift operators (just like the C++ streams that you wo

Dec 24, 2022