Adds proper Apple Pencil support to Unity's InputSystem.

This project aims to create an Apple Pencil device for Unity's InputSystem package.

The problem: Unity does not fully integrate with Apple Pencil. Apple Pencil can provide events at 240hz, but only does 60hz by default with an extra API for getting all the events. Apple also provides an API for estimating values like pen pressure before the real values are ready (filling in the correct values later), as well as predicting future events in order to reduce visual latency when drawing, not to mention barrel-tap events with the Apple Pencil 2. As of writing this, Unity doesn't support any of that, which means that it's impossible to draw a smooth curve with Apple Pencil using Unity.

Unity's Input Situation: As of writing this, Unity has two built-in Input methods. The original Input (Old Input), and the newer InputSystem package which is still in development but is packaged with newer versions of Unity. The old Input system is capped to framerate, so even if we could get 240z from Apple Pencil we still wouldn't be able to consume the events since Input would throw most of them away. (Note - A very recent alpha build adds support for getting coalesced events with Old Input on Windows, but in my testing it is still weirdly giving me fewer events on lower framerates.) The newer InputSystem package is able to give us uncapped events using its lower level onEvent() API, but it doesn't call the full Apple API so we still only get 60hz on the device.

The solution: We need to write native code that can retrieve Apple Pencil events using the full API, and then make an InputSystem Device subclass that provides them in Unity. There are a few parts to this:

  1. A replacement UnityView+iOS.mm that needs to be copied into your built XCode project. This lets us modify Unity's touch event handlers to call our own code instead.
  2. A native code bridge, iOSApplePencil.hpp, which exposes methods we can call from Unity code. UnityView+iOS.mm sends touch events to the bridge, which puts them in a buffer that both it and Unity can use directly. The bridge then tells Unity when new events have been put in the buffer. (Note - The bridge is unfortunately written in C++ instead of Objective-C because I don't have a Mac and can't compile Objective-C code without going through Unity Cloud Build, meaning 20 minute compile times. Writing it in C++ allows me to test it locally using fake events by compiling it into a DLL.)
  3. An ApplePencil Device that acts as an InputSystem Pointer.

What's working currently: Currently I can get Apple Pencil events at 240hz on my iPad, with the original Apple Pencil (I don't have an Apple Pencil 2 to test with). Right now it only sends position, pressure and tilt to Unity.

To use: Follow these steps:

  1. In Player Settings in Unity, check the allow unsafe code option. This lets us get a pointer to a memory buffer that we can share with the native code bridge.

  2. If not already done, make sure you have the new InputSystem enabled in Player Settings (under Active Input Handling).

  3. Put ApplePencil.cs in your Unity project. To get the full 240hz, you'll need to use it with onEvent() like this:

     InputSystem.onEvent += OnInputEvent;
     void OnInputEvent(InputEventPtr eventPtr, InputDevice device) {
         if (!eventPtr.IsA<StateEvent>() && !eventPtr.IsA<DeltaStateEvent>())
             return;
         var pencil = device as ApplePencil;
         if (pencil == null)
             return;
         var pos = pencil.position.ReadValueFromEvent(eventPtr);
         var pressure = pencil.pressure.ReadValueFromEvent(eventPtr);
         // draw at a smooth 240hz
     }
    
  4. After building your iOS project in Unity, copy UnityView+iOS.mm and iOSApplePencil.hpp to {BUILD}/Classes/UI. I'm using Cloud Build and a post-export hook to do this automatically, but maybe you could do it with a Unity post-build hook too.

Similar Resources

Protobuf for Proxyman app - Include Apple Silicon & Intel architecture

Protobuf for Proxyman macOS app Protobuf for Proxyman app - Include Apple Silicon & Intel architecture How to build Open the project on the latest Xco

Nov 29, 2021

Get the current average CPU frequency on Apple M1 chips.

M1 CPU Frequency Get the current average CPU frequency on Apple M1 chips. Usage Download the precompiled binary from the releases and run it in the te

Nov 30, 2022

My experiments with Linux for the Apple M1 platform, based on Asahi and Corellium patches

Linux kernel ============ There are several guides for kernel developers and users. These guides can be rendered in a number of formats, like HTML an

Oct 24, 2021

A pre-boot execution environment for Apple boards built on top of checkra1n

archOS A pre-boot execution environment for Apple boards built on top of checkra1n - currently based off the Checkra1n/PongoOS Repo. Building on macOS

Jan 17, 2022

Fairplay research - Some RE work on Apple's Fairplay DRM

Poor Man's Kernel Debuger This project loads FairplayIOKit kernel driver into userspace and make it possible for LLDB to debug How to Compile In proje

Nov 24, 2022

Allows to swap the Fn key and left Control key and other tweaks on Macbook Pro and Apple keyboards in GNU/Linux

A patched hid-apple kernel module UPDATE August 2020: swap_fn_leftctrl is now built-in in Linux 5.8 πŸŽ‰ UPDATE Jun 2020: New feature added (swap_fn_f13

Nov 23, 2022

A library to handle Apple Property List format in binary or XML

libplist A small portable C library to handle Apple Property List files in binary or XML format. Features The project provides an interface to read an

Nov 25, 2022

C Library for Apple Firmware (Amalgam)

DeviceTree Minimal code to deal with Apple's DeviceTree format. The Makefile builds dt and pmgr binaries, but you should also be able to use dt.c + dt

Sep 30, 2022

Filter driver which support changing DPI of mouse that does not support hardware dpi changing.

Custom Mouse DPI Driver ν•˜λ“œμ›¨μ–΄ DPI 변경이 λΆˆκ°€λŠ₯ν•œ λ§ˆμš°μŠ€λ“€μ˜ DPI 변경을 κ°€λŠ₯ν•˜κ²Œ ν•˜λŠ” ν•„ν„° λ“œλΌμ΄λ²„ κ²½κ³ : ν•΄λ‹Ή λ“œλΌμ΄λ²„λŠ” μ™„μ „νžˆ ν…ŒμŠ€νŠΈ λ˜μ§€ μ•Šμ•˜μŠ΅λ‹ˆλ‹€ Install ν•΄λ‹Ή λ“œλΌμ΄λ²„λŠ” μ„œλͺ…이 λ˜μ–΄μžˆμ§€μ•ŠμŠ΅λ‹ˆλ‹€. λ“œλΌμ΄λ²„λ₯Ό μ‚¬μš©ν•˜λ €λ©΄ tests

Sep 23, 2022
Automatic adds AFIX instructions for hydrogen atoms to SHELX input file based on geometry and residual desity

autoHFIX Adds AFIX instructions for hydrogen atoms to SHELX input files based on geometry and residual desity automatically Usage: autoHFIX.exe [-opti

Oct 7, 2021
Adds MSAAx4 antialiasing to Geometry Dash.
Adds MSAAx4 antialiasing to Geometry Dash.

GDAntiAliasing Adds MSAAx4 antialiasing to Geometry Dash. Currently only on Windows. Example Boom Slayer endscreen without antialiasing: Boom Slayer e

Sep 4, 2022
This program adds, displays, updates, searches and deletes records as per user preference.
This program adds, displays, updates, searches and deletes records as per user preference.

Store Management System This program adds, displays, updates, searches and deletes records as per user preference. Project Description And Example Inf

Feb 11, 2022
An REFramework plugin that adds Direct2D scripting APIs

REFramework Direct 2D This is an REFramework plugin that adds a Direct2D scripting API. Building Currently suggest building in RelWithDebInfo so that

Oct 24, 2022
A bootloader and experimentation playground for Apple Silicon

m1n1: an experimentation playground for Apple Silicon (And to some extent a Linux bootloader) Building You need an aarch64-linux-gnu-gcc cross-compile

Nov 25, 2022
Upload arbitrary data via Apple's Find My network.
Upload arbitrary data via Apple's Find My network.

Send My Send My allows you to to upload abritrary data from devices without an internet connection by (ab)using Apple's Find My network. The data is b

Nov 23, 2022
Apple TV 2/3 Jailbreak
Apple TV 2/3 Jailbreak

Blackb0x Apple TV 2/3 Jailbreak Untethered jailbreak tool that runs on modern versions of macOS. Devices supported: Apple TV 3,2 (A1469) (iOS 8.4.x un

Nov 29, 2022
An apple IIe emulator for the Raspberry Pi.
An apple IIe emulator for the Raspberry Pi.

ApplePi "applepi" is an apple IIe emulator for the Raspberry Pi, but can be built (I think) on most Linux distributions. It runs easily on a Pi4B, and

Jul 31, 2022
Apple cctools and ld64 port for Linux, *BSD and macOS

Apple cctools and ld64 port for Linux, *BSD and macOS

Nov 29, 2022
Swift Framework for Apple Health Integration with Terra Enabling Developers LTD.

TerraSwift This library allows developers to connect to TERRA ENABLING DEVELOPERS LTD. through Swift (implemented with Swift5.0). This library uses He

Oct 14, 2022