Shader cross compiler to translate HLSL (Shader Model 4 and 5) to GLSL

Build status

XShaderCompiler ("Cross Shader Compiler")

Features

  • Cross compiles HLSL shader code (Shader Model 4 and 5) into GLSL
  • Simple to integrate into other projects
  • Low overhead translation (i.e. avoidance of unnecessary wrapper functions)
  • Dead code removal
  • Code reflection
  • Meaningful report output
  • Commentary preserving
  • Written in C++11

Language Bindings

  • C Wrapper (ISO C99)
  • C# Wrapper

License

3-Clause BSD License

Documentation

Progress

Version: 0.11 Alpha (Do not use in production code!)

See the TODO.md file for more information.

 Feature  Progress  Remarks
Vertex Shader ~80% Few language features are still left
Tessellation Control Shader ~20% InputPatch and patch-constant-function translation in progress
Tessellation Evaluation Shader ~20% OutputPatch translation in progress
Geometry Shader ~60% Code generation is incomplete
Fragment Shader ~80% Few language features are still left
Compute Shader ~80% Few language features are still left

Offline Compiler

The following command line translates the "Example.hlsl" file with the vertex shader entry point "VS", and the fragment shader entry point "PS":

xsc -E VS -T vert Example.hlsl -E PS -T frag Example.hlsl

The result are two GLSL shader files: "Example.VS.vert" and "Example.PS.frag".

Library Usage (C++)

#include <Xsc/Xsc.h>
#include <fstream>
#include <iostream>

int main()
{
    // Input file stream (use std::stringstream for in-code shaders).
    auto inputStream = std::make_shared<std::ifstream>("Example.hlsl");

    // Output file stream (GLSL vertex shader)
    std::ofstream outputStream("Example.VS.vert");

    // Fill the shader input descriptor structure
    Xsc::ShaderInput inputDesc;
    inputDesc.sourceCode     = inputStream;
    inputDesc.shaderVersion  = Xsc::InputShaderVersion::HLSL5;
    inputDesc.entryPoint     = "VS";
    inputDesc.shaderTarget   = Xsc::ShaderTarget::VertexShader;

    // Fill the shader output descriptor structure
    // Use outputDesc.options, outputDesc.formatting, and outputDesc.nameMangling for more settings
    Xsc::ShaderOutput outputDesc;
    outputDesc.sourceCode = &outputStream;

    // Optional output log (can also be a custom class)
    Xsc::StdLog log;

    // Optional shader reflection data (for shader code feedback)
    Xsc::Reflection::ReflectionData reflectData;

    // Translate HLSL code into GLSL
    try
    {
        bool result = Xsc::CompileShader(inputDesc, outputDesc, &log, &reflectData);
    }
    catch (const std::exception& e)
    {
        std::cerr << e.what() << std::endl;
    }

    return 0;
}

Library Usage (C#)

using System;
using System.IO;

namespace Example
{
    class Program
    {
        static void Main()
        {
            // Create instance of the XShaderCompiler
            var compiler = new XscCompiler();

            // Fill the shader input descriptor structure
            var inputDesc = new XscCompiler.ShaderInput();
            inputDesc.SourceCode     = File.ReadAllText("Example.hlsl");
            inputDesc.ShaderVersion  = XscCompiler.InputShaderVersion.HLSL5;
            inputDesc.EntryPoint     = "VS";
            inputDesc.Target         = XscCompiler.ShaderTarget.VertexShader;

            // Fill the shader output descriptor structure
            // Use outputDesc.Options, outputDesc.Formatting, and outputDesc.MameMangling for more settings
            var outputDesc = new XscCompiler.ShaderOutput();

            // Optional output log (can also be a custom class)
            var log = compiler.StandardLog;

            // Optional shader reflection data (for shader code feedback)
            var reflectData = new XscCompiler.ReflectionData();

            // Translate HLSL code into GLSL
            try
            {
                bool result = compiler.CompileShader(inputDesc, outputDesc, log, reflectData);

                if (result)
                {
                    // Write output shader code
                    File.WriteAllText("Example.VS.vert", outputDesc.SourceCode);
                }
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }
        }
    }
}

Real-time Debugger

A real-time debugger with UI is provided. Although this debugger is mainly used for developement of the compiler itself, it can also be used for a quick translation overview, to see how language constructs are being cross compiled.

Example of the real-time debugger (requires wxWidgets 3.1.0 or later):

docu/xsc_debugger_example_02.png

Language Differences

Here is a brief outline of High-Level differences between HLSL and GLSL, which XShaderCompiler is able to translate properly:

 Feature  HLSL  GLSL
Separation of Textures and Samplers Yes Only for Vulkan
Structure Inheritance Yes No
Nested Structures Yes No
Anonymous Structures Yes No
Structure Member Functions Yes No
Default Parameters Yes No
Object-Oriented Intrinsics Yes No
Multiple Entry Points Yes No
Type Aliasing Yes No
L-Values of Input Semantics Yes No
Implicit Type Conversion Extensive Restricted
Owner
Lukas Hermanns
Shader Tools Programmer @ Epic Games; Previously @ MA Lighting; Opinions are my own
Lukas Hermanns
Comments
  • Feature request: Option to add all uniforms to a uniform block

    Feature request: Option to add all uniforms to a uniform block

    The compiler I'm working with expects all uniforms (except for images and samplers) to be in a uniform block. It would be great if there would be an option to support this. If you are not planning on supporting this maybe you could give me some pointers as to how I could add it myself.

  • Mapping between HLSL and GLSL matrix conventions

    Mapping between HLSL and GLSL matrix conventions

    HLSL matrices are named as "row X column", i.e. float3x4 has 3 rows and 4 columns. GLSL matrices on the other hand are named as "column X row", i.e. mat3x4 has 3 columns and 4 rows.

    This same naming scheme also applies to swizzles and array indices when accessing matrices, as well as the order in which the initializers interpret their arguments.

    Currently the translation between matrix types is by directly mapping their indices. This code doesn't compile when translated to GLSL:

    float3x4 mat;
    vec4 vec;
    vec3 result = mul(mat, vec);
    

    Because the matrix is translated to mat3x4, and multiplication of a 3-column and 4-row matrix with a 4-column vector is not defined.


    I can see two possible solutions to the problem above:

    1. Keep the matrices as they are, but require external code to transpose them before passing them to GLSL. Also switch the order of arguments of any mul intrinsics.
    2. Map the matrices to their row/column equivalents in GLSL, flip swizzle access indices, flip array access indices and re-map initializers.

    I have actually already started working on option 2 but am running into a problem that I'm not sure how to solve.

    In particular the problem arises when we need to translate assignment to a single matrix row mat[1] = float4(1, 2, 3, 4); into GLSL, where this needs to map to multiple accesses (one for each column), e.g.

    mat4x3 mat;
    
    // Write to second row
    vec4 tmp = vec4(1, 2, 3, 4);
    mat[0][1] = tmp[0];
    mat[1][1] = tmp[1];
    mat[2][1] = tmp[2];
    mat[3][1] = tmp[2];
    

    I can do this fine for ExprStmnt, but what is the right way to handle this when such an assignment occurs in a loop conditional? Such matrix assignments are unlikely to be used in real world, so for now I just plan to print out errors, but if possible I'd like to solve it properly, eventually.

    Because if this problem I am considering that maybe we should go with option 1 instead (transposing matrices & reversing multiply order). What do you think?

  • Request:

    Request: "carried define"

    So, for specific reasons I don't want to get into, the OpenGL part of an engine I'm working on has a very specific naming convention I need to use (basically, I have only a number to deduce the name of the uniform I'm setting a value to - OpenGL 3.3 Core, cannot use Compatibility).

    The only way I've found to do this and use XSC, is to manually change the output's uniform names, from (for example)

    uniform mat4 vreg_0;
    #define ModelToWorld vreg_0
    

    And then the meat of the code translated via XSC can be used as-is.

    My request: Can we somehow have a way to tell XSC to "carry a define through"?

    • Silly implementation idea: "@#define" forces the define to be carried through
  • Ambiguous function call

    Ambiguous function call

    http://shader-playground.timjones.io/124766c34cc34e47ca09c3df1038889d

    The above link demonstrates the issue. FXC can compile fine but switching to XShaderCompiler results in errors. There are actually more errors visible in the shader playground than the ones I'm getting but that might be because I couldn't paste the whole shader

  • Feature request: Improve reflection

    Feature request: Improve reflection

    I am adding one request with multiple things to keep them in one place. If you prefer separate requests let me know.

    The improvements that are necessary for my use cases:

    • each constant buffer should have a vector of constants (uniforms) that are contained within
    • uniforms should provide information about the data type (float, int, bool, etc...) and the number of values (float, float2, float3, etc...)
    • uniforms contained in constant buffers should provide information about their byte offsets within the buffer (taking into consideration the packing)
    • all resources and constants should provide information if they are used in the currently compiled shader or not
    • input attributes should provide information about both the original HLSL name and the original usage of the input semantic (while keeping the provided info about the GLSL name)
  • Added reflection code for shader uniforms

    Added reflection code for shader uniforms

    I've added reflection code to handle shader uniforms since I have this need. One thing I didn't handle is the uniform binding location since, looking at the code, it seems it isn't handled by the compiler.

  • Buffer, RWBuffer -> samplerBuffer, imageBuffer

    Buffer, RWBuffer -> samplerBuffer, imageBuffer

    Hi. As we discussed for a bit previously, Buffer and RWBuffer no longer map to buffer (SSBO) in GLSL, and instead map to samplerBuffer and imageBuffer, respectively. imageBuffer is treated the same as image textures, while samplerBuffer is treated the same as read-only textures (except with no support for sampling, and support for operator[]).

  • Getting information on what triggers a given extension or other

    Getting information on what triggers a given extension or other

    Is there an option/way for this shader compiler to output reasons for a given extension or another ( GL_ARB_shading_language_420pack for example ) being pulled in?

  • Linking errors

    Linking errors

    Including Xsc.h and linking with xsc_core.lib gives the following errors:

    error LNK2019: unresolved external symbol "public: __cdecl Xsc::IndentHandler::IndentHandler(class std::basic_string<char,struct std::char_traits<char>,class std::allocator<char> > const &)" ([email protected]@@[email protected][email protected][email protected]@[email protected]@[email protected]@[email protected]@[email protected]@@Z) referenced in function "protected: __cdecl Xsc::Log::Log(void)" ([email protected]@@[email protected])
    error LNK2001: unresolved external symbol "public: virtual void __cdecl Xsc::StdLog::SubmitReport(class Xsc::Report const &)" ([email protected]@[email protected]@[email protected]@@Z)
    error LNK2019: unresolved external symbol "bool __cdecl Xsc::CompileShader(struct Xsc::ShaderInput const &,struct Xsc::ShaderOutput const &,class Xsc::Log *,struct Xsc::Reflection::ReflectionData *)" ([email protected]@@[email protected]@[email protected]@[email protected]@[email protected]@[email protected]@Z) referenced in function "public: static bool __cdecl Renderer::RCompileShader(struct RShaderCompilerInput const &,struct RShaderCompilerOutput &)" ([email protected]@@[email protected]@[email protected]@@Z)
    
  • Multiple outputs in ESSL3.0

    Multiple outputs in ESSL3.0

    Input shader:

    struct Output {
        float4 SV_A : SV_Target0;
        float4 SV_B : SV_Target1;
        float4 SV_C : SV_Target2;
    };
    
    Output main() {
        Output output;
        return output;
    }
    

    Output shader (ESSL300):

    #version 300 es
    
    out highp vec4 SV_Target0;
    out highp vec4 SV_Target1;
    out highp vec4 SV_Target2;
    
    void main()
    {
        
    }
    

    When compiling:

    • 'SV_Target0' : must explicitly specify all locations when using multiple fragment outputs
    • 'SV_Target1' : must explicitly specify all locations when using multiple fragment outputs
    • 'SV_Target2' : must explicitly specify all locations when using multiple fragment outputs

    In ESSL 3.0 specification (4.3.8.2):

    If there is more than one output, the location must be specified for all outputs.

    So it should specify the location if there is more than one output (or the auto binding is enabled).

    Sidenote: The spec also says:

    Fragment shaders cannot have input layout qualifiers. ... Vertex shaders cannot have output layout qualifiers.

    But the autoBinding add the layout qualifier anyway.

    Spec: https://www.khronos.org/registry/OpenGL/specs/es/3.0/GLSL_ES_Specification_3.00.pdf

  • Oddity when using an Input's variables in a define

    Oddity when using an Input's variables in a define

    I'm not sure if this will make sense, but here we go.

    #include "stuff.h"
    struct Input
    {
    	float4 thing : SV_POSITION;
    	float2 OtherThing : TEXCOORD0;
    };
    
    [stuff]
    
    float4 main(Input input) : SV_TARGET
    {
    	float4 output = float4(0,0,0,1);
    	float3 src = TEX2DLOD(Frame, float4(input.OtherThing, 0, 0)).rgb;
    
    	float3 luminance = otherMethodInA(src);
    [...]
    	return output;
    }
    
    

    In the above context, xsc will not convert the shader, and will explode with:

    2 ERRORS
    --------
    code generation error (../../shaders/directx9/AShader.psh:xx:yy) : missing array prefix expression for input/output semantic 'OtherThing'
    			tex2Dlod(Frame, (float4(input.OtherThing, 0.0f, 0.0f)));
    			                       ^~~~~~~~~
    generating output code failed
    compilation failed
    

    But assigning input.OtherThing to a float2, and using that new float2, will just work.

  • Interpolation modifier being recognized as keyword

    Interpolation modifier being recognized as keyword

    Cannot use certain words when naming a function or variable.

    XSC fails, allowed in FXC.

    Example shader:

    //////////////////////////////////////////////////////////////////////////////////////////
    // Pixel
    
    float sample(float2 tc)
    {
    	return 0.f;
    }
    
    float4 main() : SV_Target
    {
    	float sample = 0; // if we comment out the 'sample' function, XSC will fail on this variable..
    	return	0;
    }
    

    Error log:

    syntax error (test.ps:4:7) : unexpected token: interpolation modifier (expected identifier)
    float sample(float2 tc)
          ^~~~~~
    
  • Known Issues

    Known Issues

    Here is a brief overview of some known bugs:

    • [ ] Intrinsic argument type matching: Type matching is incomplete for intrinsic function calls.
    • [ ] Struct used as input and ouput: Structures can currently not be used for shader input and output semantics simultaneously.
    • [ ] Unicode: The compiler uses the C++ ASCII file streams, that are not aware of unicode characters, neither in the file contents, nor in their filenames.
    • [ ] Texture Operator[]: The bracket operator is currently not translated for Texture objects, neither Operator[] nor mips.Operator[][].
    • [ ] GetDimensions(...): Cannot be translated propery (for all texture types) -> use cbuffer to pass dimension data.
    • [ ] Boolean in lerp/mix: Third argument in lerp/mix intrinsic must not be casted to floating-points.
    • [ ] sampler as local variables: Sampler states for Vulkan can only appear as global uniforms or parameters and must be somehow removed when they appear as local variables in HLSL.
    • [ ] Static member variables: Definition of static member variables in HLSL can appear after the struct declaration, but in GLSL they must be moved upwards before the struct declaration.
  • GetDimensions() -> textureSize()

    GetDimensions() -> textureSize()

    Looks like this is trying to substitute the function directly.

    HLSL input: MainTexture.GetDimensions(0, size.x, size.y);

    GLSL output: textureSize(MainTexture, 0, size.x, size.y);

    Had some difficulty tracking down where the actual transform happens for this.

  • Metal target

    Metal target

    Any plans for Metal output target?

    Apple devices are stuck on OpenGL 4.1, which is fairly ancient at this point. They also seem to have zero interest in supporting Vulkan. The only way to create high-end and fully-cross platform graphical applications is to use Metal on Apple devices, which comes with its own shading language.

    I'd love to see XShaderCompiler become a one-stop shop for shader cross compilation, and support Metal as well.

    I'v noticed SPIR-V Cross project has added an experimental Metal target, but cross compiling through two (or even three, if starting with HLSL) intermedia languages feels really dirty. Unity also has glsloptimizer and Unreal hlslcc but neither are a match for XShaderCompiler's level of quality, by far.

  • Translation of

    Translation of "GroupMemoryBarrierWithGroupSync" Intrinsic

    @BearishSun Glslang seems to translate the GroupMemoryBarrierWithGroupSync intrinsic into this:

    memoryBarrierShared();
    barrier();
    

    But your proposal for the wrapper function translate it into this:

    groupMemoryBarrier();
    barrier();
    

    Do you know which of the two approaches are wrong?

    Greetings, Lukas

  • Regression: conversion of Half4x3

    Regression: conversion of Half4x3

    Before, half4x3 name = {name2[0*2 + 1], name2[1*2 + 1], name2[2*2 + 1], name2[3*2 + 1]}; was translated to mat4x3 name = mat4x3(name2[0 * 2 + 1], name2[1 * 2 + 1], name2[2 * 2 + 1], name2[3 * 2 + 1]);

    Now it translates to

    context error (somefile.psh:109:22) : invalid number of elements in initializer expression for type 'half4x3' (expected 12, but got 4)
    	half4x3 name = {name2[0*2 + 1], name2[1*2 + 1], name2[2*2 + 1], name2[3*2 + 1]};
                           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    analyzing input code failed
    compilation failed
    
⚔️ A tool for cross compiling shaders. Convert between GLSL, HLSL, Metal Shader Language, or older versions of GLSL.
⚔️ A tool for cross compiling shaders. Convert between GLSL, HLSL, Metal Shader Language, or older versions of GLSL.

A cross compiler for shader languages. Convert between SPIR-V, GLSL / GLSL ES, HLSL, Metal Shader Language, or older versions of a given language. Cross Shader wraps glslang and SPIRV-Cross, exposing a simpler interface to transpile shaders.

Dec 30, 2022
HLSL Parser and Translator for HLSL, GLSL, and MSL.

HLSLParser This is a fork of Unknownworld's hlslparser adapted to our needs in The Witness. We currently use it to translate pseudo-HLSL shaders (usin

Jul 2, 2022
HLSL Parser and Translator for HLSL, GLSL, and MSL.

HLSLParser This is a fork of Unknownworld's hlslparser adapted to our needs in The Witness. We currently use it to translate pseudo-HLSL shaders (usin

Dec 25, 2022
GLSL optimizer based on Mesa's GLSL compiler. Used to be used in Unity for mobile shader optimization.

GLSL optimizer ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a different shader compilati

Jan 3, 2023
Minify and obfuscate GLSL or HLSL code

Shader Minifier Shader Minifier is a tool that minifies and obfuscates shader code (GLSL and HLSL). Its original use-case is for the demoscene, for op

Jan 2, 2023
Khronos-reference front end for GLSL/ESSL, partial front end for HLSL, and a SPIR-V generator.

News Visual Studio 2013 is no longer supported As scheduled, Microsoft Visual Studio 2013 is no longer officially supported. Please upgrade to at leas

Jan 9, 2023
LLVM IR and optimizer for shaders, including front-end adapters for GLSL and SPIR-V and back-end adapter for GLSL

Licensing LunarGLASS is available via a three clause BSD-style open source license. Goals The primary goals of the LunarGLASS project are: Reduce the

Dec 8, 2022
HLSL to GLSL language translator based on ATI's HLSL2GLSL. Used in Unity.

HLSL to GLSL shader language translator ⚠️ As of mid-2016, the project is unlikely to have any significant developments. At Unity we are moving to a d

Dec 18, 2022
A cross platform shader language with multi-threaded offline compilation or platform shader source code generation
A cross platform shader language with multi-threaded offline compilation or platform shader source code generation

A cross platform shader language with multi-threaded offline compilation or platform shader source code generation. Output json reflection info and c++ header with your shaders structs, fx-like techniques and compile time branch evaluation via (uber-shader) "permutations".

Dec 14, 2022
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.
ESP32-Skid-Steer - Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on the model.

ESP32-Skid-Steer Bruder Catepillar Skid Steer model converted to RC, controlled by an ESP32 with 2 analog joysticks and a receiver that is an ESP32 on

Oct 27, 2022
ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages

ShaderConductor ShaderConductor is a tool designed for cross-compiling HLSL to other shading languages. Features Converts HLSL to readable, usable and

Dec 29, 2022
Shader Playground is a website for exploring shader compilers.
Shader Playground is a website for exploring shader compilers.

Shader Playground is a website for exploring shader compilers. Visit website Supported backends Compilers ANGLE Clspv DXC FXC Glslan

Dec 30, 2022
Skyline's fork of yuzu's shader compiler

Skyline Shader Compiler (SSC) A fork of yuzu's shader compiler modified for Skyline's needs with all other changes being upstreamed and changes from y

Dec 9, 2022
A small DLL that fixes tool's usage of the Halo 3 shader compiler.

h3-shader-compiler-fix A small DLL that fixes tool's usage of the Halo 3 shader compiler. Tool forgot to initialise the compiler before using it, so t

Jun 20, 2022
『HLSL シェーダーの魔導書』(ISBN978-4-7981-6428-1)のサンプルファイル

# サンプルデータについて 本データは、『HLSL シェーダーの魔導書』(清原 隆行 著、翔泳社 刊)の付属データです。 なお、本データは以下のサイトから入手可能です。 - Github:https://github.com/shoeisha-books/hlsl-grimoire-sampl

Dec 15, 2022
A Visual Studio extension that provides enhanced support for editing High Level Shading Language (HLSL) files
A Visual Studio extension that provides enhanced support for editing High Level Shading Language (HLSL) files

HLSL Tools for Visual Studio This extension is for Visual Studio 2017 / 2019. Go here for the Visual Studio Code extension. HLSL Tools is a Visual Stu

Dec 27, 2022
SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages.

SPIRV-Cross SPIRV-Cross is a tool designed for parsing and converting SPIR-V to other shader languages. Features Convert SPIR-V to readable, usable an

Jan 2, 2023
this is my simple voxel engine, appart from librairies like glad it is entierly written in C++ and GLSL

simple-voxel-raycaster this is my simple voxel engine, appart from librairies like glad it is entierly written in C++ and GLSL here is a gif: https://

Sep 26, 2022
Lightweight, cross-platform & full-featured shader IDE
Lightweight, cross-platform & full-featured shader IDE

SHADERed is a lightweight tool for writing and debugging shaders. It is easy to use, open source, cross-platform (runs on Windows, Linux & Web).

Dec 30, 2022