Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS)

Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Various tools could be available for fast experimentation, for example sklearn, CNTK, Tensorflow, PyTorch and etc. However, when it comes to deployement, problems will emerge:

  • Is it optimized, fast or memory efficient?
  • Is the runtime or model compact enough for edge devices?
  • Is it easy to learn or cross-platform?

To solve those puzzles, the Python Inference Script(PyIS) is introduced.

Image

Installation

Build and install from source

Instruction

from pip source (Coming Soon)

python -m pip install pyis-python --upgrade 

Verification

Python Backend

    # Python backend
    from pyis.python import ops
    from pyis.python.model_context import save, load

    # create trie op
    trie = ops.CedarTrie()
    trie.insert('what time is it in Seattle?')
    trie.insert('what is the time in US?')

    # run trie match
    query = 'what is the time in US?'
    is_matched = trie.contains(query)

    # serialize
    save(trie, 'tmp/trie.pkl')

    # load and run
    trie = load('tmp/trie.pkl')
    is_matched = trie.contains(query)

LibTorch Backend

    # LibTorch backend
    import torch
    from pyis.torch import ops
    from pyis.torch.model_context import save, load

    # define torch model
    class TrieMatcher(torch.nn.Module):
        def __init__(self):
            super().__init__()
            self.trie = ops.CedarTrie()
            self.trie.insert('what time is it in Seattle?')
            self.trie.insert('what is the time in US?')

        def forward(self, query: str) -> bool:
            return self.trie.contains(query)

    # create torch model
    model = torch.jit.script(TrieMatcher())

    # run trie match
    query = 'what is the time in US?'
    is_matched = model.forward(query)

    # serialize
    save(model, 'tmp/trie.pt')

    # load and run
    model = load('tmp/trie.pt')
    is_matched = model.forward(query)

ONNXRuntime Backend

COMING SOON...

Build the Docs

Run the following commands and open docs/_build/html/index.html in browser.

    pip install sphinx myst-parser sphinx-rtd-theme sphinxemoji
    cd docs/

    make html         # for linux
    .\make.bat html   # for windows

Contributing

Please refer to CONTRIBUTING.md for the agreement and the instructions if you want to participate in this project.

Data Collection

The software may collect information about you and your use of the software and send it to Microsoft. Microsoft may use this information to provide services and improve our products and services. You may turn off the telemetry as described in the repository. There are also some features in the software that may enable you and Microsoft to collect data from users of your applications. If you use these features, you must comply with applicable law, including providing appropriate notices to users of your applications together with a copy of Microsoft’s privacy statement. Our privacy statement is located at https://go.microsoft.com/fwlink/?LinkID=824704. You can learn more about data collection and use in the help documentation and our privacy statement. Your use of the software operates as your consent to these practices.

Trademarks

This project may contain trademarks or logos for projects, products, or services. Authorized use of Microsoft trademarks or logos is subject to and must follow Microsoft’s Trademark & Brand Guidelines. Use of Microsoft trademarks or logos in modified versions of this project must not cause confusion or imply Microsoft sponsorship. Any use of third-party trademarks or logos are subject to those third-party’s policies.

Code of Conduct

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

License

Copyright (c) Microsoft Corporation. All rights reserved.

Licensed under the MIT license.

Owner
Microsoft
Open source projects and samples from Microsoft
Microsoft
Similar Resources

TFCC is a C++ deep learning inference framework.

TFCC is a C++ deep learning inference framework.

Sep 28, 2022

KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite English | 简体中文 KSAI Lite是一个轻量级、灵活性强、高性能且易于扩展的深度学习推理框架,底层基于tensorflow lite,定位支持包括移动端、嵌入式以及服务器端在内的多硬件平台。 当前KSAI Lite已经应用在金山office内部业务中,并逐步支持金山

Nov 2, 2022

PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

PPLNN, which is short for "PPLNN is a Primitive Library for Neural Network", is a high-performance deep-learning inference engine for efficient AI inferencing.

Nov 24, 2022

Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla

Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla

InferenceHelper This is a helper class for deep learning frameworks especially for inference This class provides an interface to use various deep lear

Nov 16, 2022

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

MACE is a deep learning inference framework optimized for mobile heterogeneous computing platforms.

Mobile AI Compute Engine (or MACE for short) is a deep learning inference framework optimized for mobile heterogeneous computing on Android, iOS, Linux and Windows devices.

Nov 27, 2022

An open-source, low-code machine learning library in Python

An open-source, low-code machine learning library in Python

An open-source, low-code machine learning library in Python 🚀 Version 2.3.6 out now! Check out the release notes here. Official • Docs • Install • Tu

Nov 28, 2022

Toolkit for software engineers to Deploy AI Systems Yourself (DAISY). DaisyKit SDK is the core of models and algorithms, which can be used to develop wrappers and applications for different platforms: mobile, embedded or web browsers.

Toolkit for software engineers to Deploy AI Systems Yourself (DAISY). DaisyKit SDK is the core of models and algorithms, which can be used to develop wrappers and applications for different platforms: mobile, embedded or web browsers.

Daisykit SDK DaisyKit SDK is the core of models and algorithms, which can be used to develop wrappers and applications for different platforms: mobile

Nov 30, 2022

Deploy SCRFD, an efficient high accuracy face detection approach, in your web browser with ncnn and webassembly

ncnn-webassembly-scrfd open https://nihui.github.io/ncnn-webassembly-scrfd and enjoy build and deploy Install emscripten

Nov 16, 2022

Deploy OcrLite in your web browser with ncnn and webassembly

Deploy OcrLite in your web browser with ncnn and webassembly

ncnn-webassembly-ocrlite Requirements ncnn webassembly opencv-mobile webassembly 3.4.13 Build Install emscripten git clone https://github.com/emscript

Nov 16, 2022
Comments
  • Features/ortsession update

    Features/ortsession update

    Update ORT Session

    • Separate Ort Env initialization to static initializer;
    • Fix minor issues in tests;

    Known Issues

    • def_static in torch binding does not support named arguments and default values, thus for OrtSession InitializeOrt in torch, the ort_dll_file must be passed (empty string to use PyIS default)
  • Update document and readme

    Update document and readme

    Update project readme and document

    • Update samples/figures in Project Readme and Docs;
    • Move document source from docs directory to docs_src directory for Github Pages hosting;
  • Enable Document Hosting with Github Pages

    Enable Document Hosting with Github Pages

    This PR makes below changes

    • Move all the documentation sources and build script to /docs_src folder;

    • Append command in /docs_src/make.bat to copy built doc files to /docs folder for Github Pages to host;

  • Add `$schema` to `cgmanifest.json`

    Add `$schema` to `cgmanifest.json`

    This pull request adds the JSON schema for cgmanifest.json.

    FAQ

    Why?

    A JSON schema helps you to ensure that your cgmanifest.json file is valid. JSON schema validation is a build-in feature in most modern IDEs like Visual Studio and Visual Studio Code. Most modern IDEs also provide code-completion for JSON schemas.

    How can I validate my cgmanifest.json file?

    Most modern IDEs like Visual Studio and Visual Studio Code have a built-in feature to validate JSON files. You can also use this small script to validate your cgmanifest.json file.

    Why does it suggest camel case for the properties?

    Component Detection is able to read camel case and pascal case properties. However, the JSON schema doesn't have a case-insensitive mode. We therefore suggest camel case as it's the most common format for JSON.

    Why is the diff so large?

    To deserialize the cgmanifest.json file, we use JSON.parse(). However, to serialize the JSON again we use prettier. We found that, in general, it gave smaller diffs than the default JSON.stringify() function.

High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Nov 25, 2022
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference
 Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.

Nov 25, 2022
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.
A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms.

iNeural A library for creating Artificial Neural Networks, for use in Machine Learning and Deep Learning algorithms. What is a Neural Network? Work on

Apr 5, 2022
Deep Scalable Sparse Tensor Network Engine (DSSTNE) is an Amazon developed library for building Deep Learning (DL) machine learning (ML) models

Amazon DSSTNE: Deep Scalable Sparse Tensor Network Engine DSSTNE (pronounced "Destiny") is an open source software library for training and deploying

Nov 30, 2022
TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for Tens

Dec 3, 2022
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

Nov 26, 2022
In-situ data analyses and machine learning with OpenFOAM and Python

PythonFOAM: In-situ data analyses with OpenFOAM and Python Using Python modules for in-situ data analytics with OpenFOAM 8. NOTE that this is NOT PyFO

Nov 12, 2022
Forward - A library for high performance deep learning inference on NVIDIA GPUs
 Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Mar 17, 2021
A library for high performance deep learning inference on NVIDIA GPUs.
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Nov 21, 2022