An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

Read this in other languages: English, 简体中文.

News:

Tutorial for exporting CenterNet from pytorch to tensorRT is released.

Tutorial Video

An Out-of-the-Box TensorRT-based Framework for High Performance Inference with C++/Python Support

  • C++ Interface: 3 lines of code is all you need to run a YoloX

    commit(image).get(); // return vector ">
    // create inference engine on gpu-0
    //auto engine = Yolo::create_infer("yolov5m.fp32.trtmodel", Yolo::Type::V5, 0);
    auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel", Yolo::Type::X, 0);
    
    // load image
    auto image = cv::imread("1.jpg");
    
    // do inference and get the result
    auto box = engine->commit(image).get();  // return vector
          
  • Python Interface:

    import trtpy
    
    model     = models.resnet18(True).eval().to(device)
    trt_model = tp.from_torch(model, input)
    trt_out   = trt_model(input)

INTRO

  1. High level interface for C++/Python.
  2. Based on TensorRT8.0.
  3. Simplify the implementation of custom plugin. And serialization and deserialization have been encapsulated for easier usage.
  4. Simplify the compile of fp32, fp16 and int8 for facilitating the deployment with C++/Python in server or embeded device.
  5. Models ready for use also with examples are RetinaFace, Scrfd, YoloV5, YoloX, Arcface, AlphaPose, CenterNet and DeepSORT(C++)

YoloX and YoloV5-series Model Test Report

app_yolo.cpp speed testing
  1. Resolution (YoloV5P5, YoloX) = (640x640), (YoloV5P6) = (1280x1280)
  2. max batch size = 16
  3. preprocessing + inference + postprocessing
  4. cuda10.2, cudnn8.2.2.26, TensorRT-8.0.1.6
  5. RTX2080Ti
  6. num of testing: take the average on the results of 100 times but excluding the first time for warmup
  7. Testing log: [workspace/perf.result.std.log (workspace/perf.result.std.log)
  8. code for testing: src/application/app_yolo.cpp
  9. images for testing: 6 images in workspace/inference
    • with resolution 810x1080,500x806,1024x684,550x676,1280x720,800x533 respetively
  10. Testing method: load 6 images. Then do the inference on the 6 images, which will be repeated for 100 times. Note that each image should be preprocessed and postprocessed.

Model Resolution Type Precision Elapsed Time FPS
yolox_x 640x640 YoloX FP32 21.879 45.71
yolox_l 640x640 YoloX FP32 12.308 81.25
yolox_m 640x640 YoloX FP32 6.862 145.72
yolox_s 640x640 YoloX FP32 3.088 323.81
yolox_x 640x640 YoloX FP16 6.763 147.86
yolox_l 640x640 YoloX FP16 3.933 254.25
yolox_m 640x640 YoloX FP16 2.515 397.55
yolox_s 640x640 YoloX FP16 1.362 734.48
yolox_x 640x640 YoloX INT8 4.070 245.68
yolox_l 640x640 YoloX INT8 2.444 409.21
yolox_m 640x640 YoloX INT8 1.730 577.98
yolox_s 640x640 YoloX INT8 1.060 943.15
yolov5x6 1280x1280 YoloV5_P6 FP32 68.022 14.70
yolov5l6 1280x1280 YoloV5_P6 FP32 37.931 26.36
yolov5m6 1280x1280 YoloV5_P6 FP32 20.127 49.69
yolov5s6 1280x1280 YoloV5_P6 FP32 8.715 114.75
yolov5x 640x640 YoloV5_P5 FP32 18.480 54.11
yolov5l 640x640 YoloV5_P5 FP32 10.110 98.91
yolov5m 640x640 YoloV5_P5 FP32 5.639 177.33
yolov5s 640x640 YoloV5_P5 FP32 2.578 387.92
yolov5x6 1280x1280 YoloV5_P6 FP16 20.877 47.90
yolov5l6 1280x1280 YoloV5_P6 FP16 10.960 91.24
yolov5m6 1280x1280 YoloV5_P6 FP16 7.236 138.20
yolov5s6 1280x1280 YoloV5_P6 FP16 3.851 259.68
yolov5x 640x640 YoloV5_P5 FP16 5.933 168.55
yolov5l 640x640 YoloV5_P5 FP16 3.450 289.86
yolov5m 640x640 YoloV5_P5 FP16 2.184 457.90
yolov5s 640x640 YoloV5_P5 FP16 1.307 765.10
yolov5x6 1280x1280 YoloV5_P6 INT8 12.207 81.92
yolov5l6 1280x1280 YoloV5_P6 INT8 7.221 138.49
yolov5m6 1280x1280 YoloV5_P6 INT8 5.248 190.55
yolov5s6 1280x1280 YoloV5_P6 INT8 3.149 317.54
yolov5x 640x640 YoloV5_P5 INT8 3.704 269.97
yolov5l 640x640 YoloV5_P5 INT8 2.255 443.53
yolov5m 640x640 YoloV5_P5 INT8 1.674 597.40
yolov5s 640x640 YoloV5_P5 INT8 1.143 874.91
app_yolo_fast.cpp speed testing. Never stop desiring for being faster
  • Highlight: 0.5 ms faster without any loss in precision compared with the above. Specifically, we remove the Focus and some transpose nodes etc, and implement them in CUDA kenerl function. But the rest remains the same.
  • Test log: workspace/perf.result.std.log
  • Code for testing: src/application/app_yolo_fast.cpp
  • Tips: you can do the modification while refering to the downloaded onnx. Any questions are welcomed through any kinds of contact.
  • Conclusion: the main idea of this work is to optimize the pre-and-post processing. If you go for yolox, yolov5 small version, the optimization might help you.
Model Resolution Type Precision Elapsed Time FPS
yolox_x_fast 640x640 YoloX FP32 21.598 46.30
yolox_l_fast 640x640 YoloX FP32 12.199 81.97
yolox_m_fast 640x640 YoloX FP32 6.819 146.65
yolox_s_fast 640x640 YoloX FP32 2.979 335.73
yolox_x_fast 640x640 YoloX FP16 6.764 147.84
yolox_l_fast 640x640 YoloX FP16 3.866 258.64
yolox_m_fast 640x640 YoloX FP16 2.386 419.16
yolox_s_fast 640x640 YoloX FP16 1.259 794.36
yolox_x_fast 640x640 YoloX INT8 3.918 255.26
yolox_l_fast 640x640 YoloX INT8 2.292 436.38
yolox_m_fast 640x640 YoloX INT8 1.589 629.49
yolox_s_fast 640x640 YoloX INT8 0.954 1048.47
yolov5x6_fast 1280x1280 YoloV5_P6 FP32 67.075 14.91
yolov5l6_fast 1280x1280 YoloV5_P6 FP32 37.491 26.67
yolov5m6_fast 1280x1280 YoloV5_P6 FP32 19.422 51.49
yolov5s6_fast 1280x1280 YoloV5_P6 FP32 7.900 126.57
yolov5x_fast 640x640 YoloV5_P5 FP32 18.554 53.90
yolov5l_fast 640x640 YoloV5_P5 FP32 10.060 99.41
yolov5m_fast 640x640 YoloV5_P5 FP32 5.500 181.82
yolov5s_fast 640x640 YoloV5_P5 FP32 2.342 427.07
yolov5x6_fast 1280x1280 YoloV5_P6 FP16 20.538 48.69
yolov5l6_fast 1280x1280 YoloV5_P6 FP16 10.404 96.12
yolov5m6_fast 1280x1280 YoloV5_P6 FP16 6.577 152.06
yolov5s6_fast 1280x1280 YoloV5_P6 FP16 3.087 323.99
yolov5x_fast 640x640 YoloV5_P5 FP16 5.919 168.95
yolov5l_fast 640x640 YoloV5_P5 FP16 3.348 298.69
yolov5m_fast 640x640 YoloV5_P5 FP16 2.015 496.34
yolov5s_fast 640x640 YoloV5_P5 FP16 1.087 919.63
yolov5x6_fast 1280x1280 YoloV5_P6 INT8 11.236 89.00
yolov5l6_fast 1280x1280 YoloV5_P6 INT8 6.235 160.38
yolov5m6_fast 1280x1280 YoloV5_P6 INT8 4.311 231.97
yolov5s6_fast 1280x1280 YoloV5_P6 INT8 2.139 467.45
yolov5x_fast 640x640 YoloV5_P5 INT8 3.456 289.37
yolov5l_fast 640x640 YoloV5_P5 INT8 2.019 495.41
yolov5m_fast 640x640 YoloV5_P5 INT8 1.425 701.71
yolov5s_fast 640x640 YoloV5_P5 INT8 0.844 1185.47

Setup and Configuration

Linux
  1. VSCode (highly recommended!)
  2. Configure your path for cudnn, cuda, tensorRT8.0 and protobuf.
  3. Configure the compute capability matched with your nvidia graphics card in Makefile/CMakeLists.txt
  4. Configure your library path in .vscode/c_cpp_properties.json
  5. CUDA version: CUDA10.2
  6. CUDNN version: cudnn8.2.2.26. Note that dev(.h file) and runtime(.so file) should be downloaded.
  7. tensorRT version:tensorRT-8.0.1.6-cuda10.2
  8. protobuf version(for onnx parser):protobufv3.11.4
  • CMake:
    • mkdir build && cd build
    • cmake ..
    • make yolo -j8
  • Makefile:
    • make yolo -j8
Linux: Compile for Python
  • compile and install
    • Makefile:
      • set use_python := true in Makefile
    • CMakeLists.txt:
      • set(HAS_PYTHON ON) in CMakeLists.txt
    • Type in make pyinstall -j8
    • Complied files are in python/trtpy/libtrtpyc.so
Windows
  1. Please check the lean/README.md for the detailed dependency

  2. In TensorRT.vcxproj, replace the with your own CUDA path

  3. In TensorRT.vcxproj, replace the with your own CUDA path

  4. In TensorRT.vcxproj, replace the compute_61,sm_61 with your compute capability.

  5. Configure your dependency or download it to the foler /lean. Configure VC++ dir (include dir and refence)

  6. Configure your env, debug->environment

  7. Compile and run the example, where 3 options are available.

Windows: Compile for Python
  1. Compile trtpyc.pyd. Choose python in visual studio to compile
  2. Copy dll and execute 'python/copy_dll_to_trtpy.bat'
  3. Execute the example in python dir by 'python test_yolov5.py'
  • if installation is needed, switch to target env(e.g. your conda env) then 'python setup.py install', which has to be followed by step 1 and step 2.
  • the compiled files are in python/trtpy/libtrtpyc.pyd
Other Protobuf Version
  • in onnx/make_pb.sh, replace the path protoc=/data/sxai/lean/protobuf3.11.4/bin/protoc in protoc with the protoc of your own version
#cd the path in terminal to /onnx
cd onnx

#execuete the command to make pb files
bash make_pb.sh
  • CMake:
    • replace the set(PROTOBUF_DIR "/data/sxai/lean/protobuf3.11.4") in CMakeLists.txt with the same path of your protoc.
mkdir build && cd build
cmake ..
make yolo -j64
  • Makefile:
    • replace the path lean_protobuf := /data/sxai/lean/protobuf3.11.4 in Makefile with the same path of protoc
make yolo -j64
TensorRT 7.x support
  • The default is tensorRT8.x
  1. Replace onnx_parser_for_7.x/onnx_parser to src/tensorRT/onnx_parser
    • bash onnx_parser/use_tensorrt_7.x.sh
  2. Configure Makefile/CMakeLists.txt path to TensorRT7.x
  3. Execute make yolo -j64
TensorRT 8.x support
  • The default is tensorRT8.x
  1. Replace onnx_parser_for_8.x/onnx_parser to src/tensorRT/onnx_parser
    • bash onnx_parser/use_tensorrt_8.x.sh
  2. Configure Makefile/CMakeLists.txt path to TensorRT8.x
  3. Execute make yolo -j64

Guide for Different Tasks/Model Support

YoloV5 Support
  • if pytorch >= 1.7, and the model is 5.0+, the model is suppored by the framework
  • if pytorch < 1.7 or yolov5(2.0, 3.0 or 4.0), minor modification should be done in opset.
  • if you want to achieve the inference with lower pytorch, dynamic batchsize and other advanced setting, please check our blog (http://zifuture.com:8090)(now in Chinese) and scan the QRcode via Wechat to join us.
  1. Download yolov5
git clone [email protected]:ultralytics/yolov5.git
  1. Modify the code for dynamic batchsize
# line 55 forward function in yolov5/models/yolo.py 
# bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
# x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()
# modified into:

bs, _, ny, nx = x[i].shape  # x(bs,255,20,20) to x(bs,3,20,20,85)
bs = -1
ny = int(ny)
nx = int(nx)
x[i] = x[i].view(bs, self.na, self.no, ny, nx).permute(0, 1, 3, 4, 2).contiguous()

# line 70 in yolov5/models/yolo.py
#  z.append(y.view(bs, -1, self.no))
# modified into:
z.append(y.view(bs, self.na * ny * nx, self.no))

# line 52 in yolov5/export.py
# torch.onnx.export(dynamic_axes={'images': {0: 'batch', 2: 'height', 3: 'width'},  # shape(1,3,640,640)
#                                'output': {0: 'batch', 1: 'anchors'}  # shape(1,25200,85)  修改为
# modified into:
torch.onnx.export(dynamic_axes={'images': {0: 'batch'},  # shape(1,3,640,640)
                                'output': {0: 'batch'}  # shape(1,25200,85) 
  1. Export to onnx model
cd yolov5
python export.py --weights=yolov5s.pt --dynamic --include=onnx --opset=11
  1. Copy the model and execute it
cp yolov5/yolov5s.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32
YoloX Support
  1. Download YoloX
git clone [email protected]:Megvii-BaseDetection/YOLOX.git
cd YOLOX
  1. Modify the code The modification ensures a successful int8 compilation and inference, otherwise Missing scale and zero-point for tensor (Unnamed Layer* 686) will be raised.
# line 206 forward fuction in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# self.hw = [x.shape[-2:] for x in outputs] 
self.hw = [list(map(int, x.shape[-2:])) for x in outputs]


# line 208 forward function in yolox/models/yolo_head.py. Replace the commented code with the uncommented code
# [batch, n_anchors_all, 85]
# outputs = torch.cat(
#     [x.flatten(start_dim=2) for x in outputs], dim=2
# ).permute(0, 2, 1)
proc_view = lambda x: x.view(-1, int(x.size(1)), int(x.size(2) * x.size(3)))
outputs = torch.cat(
    [proc_view(x) for x in outputs], dim=2
).permute(0, 2, 1)


# line 253 decode_output function in yolox/models/yolo_head.py Replace the commented code with the uncommented code
#outputs[..., :2] = (outputs[..., :2] + grids) * strides
#outputs[..., 2:4] = torch.exp(outputs[..., 2:4]) * strides
#return outputs
xy = (outputs[..., :2] + grids) * strides
wh = torch.exp(outputs[..., 2:4]) * strides
return torch.cat((xy, wh, outputs[..., 4:]), dim=-1)

# line 77 in tools/export_onnx.py
model.head.decode_in_inference = True
  1. Export to onnx
# download model
wget https://github.com/Megvii-BaseDetection/YOLOX/releases/download/0.1.1rc0/yolox_m.pth

# export
python tools/export_onnx.py -c yolox_m.pth -f exps/default/yolox_m.py --output-name=yolox_m.onnx --dynamic --no-onnxsim
  1. Execute the command
cp YOLOX/yolox_m.onnx tensorRT_cpp/workspace/
cd tensorRT_cpp
make yolo -j32
Retinaface Support
  1. Download Pytorch_Retinaface Repo
git clone [email protected]:biubug6/Pytorch_Retinaface.git
cd Pytorch_Retinaface
  1. Download model from the Training of README.md in https://github.com/biubug6/Pytorch_Retinaface#training .Then unzip it to the /weights . Here, we use mobilenet0.25_Final.pth

  2. Modify the code

# line 24 in models/retinaface.py
# return out.view(out.shape[0], -1, 2) is modified into 
return out.view(-1, int(out.size(1) * out.size(2) * 2), 2)

# line 35 in models/retinaface.py
# return out.view(out.shape[0], -1, 4) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 4)

# line 46 in models/retinaface.py
# return out.view(out.shape[0], -1, 10) is modified into
return out.view(-1, int(out.size(1) * out.size(2) * 2), 10)

# The following modification ensures the output of resize node is based on scale rather than shape such that dynamic batch can be achieved.
# line 89 in models/net.py
# up3 = F.interpolate(output3, size=[output2.size(2), output2.size(3)], mode="nearest") is modified into
up3 = F.interpolate(output3, scale_factor=2, mode="nearest")

# line 93 in models/net.py
# up2 = F.interpolate(output2, size=[output1.size(2), output1.size(3)], mode="nearest") is modified into
up2 = F.interpolate(output2, scale_factor=2, mode="nearest")

# The following code removes softmax (bug sometimes happens). At the same time, concatenate the output to simplify the decoding.
# line 123 in models/retinaface.py
# if self.phase == 'train':
#     output = (bbox_regressions, classifications, ldm_regressions)
# else:
#     output = (bbox_regressions, F.softmax(classifications, dim=-1), ldm_regressions)
# return output
# the above is modified into:
output = (bbox_regressions, classifications, ldm_regressions)
return torch.cat(output, dim=-1)

# set 'opset_version=11' to ensure a successful export
# torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False,
#     input_names=input_names, output_names=output_names)
# is modified into:
torch_out = torch.onnx._export(net, inputs, output_onnx, export_params=True, verbose=False, opset_version=11,
    input_names=input_names, output_names=output_names)


  1. Export to onnx
python convert_to_onnx.py
  1. Execute
cp FaceDetector.onnx ../tensorRT_cpp/workspace/mb_retinaface.onnx
cd ../tensorRT_cpp
make retinaface -j64
DBFace Support
make dbface -j64
Scrfd Support
Arcface Support
commit(make_tuple(face, landmarks)).get(); cout << feature << endl; // 1x512 ">
auto arcface = Arcface::create_infer("arcface_iresnet50.fp32.trtmodel", 0);
auto feature = arcface->commit(make_tuple(face, landmarks)).get();
cout << feature << endl;  // 1x512
  • In the example of Face Recognition, workspace/face/library is the set of faces registered.
  • workspace/face/recognize is the set of face to be recognized.
  • the result is saved in workspace/face/resultworkspace/face/library_draw
CenterNet Support

check the great details in tutorial/2.0

Bert Support(Chinese Classification)

the INTRO to Interface

Python Interface:Get onnx and trtmodel from pytorch model more easily
  • Just one line of code to export onnx and trtmodel. And save them for usage in the future.
import trtpy

model = models.resnet18(True).eval()
trtpy.from_torch(
    model, 
    dummy_input, 
    max_batch_size=16, 
    onnx_save_file="test.onnx", 
    engine_save_file="engine.trtmodel"
)
Python Interface:TensorRT Inference
  • YoloX TensorRT Inference
import trtpy

yolo   = tp.Yolo(engine_file, type=tp.YoloType.X)   # engine_file is the trtmodel file
image  = cv2.imread("inference/car.jpg")
bboxes = yolo.commit(image).get()
  • Seamless Inference from Pytorch to TensorRT
import trtpy

model     = models.resnet18(True).eval().to(device) # pt model
trt_model = tp.from_torch(model, input)
trt_out   = trt_model(input)
C++ Interface:YoloX Inference
commit(image).get(); ">
// create infer engine on gpu 0
auto engine = Yolo::create_infer("yolox_m.fp32.trtmodel", Yolo::Type::X, 0);

// load image
auto image = cv::imread("1.jpg");

// do inference and get the result
auto box = engine->commit(image).get();
C++ Interface:Comple Model in FP32/FP16
TRT::compile(
  TRT::Mode::FP32,   // compile model in fp32
  3,                          // max batch size
  "plugin.onnx",              // onnx file
  "plugin.fp32.trtmodel",     // save path
  {}                         //  redefine the shape of input when needed
);
  • For fp32 compilation, all you need is offering onnx file whose input shape is allowed to be redefined.
C++ Interface:Compile in int8
  • The in8 inference performs slightly worse than fp32 in precision(about -5% drop down), but stunningly faster. In the framework, we offer int8 inference
// define int8 calibration function to read data and handle it to tenor.
auto int8process = [](int current, int count, vector
    & images, shared_ptr
    
     & tensor){
    
     for(
     int i = 
     0; i < images.
     size(); ++i){
    
     // int8 compilation requires calibration. We read image data and set_norm_mat. Then the data will be transfered into the tensor.
        
     auto image = 
     cv::imread(images[i]);
        
     cv::resize(image, image, 
     cv::Size(
     640, 
     640));
        
     float mean[] = {
     0, 
     0, 
     0};
        
     float std[]  = {
     1, 
     1, 
     1};
        tensor->
     set_norm_mat(i, image, mean, std);
    }
};



     // Specify TRT::Mode as INT8

     auto model_file = 
     "yolov5m.int8.trtmodel";

     TRT::compile(
  TRT::Mode::INT8,            
     // INT8
  
     3,                          
     // max batch size
  
     "yolov5m.onnx",             
     // onnx
  model_file,                 
     // saved filename
  {},                         
     // redefine the input shape
  int8process,                
     // the recall function for calibration
  
     ".",                        
     // the dir where the image data is used for calibration
  
     ""                          
     // the dir where the data generated from calibration is saved(a.k.a where to load the calibration data.)
);
    
   
  • We integrate into only one int8process function to save otherwise a lot of issues that might happen in tensorRT official implementation.
C++ Interface:Inference
  • We introduce class Tensor for easier inference and data transfer between host to device. So that as a user, the details wouldn't be annoying.

  • class Engine is another facilitator.

print(); // load image auto image = imread("demo.jpg"); // get the model input and output node, which can be accessed by name or index auto input = engine->input(0); // or auto input = engine->input("images"); auto output = engine->output(0); // or auto output = engine->output("output"); // put the image into input tensor by calling set_norm_mat() float mean[] = {0, 0, 0}; float std[] = {1, 1, 1}; input->set_norm_mat(i, image, mean, std); // do the inference. Here sync(true) or async(false) is optional engine->forward(); // engine->forward(true or false) // get the outut_ptr, which can used to access the output float* output_ptr = output->cpu (); ">
// load model and get a shared_ptr. get nullptr if fail to load.
auto engine = TRT::load_infer("yolov5m.fp32.trtmodel");

// print model info
engine->print();

// load image
auto image = imread("demo.jpg");

// get the model input and output node, which can be accessed by name or index
auto input = engine->input(0);   // or auto input = engine->input("images");
auto output = engine->output(0); // or auto output = engine->output("output");

// put the image into input tensor by calling set_norm_mat()
float mean[] = {0, 0, 0};
float std[]  = {1, 1, 1};
input->set_norm_mat(i, image, mean, std);

// do the inference. Here sync(true) or async(false) is optional
engine->forward(); // engine->forward(true or false)

// get the outut_ptr, which can used to access the output
float* output_ptr = output->cpu<float>();
C++ Interface:Plugin
  • You only need to define kernel function and inference process. The details of code(e.g the serialization, deserialization and injection of plugin etc) are under the hood.
  • Easy to implement a new plugin in FP32 and FP16. Refer to HSwish.cu for details.
template<>
__global__ void HSwishKernel(float* input, float* output, int edge) {

    KernelPositionBlock;
    float x = input[position];
    float a = x + 3;
    a = a < 0 ? 0 : (a >= 6 ? 6 : a);
    output[position] = x * a / 6;
}

int HSwish::enqueue(const std::vector
    & inputs, std::vector
    
     & outputs, 
     const std::vector
     
      & weights, 
      void* workspace, cudaStream_t stream) {

    
      int count = inputs[
      0].
      count();
    
      auto grid = 
      CUDATools::grid_dims(count);
    
      auto block = 
      CUDATools::block_dims(count);
    HSwishKernel <<
      
       0, stream >>> (inputs[
       0].
       ptr<
       float>(), outputs[
       0].
       ptr<
       float>(), count);
    
       return 
       0;
}



       RegisterPlugin(HSwish);
      
     
    
   

About Us

Comments
  • no result in int8 mode

    no result in int8 mode

    Hi, thanks for your awesome code share!

    I test this repo on my Ubuntu 18 Linux with CUDA V11.2.152, TensorRT-8.0.0.3 and cudnn8.2.0.

    I tested yolox_s model in FP32、FP16 and INT8 mode, but in INT8 mode, the network output has no results.

    FP16 test code is below:

    static void test_fp16(Yolo::Type type){
    
        TRT::set_device(0);
        INFO("===================== test %s fp16 ==================================", Yolo::type_name(type));
    
        const char* name = nullptr;
        if(type == Yolo::Type::V5){
            name = "yolov5m";
        }else if(type == Yolo::Type::X){
            name = "yolox_s";
        }
    
        if(not requires(name))
            return;
    
        string onnx_file = iLogger::format("%s.onnx", name);
        string model_file = iLogger::format("%s.fp16.trtmodel", name);
        int test_batch_size = 1;  // 当你需要修改batch大于1时,请注意你的模型是否修改(看readme.md代码修改部分),否则会有错误
        
        // 动态batch和静态batch,如果你想要弄清楚,请打开http://www.zifuture.com:8090/
        // 找到右边的二维码,扫码加好友后进群交流(免费哈,就是技术人员一起沟通)
        if(not iLogger::exists(model_file)){
            TRT::compile(
                TRT::TRTMode_FP16,   // 编译方式有,FP32、FP16、INT8
                {},                         // onnx时无效,caffe的输出节点标记
                test_batch_size,            // 指定编译的batch size
                onnx_file,                  // 需要编译的onnx文件
                model_file,                 // 储存的模型文件
                {},                         // 指定需要重定义的输入shape,这里可以对onnx的输入shape进行重定义
                false                       // 是否采用动态batch维度,true采用,false不采用,使用静态固定的batch size
            );
        }
    
        forward_engine(model_file, type);
    }
    

    Below is the output of the log:

    [2021-09-01 19:38:45][info][app_yolo.cpp:240]:===================== test YoloX fp32 ==================================
    [2021-09-01 19:38:45][info][trt_builder.cpp:473]:Compile FP32 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:38:45][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:38:45][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:38:45][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:38:45][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:38:45][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:38:45][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:38:45][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:38:45][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:38:45][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:38:45][warn][trt_builder.cpp:33]:NVInfer WARNING: Convolution + generic activation fusion is disable due to incompatible driver or nvrtc
    [2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:38:46][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:18][info][trt_builder.cpp:690]:Build done 32689 ms !
    [2021-09-01 19:39:18][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][info][trt_infer.cpp:169]:Infer 0x7fe3f8000c40 detail
    [2021-09-01 19:39:19][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:39:19][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:39:19][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:39:19][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.10 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.19 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 8.72 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.03 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 5.95 ms
    [2021-09-01 19:39:19][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 5.92 ms
    [2021-09-01 19:39:19][info][yolo.cpp:214]:Engine destroy.
    [2021-09-01 19:39:19][info][app_yolo.cpp:277]:===================== test YoloX fp16 ==================================
    [2021-09-01 19:39:19][info][trt_builder.cpp:473]:Compile FP16 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:39:19][warn][trt_builder.cpp:483]:Platform not have fast fp16 support
    [2021-09-01 19:39:19][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:19][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:39:19][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:39:19][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:39:19][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:39:19][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:19][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:39:19][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:39:19][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Half2 support requested on hardware without native FP16 support, performance will be negatively affected.
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:19][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:47][info][trt_builder.cpp:690]:Build done 28282 ms !
    [2021-09-01 19:39:47][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:48][info][trt_infer.cpp:169]:Infer 0x7fe3f8015650 detail
    [2021-09-01 19:39:48][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:39:48][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:39:48][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:39:48][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 2 object, 10.75 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 1 object, 6.38 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 2 object, 9.12 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 3 object, 6.10 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 1 object, 6.07 ms
    [2021-09-01 19:39:48][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 1 object, 6.01 ms
    [2021-09-01 19:39:48][info][yolo.cpp:214]:Engine destroy.
    [2021-09-01 19:39:48][info][app_yolo.cpp:190]:===================== test YoloX int8 ==================================
    [2021-09-01 19:39:48][info][trt_builder.cpp:473]:Compile INT8 Onnx Model 'yolox_s.onnx'.
    [2021-09-01 19:39:48][info][trt_builder.cpp:593]:Using image list[6 files]: inference
    [2021-09-01 19:39:48][info][trt_builder.cpp:602]:Input shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:48][info][trt_builder.cpp:603]:Set max batch size = 1
    [2021-09-01 19:39:48][info][trt_builder.cpp:604]:Set max workspace size = 1024.00 MB
    [2021-09-01 19:39:48][info][trt_builder.cpp:605]:Dynamic batch dimension is false
    [2021-09-01 19:39:48][info][trt_builder.cpp:608]:Network has 1 inputs:
    [2021-09-01 19:39:48][info][trt_builder.cpp:614]:      0.[images] shape is 1 x 3 x 640 x 640
    [2021-09-01 19:39:48][info][trt_builder.cpp:620]:Network has 1 outputs:
    [2021-09-01 19:39:48][info][trt_builder.cpp:625]:      0.[output] shape is 1 x 8400 x 85
    [2021-09-01 19:39:48][info][trt_builder.cpp:670]:Building engine...
    [2021-09-01 19:39:48][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 1 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 2 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 3 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 4 / 6
    [2021-09-01 19:39:49][info][app_yolo.cpp:193]:Int8 5 / 6
    [2021-09-01 19:39:50][info][app_yolo.cpp:193]:Int8 6 / 6
    [2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:27][warn][trt_builder.cpp:33]:NVInfer WARNING: Detected invalid timing cache, setup a local cache instead
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][info][trt_builder.cpp:685]:No set entropyCalibratorFile, and entropyCalibrator will not save.
    [2021-09-01 19:40:59][info][trt_builder.cpp:690]:Build done 70917 ms !
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: The logger passed into createInferRuntime differs from one already assigned, 0x557f9ae330b0, logger not updated.
    
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][warn][trt_builder.cpp:33]:NVInfer WARNING: TensorRT was linked against cuBLAS/cuBLAS LT 11.4.2 but loaded cuBLAS/cuBLAS LT 11.4.1
    [2021-09-01 19:40:59][info][trt_infer.cpp:169]:Infer 0x7fe3b4000c40 detail
    [2021-09-01 19:40:59][info][trt_infer.cpp:170]: Max Batch Size: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:171]: Dynamic Batch Dimension: false
    [2021-09-01 19:40:59][info][trt_infer.cpp:172]: Inputs: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:176]:         0.images : shape {1 x 3 x 640 x 640}
    [2021-09-01 19:40:59][info][trt_infer.cpp:179]: Outputs: 1
    [2021-09-01 19:40:59][info][trt_infer.cpp:183]:         0.output : shape {1 x 8400 x 85}
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/car.jpg, 0 object, 7.75 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/group.jpg, 0 object, 4.04 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zand.jpg, 0 object, 6.04 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/zgjr.jpg, 0 object, 3.74 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/gril.jpg, 0 object, 3.72 ms
    [2021-09-01 19:40:59][info][app_yolo.cpp:94]:Save to YoloX_result/yq.jpg, 0 object, 3.74 ms
    [2021-09-01 19:40:59][info][yolo.cpp:214]:Engine destroy.
    

    Time-consuming in the FP16/FP32 modes is close, is it because the code I wrote has a problem?

    Your sincerely!

  • AffineMatrix求解?

    AffineMatrix求解?

    正常图像作缩放是不需要偏移的吧? M = [ scale, 0, -scale * from.width * 0.5 + to.width * 0.5 0, scale, -scale * from.height * 0.5 + to.height * 0.5 0, 0, 1 ] -scale * from.width * 0.5 + to.width * 0.5这个偏移量抵消了,

    但是第一次平移矩阵不是 P = [ 1, 0, -scale * from.width * 0.5 0, -1, scale * from.height * 0.5 0, 0, 1 ] 第二次平移矩阵不应该是: T = [ 1, 0, to.width * 0.5, 0, - 1, to.height * 0.5, 0, 0, 1 ]

    ??? 参考的矩阵: https://www.cnblogs.com/xuanyuyt/p/7112876.html

  • Environment Configuration Issues!

    Environment Configuration Issues!

    QQ截图20211119174416 QQ截图20211119173519 I have never used VS Code before. I want to learn from the video of the blogger, but there are always errors. Is there any solution?(I have installed Tensorrt on Windows)

  • arcface人脸特征提取不同人脸,计算的相似度都一样,是怎么回事呢?

    arcface人脸特征提取不同人脸,计算的相似度都一样,是怎么回事呢?

    余弦距离计算代码: def cosine_distance(matrix1, matrix2): matrix1_matrix2 = np.dot(matrix1, matrix2.transpose()) matrix1_norm = np.sqrt(np.multiply(matrix1, matrix1).sum(axis=1)) matrix1_norm = matrix1_norm[:, np.newaxis] matrix2_norm = np.sqrt(np.multiply(matrix2, matrix2).sum(axis=1)) matrix2_norm = matrix2_norm[:, np.newaxis] cosine_distance = np.divide(matrix1_matrix2, np.dot(matrix1_norm, matrix2_norm.transpose())) return cosine_distance

    是不是没有进行归一化呢,归一化有没有好的函数推荐呢?

  • Error,arcface onnx 转tensorrt 错误

    Error,arcface onnx 转tensorrt 错误

    使用了本项目提供给的arcface_iresnet50.onnx 模型
    错误如下:
    While parsing node number 182 [BatchNormalization]:
    ERROR: /home/Project/tensorRT_Pro-main/src/tensorRT/onnx_parser/onnx2trt_utils.cpp:1523 In function scaleHelper:
    [8] Assertion failed: dims.nbDims == 4 || dims.nbDims == 5
    [2021-10-04 16:39:25][error][trt_builder.cpp:517]:Can not parse OnnX file: arcface_iresnet50_iii.onnx
    

    计算机环境如下:

    tensorrt 7.2
    cudnn8.1
    cuda 11.2
    protobufv3.11.4
    gpu 3080  arch86
    

    BatchNormalization_182 是模型的倒数第二层。

  • preprocess  engine->forward  decode_kernel_invoker  线程问题

    preprocess engine->forward decode_kernel_invoker 线程问题

    老师,您好,preprocess 在commits,是在主线程执行,而decode_kernel_invoker是在work 函数中和forward在一个线程中执行,为什么会这么设计呀? 如果送入的图片存在高并发问题是否会导致主线程卡死在commits中,为什么不把preprocess过程和decode一样放在work线程中执行?

    然后我有个想法是不是可以把preprocess 和decode 剥离出来,通过多线程实现, work只做forward的工作,这样是不是更高效?

  • cmake编译失败

    cmake编译失败

    cmake能全部正常build,但是在make的时候正常编译到85%的时候报错了,以下是报错内容(我未修改报错文件内的代码)

    [ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_yolo_fast.cpp.o [ 83%] Building CXX object CMakeFiles/pro.dir/src/application/app_alphapose.cpp.o [ 85%] Building CXX object CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o In file included from /xxx/datav/projects/tensorRT_Pro/src/application/app_python/interface.cpp:6:0: /xxx/datav/projects/tensorRT_Pro/src/application/tools/pybind11.hpp:159:20: fatal error: Python.h: No such file or directory #include <Python.h> ^ compilation terminated. CMakeFiles/pro.dir/build.make:1142: recipe for target 'CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o' failed make[3]: *** [CMakeFiles/pro.dir/src/application/app_python/interface.cpp.o] Error 1 make[3]: *** Waiting for unfinished jobs.... CMakeFiles/Makefile2:328: recipe for target 'CMakeFiles/pro.dir/all' failed make[2]: *** [CMakeFiles/pro.dir/all] Error 2 CMakeFiles/Makefile2:537: recipe for target 'CMakeFiles/yolo.dir/rule' failed make[1]: *** [CMakeFiles/yolo.dir/rule] Error 2 Makefile:300: recipe for target 'yolo' failed make: *** [yolo] Error 2

  • protobuf编译有问题

    protobuf编译有问题

    make报错

    g++: error: google/protobuf/util/internal/.libs/proto_writer.o: No such file or directory
    Makefile:2372: recipe for target 'libprotobuf.la' failed
    make[2]: *** [libprotobuf.la] Error 1
    

    是不是要先下载googletest,如果是的话,是不是哪个版本呢? ubuntu18.04 jetson xavier nx

  • ][trt_builder.cpp:36]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.1

    ][trt_builder.cpp:36]:NVInfer: TensorRT was linked against cuBLAS/cuBLAS LT 11.6.3 but loaded cuBLAS/cuBLAS LT 11.3.1

    warning, errors are showed above, it is annoying and weird to concern the compatibility issues among tensorrt version and cuda, and cuda toolkit versions. i cannot figure out the difference among them, any help will be approciated!!!

    i have installed tensorrt !!!

    dpkg -l | grep tensorrt ii nv-tensorrt-repo-ubuntu1804-cuda10.0-trt5.0.2.6-ga-20181009 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda10.1-trt5.1.5.0-ga-20190427 1-1 amd64 nv-tensorrt repository configuration files ii nv-tensorrt-repo-ubuntu1804-cuda11.4-trt8.2.1.8-ga-20211117 1-1 amd64 nv-tensorrt repository configuration files ii tensorrt 8.2.1.8-1+cuda11.4 amd64

    it should be 8.2.18 cuda11.4 as i am concerned.

    and after i typed nvcc -V , the cuda version is nvcc as follows:

    NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2020 NVIDIA Corporation Built on Mon_Nov_30_19:08:53_PST_2020 Cuda compilation tools, release 11.2, V11.2.67 Build cuda_11.2.r11.2/compiler.29373293_0

    so which version of toolkit should i install??? currently, my cuda toolkit is v11.5.0 i think

  • yolox master分支的模型能够正常导出到onnx但生成engine失败

    yolox master分支的模型能够正常导出到onnx但生成engine失败

    [2021-10-26 14:29:01][info][app_yolo.cpp:121]:===================== test YoloX FP32 yolox_s ================================== [2021-10-26 14:29:01][info][trt_builder.cpp:471]:Compile FP32 Onnx Model 'yolox_s.onnx'. [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: Slice_562: slice size must be positive, size = [0] [2021-10-26 14:29:02][error][trt_builder.cpp:30]:NVInfer: INVALID_ARGUMENT: getPluginCreator could not find plugin ScatterND version 1 While parsing node number 565 [ScatterND]: ERROR: /home/work/tracking/tensorRT_Pro/src/tensorRT/onnx_parser/builtin_op_importers.cpp:4013 In function importFallbackPluginImporter: [8] Assertion failed: creator && "Plugin not found, are the plugin name, version, and namespace correct?" [2021-10-26 14:29:02][error][trt_builder.cpp:517]:Can not parse OnnX file: yolox_s.onnx [2021-10-26 14:29:02][error][yolo.cpp:138]:Engine yolox_s.FP32.trtmodel load failed [2021-10-26 14:29:02][error][app_yolo.cpp:42]:Engine is nullptr [100%] Built target yolo

  • 对输入输出的内存申请为啥要加个size_matrix 仿射矩阵的大小,而且对输出affine_matrix_device的内存分配是sizeof(job.additional.d2i)

    对输入输出的内存申请为啥要加个size_matrix 仿射矩阵的大小,而且对输出affine_matrix_device的内存分配是sizeof(job.additional.d2i)

            uint8_t* gpu_workspace        = (uint8_t*)workspace->gpu(size_matrix + size_image);
            float*   affine_matrix_device = (float*)gpu_workspace;
            uint8_t* image_device         = size_matrix + gpu_workspace;
    
            uint8_t* cpu_workspace        = (uint8_t*)workspace->cpu(size_matrix + size_image);
            float* affine_matrix_host     = (float*)cpu_workspace;
            uint8_t* image_host           = size_matrix + cpu_workspace;
           memcpy(image_host, image.data, size_image);
            memcpy(affine_matrix_host, job.additional.d2i, sizeof(job.additional.d2i));
            checkCudaRuntime(cudaMemcpyAsync(image_device, image_host, size_image, cudaMemcpyHostToDevice, preprocess_stream));
            checkCudaRuntime(cudaMemcpyAsync(affine_matrix_device, affine_matrix_host, sizeof(job.additional.d2i), cudaMemcpyHostToDevice, preprocess_stream));
    
  • 在Jetson AGX ORIN中遇到未知设备错误

    在Jetson AGX ORIN中遇到未知设备错误

    NVInfer: 2: [utils.cpp::checkMemLimit::380] Error Code 2: Internal Error (Assertion upperBound != 0 failed. Unknown embedded device detected. Please update the table with the entry: {{2055, 16, 32}, 24498},)

  • 佬,之前使用yolox推导没有问题,然后现在突然报错:Tensor allocator query failed

    佬,之前使用yolox推导没有问题,然后现在突然报错:Tensor allocator query failed

    佬,去年使用yolox推导没有问题,然后现在突然报错:Tensor allocator query failed. 而且我要是从加载onnx开始一套走下来,不会报错,但是单独加载引擎文件也会报上面的错:Tensor allocator query failed.

    去年是用的老版的yolox训练然后来推导,今年使用较新版的yolox(归一化有区别 @Y%HA5OG1655(79B}U@9S

  • simple_yolo中的compile转换带nms的yolov7_tiny.onnx, 失败!

    simple_yolo中的compile转换带nms的yolov7_tiny.onnx, 失败!

    使用如下命令生成带nms的yolov7_tiny版本onnx: python export.py --weights yolov7-tiny.pt --grid --end2end --simplify --topk-all 100 --iou-thres 0.65 --conf-thres 0.35 --img-size 640 640 --max-wh 640 然后使用simple_yolo中的compile转换onnx成tensorrt, 转换的时候: output shape is 100x7 图片 转换完加载模型的时候,反序列化时, output is 1x7 图片

TensorRT is a C++ library for high performance inference on NVIDIA GPUs and deep learning accelerators.

TensorRT Open Source Software This repository contains the Open Source Software (OSS) components of NVIDIA TensorRT. Included are the sources for Tens

Aug 8, 2022
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.
Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and deploy without Python.

Python Inference Script(PyIS) Python Inference Script is a Python package that enables developers to author machine learning workflows in Python and d

Feb 23, 2022
ncnn is a high-performance neural network inference framework optimized for the mobile platform
ncnn is a high-performance neural network inference framework optimized for the mobile platform

ncnn ncnn is a high-performance neural network inference computing framework optimized for mobile platforms. ncnn is deeply considerate about deployme

Aug 12, 2022
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.
Hardware-accelerated DNN model inference ROS2 packages using NVIDIA Triton/TensorRT for both Jetson and x86_64 with CUDA-capable GPU.

Isaac ROS DNN Inference Overview This repository provides two NVIDIA GPU-accelerated ROS2 nodes that perform deep learning inference using custom mode

Jul 18, 2022
Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla
Helper Class for Deep Learning Inference Frameworks: TensorFlow Lite, TensorRT, OpenCV, ncnn, MNN, SNPE, Arm NN, NNAbla

InferenceHelper This is a helper class for deep learning frameworks especially for inference This class provides an interface to use various deep lear

Aug 1, 2022
Forward - A library for high performance deep learning inference on NVIDIA GPUs
 Forward - A library for high performance deep learning inference on NVIDIA GPUs

a library for high performance deep learning inference on NVIDIA GPUs.

Mar 17, 2021
A library for high performance deep learning inference on NVIDIA GPUs.
A library for high performance deep learning inference on NVIDIA GPUs.

Forward - A library for high performance deep learning inference on NVIDIA GPUs Forward - A library for high performance deep learning inference on NV

Jul 31, 2022
Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference
 Edge ML Library - High-performance Compute Library for On-device Machine Learning Inference

Edge ML Library (EMLL) offers optimized basic routines like general matrix multiplications (GEMM) and quantizations, to speed up machine learning (ML) inference on ARM-based devices. EMLL supports fp32, fp16 and int8 data types. EMLL accelerates on-device NMT, ASR and OCR engines of Youdao, Inc.

Jul 21, 2022
PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.
PPLNN is a high-performance deep-learning inference engine for efficient AI inferencing.

PPLNN, which is short for "PPLNN is a Primitive Library for Neural Network", is a high-performance deep-learning inference engine for efficient AI inferencing.

Aug 17, 2022
Benchmark framework of 3D integrated CIM accelerators for popular DNN inference, support both monolithic and heterogeneous 3D integration

3D+NeuroSim V1.0 The DNN+NeuroSim framework was developed by Prof. Shimeng Yu's group (Georgia Institute of Technology). The model is made publicly av

Dec 21, 2021
Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.
Implement yolov5 with Tensorrt C++ api, and integrate batchedNMSPlugin. A Python wrapper is also provided.

yolov5 Original codes from tensorrtx. I modified the yololayer and integrated batchedNMSPlugin. A yolov5s.wts is provided for fast demo. How to genera

Jul 6, 2022
Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI
Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI

High-Performance-Computing-Experiments Experimental and Comparative Performance Measurements of High Performance Computing Based on OpenMP and MPI 实验结

Nov 27, 2021
Deep Learning API and Server in C++11 support for Caffe, Caffe2, PyTorch,TensorRT, Dlib, NCNN, Tensorflow, XGBoost and TSNE

Open Source Deep Learning Server & API DeepDetect (https://www.deepdetect.com/) is a machine learning API and server written in C++11. It makes state

Aug 14, 2022
Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt
Support Yolov4/Yolov3/Centernet/Classify/Unet. use darknet/libtorch/pytorch to onnx to tensorrt

ONNX-TensorRT Yolov4/Yolov3/CenterNet/Classify/Unet Implementation Yolov4/Yolov3 centernet INTRODUCTION you have the trained model file from the darkn

Aug 13, 2022
APFS module for linux, with experimental write support (out-of-tree repository)

Apple File System ================= The Apple File System (APFS) is the copy-on-write filesystem currently used on all Apple devices. This module pro

Aug 4, 2022
KSAI Lite is a deep learning inference framework of kingsoft, based on tensorflow lite

KSAI Lite English | 简体中文 KSAI Lite是一个轻量级、灵活性强、高性能且易于扩展的深度学习推理框架,底层基于tensorflow lite,定位支持包括移动端、嵌入式以及服务器端在内的多硬件平台。 当前KSAI Lite已经应用在金山office内部业务中,并逐步支持金山

Aug 11, 2022
Benchmark framework of compute-in-memory based accelerators for deep neural network (inference engine focused)

DNN+NeuroSim V1.3 The DNN+NeuroSim framework was developed by Prof. Shimeng Yu's group (Georgia Institute of Technology). The model is made publicly a

Aug 5, 2022
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.

Website | Documentation | Tutorials | Installation | Release Notes CatBoost is a machine learning method based on gradient boosting over decision tree

Aug 12, 2022
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.
High performance, easy-to-use, and scalable machine learning (ML) package, including linear model (LR), factorization machines (FM), and field-aware factorization machines (FFM) for Python and CLI interface.

What is xLearn? xLearn is a high performance, easy-to-use, and scalable machine learning package that contains linear model (LR), factorization machin

Aug 1, 2022