三、將best.onnx轉為RKNN格式

? ? ? ? 這一步就需要我們進入到Ubuntu20.04系統(tǒng)中了，我的Ubuntu系統(tǒng)中已經下載好了anaconda，使用anaconda的好處就是可以方便的安裝一些庫，而且還可以利用conda來配置虛擬環(huán)境，做到環(huán)境與環(huán)境之間相互獨立。

??????? 對于我來說，使用了以下命令創(chuàng)建了一個名為rknn_ceshi的虛擬環(huán)境

conda create -n rknn_ceshi python=3.8

??????? 之后，點擊GitHub - rockchip-linux/rknn-toolkit2把整個項目給下載下來，解壓后，我們進入剛剛創(chuàng)立虛擬環(huán)境下安裝rknn-toolkit2。進入doc目錄后，輸入命令

pip install -r requirements_cp38-1.4.0.txt -i https://mirror.baidu.com/pypi/simple

??????? 這里一定要帶上百度的鏡像源，要不然會報錯，這個坑折磨了我整整3小時。

??????? 安裝完成后，出現(xiàn)下面的界面就說明已經安裝好了

????????接下來，我們進入packages文件夾，輸入一下命令

pip install rknn_toolkit2-1.4.0_22dcfef4-cp38-cp38-linux_x86_64.whl

??????? 出現(xiàn)以下界面，但是這里一定要注意，有一個超級大坑，超級大坑，超級大坑，后續(xù)我也是詢問了大佬才解決的這個問題，這個地方安裝的是rknn-toolkit2-1.4.0-22dcfef4！！！

??????? 出現(xiàn)上面那個界面后，我們在終端輸入python，再輸入以下命令，如果沒有報錯，則證明我們的環(huán)境已經搭載好了

from rknn.api import RKNN

??????? 接下來，我們要做的就是修改test.py里面的一些內容

????????我第一次做的時候，報了錯誤，當時提示的是沒有這個2-1.4.0-22dcfef4版本，我就跑到conda的環(huán)境包下，把所有的2-1.4.0-22dcfef4版本改成了2-1.4.0，才解決這個問題。

??????? 可是當我這一次在執(zhí)行這個文件的時候，就沒報這個錯誤，直接就跑起來了。

??????? 之后在我們的文件夾下出現(xiàn)了best.rknn這樣就可以到香橙派5上部署了！！！

四、香橙派5部署rknn實現(xiàn)NPU加速YOLOV5視頻推理

? ? ? ? 這里給大家強調一下，我使用的是RKNN的python版本來實現(xiàn)NPU加速的，這里我們需要到Github上下載RKNN官方教程，下載完成后進入該文件夾，輸入指令

cd /examples/onnx/yolov5

? ? ? ? 進入文件夾后，創(chuàng)建一個名為demo.py的文件，將以下代碼復制即可，我已經實現(xiàn)了實時視頻為了保護隱私，這里我依舊采用官方的yolov5s.rknn模型，而沒有用我自己的

import os
import urllib
import traceback
import time
import datetime as dt
import sys
import numpy as np
import cv2
from rknnlite.api import RKNNLiteRKNN_MODEL = 'yolov5s.rknn'
DATASET = './dataset.txt'QUANTIZE_ON = TrueOBJ_THRESH = 0.25
NMS_THRESH = 0.45
IMG_SIZE = 640CLASSES = ("person", "bicycle", "car", "motorbike ", "aeroplane ", "bus ", "train", "truck ", "boat", "traffic light","fire hydrant", "stop sign ", "parking meter", "bench", "bird", "cat", "dog ", "horse ", "sheep", "cow", "elephant","bear", "zebra ", "giraffe", "backpack", "umbrella", "handbag", "tie", "suitcase", "frisbee", "skis", "snowboard", "sports ball", "kite","baseball bat", "baseball glove", "skateboard", "surfboard", "tennis racket", "bottle", "wine glass", "cup", "fork", "knife ","spoon", "bowl", "banana", "apple", "sandwich", "orange", "broccoli", "carrot", "hot dog", "pizza ", "donut", "cake", "chair", "sofa","pottedplant", "bed", "diningtable", "toilet ", "tvmonitor", "laptop	", "mouse	", "remote ", "keyboard ", "cell phone", "microwave ","oven ", "toaster", "sink", "refrigerator ", "book", "clock", "vase", "scissors ", "teddy bear ", "hair drier", "toothbrush ")def sigmoid(x):return 1 / (1 + np.exp(-x))def xywh2xyxy(x):# Convert [x, y, w, h] to [x1, y1, x2, y2]y = np.copy(x)y[:, 0] = x[:, 0] - x[:, 2] / 2  # top left xy[:, 1] = x[:, 1] - x[:, 3] / 2  # top left yy[:, 2] = x[:, 0] + x[:, 2] / 2  # bottom right xy[:, 3] = x[:, 1] + x[:, 3] / 2  # bottom right yreturn ydef process(input, mask, anchors):anchors = [anchors[i] for i in mask]grid_h, grid_w = map(int, input.shape[0:2])box_confidence = sigmoid(input[..., 4])box_confidence = np.expand_dims(box_confidence, axis=-1)box_class_probs = sigmoid(input[..., 5:])box_xy = sigmoid(input[..., :2])*2 - 0.5col = np.tile(np.arange(0, grid_w), grid_w).reshape(-1, grid_w)row = np.tile(np.arange(0, grid_h).reshape(-1, 1), grid_h)col = col.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)row = row.reshape(grid_h, grid_w, 1, 1).repeat(3, axis=-2)grid = np.concatenate((col, row), axis=-1)box_xy += gridbox_xy *= int(IMG_SIZE/grid_h)box_wh = pow(sigmoid(input[..., 2:4])*2, 2)box_wh = box_wh * anchorsbox = np.concatenate((box_xy, box_wh), axis=-1)return box, box_confidence, box_class_probsdef filter_boxes(boxes, box_confidences, box_class_probs):"""Filter boxes with box threshold. It's a bit different with origin yolov5 post process!# Argumentsboxes: ndarray, boxes of objects.box_confidences: ndarray, confidences of objects.box_class_probs: ndarray, class_probs of objects.# Returnsboxes: ndarray, filtered boxes.classes: ndarray, classes for boxes.scores: ndarray, scores for boxes."""boxes = boxes.reshape(-1, 4)box_confidences = box_confidences.reshape(-1)box_class_probs = box_class_probs.reshape(-1, box_class_probs.shape[-1])_box_pos = np.where(box_confidences >= OBJ_THRESH)boxes = boxes[_box_pos]box_confidences = box_confidences[_box_pos]box_class_probs = box_class_probs[_box_pos]class_max_score = np.max(box_class_probs, axis=-1)classes = np.argmax(box_class_probs, axis=-1)_class_pos = np.where(class_max_score >= OBJ_THRESH)boxes = boxes[_class_pos]classes = classes[_class_pos]scores = (class_max_score* box_confidences)[_class_pos]return boxes, classes, scoresdef nms_boxes(boxes, scores):"""Suppress non-maximal boxes.# Argumentsboxes: ndarray, boxes of objects.scores: ndarray, scores of objects.# Returnskeep: ndarray, index of effective boxes."""x = boxes[:, 0]y = boxes[:, 1]w = boxes[:, 2] - boxes[:, 0]h = boxes[:, 3] - boxes[:, 1]areas = w * horder = scores.argsort()[::-1]keep = []while order.size > 0:i = order[0]keep.append(i)xx1 = np.maximum(x[i], x[order[1:]])yy1 = np.maximum(y[i], y[order[1:]])xx2 = np.minimum(x[i] + w[i], x[order[1:]] + w[order[1:]])yy2 = np.minimum(y[i] + h[i], y[order[1:]] + h[order[1:]])w1 = np.maximum(0.0, xx2 - xx1 + 0.00001)h1 = np.maximum(0.0, yy2 - yy1 + 0.00001)inter = w1 * h1ovr = inter / (areas[i] + areas[order[1:]] - inter)inds = np.where(ovr <= NMS_THRESH)[0]order = order[inds + 1]keep = np.array(keep)return keepdef yolov5_post_process(input_data):masks = [[0, 1, 2], [3, 4, 5], [6, 7, 8]]anchors = [[10, 13], [16, 30], [33, 23], [30, 61], [62, 45],[59, 119], [116, 90], [156, 198], [373, 326]]boxes, classes, scores = [], [], []for input, mask in zip(input_data, masks):b, c, s = process(input, mask, anchors)b, c, s = filter_boxes(b, c, s)boxes.append(b)classes.append(c)scores.append(s)boxes = np.concatenate(boxes)boxes = xywh2xyxy(boxes)classes = np.concatenate(classes)scores = np.concatenate(scores)nboxes, nclasses, nscores = [], [], []for c in set(classes):inds = np.where(classes == c)b = boxes[inds]c = classes[inds]s = scores[inds]keep = nms_boxes(b, s)nboxes.append(b[keep])nclasses.append(c[keep])nscores.append(s[keep])if not nclasses and not nscores:return None, None, Noneboxes = np.concatenate(nboxes)classes = np.concatenate(nclasses)scores = np.concatenate(nscores)return boxes, classes, scoresdef draw(image, boxes, scores, classes, fps):"""Draw the boxes on the image.# Argument:image: original image.boxes: ndarray, boxes of objects.classes: ndarray, classes of objects.scores: ndarray, scores of objects.fps: int.all_classes: all classes name."""for box, score, cl in zip(boxes, scores, classes):top, left, right, bottom = boxprint('class: {}, score: {}'.format(CLASSES[cl], score))print('box coordinate left,top,right,down: [{}, {}, {}, {}]'.format(top, left, right, bottom))top = int(top)left = int(left)right = int(right)bottom = int(bottom)cv2.rectangle(image, (top, left), (right, bottom), (255, 0, 0), 2)cv2.putText(image, '{0} {1:.2f}'.format(CLASSES[cl], score),(top, left - 6),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 0, 255), 2)def letterbox(im, new_shape=(640, 640), color=(0, 0, 0)):# Resize and pad image while meeting stride-multiple constraintsshape = im.shape[:2]  # current shape [height, width]if isinstance(new_shape, int):new_shape = (new_shape, new_shape)# Scale ratio (new / old)r = min(new_shape[0] / shape[0], new_shape[1] / shape[1])# Compute paddingratio = r, r  # width, height ratiosnew_unpad = int(round(shape[1] * r)), int(round(shape[0] * r))dw, dh = new_shape[1] - new_unpad[0], new_shape[0] - new_unpad[1]  # wh paddingdw /= 2  # divide padding into 2 sidesdh /= 2if shape[::-1] != new_unpad:  # resizeim = cv2.resize(im, new_unpad, interpolation=cv2.INTER_LINEAR)top, bottom = int(round(dh - 0.1)), int(round(dh + 0.1))left, right = int(round(dw - 0.1)), int(round(dw + 0.1))im = cv2.copyMakeBorder(im, top, bottom, left, right, cv2.BORDER_CONSTANT, value=color)  # add borderreturn im, ratio, (dw, dh)# ==================================
# 如下為改動部分，主要就是去掉了官方 demo 中的模型轉換代碼，直接加載 rknn 模型，并將 RKNN 類換成了 rknn_toolkit2_lite 中的 RKNNLite 類
# ==================================rknn = RKNNLite()# load RKNN model
print('--> Load RKNN model')
ret = rknn.load_rknn(RKNN_MODEL)# Init runtime environment
print('--> Init runtime environment')
# use NPU core 0 1 2
ret = rknn.init_runtime(core_mask=RKNNLite.NPU_CORE_0_1_2)
if ret != 0:print('Init runtime environment failed!')exit(ret)
print('done')# Create a VideoCapture object and read from input file
# If the input is the camera, pass 0 instead of the video file name
cap = cv2.VideoCapture(0)# Check if camera opened successfully
if (cap.isOpened()== False): print("Error opening video stream or file")# Read until video is completed
while(cap.isOpened()):start = dt.datetime.utcnow()# Capture frame-by-frameret, img = cap.read()if not ret:breakimg = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)img = cv2.resize(img, (IMG_SIZE, IMG_SIZE))# Inferenceprint('--> Running model')outputs = rknn.inference(inputs=[img])print('done')# post processinput0_data = outputs[0]input1_data = outputs[1]input2_data = outputs[2]input0_data = input0_data.reshape([3, -1]+list(input0_data.shape[-2:]))input1_data = input1_data.reshape([3, -1]+list(input1_data.shape[-2:]))input2_data = input2_data.reshape([3, -1]+list(input2_data.shape[-2:]))input_data = list()input_data.append(np.transpose(input0_data, (2, 3, 0, 1)))input_data.append(np.transpose(input1_data, (2, 3, 0, 1)))input_data.append(np.transpose(input2_data, (2, 3, 0, 1)))boxes, classes, scores = yolov5_post_process(input_data)duration = dt.datetime.utcnow() - startfps = round(10000000 / duration.microseconds)# draw process result and fpsimg_1 = cv2.cvtColor(img, cv2.COLOR_RGB2BGR)cv2.putText(img_1, f'fps: {fps}',(20, 20),cv2.FONT_HERSHEY_SIMPLEX,0.6, (0, 125, 125), 2)if boxes is not None:draw(img_1, boxes, scores, classes, fps)# show outputcv2.imshow("post process result", img_1)# Press Q on keyboard to  exitif cv2.waitKey(25) & 0xFF == ord('q'):break# When everything done, release the video capture object
cap.release()# Closes all the frames
cv2.destroyAllWindows()

? ? ? ? 這里我開啟了CPU定頻，可是并沒有提高多少NPU的使用率，干脆我也不再把代碼放上來了，等到以后有時間我再研究一下。

? ? ? ? 之后在終端中，運行命令

python demo.py

? ? ? ? 效果我放到了B站，感興趣的小伙伴可以點進去看一下B站視頻