1.安裝torch

pip install torchvision torch

PyTorch的torchvision.models模塊中自帶的很多預(yù)定義模型。torchvision 是PyTorch的一個(gè)官方庫(kù)，專(zhuān)門(mén)用于處理計(jì)算機(jī)視覺(jué)任務(wù)。在這個(gè)庫(kù)中，可以找到許多常用的卷積神經(jīng)網(wǎng)絡(luò)模型，包括ResNet、VGG、AlexNet等，以及它們的不同變體，如resnet50、vgg16等

2.準(zhǔn)備模型

1.導(dǎo)出resnet50模型

import torch
import torchvision.models as modelsresnet50 = models.resnet50(pretrained=True)
resnet50.eval()
image = torch.randn(1, 3, 244, 244)
resnet50_traced = torch.jit.trace(resnet50, image)
resnet50(image)
resnet50_traced.save('model.pt')

創(chuàng)建resnet50_pytorch目錄，目錄下創(chuàng)建目錄1(1表示版本號(hào))，然后將model.pt模型放到resnet50_pytorch/1目錄下

執(zhí)行該P(yáng)ython文件的時(shí)候會(huì)從https://download.pytorch.org/models/resnet50-0676ba61.pth下載模型文件，保存到本地的.cache/torch/hub/checkoutpoints

如我是在容器中執(zhí)行的，保存路徑為/root/.cache/torch/hub/checkpoints/resnet50-0676ba61.pth

2.準(zhǔn)備模型配置

name: "resnet50_pytorch"
platform: "pytorch_libtorch"
max_batch_size: 128
input [{name: "INPUT__0"data_type: TYPE_FP32dims: [ 3, -1, -1 ]}
]
output [{name: "OUTPUT__0"data_type: TYPE_FP32dims: [ 1000 ]label_filename: "labels.txt"}
]
instance_group [{count: 1kind: KIND_GPU}
]

此時(shí)目錄結(jié)構(gòu)為

模型目錄的名稱(chēng)必須與config.pbtxt中指定的模型名稱(chēng)完全匹配。這是為了確保 Triton 能夠正確地識(shí)別和加載模型

3.加載模型

此時(shí)已經(jīng)可以通過(guò)triton加載模型，需要注意的model-repository指出resnet50_pytorch的上一級(jí)目錄即可(否則會(huì)報(bào)錯(cuò))，Triton會(huì)加載model-repo路徑下的所有模型

 /opt/tritonserver/bin/tritonserver --model-repository=/triton

?4.發(fā)送請(qǐng)求

想要獲取分類(lèi)的結(jié)果，可以設(shè)置?class_count=k，表示獲取 TopK 分類(lèi)預(yù)測(cè)結(jié)果。如果沒(méi)有設(shè)置這個(gè)選項(xiàng)，那么將會(huì)得到一個(gè) 1000 維的向量。

import numpy as np
import tritonclient.http as httpclient
import torch
from PIL import Imageif __name__ == '__main__':#1.創(chuàng)建triton clienttriton_client = httpclient.InferenceServerClient(url='127.0.0.1:8000')#2.加載圖片image = Image.open('/test_triton/24poJOgl7m_small.jpg')#3.對(duì)圖片進(jìn)行預(yù)處理，以滿(mǎn)足resnet50的input要求image = image.resize((224, 224), Image.ANTIALIAS)image = np.asarray(image)image = image / 255image = np.expand_dims(image, axis=0)image = np.transpose(image, axes=[0, 3, 1, 2])image = image.astype(np.float32)#4.創(chuàng)建inputsinputs = []inputs.append(httpclient.InferInput('INPUT__0', image.shape, "FP32"))inputs[0].set_data_from_numpy(image, binary_data=False)#5.創(chuàng)建outputsoutputs = []outputs.append(httpclient.InferRequestedOutput('OUTPUT__0', binary_data=False, class_count=1))#6.向triton server發(fā)送請(qǐng)求results = triton_client.infer('resnet50_pytorch', inputs=inputs, outputs=outputs)output_data0 = results.as_numpy('OUTPUT__0')print(output_data0.shape)print(output_data0)

AttributeError: module 'PIL.Image' has no attribute 'ANTIALIAS'

則降低PIL版本

pip uninstall Pillow
pip install Pillow==9.5.0

結(jié)果如下：

test_triton.py:12: DeprecationWarning: ANTIALIAS is deprecated and will be removed in Pillow 10 (2023-07-01). Use LANCZOS or Resampling.LANCZOS instead.image = image.resize((224, 224), Image.ANTIALIAS)
(1, 1)
[['10.245845:283']]

輸出的幾個(gè)數(shù)字的含義如下：

1. (1, 1)：這是輸出數(shù)據(jù)的形狀。這個(gè)元組表示輸出數(shù)據(jù)的維度，第一個(gè)數(shù)字表示批處理大小（batch size），第二個(gè)數(shù)字表示每個(gè)樣本的輸出數(shù)目。在這個(gè)結(jié)果中，批處理大小是1，每個(gè)樣本有1個(gè)輸出。

2. ['10.245845:283']：這是模型的輸出值。它是一個(gè)字符串?dāng)?shù)組，通常包含了一個(gè)或多個(gè)浮點(diǎn)數(shù)值，以字符串形式表示。在這個(gè)結(jié)果中，字符串 '10.245845:283' 可以分為兩部分：

   • 10.245845：這是模型對(duì)輸入圖像的分類(lèi)概率得分。它表示模型認(rèn)為輸入圖像屬于某個(gè)特定類(lèi)別的概率得分。通常，這個(gè)值越高，模型越確信輸入圖像屬于這個(gè)類(lèi)別。
   • 283：這通常是與類(lèi)別標(biāo)簽相關(guān)的索引或標(biāo)識(shí)符。這個(gè)索引可以用來(lái)查找與模型輸出的概率得分對(duì)應(yīng)的類(lèi)別名稱(chēng)。具體來(lái)說(shuō)，索引 283 對(duì)應(yīng)于 ImageNet 數(shù)據(jù)集中的一個(gè)類(lèi)別。您可以使用相應(yīng)的 labels.txt 文件來(lái)查找該索引對(duì)應(yīng)的類(lèi)別名稱(chēng)。

5.準(zhǔn)備標(biāo)簽

在第4步無(wú)論是使用class_count與否，都沒(méi)有直接返回分類(lèi)結(jié)果。這是因?yàn)镽esNet-50本身不包含與標(biāo)簽（labels）相關(guān)的信息，因?yàn)樗且粋€(gè)圖像分類(lèi)模型，它將輸入圖像分為一組預(yù)定義的類(lèi)別，但它并不知道這些類(lèi)別的名稱(chēng)。標(biāo)簽信息通常是根據(jù)您的具體任務(wù)和數(shù)據(jù)集來(lái)定義的。

不同的labels.txt會(huì)導(dǎo)致最終的分類(lèi)結(jié)果不一樣

wget https://raw.githubusercontent.com/pytorch/hub/master/imagenet_classes.txt

下載之后重命名為labels.txt,

將config.pbtxt的內(nèi)容改為如下：

name: "resnet50_pytorch"
platform: "pytorch_libtorch"
max_batch_size: 128
input [{name: "INPUT__0"data_type: TYPE_FP32dims: [ 3, -1, -1 ]}
]
output [{name: "OUTPUT__0"data_type: TYPE_FP32dims: [ 1000 ]label_filename: "labels.txt"}
]
instance_group [{count: 1kind: KIND_GPU}
]

?重新啟動(dòng)服務(wù)，重新發(fā)送請(qǐng)求，結(jié)果為

(1, 1)
[['10.245845:283:Persian cat']]

查詢(xún)labels.txt，283對(duì)應(yīng)的類(lèi)別是Persian cat(索引從0開(kāi)始)

3.使用ensemble

第2部分的client.py里可以看到進(jìn)行了數(shù)據(jù)處理，現(xiàn)在我們專(zhuān)門(mén)使用一個(gè)模型來(lái)進(jìn)行數(shù)據(jù)處理

首先創(chuàng)建resnet50_ensemble目錄，并把resnet50_pytorch拷貝到resnet50_ensemble目錄下

1.python script model

使用Python Script Model來(lái)完成image的數(shù)據(jù)處理，以符合input需求(正式叫法是前處理)，該類(lèi)型的model通過(guò)python backend來(lái)進(jìn)行execute。編寫(xiě)Python script model，需要實(shí)現(xiàn)如下接口供triton server調(diào)用

• initialize:加載model config；創(chuàng)建image預(yù)處理所需要的對(duì)象

• execute:有兩種模式：

  • Default model：execute輸入為batch request，返回的結(jié)果也應(yīng)該是相同order和number的batch response

  • Decoupled model：這里對(duì)返回的order和number都沒(méi)有限制，主要應(yīng)用在Automated Speech Recognition (ASR）

• finalize:是可選的。該函數(shù)允許在從Triton服務(wù)器卸載模型之前進(jìn)行任何必要的清理。

看不懂不要緊，先跑就行

創(chuàng)建一個(gè)model.py文件，內(nèi)容如下

import numpy as np
import sys
import json
import ioimport triton_python_backend_utils as pb_utilsfrom PIL import Image
import torchvision.transforms as transforms
import os
class TritonPythonModel:def initialize(self, args):# You must parse model_config. JSON string is not parsed hereself.model_config = model_config = json.loads(args['model_config'])# Get OUTPUT0 configurationoutput0_config = pb_utils.get_output_config_by_name(model_config, "OUTPUT_0")# Convert Triton types to numpy typesself.output0_dtype = pb_utils.triton_string_to_numpy(output0_config['data_type'])self.normalize = transforms.Normalize(mean=[0.485, 0.456, 0.406],std=[0.229, 0.224, 0.225])self.loader = transforms.Compose([transforms.Resize([224, 224]),transforms.CenterCrop(224),transforms.ToTensor(), self.normalize])def _image_preprocess(self, image_name):image = self.loader(image_name)#expand the dimension to nchwimage = image.unsqueeze(0)return imagedef execute(self, requests):output0_dtype = self.output0_dtyperesponses = []# Every Python backend must iterate over everyone of the requests# and create a pb_utils.InferenceResponse for each of them.for request in requests:# 1) 獲取request中name為INPUT_0的tensor數(shù)據(jù), 并轉(zhuǎn)換為image類(lèi)型in_0 = pb_utils.get_input_tensor_by_name(request, "INPUT_0")img = in_0.as_numpy()image = Image.open(io.BytesIO(img.tobytes()))       # 2) 進(jìn)行圖片的transformer，并將結(jié)果設(shè)置為numpy類(lèi)型img_out = self._image_preprocess(image)img_out = np.array(img_out)# 3) 構(gòu)造output tesnorout_tensor_0 = pb_utils.Tensor("OUTPUT_0", img_out.astype(output0_dtype))# 4) 設(shè)置resposneinference_response = pb_utils.InferenceResponse(output_tensors=[out_tensor_0])responses.append(inference_response)return responsesdef finalize(self):print('Cleaning up...')

該model.py的主要功能是對(duì)圖像進(jìn)行預(yù)處理，并生成推理響應(yīng)

對(duì)應(yīng)的config.pbtxt為

name: "preprocess"
backend: "python"
max_batch_size: 256
input [
{name: "INPUT_0"data_type: TYPE_UINT8 dims: [ -1 ]
}
]output [
{name: "OUTPUT_0"data_type: TYPE_FP32dims: [ 3, 224, 224 ]
}
]instance_group [{ kind: KIND_CPU }]

我將這個(gè)模塊放在了preprocess

2.ensemble model

ensemble model是用來(lái)描述Triton server模型處理的pipeline,其中僅有一個(gè)配置文件，并不存在真實(shí)的model

config.pbtxt內(nèi)容如下：

其中通過(guò)platform設(shè)置當(dāng)前model的類(lèi)型為ensemble

通過(guò)ensemble_scheduling來(lái)指明model間的調(diào)用關(guān)系，其中step指定了執(zhí)行的前后依賴(lài)關(guān)系

name: "ensemble_python_resnet50"
platform: "ensemble"
max_batch_size: 256
input [{name: "INPUT"data_type: TYPE_UINT8dims: [ -1 ]}
]
output [{name: "OUTPUT"data_type: TYPE_FP32dims: [ 1000 ]}
]
ensemble_scheduling {step [{model_name: "preprocess"model_version: -1input_map {key: "INPUT_0"value: "INPUT"  # 指向ensemble的input}output_map {key: "OUTPUT_0"value: "preprocessed_image"}},{model_name: "resnet50_pytorch"model_version: -1input_map {key: "INPUT__0"    #對(duì)應(yīng)resnet50_pytorch里的input名字value: "preprocessed_image" # 指向preprocess的output}output_map {key: "OUTPUT__0" #對(duì)應(yīng)resnet50_pytorch里的outputvalue: "OUTPUT"  # 指向ensemble的output}}]
}

此時(shí)resnet50_ensemble的目錄結(jié)構(gòu)為：

?3.啟動(dòng)程序并測(cè)試

啟動(dòng)程序

/opt/tritonserver/bin/tritonserver --model-repository=/triton/resnet50_ensemble

測(cè)試代碼為

import numpy as np
import tritonclient.http as httpclient
import torch
from PIL import Imageif __name__ == '__main__':triton_client = httpclient.InferenceServerClient(url='127.0.0.1:8000')img_path = '/test_triton/24poJOgl7m_small.jpg'image = np.fromfile(img_path, dtype='uint8')image = np.expand_dims(image, axis=0)#設(shè)置inputinputs = []inputs.append(httpclient.InferInput('INPUT', image.shape, "UINT8"))inputs[0].set_data_from_numpy(image)#設(shè)置outputoutputs = []outputs.append(httpclient.InferRequestedOutput('OUTPUT', binary_data=False, class_count=1))#發(fā)送請(qǐng)求results = triton_client.infer('ensemble_python_resnet50', inputs=inputs, outputs=outputs)output_data0 = results.as_numpy('OUTPUT')print(output_data0.shape)print(output_data0)

運(yùn)行結(jié)果為

(1, 1)
[['9.462329:434:bath towel']]

4.dali model

在第3部分，把數(shù)據(jù)處理放到了triton server進(jìn)行，但問(wèn)題在于數(shù)據(jù)處理的操作并沒(méi)有充分利用硬件資源。為了加速模型的推理速度，一般將triton server部署在GPU節(jié)點(diǎn)上(第3部分的數(shù)據(jù)處理是在CPU上進(jìn)行的)。將數(shù)據(jù)處理轉(zhuǎn)移到GPU上，可以使用nvidia提供的dali數(shù)據(jù)處理庫(kù)

首先創(chuàng)建resnet50_ensemble_dali目錄，并把resnet50_pytorch模型拷貝到resnet50_ensemble_dali路徑下

1.準(zhǔn)備dali模型

安裝依賴(lài)

curl -O https://developer.download.nvidia.com/compute/redist/nvidia-dali-cuda110/nvidia_dali_cuda110-1.28.0-8915299-py3-none-manylinux2014_x86_64.whlpip install?nvidia_dali_cuda110-1.28.0-8915299-py3-none-manylinux2014_x86_64.whl

在Releases · NVIDIA/DALI · GitHub下載與自己系統(tǒng)適配的whl

Python文件如下

import nvidia.dali as dali
import nvidia.dali.fn as fn@dali.pipeline_def(batch_size=128, num_threads=4, device_id=0)
def pipeline():images = fn.external_source(device='cpu', name='DALI_INPUT_0')images = fn.resize(images, resize_x=224, resize_y=224)images = fn.transpose(images, perm=[2, 0, 1])images = images / 255return imagespipeline().serialize(filename='./model.dali')

執(zhí)行該P(yáng)ython文件將得到model.dali模型

在resnet50_ensemble_dali目錄下創(chuàng)建resnet50_dali,把model.dali放到該目錄下

對(duì)應(yīng)的config.pbtxt文件為

name: "resnet50_dali"
backend: "dali"
max_batch_size: 128
input [{name: "DALI_INPUT_0"data_type: TYPE_FP32dims: [ -1, -1, 3 ]}
]output [{name: "DALI_OUTPUT_0"data_type: TYPE_FP32dims: [ 3, 224, 224 ]}
]
instance_group [{count: 1kind: KIND_GPUgpus: [ 0 ]}
]

2.創(chuàng)建pipeline

創(chuàng)建ensemble_python_resnet50目錄，和3.2一樣，對(duì)應(yīng)的config.pbtxt內(nèi)容為

name: "ensemble_python_resnet50"
platform: "ensemble"
max_batch_size: 128
input [{name: "INPUT"data_type: TYPE_FP32dims: [ -1, -1, 3 ]}
]
output [{name: "OUTPUT"data_type: TYPE_FP32dims: [ 1000 ]}
]
ensemble_scheduling {step [{model_name: "resnet50_dali"model_version: -1input_map {key: "DALI_INPUT_0"value: "INPUT"  # 指向ensemble的input}output_map {key: "DALI_OUTPUT_0"value: "preprocessed_image"}},{model_name: "resnet50_pytorch"model_version: -1input_map {key: "INPUT__0"value: "preprocessed_image" # 指向resnet50_dali的output}output_map {key: "OUTPUT__0"value: "OUTPUT"  # 指向ensemble的output}}]
}

現(xiàn)在整個(gè)resnet50_ensemble_dali目錄結(jié)構(gòu)為

?3.啟動(dòng)并測(cè)試

啟動(dòng)Triton加載模型

/opt/tritonserver/bin/tritonserver --model-repository=/triton/resnet50_ensemble_dali/

測(cè)試代碼為

import numpy as np
import tritonclient.http as httpclient
import torch
from PIL import Imageif __name__ == '__main__':triton_client = httpclient.InferenceServerClient(url='127.0.0.1:8000')img_path = '/test_triton/24poJOgl7m_small.jpg'image = Image.open(img_path)image = np.asarray(image)image = np.expand_dims(image, axis=0)image = image.astype(np.float32)inputs = []inputs.append(httpclient.InferInput('INPUT', image.shape, "FP32"))inputs[0].set_data_from_numpy(image, binary_data=False)outputs = []outputs.append(httpclient.InferRequestedOutput('OUTPUT', binary_data=False, class_count=1))#發(fā)送請(qǐng)求results = triton_client.infer('ensemble_python_resnet50', inputs=inputs, outputs=outputs)output_data0 = results.as_numpy('OUTPUT')print(output_data0.shape)print(output_data0)

結(jié)果為

root@aea5f00fde8d:/triton/resnet50_ensemble_dali# python3 /test_triton/dali/client.py
(1, 1)
[['10.661538:283:Persian cat']]

結(jié)束！?