雙目深度估計開源數(shù)據(jù)集很多都是用UE制作的，那么我們自己能否通過UE制作自己想要的場景的數(shù)據(jù)集呢。最近花了點時間研究了一下，分享給需要的小伙伴。

主要使用的是UnrealCV插件，UnrealCV是一個開源項目，旨在幫助計算機視覺研究人員使用虛幻引擎（UE）構(gòu)建虛擬世界。

下載UnrealCV

GitHub - unrealcv/unrealcv: UnrealCV: Connecting Computer Vision to Unreal Engine

下載并安裝對應(yīng)版本的UE5，參考這個鏈接：

https://blog.csdn.net/ButDanJi/article/details/133919089

注意UnrealCV的版本和UE5的版本必須一致，例如UnrealCV5.2 必須對應(yīng)UE5.2，否則可能會報錯

進入UE，新建項目，例如這里可以創(chuàng)建一個第一人稱游戲的項目：

項目創(chuàng)建完成后，關(guān)閉UE。在對應(yīng)項目下新建Plugins文件夾，并把unrealcv放在項目的Plugins下，例如：E:\UE_Project\testproject5\Plugins\unrealcv-5.2

打開UE下的unrealcv.ini文件，E:\UnrealEngine-5.2.0-release\Engine\Binaries\Win64\unrealcv.ini

將EnableRightEye設(shè)置為True

再次打開UE，打開這個項目，此時會提示安裝UnrealCV

點擊yes安裝UnrealCV，等待一段時間后會進入項目，點擊編輯-插件，搜索UnrealCV，如果安裝成功能搜到UnrealCV且處于啟動狀態(tài)

點擊窗口-加載布局-UE4經(jīng)典布局

在放置Actor下搜索fusion camera actor，放置2個相機到場景中

點擊play 運行關(guān)卡

按下`輸入vget /unrealcv/status

會得到以下日志：

LogUnrealCV: Warning: vget helper function, the real command is vget /unrealcv/status
LogUnrealCV: Warning: Is Listening
No Client Connected
9001
Configuration
Config file: E:/UnrealEngine-5.2.0-release/Engine/Binaries/Win64/unrealcv.ini
Port: 9001
Width: 640
Height: 480
FOV: 90.000000
EnableInput: true
EnableRightEye: true

此時UnrealCV已準備完畢，UnrealCV服務(wù)器正處于監(jiān)聽狀態(tài)，接下來我們通過python構(gòu)建客戶端連接到UnrealCV進行采圖

下載

https://github.com/ibaiGorordo/UnrealCV-stereo-depth-generation

注意直接運行會報錯，UnrealCV的用法有改變，不能直接使用client.connect()?

需要在代碼開頭加上

ip = '127.0.0.1'
port = 9001?
client = Client((ip, port))

至于原因可以參考我在UnrealCV下問的帖子：

Can not connect to localhost · Issue #258 · unrealcv/unrealcv

這個項目可以獲得平面深度，但不是視差圖，我用以下代碼獲得視差圖：

def convert_plane_depth_to_disp(plane_depth, f=320.0, baseline_meters=1.0):disp = f * baseline_meters * (1.0 / plane_depth)return disp

這個代碼是參考自以下鏈接：https://github.com/wuwushrek/AirSim/blob/56e2c5c3ec461f2d95c6a9e80c98767078e718ac/PythonClient/generate_stereo_data.py#L67

于是最后的代碼為(這里是示例，相機的姿態(tài)等參數(shù)需要自己修改)：

from unrealcv import Client
import sys
import numpy as np
import cv2
import io
ip = '127.0.0.1'
port = 9001 
client = Client((ip, port))camera_poses=np.array([[-106.933, 459.372, 167.895, 0.213, -80.610, 0.000],
[-97.576, 413.807, 168.308, 2.901, -79.483, 0.000],
[-88.197, 346.847, 166.356, 3.644, -89.711, 0.000],
[-82.595, 278.711, 172.572, 5.711, -85.554, 0.000],
[-73.239, 149.936, 176.386, 0.058, -89.777, 0.000],
[-71.879, 58.805, 175.112, 1.199, -89.030, 0.000],
[-69.923, 10.021, 161.958, 4.062, -59.268, 0.000],
[-28.289, -68.530, 159.251, 2.186, -61.090, 0.000],
[-28.289, -68.530, 159.251, 2.831, -43.937, 0.000],
[-28.289, -68.530, 159.251, 1.782, 0.917, 0.000],
[-28.289, -68.530, 159.251, 3.708, 33.667, 0.000],
[-28.289, -68.530, 159.251, 0.167, 92.277, 0.000],
[-32.458, 5.207, 157.922, 2.922, 93.428, 0.000],
[-35.463, 90.040, 156.689, 1.045, 97.168, 0.000],
[-46.087, 180.173, 155.370, 1.167, 96.643, 0.000],
[-52.370, 234.121, 154.580, 1.167, 96.315, 0.000],
[-52.370, 234.121, 154.580, 3.425, 54.474, 0.000],
[-52.370, 234.121, 154.580, 5.985, 18.172, 0.000],
[-52.370, 234.121, 154.580, 5.675, -10.430, 0.000],
[-52.370, 234.121, 154.580, 11.879, -34.452, 0.000],
[-52.370, 234.121, 154.580, 13.122, -66.362, 0.000],
[-52.370, 234.121, 154.580, 14.454, -81.988, 0.000]])fps = 45
times = np.arange(0,camera_poses.shape[0]*fps,fps)
filled_times = np.arange(0,camera_poses.shape[0]*fps)filtered_poses = np.array([np.interp(filled_times, times, axis) for axis in camera_poses.T]).Tclass UnrealcvStereo():def __init__(self):client.connect() if not client.isconnected():print('UnrealCV server is not running. Run the game downloaded from http://unrealcv.github.io first.')sys.exit(-1)def __str__(self):return client.request('vget /unrealcv/status')@staticmethoddef set_position(pose):# Set position of the first cameraclient.request(f'vset /camera/1/location {pose[0]} {pose[1]} {pose[2]}')client.request(f'vset /camera/1/rotation {pose[3]} {pose[4]} {pose[5]}')client.request(f'vset /camera/2/location {pose[0]} {pose[1]} {pose[2]}')client.request(f'vset /camera/2/rotation {pose[3]} {pose[4]} {pose[5]}')@staticmethoddef get_stereo_pair(eye_distance):res = client.request('vset /action/eyes_distance %d' % eye_distance)res = client.request('vget /camera/1/lit png')left = cv2.imdecode(np.frombuffer(res, dtype='uint8'), cv2.IMREAD_UNCHANGED)res = client.request('vget /camera/2/lit png')right = cv2.imdecode(np.frombuffer(res, dtype='uint8'), cv2.IMREAD_UNCHANGED)return left, right@staticmethoddef convert_depth(PointDepth, f=320):H = PointDepth.shape[0]W = PointDepth.shape[1]i_c = float(H) / 2 - 1j_c = float(W) / 2 - 1columns, rows = np.meshgrid(np.linspace(0, W-1, num=W), np.linspace(0, H-1, num=H))DistanceFromCenter = ((rows - i_c)**2 + (columns - j_c)**2)**(0.5)PlaneDepth = PointDepth / (1 + (DistanceFromCenter / f)**2)**(0.5)return PlaneDepth@staticmethoddef get_depth():res = client.request('vget /camera/1/depth npy')point_depth = np.load(io.BytesIO(res))return UnrealcvStereo.convert_depth(point_depth)@staticmethoddef color_depth(depth_map, max_dist):norm_depth_map = 255*(1-depth_map/max_dist)norm_depth_map[norm_depth_map < 0] =0norm_depth_map[depth_map == 0] =0return cv2.applyColorMap(cv2.convertScaleAbs(norm_depth_map,1), cv2.COLORMAP_MAGMA)def convert_plane_depth_to_disp(plane_depth, f=320.0, baseline_meters=1.0):disp = f * baseline_meters * (1.0 / plane_depth)return disp
if __name__ == '__main__':eye_distance = 10max_depth = 5stereo_generator = UnrealcvStereo()for pose in filtered_poses:stereo_generator.set_position(pose)# Set the eye distanceleft, right = stereo_generator.get_stereo_pair(eye_distance)depth_map = stereo_generator.get_depth()baseline_cm =25# Parameters for cameracx = float(depth_map.shape[1]) / 2.0 - 1.0cy = float(depth_map.shape[0]) / 2.0 - 1.0f = cxdisparity = convert_plane_depth_to_disp(plane_depth=depth_map, f=f, baseline_meters=baseline_cm/100.0)color_depth_map = stereo_generator.color_depth(disparity, max_depth)left = cv2.cvtColor(left, cv2.COLOR_BGRA2BGR)right = cv2.cvtColor(right, cv2.COLOR_BGRA2BGR)output_path = "C:/Users/chen/Desktop/output_image.jpg"output_path1 = "C:/Users/chen/Desktop/output_image1.jpg"output_path2 = "C:/Users/chen/Desktop/output_image2.jpg"cv2.imwrite(output_path, color_depth_map)        cv2.imwrite(output_path1, left)cv2.imwrite(output_path2, right)combined_image = np.hstack((left, right, color_depth_map))cv2.imshow("stereo", combined_image)# Press key q to stopif cv2.waitKey(1) == ord('q'):breakcv2.destroyAllWindows()

運行python文件（運行時，UE的項目必須處于運行狀態(tài)，即play狀態(tài)）

這時就能獲得雙目圖像和視差圖了。

再往后就是換成自己想要的場景并修改兩個相機的姿態(tài)以及baseline_meters等參數(shù)，修改完就可以得到想要的圖像了