Color/Render a 3D Point Cloud in Python 🎨

Let’s use the powerful vectorization capabilities of NumPy to switch between 2D spherical images and 3D point clouds

Photo by Robert Katzki on Unsplash

1. Point Clouds and Spherical Images

Introduction

By harnessing the duality between 3D point clouds and 2D spherical colored images, we can synergize robust 3D reconstruction algorithms with powerful 2D object detection techniques.

To simultaneously capture spherical colored images and point clouds, specialized devices such as depth sensors or LiDAR scanners are commonly equipped with spherical cameras.

Let’s establish a precise definition of the objects we’re talking about:

3D point cloud

• it provides a direct representation of the spatial layout of the scene
• each point is defined by its 3D spatial coordinates (x, y, z)
• additional attributes such as color, intensity, or other properties can be associated with each point

Colored point cloud of a room —Image by the author

2D spherical image

• It provides a panoramic view of the scene, capturing a wide field of view in all directions around a central viewpoint. For instance, it could be provided by two back-to-back fisheye cameras (See my previous Medium article Understanding 360 images 🌎)
• Each pixel is defined by its 2D coordinates (latitude φ, longitude θ) and refers to a specific viewing direction, like on the earth’s surface.

Colored 360 image of a Room — Image by the author

Hidden potential of 2D algorithms in 3D realm

Object detection algorithms are usually more robust in 2D than in 3D. Hence, the ability to transition to a 2D representation for efficient detection and segmentation and return to the 3D domain with the identified objects holds tremendous promise and excitement.