Volumetric Rendering¶
In nerfacc, the volumetric rendering pipeline is broken down into 2 steps:
Raymarching: This is the process of shooting a ray through the scene and generate samples along the way. To perform efficient volumetric rendering, here we aim at skipping as many areas as possible. The emtpy space is skipped by using the cached occupancy grid (see
nerfacc.OccupancyGrid
), and the invisible space is skipped by checking the transmittance of the ray while marching. Almost in all cases, those skipping won’t result in a noticeable loss of quality as they would contribute very little to the final rendered image. But they will bring a significant speedup.Rendering: This is the process of accumulating samples along the rays into final image. In this step we also need to query the attributes (a.k.a. color and density) of those samples generated by raymarching. Early stoping is supported in this step.
- nerfacc.ray_marching(rays_o, rays_d, t_min=None, t_max=None, scene_aabb=None, grid=None, sigma_fn=None, alpha_fn=None, early_stop_eps=0.0001, alpha_thre=0.0, near_plane=None, far_plane=None, render_step_size=0.001, stratified=False, cone_angle=0.0)¶
Ray marching with space skipping.
Note
The logic for computing t_min and t_max: 1. If t_min and t_max are given, use them with highest priority. 2. If t_min and t_max are not given, but scene_aabb is given, use
ray_aabb_intersect()
to compute t_min and t_max. 3. If t_min and t_max are not given, and scene_aabb is not given, set t_min to 0.0, and t_max to 1e10. (the case of unbounded scene) 4. Always clip t_min with near_plane and t_max with far_plane if given.Warning
This function is not differentiable to any inputs.
- Parameters:
rays_o (Tensor) – Ray origins of shape (n_rays, 3).
rays_d (Tensor) – Normalized ray directions of shape (n_rays, 3).
t_min (Optional[Tensor]) – Optional. Per-ray minimum distance. Tensor with shape (n_rays).
t_max (Optional[Tensor]) – Optional. Per-ray maximum distance. Tensor with shape (n_rays).
scene_aabb (Optional[Tensor]) – Optional. Scene bounding box for computing t_min and t_max. A tensor with shape (6,) {xmin, ymin, zmin, xmax, ymax, zmax}. scene_aabb will be ignored if both t_min and t_max are provided.
grid (Optional[Grid]) – Optional. Grid that idicates where to skip during marching. See
nerfacc.Grid
for details.sigma_fn (Optional[Callable]) – Optional. If provided, the marching will skip the invisible space by evaluating the density along the ray with sigma_fn. It should be a function that takes in samples {t_starts (N, 1), t_ends (N, 1), ray indices (N,)} and returns the post-activation density values (N, 1). You should only provide either sigma_fn or alpha_fn.
alpha_fn (Optional[Callable]) – Optional. If provided, the marching will skip the invisible space by evaluating the density along the ray with alpha_fn. It should be a function that takes in samples {t_starts (N, 1), t_ends (N, 1), ray indices (N,)} and returns the post-activation opacity values (N, 1). You should only provide either sigma_fn or alpha_fn.
early_stop_eps (float) – Early stop threshold for skipping invisible space. Default: 1e-4.
alpha_thre (float) – Alpha threshold for skipping empty space. Default: 0.0.
near_plane (Optional[float]) – Optional. Near plane distance. If provided, it will be used to clip t_min.
far_plane (Optional[float]) – Optional. Far plane distance. If provided, it will be used to clip t_max.
render_step_size (float) – Step size for marching. Default: 1e-3.
stratified (bool) – Whether to use stratified sampling. Default: False.
cone_angle (float) – Cone angle for linearly-increased step size. 0. means constant step size. Default: 0.0.
- Returns:
A tuple of tensors.
ray_indices: Ray index of each sample. IntTensor with shape (n_samples).
t_starts: Per-sample start distance. Tensor with shape (n_samples, 1).
t_ends: Per-sample end distance. Tensor with shape (n_samples, 1).
- Return type:
Tuple[Tensor, Tensor, Tensor]
Examples:
import torch from nerfacc import OccupancyGrid, ray_marching, unpack_info device = "cuda:0" batch_size = 128 rays_o = torch.rand((batch_size, 3), device=device) rays_d = torch.randn((batch_size, 3), device=device) rays_d = rays_d / rays_d.norm(dim=-1, keepdim=True) # Ray marching with near far plane. ray_indices, t_starts, t_ends = ray_marching( rays_o, rays_d, near_plane=0.1, far_plane=1.0, render_step_size=1e-3 ) # Ray marching with aabb. scene_aabb = torch.tensor([0.0, 0.0, 0.0, 1.0, 1.0, 1.0], device=device) ray_indices, t_starts, t_ends = ray_marching( rays_o, rays_d, scene_aabb=scene_aabb, render_step_size=1e-3 ) # Ray marching with per-ray t_min and t_max. t_min = torch.zeros((batch_size,), device=device) t_max = torch.ones((batch_size,), device=device) ray_indices, t_starts, t_ends = ray_marching( rays_o, rays_d, t_min=t_min, t_max=t_max, render_step_size=1e-3 ) # Ray marching with aabb and skip areas based on occupancy grid. scene_aabb = torch.tensor([0.0, 0.0, 0.0, 1.0, 1.0, 1.0], device=device) grid = OccupancyGrid(roi_aabb=[0.0, 0.0, 0.0, 0.5, 0.5, 0.5]).to(device) ray_indices, t_starts, t_ends = ray_marching( rays_o, rays_d, scene_aabb=scene_aabb, grid=grid, render_step_size=1e-3 ) # Convert t_starts and t_ends to sample locations. t_mid = (t_starts + t_ends) / 2.0 sample_locs = rays_o[ray_indices] + t_mid * rays_d[ray_indices]
- nerfacc.rendering(t_starts, t_ends, ray_indices, n_rays, rgb_sigma_fn=None, rgb_alpha_fn=None, render_bkgd=None)¶
Render the rays through the radience field defined by rgb_sigma_fn.
This function is differentiable to the outputs of rgb_sigma_fn so it can be used for gradient-based optimization.
Note
Either rgb_sigma_fn or rgb_alpha_fn should be provided.
Warning
This function is not differentiable to t_starts, t_ends and ray_indices.
- Parameters:
t_starts (Tensor) – Per-sample start distance. Tensor with shape (n_samples, 1).
t_ends (Tensor) – Per-sample end distance. Tensor with shape (n_samples, 1).
ray_indices (Tensor) – Ray index of each sample. IntTensor with shape (n_samples).
n_rays (int) – Total number of rays. This will decide the shape of the ouputs.
rgb_sigma_fn (Optional[Callable]) – A function that takes in samples {t_starts (N, 1), t_ends (N, 1), ray indices (N,)} and returns the post-activation rgb (N, 3) and density values (N, 1).
rgb_alpha_fn (Optional[Callable]) – A function that takes in samples {t_starts (N, 1), t_ends (N, 1), ray indices (N,)} and returns the post-activation rgb (N, 3) and opacity values (N, 1).
render_bkgd (Optional[Tensor]) – Optional. Background color. Tensor with shape (3,).
- Returns:
Ray colors (n_rays, 3), opacities (n_rays, 1) and depths (n_rays, 1).
- Return type:
Tuple[Tensor, Tensor, Tensor]
Examples:
>>> rays_o = torch.rand((128, 3), device="cuda:0") >>> rays_d = torch.randn((128, 3), device="cuda:0") >>> rays_d = rays_d / rays_d.norm(dim=-1, keepdim=True) >>> ray_indices, t_starts, t_ends = ray_marching( >>> rays_o, rays_d, near_plane=0.1, far_plane=1.0, render_step_size=1e-3) >>> def rgb_sigma_fn(t_starts, t_ends, ray_indices): >>> # This is a dummy function that returns random values. >>> rgbs = torch.rand((t_starts.shape[0], 3), device="cuda:0") >>> sigmas = torch.rand((t_starts.shape[0], 1), device="cuda:0") >>> return rgbs, sigmas >>> colors, opacities, depths = rendering( >>> t_starts, t_ends, ray_indices, n_rays=128, rgb_sigma_fn=rgb_sigma_fn) >>> print(colors.shape, opacities.shape, depths.shape) torch.Size([128, 3]) torch.Size([128, 1]) torch.Size([128, 1])