Neural Light Spheres for Implicit Image Stitching and View Synthesis

This is the official code repository for the SIGGRAPH Asia 2024 work: Neural Light Spheres for Implicit Image Stitching and View Synthesis. If you use parts of this work, or otherwise take inspiration from it, please considering citing our paper:

@inproceedings{chugunov2024light,
author = {Chugunov, Ilya and Joshi, Amogh and Murthy, Kiran and Bleibel, Francois and Heide, Felix},
title = {Neural Light Spheres for {Implicit Image Stitching and View Synthesis}},
booktitle = {Proceedings of the ACM SIGGRAPH Asia 2024},
year = {2024},
publisher = {ACM},
doi = {10.1145/3680528.3687660},
url = {https://doi.org/10.1145/3680528.3687660}
}

Requirements:

• Code was written in PyTorch 2.2.1 and Pytorch Lightning 2.0.1 on an Ubuntu 22.04 machine.
• Condensed package requirements are in \requirements.txt. Note that this contains the exact package versions at the time of publishing. Code will most likely work with newer versions of the libraries, but you will need to watch out for changes in class/function calls.
• The non-standard packages you may need are pytorch_lightning, commentjson, rawpy, pygame, and tinycudann. See NVlabs/tiny-cuda-nn for installation instructions. Depending on your system you might just be able to do pip install git+https://github.com/NVlabs/tiny-cuda-nn/#subdirectory=bindings/torch, or might have to cmake and build it from source.

Project Structure:

NSF
  ├── checkpoints  
  │   └── // folder for network checkpoints
  ├── config
  │   └── // network and encoding configurations for different sizes of MLPs
  ├── data  
  │   └── // folder for image sequence data
  ├── lightning_logs  
  │   └── // folder for tensorboard logs
  ├── outputs  
  │   └── // folder for model outputs (e.g., final reconstructions) 
  ├── utils  
  │   └── utils.py  // model helper functions (e.g., RAW demosaicing, quaternion math)
  ├── LICENSE  // legal stuff
  ├── README.md  // <- recursion errors
  ├── render.py  // interactive render demo
  ├── requirements.txt  // frozen package requirements
  ├── train.py  // dataloader, network, visualization, and trainer code
  └── tutorial.ipynb // interactive tutorial for training the model

Getting Started:

We highly recommend you start by going through tutorial.ipynb, either on your own machine or with this Google Colab link.

TLDR: models can be trained with:

python3 train.py --data_path {path_to_data} --name {checkpoint_name}

For a full list of training arguments, we recommend looking through the argument parser section at the bottom of \train.py.

The final model checkpoint will be saved to checkpoints/{checkpoint_name}/last.ckpt

And you can launch an interactive demo for the scene via:

python3 render.py --checkpoint {path_to_checkpoint_folder}

Press P to turn the view-dependent color model off/on, O to turn the ray offset model off/on, + and - to raise and lower the image brightness, click and drag to rotate the camera, hold shift and click and drag to translate the camera, scroll to zoom in/out, press R to reset the view, and Escape to quit.

Data:

This models trains on data recorded by our Android RAW capture app Pani, see that repo for details on collecting/processing your own captures.

You can download the full 30gb dataset via this link, or the individual scenes from the paper via following links:

Image	Name	Num Frames	Camera Lens	ISO	Exposure Time (s)	Aperture	Focal Length	Height	Width
	360Road	77	main	21	1/1110	1.68	6.9	3072	4080
	Beppu	40	main	21	1/630	1.68	6.9	3072	4080
	BikeRacks	48	main	21	1/1960	1.68	6.9	3072	4080
	BikeShelf	37	main	8065	1/100	1.68	6.9	3072	4080
	BluePit	32	main	21	1/199	1.68	6.9	3072	4080
	BluePlane	59	main	1005	1/120	1.68	6.9	3072	4080
	Bridge	49	main	21	1/1384	1.68	6.9	3072	4080
	CityCars	46	main	21	1/1744	1.68	6.9	3072	4080
	Construction	53	main	21	1/1653	1.68	6.9	3072	4080
	DarkDistillery	57	main	10667	1/100	1.68	6.9	3072	4080
	DarkPeace	51	main	10667	1/100	1.68	6.9	3072	4080
	DarkShrine	34	main	5065	1/80	1.68	6.9	3072	4080
	DarkTruck	43	main	10667	1/60	1.68	6.9	3072	4080
	Eiffel	73	main	21	1/1183	1.68	6.9	3072	4080
	Escalatosaur	49	main	589	1/100	1.68	6.9	3072	4080
	Fireworks	78	main	5000	1/100	1.68	6.9	3072	4080
	Fukuoka	40	main	21	1/1312	1.68	6.9	3072	4080
	LanternDeer	34	main	42	1/103	1.68	6.9	3072	4080
	MountainTop	59	main	21	1/2405	1.68	6.9	3072	4080
	Ocean	44	main	110	1/127	1.68	6.9	3072	4080
	ParisCity	55	main	21	1/1265	1.68	6.9	3072	4080
	PlaneHall	77	main	1005	1/120	1.68	6.9	3072	4080
	PondHouse	51	main	21	1/684	1.68	6.9	3072	4080
	RainyPath	38	main	600	1/100	1.68	6.9	3072	4080
	RedShrine	40	main	21	1/499	1.68	6.9	3072	4080
	RockStream	31	main	21	1/1110	1.68	6.9	3072	4080
	Seafood	44	main	21	1/193	1.68	6.9	3072	4080
	ShinyPlane	50	main	21	1/210	1.68	6.9	3072	4080
	ShinySticks	37	main	21	1/1417	1.68	6.9	3072	4080
	SnowTree	42	main	21	1/320	1.68	6.9	3072	4080
	Stalls	52	main	49	1/79	1.68	6.9	3072	4080
	StatueLeft	22	main	805	1/60	1.68	6.9	3072	4080
	StatueRight	26	main	802	1/60	1.68	6.9	3072	4080
	Tenjin	36	main	602	1/100	1.68	6.9	3072	4080
	Tigers	42	main	507	1/200	1.68	6.9	3072	4080
	Toronto	31	main	21	1/1250	1.68	6.9	3072	4080
	Vending	42	main	21	1/352	1.68	6.9	3072	4080
	WoodOffice	83	main	206	1/120	1.68	6.9	3072	4080
	GlassGarden	59	telephoto	24	1/231	2.8	18.0	3024	4032
	NaraCity	54	telephoto	17	1/327	2.8	18.0	3024	4032
	360Beach	56	ultrawide	42	1/3175	1.95	2.23	3000	4000
	360Garden	77	ultrawide	41	1/1104	1.95	2.23	3000	4000
	360Siegen	67	ultrawide	41	1/1029	1.95	2.23	3000	4000
	CatBar	37	ultrawide	88	1/110	1.95	2.23	3000	4000
	Convocation	43	ultrawide	41	1/2309	1.95	2.23	3000	4000
	MellonDoor	40	ultrawide	41	1/564	1.95	2.23	3000	4000
	RedPit	53	ultrawide	41	1/418	1.95	2.23	3000	4000
	River	65	ultrawide	48	1/78	1.95	2.23	3000	4000
	UniversityCollege	55	ultrawide	42	1/2177	1.95	2.23	3000	4000
	Waterfall	74	ultrawide	41	1/1621	1.95	2.23	3000	4000

Glhf,
Ilya