Remote sensing imagery meets generative AI
Is it possible to rely on artificial yet realistic samples when studying natural geographic phenomena?
Remote sensing and diffusion models
Is it possible to rely on artificial yet realistic samples when studying natural geographic phenomena?
For remote sensing practitioners, Generative Artificial Intelligence is nothing new. For several years now, we have been using it successfully in our research to improve our models: filling in areas hidden under clouds, removing noise, upscaling and downscaling images, and even producing artificial samples. All of these tasks were handled by carefully tuned algorithms, frequently backed by AutoEncoders (AE) or Generative Adversarial Networks (GAN).
I was involved in introducing new aerial imagery data generation techniques. At first, I was skeptical about whether generative methods could help tackle problems connected with natural phenomena and the environment. In the end, after weighing every pro and con, I came to the conclusion that generative models – provided their output is realistic – can support us in remote sensing work. Numerous tests and model benchmarks back this up as well.
Sadly, the realism of the artificial samples produced by those models had its limits. Most spatial features were reproduced correctly and could not even be told apart by a human expert. Yet a handful of small defects marred the overall outcome: shadow aberrations, overly smooth edges in the geometry of man-made objects, invisible water flow, and color problems in depicting different plants. They did not drastically reduce the model’s usefulness, but they looked far too synthetic for a geographer to accept in every scenario. That was about to change.
Not long ago, with open fascination, I caught up on the scientific literature and code repositories dealing with diffusion. You have surely heard of DALL-E or Midjourney, both practical applications of the models mentioned above. I was amazed by how much progress had been achieved in so little time. Had someone asked me a year ago about the role of generative AI in remote sensing, I would have called it an interesting complement to the classic methods. Now I am convinced that our GIS toolboxes are in for a major upgrade.
Without a doubt, the Generative AI revolution has arrived, and it will have a major impact on remote sensing. Our field is about to undergo huge changes, so it pays to be ready!
Generative AI techniques are powerful, yet complex at the same time. To make the most of them and understand where they fall short, it is best to study the scientific literature while working through hands-on exercises in parallel. If Generative AI is new to you, the following papers and their accompanying code make a good starting point:
- Pix2Pix - a GAN-based architecture for conditional image-to-image translation.
- BigGAN - a generative adversarial network built to scale generation up to high-resolution, high-fidelity images.
- Diffusion - models that produce samples by progressively stripping noise from a signal.
Let’s move on to a simple example. Suppose we are dealing with a classification problem. Our job is to process patches of satellite imagery and assign a label to each one. The label gives a rough description of what the patch shows: a bridge, a forest, a stadium, an industrial complex, and so on.
Image samples from the AID dataset: the river, mountain, and forest classes.
Even though it is a classic machine learning problem, in remote sensing the task is far from trivial. There are many distinct classes, and the samples are highly imbalanced. It is easy to picture that forests cover far more ground than power plants do (fortunately). Gathering samples of common classes is no problem. Trouble begins when our hungry neural network classifier needs patches showing rare objects. Searching for particular samples can be tiresome, and it gets even harder when only a handful of such objects exist at all.
We can always attempt to produce additional samples with Generative AI and use them to augment our imbalanced training dataset. To demonstrate a solution, we will use a diffusion model together with the AID dataset.
Before you run the code, make sure you have a reasonable GPU (8GB+ VRAM) and that CUDA is properly installed in your environment. If the setup gives you trouble, Colab will make things easier – just remember to choose the right runtime type. Once you are ready, install PyTorch and the open-source Denoising Diffusion library for PyTorch.
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cu118
pip install denoising_diffusion_pytorch
Next, we need to prepare the training samples. The implementation supports RGB imagery in a variety of formats (TIF included). While it is not tailored to multi-spectral imagery, diffusion models are not restricted to just three channels – Generative AI has already been used successfully to synthesize near-infrared data. Open-source software supporting multiple spectral channels is only a matter of time.
Returning to the dataset: you may feel like feeding the entire AID dataset into a neural network. In ML, more is better, right? Under some conditions that might work, but here I will spare you the time. The reality is that the more complex the phenomena and the more varied the data, the more time and examples the generator needs to produce sensible samples. Take a look at the example: the result is easy on the eye, but it does not resemble realistic satellite imagery patches.
Training a diffusion model on every available AID class.
Let’s concentrate on single classes and begin by generating artificial mountains. Extract the AID mountains catalog and put it in a folder of your choice. In the code, we set up a backbone model (Unet) and configure the parameters of the diffusion process. Note that we operate on 256 px rectangular patches; if your resources are limited, you can lower the image_size value to 128 px or even 64 px. The final part concerns training. Most parameters keep the default values defined in the denoising_diffusion_pytorch repository. Pay particular attention to batch_size, num_samples, and amp. The first determines how many samples are processed at once and depends directly on your computational resources – in my case, 10 patches fit on my Titan RTX card. The second specifies how many samples are generated every 1000 iterations; lowering it speeds the process up. The third sets the acceleration type and is rather troublesome: the project’s issue tracker contains several reports concerning it. Training takes a long time, so turning acceleration off is the safer option to avoid nan loss.
from denoising_diffusion_pytorch import Unet, GaussianDiffusion, Trainer
if __name__ == '__main__':
model = Unet(
dim=64,
dim_mults=(1, 2, 4, 8)
)
diffusion = GaussianDiffusion(
model,
image_size=256,
timesteps=1000,
loss_type='l1'
)
trainer = Trainer(
diffusion,
'data/AID_mountains',
train_batch_size=10,
train_lr=8e-5,
train_num_steps=10000,
gradient_accumulate_every=2,
ema_decay=0.995,
amp=False,
calculate_fid=False,
convert_image_to='RGB',
save_and_sample_every=1000,
num_samples=36
)
trainer.train()
Roughly 8h later, the first results are ready. Impressive, isn’t it? An experienced eye will still recognize these samples as synthetic, but we are not far from a high-quality outcome! Just imagine the results possible with a bigger remote sensing imagery dataset and more resources.
Mountain samples created artificially with a diffusion model.
Now is the ideal moment to explain how diffusion-based models operate. Their primary goal is to predict (identify) noise in images that have been progressively obscured by systematically adding jitter. The noise is introduced over many time steps, and the trained model learns to remove the image distortions. How does that help generate new samples? The model is able to denoise an image made up entirely of noise. At every time step, it subtracts a little distortion until it reaches the end, i.e., a clean image. The clever part is that the starting noise does not have to come from obscuring a real image – it can be entirely random. Denoising random noise yields a random, artificial image. Brilliant idea!
How a diffusion model is trained
I think this example illustrates generative AI techniques in remote sensing well. Naturally, it does not have to stop at producing new samples and augmenting data. I see a lot of potential in correcting real samples, and not only where clouds are concerned. These techniques can prove very valuable when merging different materials. We all know how difficult it is to process and assemble long time series of remote sensing images, particularly when a time sample is missing, or how tricky it is to merge point clouds captured by LIDAR. Watch the world of science and engineering closely – in the near future, you will come across plenty of exciting Generative AI projects. I am confident you will find something that can give your research a boost. And if you have any interesting ideas, don’t hesitate to get in touch :)