Designing interiors with stable diffusion
Can stable diffusion help anyone come up with a new look for their apartment? A brief overview of what stable diffusion is and how it operates.
Can stable diffusion help any of us come up with a new layout for our apartment? This article starts with a brief overview of what stable diffusion is and how it operates. Next, I explain how the shapes produced by stable diffusion can be controlled. To wrap up, I present a toy project showing how the technology can generate ideas for room design.
Stable Diffusion
Stable diffusion is a generative text-to-image neural network that produces photo-realistic images from text prompts and noise. The original model, released in August 2022, has 859 million parameters and takes input images of 512x512 pixels. Compared to GANs, stable diffusion trains far more stably while producing images of the same or better quality. Its architecture follows the U-Net design with skip connections, and text embeddings are fed into the model through cross-attention layers. These text embeddings give the model controllability, so the user can request a particular image. The image is produced iteratively: the network predicts a small amount of noise present in the image and subtracts it from the input image. The result of that subtraction is passed back into the model as the next input. This loop continues until a final, high-quality image emerges, which typically takes around 1,000 recurrent calls.
Table 1. Stable diffusion examples. The model produced images from prompts provided by the user. The images come from the official Stable Diffusion demo.
By design, stable diffusion is steered only by the text prompt, but its architecture does not restrict it to that. After small changes, it can also be guided by user sketches, segmentation masks, or edges. Here I concentrate specifically on guiding the stable diffusion model with lines, since this makes it possible to generate many different interior design concepts from a single photo of a room. The difficulty with conditioning the model on modalities other than text lies in the training time and the resources needed. The original stable diffusion model was trained on 2 billion images using 256 Nvidia A100 GPUs, taking roughly 150,000 hours. The electricity used during training corresponds to a market price of approximately $600,000. Changing the model architecture and retraining it from scratch is costly and out of reach for most people. This is where ControlNet comes in. Its authors designed an architecture that tunes the parameters of the original stable diffusion model efficiently. Training can be done on just a single RTX 3090, and good results appear after only 16 hours of training on a dataset of 50k images.
ControlNet
ControlNet is built on top of the Stable Diffusion model. As Figure 1 shows, the U-Net model along with its text encoder and time encoder is an exact copy of the original Stable Diffusion model. It uses the same weights as the original, and they stay frozen throughout training. On top of that, ControlNet adds a trainable encoder that copies the U-Net Stable Diffusion encoder. This trainable encoder handles spatial control over the output image. It receives noise and edges as input, where the edges are the contours of the target image. Lastly, ControlNet introduces zero convolution layers that link the controlling encoder with the stable diffusion U-Net. The term zero-convolution refers to how the weights are initialized – to zeros. The authors demonstrate in their paper that initializing the weights to zero lets the influence of the control encoder on the controlled U-Net grow smoothly. In the first iteration, the control encoder does not affect the final output at all. As training progresses, it gains more and more influence, and the model’s output increasingly follows the condition shape.

Figure 1. The ControlNet architecture.
Using ControlNet for Interior Design
For this project, I combined several interior design datasets:
- IKEA Interior design dataset (298 images)
- Indoor Scene Recognition dataset (2,809)
- House Roomsset (5,250)
- GeoSynth: A Photorealistic Synthetic Indoor Dataset for Scene Understanding (170)
Altogether, this produced a dataset of 8,527 images. The ControlNet paper’s authors recommend at least 50k images, but I struggled to find more interior design images of adequate quality.
Some of the images have a resolution of 256x256 pixels, while the original stable diffusion model was trained on 512x512 pixel images. To leave the original architecture untouched, I chose to upscale every image and center crop it to 512x512.
The ControlNet training dataset is made up of the original image, a condition image (lines, edges, masks), and text prompts. I generated a condition image for each original image with the Hough transform. At first I tried a canny edge detector, but the condition images it produced contained too many contours, and the model struggled to come up with new designs that fit so many unstructured edges. The Hough transform returns only the straight lines in an image, which suits interior design mapping perfectly.
Lastly, every image needed a text prompt. One option is simply to use an empty prompt label “”. Since the model was originally trained with prompts, though, this could lead to weak results. I opted for the same default prompt, “interior design”, for every image. Ideally, an automatic image captioning network such as Bootstrapping Language-Image Pre-training (BLIP) would generate a tailored text prompt for each image. I’m leaving that approach for future work.
Figure 2. Bedroom design concepts produced by the ControlNet trained on RTX 3090, batch size = 4, for 48 hours.
Figure 3. Bedroom design concepts produced by the ControlNet trained on RTX 3090, batch size = 4, for 48 hours.
Figure 4. Livingroom design concepts produced by the ControlNet trained on RTX 3090, batch size = 4, for 48 hours.
Figure 5. Bedroom design concepts produced by the ControlNet trained on RTX 3090, batch size = 4, for 48 hours.
Summary
Figures 2-5 show that ControlNet delivers reasonable results even with a small dataset. The control encoder successfully forces the network to produce images with similar shapes. Thanks to the extensive knowledge of how the spatial world looks, already encoded in the pre-trained stable diffusion model, high-quality images appear from the very start of training. Because only Control-Net is trained and the weights of the Stable Diffusion U-Net model stay untouched, the model learns only the new image condition while keeping the knowledge of the world it already has.
Better results would require building a larger dataset of higher-quality images. Text prompts could be improved with the BLIP network, which would allow a tailored prompt for every image and further boost the final model’s results. The authors also point out that training benefits from larger batch sizes, whereas longer training does not improve the results. That approach, however, would need a GPU cluster, gradient accumulation, or a GPU with more than 24GB of VRAM.