Neural Networks and Bublik

All my childhood, I dreamed of a cat, but for various reasons, my family couldn’t have a pet. However, this wish was destined to come true, and for nearly five years now, I have lived side-by-side with the best cat in the world—Bublik! He is an important part of my life and often inspires me, so I decided to make him the central figure of this generative neural network training project.

The path to creating the desired style—in my case, having the neural network memorize the appearance of Bublik—can be divided into major blocks:

1. Image set formation and preparation;
2. Creating a text dataset (captions) based on the visual database;
3. Direct training using pre-configured libraries;
4. Generating results with the neural network based on the learned instructions.

The following software solutions were used in this project:

1. Stable Diffusion (training the generative model);
2. Google Colab (execution environment);
3. Hugging Face (obtaining the token and subsequently uploading the trained model);
4. ChatGPT (creating complex code snippets, verification and consultation, assistance with the prompt for generating final images);
5. Adobe Photoshop (minimal post-processing of results, collage creation, graphic editing of code screenshots);
6. Picsart (improving image quality for the cover).

The dataset consists of 46 photographs of Bublik or fragments of his body. All images were sourced from my personal gallery and manually cropped to a perfect square aspect ratio to ensure that the most important elements were preserved (although the resolution was scaled to 512×512 using code).

At this stage, the BLIP model is used to automatically generate text descriptions for each image. A general stylistic component is then added to these descriptions. The resulting data is saved in the metadata.jsonl file, ensuring that each image has its own training prompt.

After generating the captions, the BLIP model is unloaded from GPU memory to free up VRAM for subsequent LoRA training.

The «accelerate config default» command generates a standard configuration for the Accelerate library, which manages the training process. Hugging Face authentication is required for working in an external environment: it allows for model uploads and publication to a personal repository, ensuring accessibility for other users.

Upon completing the training, I confirmed that all LoRA weights were saved correctly and created a repository on Hugging Face. I added a model card to it, detailing the training process. Then, I uploaded all the model files (excluding temporary checkpoints) and obtained a link to the finished repository.

In the next phase, I connected the VAE for correct image decoding and integrated the trained LoRA into the Stable Diffusion XL pipeline, preparing the system for test generation.

Now the neural network knows what Bublik looks like and is ready to create images featuring him as the main character!

The prompts cover several thematic areas. Initially, scenes set in famous cities around the world were created, depicting Bublik in various locations—from the bustling streets of metropolises to historic landmarks. Additionally, storylines featuring rural landscapes were developed: fields, gardens, and a village atmosphere, which allowed us to test the generation of calmer, more natural scenes.

Special attention was paid to creative and unconventional scenarios, including fantastic and surreal scenes where Bublik interacts with unusual objects and environments. Furthermore, prompts were created inspired by famous works of art, allowing for a fusion of classical painting aesthetics with the characteristics of the trained model.

Thus, the diversity of themes allowed for a comprehensive evaluation of the generative model’s capabilities—ranging from realistic scenes to artistic and experimental imagery.

Based on the experiments conducted, it can be concluded that the model successfully handles the task of generating images featuring Bublik.

Specifically, the neural network accurately understands the general structure of a cat: in most cases, the images possess correct anatomy, adhere to proper proportions, and feature minimal artifacts. This indicates that the model training was effective at the level of overall form and object structure.

However, a significant issue was identified: Bublik’s face in the generated images is rarely recognizable. Despite maintaining the general «feline» form, the individual characteristics of its appearance are lost, suggesting insufficient precision in conveying unique features. It is probable that increasing the dataset size would lower the level of inaccuracies.

During the process, it was also observed that the neural network consistently generates a cat when prompted with the phrase «cute cat.» For this reason, I used this specific phrasing in all prompts, which allowed for more predictable and stable generation results.

In summary, the model demonstrates strong performance in generating images with correct anatomy and composition; however, it requires refinement to capture the specific individual traits of the object more accurately.