Umut Kocasarı

Publications

1. GCPR

   G3DST: Generalizing 3D Style Transfer with Neural Radiance Fields across Scenes and Styles

   Adil Meric, Umut Kocasari, Matthias Nießner, and Barbara Roessle

   DAGM German Conference on Pattern Recognition (GCPR) , 2024

   Abstract PDF Project Page

   Neural Radiance Fields (NeRF) have emerged as a powerful tool for creating highly detailed and photorealistic scenes. Existing methods for NeRF-based 3D style transfer need extensive per-scene optimization for single or multiple styles, limiting the applicability and efficiency of 3D style transfer. In this work, we overcome the limitations of existing methods by rendering stylized novel views from a NeRF without the need for per-scene or per-style optimization. To this end, we take advantage of a generalizable NeRF model to facilitate style transfer in 3D, thereby enabling the use of a single learned model across various scenes. By incorporating a hypernetwork into a generalizable NeRF, our approach enables on-the-fly generation of stylized novel views. Moreover, we introduce a novel flow-based multi-view consistency loss to preserve consistency across multiple views. We evaluate our method across various scenes and artistic styles and show its performance in generating high-quality and multi-view consistent stylized images without the need for a scene-specific implicit model. Our findings demonstrate that this approach not only achieves a good visual quality comparable to that of per-scene methods but also significantly enhances efficiency and applicability, marking a notable advancement in the field of 3D style transfer.

2. WACV

   Fantastic Style Channels and Where to Find Them: A Submodular Framework for Discovering Diverse Directions in GANs

   Enis Simsar, Umut Kocasari, Ezgi Gulperi Er, and Pinar Yanardag

   Winter Conference on Applications of Computer Vision (WACV) , 2023

   Abstract PDF Project Page

   The discovery of interpretable directions in the latent spaces of pre-trained GAN models has recently become a popular topic. In particular, StyleGAN2 has enabled various image generation and manipulation tasks due to its rich and disentangled latent spaces. The discovery of such directions is typically done either in a supervised manner, which requires annotated data for each desired manipulation, or in an unsupervised manner, which requires a manual effort to identify the directions. As a result, existing work typically finds only a handful of directions in which controllable edits can be made. In this paper, we attempt to find the most representative and diverse subset of directions in stylespace of StyleGAN2. We formulate the problem as a coverage of stylespace and propose a novel submodular optimization framework that can be solved efficiently with a greedy optimization scheme. We evaluate our framework with qualitative and quantitative experiments and show that our method finds more diverse and relevant channels.

3. CVPR Workshop

   PaintInStyle: One-Shot Discovery of Interpretable Directions by Painting

   Berkay Doner*, Elif Sema Balcioglu*, Merve Rabia Barin*, Umut Kocasari, Mert Tiftikci, and Pinar Yanardag

   Computer Vision for Fashion, Art, and Design (CVPR Workshop) , 2022

   Abstract Project Page

   The search for interpretable directions in latent spaces of pre-trained Generative Adversarial Networks (GANs) has become a topic of interest. These directions can be utilized to perform semantic manipulations on the GAN generated images. The discovery of such directions is performed either in a supervised way, which requires manual annotation or pre-trained classifiers, or in an unsupervised way, which requires the user to interpret what these directions represent. Our goal in this work is to find meaningful latent space directions that can be used to manipulate images in a one-shot manner where the user provides a simple drawing (such as drawing a beard or painting a red lipstick) using basic image editing tools. Our method then finds a direction that can be applied to any latent vector to perform the desired edit. We demonstrate that our method is able to find several distinct and fine-grained directions in a variety of datasets.

4. CVPR Workshop

   Rank in Style: A Ranking-based Approach to Find Interpretable Directions

   Umut Kocasari, Kerem Zaman, Mert Tiftikci, Enis Simsar, and Pinar Yanardag

   Computer Vision for Fashion, Art, and Design (CVPR Workshop) , 2022

   Abstract Project Page

   Recent work such as StyleCLIP aims to harness the power of CLIP embeddings for controlled manipulations. Although these models are capable of manipulating images based on a text prompt, the success of the manipulation often depends on careful selection of the appropriate text for the desired manipulation. This limitation makes it particularly difficult to perform text-based manipulations in domains where the user lacks expertise, such as fashion. To address this problem, we propose a method for automatically determining the most successful and relevant text-based edits using a pre-trained StyleGAN model. Our approach consists of a novel mechanism that uses CLIP to guide beam-search decoding, and a ranking method that identifies the most relevant and successful edits based on a list of keywords. We also demonstrate the capabilities of our framework in several domains, including fashion.

5. WACV

   StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulation

   Umut Kocasari, Alara Dirik, Mert Tiftikci, and Pinar Yanardag

   Winter Conference on Applications of Computer Vision (WACV) , 2022

   Abstract PDF Code Project Page

   Pre-trained GANs have shown great potential for interpretable directions in the latent space. The discovery of such directions is often done in a supervised or self-supervised manner and requires manual annotations which limits their application in practice. On the other hand, unsupervised approaches provide a way to discover interpretable directions without any supervision, but no fine-grained attribute can be discovered. Recent work such as StyleCLIP aims to overcome this limitation by leveraging the power of CLIP, a joint representational model for text and images, for text-driven image manipulation. While promising, these methods take several hours of pre-processing or training time, and require multiple text prompts. In this work, we propose a fast and efficient method for text-guided image generation and manipulation by leveraging the power of StyleGAN2 and CLIP. Our method uses a CLIP-based loss and an identity loss to manipulate images via user-supplied text prompts without changing any of the irrelevant attributes. Unlike previous work, our method requires only 12 seconds of optimization per text prompt and can be used with any pre-trained StyleGAN2 model. We demonstrate the effectiveness of our method with extensive results and comparisons to state-of-the-art approaches.

6. NeurIPS Workshop

   Exploring Latent Dimensions of Crowd-sourced Creativity

   Umut Kocasari*, Alperen Bag*, Efehan Atici, and Pinar Yanardag

   Machine Learning for Creativity and Design (NeurIPS Workshop) , 2021

   Abstract PDF Code Project Page

   Recent research showed that it is possible to find directions in the latent spaces of pre-trained GANs. These directions provide controllable generation and support a wide range of semantic editing operations such as zoom-in or rotation. While existing works focus on discovering directions for semantic image editing, we focus on an abstract property: Creativity. Can we manipulate an image to make it more or less creative? We build our work on the largest AI-based creativity platform Artbreeder where users are able to generate unique images using pre-trained GAN models. We explore the latent dimensions of the images generated on this platform and present a novel framework for manipulating images to make them more creative.