Introduction to the Causal Image Generation Model

How to create a causal image generation model?

Problem Statement

To first build a causal image generative model, we need to have a valid problem statement, In our case, we are creating an image where two game characters with certain attributes interact with each other. Their actions and reactions are affected by each other's attributes. By using a causal model, it gives us the ability to condition and intervene in the image generation process and get a corresponding image. Sample condition queries can be like, "How would the image look had the character's action was Attack"? Conditioning on the observable classes is possible via conditioned GAN's too. So, how is it different from a causal model?

Causal Model Vs Conditioned GAN

In a conditioned GAN, we can condition on the values of the observed classes but this doesn't work in a case where the other observed classes and unobserved variables are affected by the class we conditioned on. We need a way to represent the causality in the system and how different entities in the image are getting affected by each other. The relationship between the different entities in the image is described by the causal DAG.

DAG

To build a causal model, we must construct a DAG that states the causal assumptions in the image generation process. In the below image, we see that there are 2 variables from which an image is generated. The DAG is an assumption of how the different latent and/or observed variables might interact together to produce the image.

A DAG describing image generation process

Data

To start developing a causal image generative model, we need images. We use procedural generation to create a dataset of game characters. Procedural generation is an automated way of generating a dataset. We used a small set of game characters and merged them to create multiple images of two-game characters interacting. A Sample image can be found below.

A Game Character attacking another character which resulted in its death.

The above image is merged by the following individual sprites.

Actor Attacking

Reactor Dying

The reactor image is just flipped to make it seem obvious that the reaction of the reactor game character is caused due to the action of the actor game character.

Pyro

Pyro is a universal probabilistic programming language (PPL) written in Python and supported by PyTorch on the backend. Pyro enables flexible and expressive deep probabilistic modeling, unifying the best of modern deep learning and Bayesian modeling. To get started,

pip install pyro-ppl

More info about pyro can be found at

Getting Started With Pyro: Tutorials, How-to Guides and Examples — Pyro Tutorials 1.8.0 documentation

Introduction to Pyro

Pytorch

Pytorch is needed for building a deep neural network to learn the distribution of images. In this tutorial, we will use a Variational Auto Encoder architecture to build a generative model. A general understanding of how to build a neural network using PyTorch is needed to fully understand this tutorial. To get started,

pip install torch
pip install torchvision

PyTorch for Deep Learning: A Quick Guide for StartersMedium