Gram Matrix for Style Transferring

Previously we extracted all the relevant features which we wanted for our content and style images. The convolutional neural network does a good job for extracting the content element from any image which is fed into it.

The extracted style features require one additional pre-processing step to be more useful. The researcher used the gram matrix for more effective style feature extraction and made it an important step. Any feature extracted from the convolutional network still holds content related information such as object structure and positioning.

This content-related information is eliminated by applying the gram matrix to these extracted features, and it will not affect the style information. Style extraction from images is a broad topic on its own. Applying a gram matrix to features extracted from convolutional neural networks helps to create texture information related to the data.

The Gram Matrix is defined using the following simple equation:

Gram=V^T V

Here, V is an arbitrary vector and multiply with its transpose.

Defining gram_matrix() function

#Initializing gram_matrix function for our tensor image 
def gram_matrix(tensor):
   #Unwrapping the tensor dimensions into respective variables i.e. batch size, distance, height and width 
  _,d,h,w=tensor.size() 
  #Reshaping data into a two dimensional of array or two dimensional of tensor
  tensor=tensor.view(d,h*w)
  #Multiplying the original tensor with its own transpose using torch.mm 
  #tensor.t() will return the transpose of original tensor
  gram=torch.mm(tensor,tensor.t())
  #Returning gram matrix 
  return gram

Applying gram_matrix() function to the Style features

We have the feature extraction function and gram matrix function to make the process of style transfer. Now, we will apply the gram_matrix() function to the style feature which we extracted earlier.

Now we will create a dictionary for style grams and map each layer to the gram matrix of its corresponding feature. The key to our dictionary is going to be a specific layer. While the value which is going to contain the gram matrix of the respective style feature for that same layer. After that, we will iterate each layer inside of our style feature dictionary to get the gram matrix of all the features which we previously extracted.

style_grams={layer:gram_matrix(style_features[layer]) for layer in style_features}

Initialization of style weights dictionary

We have all our extracted features and a dictionary which contain the respective gram matrix of all the features which are extracted. We have chosen five layers to extract features from ii and provides a variety of ways for re-construction of the image style, which also leaves view for customizability. We will choose to prioritize certain layers over other layers by associating certain weight parameters with each layer.

Note: Layers close to the beginning of the model are usually effective at re-creating style features while later layers offer additional variety towards the style element.

#Initializing style_weights dictionary
style_weights={'conv1_1':1.,      #Key 1 with max value 1
               'conv2_1':0.75,  #Key 2 with max value 0.75
               'conv3_1':0.2,    #Key 3 with max value 0.2
               'conv4_1':0.2,   #Key 4 with max value 0.2
               'conv5_1':0.2}   #Key 5 with max value 0.2

Another weight parameter which we need to define is the amount of balance that we dictated between the content image and the style image. This allows us to customize our final target image by defining the ratio of style to content.

content_weight=1
style_weight=1e6 

Now, we will use the style and content image data to optimize our target image. So we will start with our original target image. Our target image will obtain from the content .clone statement as:

Now, we have all three of our images and will perform the optimization process later.

Next TopicOptimization process for Style Transferring

← prev next →