Convolutional NNs for CIFAR 10

In this project, a data set of (small) natural images known as CIFAR10 will be used. The goal is to present:

i) how CNN layers are used in Pytorch, and

ii) evaluate the performance of a simple CNN over this dataset.

from IPython.display import Image
from IPython.core.display import HTML 

## just showing the image from the website
Image(url= "https://pytorch.org/tutorials/_images/cifar10.png", width=400, height=300)

%matplotlib inline
%config InlineBackend.figure_format = 'retina'  #To get figures with high quality!

import numpy as np
import torch
from torch import nn
from torch import optim
import matplotlib.pyplot as plt
import time

Downlading the CIFAR10 with torchvision

The code below will download the MNIST dataset, then create training and test datasets.

import torch
from torchvision import datasets, transforms, utils

transform = transforms.Compose(
    [transforms.ToTensor(),
     ## we’re passing in three values for the mean and three values for the standard deviation, one for each channel.
     transforms.Normalize(
         mean= [0.5, 0.5, 0.5], 
         std= [0.5, 0.5, 0.5])])

## First we uploaded the train sample (train=True) by downloading it. But when we download, immediatly we do transformation.
trainset = datasets.CIFAR10(root='./data', train=True,
                                        download=True, transform=transform)

## Here, after uploading the trainset, we are creating batches with size 64 by shuffling. Because we will work on batches.
trainloader = torch.utils.data.DataLoader(trainset, batch_size=64,
                                          shuffle=True, num_workers=2)
## First we uploaded the test sample (train=False) by downloading it. But when we download, immediatly we do transformation.
testset = datasets.CIFAR10(root='./data', train=False,
                                       download=True, transform=transform)

## why we are not shuffling? bcz there is only one test batch?
testloader = torch.utils.data.DataLoader(testset, batch_size=64,
                                         shuffle=False, num_workers=2)

classes = ('plane', 'car', 'bird', 'cat',
           'deer', 'dog', 'frog', 'horse', 'ship', 'truck')
n_classes = len(classes)

Downloading https://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz to ./data/cifar-10-python.tar.gz

Extracting ./data/cifar-10-python.tar.gz to ./data
Files already downloaded and verified

torchvision splits the work up between ToTensor() and Normalize():

ToTensor() takes a PIL image (or np.int8 NumPy array) with shape (n_rows, n_cols, n_channels) as input and returns a PyTorch tensor with floats between 0 and 1 and shape (n_channels, n_rows, n_cols). Normalize() subtracts the mean and divides by the standard deviation of the floating point values in the range [0, 1].

traindata = iter(trainloader)

images, labels = next(traindata)

print(images[1].shape) ## burada 1 rakamı farketmiyor, genel olarak image ler piksel halinde.

torch.Size([3, 32, 32])

len(traindata)

782

def imshow(img):
    img = img / 2 + 0.5     # unnormalize to pot
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.show()

imshow(utils.make_grid(images))

A validation set using the 20% of train images

len(trainset) # 50000
seed_number=3467
torch.manual_seed(seed_number)
val_size = int(len(trainset)*0.20)
train_size = len(trainset) - val_size


train_ds, val_ds = torch.utils.data.random_split(trainset, [train_size, val_size])
len(train_ds), len(val_ds)

(40000, 10000)

trainloader = torch.utils.data.DataLoader(train_ds, batch_size=64,
                                          shuffle=True, num_workers=2)

valloader = torch.utils.data.DataLoader(val_ds, batch_size=64,
                                          shuffle=True, num_workers=2)

## defining the torches in 64 batches for train and validation sets
traindata = iter(trainloader)
valdata = iter(valloader)

Implementing the LeNet 5 CNN Network

In the CNN network above there are 2 convolutional layers with the following properties:

• ReLU activation functions are used as non-linear functions
• Maxpooling with $2\times 2$ kernels is used to reduce the spatial dimension in both layers.
• $5\times 5$ convolutional filters are used. Stride is 1.
• After the second convolutional layer, three dense layers are stacked.

Each of dimension in CIFAR-10 images are composed by 3 input color maps, therefore, they are $32\times32\times3$.

class Lenet5(nn.Module):
    def __init__(self,dimx,nlabels): #Nlabels will be 10 in our case
        super().__init__()

        # convolutional layer (sees 28x28x1 image tensor)
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, 
                               kernel_size=5, stride=1, padding=0)
        
        # convolutional layer (sees 12x12x16 tensor)
        self.conv2 = nn.Conv2d(6, 16, 5, padding=0)

        
        # Max pool layer
        self.pool = nn.MaxPool2d(2, 2)

        # Linear layers

        self.linear1 = nn.Linear(16*5*5,120)
        
        self.linear2 = nn.Linear(120,84)
        
        self.linear3 = nn.Linear(84, nlabels)
    
        self.relu = nn.ReLU()
        
        self.logsoftmax = nn.LogSoftmax(dim=1) 
        
        # Spatial dimension of the Tensor at the output of the 2nd CNN
        self.final_dim =int(((dimx-4)/2-4)/2) #int((dimx-5)/5+1) #
        
    def forward(self, x):
        # Pass the input tensor through the CNN operations
        x = self.conv1(x)
        x = self.relu(x) 
        x = self.pool(x)
        x = self.conv2(x)
        x = self.relu(x)
        x = self.pool(x)
        # Flatten the tensor into a vector of appropiate dimension using self.final_dim
        x = x.view(-1, 16*self.final_dim*self.final_dim)
        # Pass the tensor through the Dense Layers
        x = self.linear1(x)
        x = self.relu(x)
        x = self.linear2(x)
        x = self.relu(x)
        x = self.linear3(x)
        x = self.logsoftmax(x) 
        return x

The spatial size of the output volume as a function of the input volume size (W) can be computed, the kernel/filter size (F), the stride with which they are applied (S), and the amount of zero padding used (P) on the border. The correct formula for calculating how many neurons define the output_W is given by (W−F+2P)/S+1.

For example for a 7x7 input and a 3x3 filter with stride 1 and pad 0, the output would be 5x5. With stride 2 we would get a 3x3 output.

Extending the class to incorporate a training method, to evaluate the both the validation and train losses and to evaluate the classification performance in a set.

class Lenet5_extended(Lenet5):
    ## extending the class

    def __init__(self,dimx,nlabels,epochs=5,lr=0.001):
        
        super().__init__(dimx,nlabels)  
        
        self.lr = lr #Learning Rate
        
        self.optim = optim.Adam(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.criterion = nn.CrossEntropyLoss() #nn.NLLLoss()             
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.valid_loss_during_training = []
        

    def trainloop(self,trainloader,validloader):
        
        torch.manual_seed(seed_number)
        for e in range(int(self.epochs)):
            
            start_time = time.time()
            
            # Random data permutation at each epoch
            
            running_loss = 0.
            
            for images, labels in trainloader:
                 
                #Reset Gradients!
                self.optim.zero_grad() # zero the parameter gradients - # clear the gradients of all optimized variables
            
                out = self.forward(images)
                loss = self.criterion(out,labels) # calculate the loss 
                
                #Compute gradients
                loss.backward()  # backward pass: compute gradient of the loss with respect to model parameters
                
                #SGD stem
                self.optim.step() # perform a single optimization step (parameter update)
                
                running_loss += loss.item()

            self.loss_during_training.append(running_loss/len(trainloader))
            
            # Validation Loss
            
            # Turn off gradients for validation, saves memory and computations
            with torch.no_grad():            
                
                ## make the running loss zero again.
                running_loss = 0.
                
                for images,labels in validloader:                    
                    
                    # Compute output for input minibatch
                    out = self.forward(images)
                
                    loss = self.criterion(out,labels) # calculate the loss

                    running_loss += loss.item()   
                    
                self.valid_loss_during_training.append(running_loss/len(validloader))    
                    

            print("Epoch %d. Training loss: %f, Validation loss: %f, Time per epoch: %f seconds" 
                      %(e,self.loss_during_training[-1],self.valid_loss_during_training[-1],
                       (time.time() - start_time)))
                
    def eval_performance(self,dataloader):
        # since we're not training, we don't need to calculate the gradients for our outputs
        loss = 0
        accuracy = 0

        # Turn off gradients for validation, saves memory and computations
        with torch.no_grad():

            for images,labels in dataloader:

                probs = self.forward(images)

                top_p, top_class = probs.topk(1, dim=1)
                equals = (top_class == labels.view(images.shape[0], 1))
                accuracy += torch.mean(equals.type(torch.FloatTensor))
    
            return accuracy/len(dataloader)

Training the model for 5 epochs

my_CNN = Lenet5_extended(dimx=32,nlabels=10,epochs=5,lr=1e-3)

my_CNN.trainloop(trainloader,valloader)

Epoch 0. Training loss: 1.684646, Validation loss: 1.473145, Time per epoch: 21.936860 seconds
Epoch 1. Training loss: 1.388226, Validation loss: 1.352951, Time per epoch: 21.199301 seconds
Epoch 2. Training loss: 1.267175, Validation loss: 1.294347, Time per epoch: 21.421658 seconds
Epoch 3. Training loss: 1.185563, Validation loss: 1.198459, Time per epoch: 21.298973 seconds
Epoch 4. Training loss: 1.121921, Validation loss: 1.179710, Time per epoch: 21.153286 seconds

Train/validation loss plots during training & Train/validation performance

plt.plot(my_CNN.loss_during_training,label='Training Loss')
plt.plot(my_CNN.valid_loss_during_training,label='Validation Loss')
plt.legend()

print(my_CNN.eval_performance(trainloader))
print(my_CNN.eval_performance(valloader))

tensor(0.6161)
tensor(0.5802)

After running the model with 5 epoch, it is clear that both training and validation losses are decreasing with the number of epochs. The minimum loss is obtained in the last epoch. Therefore, in order to get further insights it would necassary to retrain the model with more epochs to evaluate the model in detail.

GPU-based training

The newtork is already quite deep and gradient evaluation becomes a heavy operation.

PyTorch, along with pretty much every other deep learning framework, uses CUDA to efficiently compute the forward and backwards passes on the GPU. In PyTorch, you move your model parameters and other tensors to the GPU memory using model.to('cuda'). You can move them back from the GPU with model.to('cpu') which you'll commonly do when you need to operate on the network output outside of PyTorch. As a demonstration of the increased speed, I'll compare how long it takes to perform a forward and backward pass with and without a GPU.

class Lenet5_extended_GPU(Lenet5):
    
    #Your code here
    
    def __init__(self,dimx,nlabels,epochs=100,lr=0.001):
        
        super().__init__(dimx,nlabels)  
        
        self.lr = lr #Learning Rate
        
        self.optim = optim.Adam(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.criterion = nn.NLLLoss()             
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.valid_loss_during_training = []
        
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        self.to(self.device)
        
    def trainloop(self,trainloader,validloader):
        
        # Optimization Loop
        
        for e in range(int(self.epochs)):
            
            start_time = time.time()
            
            # Random data permutation at each epoch
            
            running_loss = 0.
            
            for images, labels in trainloader:
                
                # Move input and label tensors to the default device
                images, labels = images.to(self.device), labels.to(self.device)  
        
                #Reset Gradients!
                self.optim.zero_grad() # zero the parameter gradients - # clear the gradients of all optimized variables
            
                out = self.forward(images)
                loss = self.criterion(out,labels) # calculate the loss 
                
                #Compute gradients
                loss.backward()  # backward pass: compute gradient of the loss with respect to model parameters
                
                #SGD stem
                self.optim.step() # perform a single optimization step (parameter update)
                
                running_loss += loss.item()
                
                
            self.loss_during_training.append(running_loss/len(trainloader))
            
            # Validation Loss

            # Turn off gradients for validation, saves memory and computations
            with torch.no_grad():            
                
                running_loss = 0.
                
                for images,labels in validloader:
                    
                    # Move input and label tensors to the default device
                    images, labels = images.to(self.device), labels.to(self.device)                    
                    
                    # Compute output for input minibatch
                    out = self.forward(images)
                
                    loss = self.criterion(out,labels) # calculate the loss

                    running_loss += loss.item()

                    
                self.valid_loss_during_training.append(running_loss/len(validloader))    
            
            if(e % 1 == 0): # Every 10 epochs

                print("Epoch %d. Training loss: %f, Validation loss: %f, Time per epoch: %f seconds" 
                      %(e,self.loss_during_training[-1],self.valid_loss_during_training[-1],
                       (time.time() - start_time)))
                
    def eval_performance(self,dataloader):
        
        loss = 0
        accuracy = 0

        # Turn off gradients for validation, saves memory and computations
        with torch.no_grad():

            for images,labels in dataloader:
                # Move input and label tensors to the default device
                images, labels = images.to(self.device), labels.to(self.device) 
                probs = self.forward(images)

                top_p, top_class = probs.topk(1, dim=1)
                equals = (top_class == labels.view(images.shape[0], 1))
                accuracy += torch.mean(equals.type(torch.FloatTensor))
    
            return accuracy/len(dataloader)

    def test_performance(self,testloader):
        correct = 0
        total = 0
        
        with torch.no_grad():
            for images,labels in testloader:
                images, labels = images.to(self.device), labels.to(self.device)
                # calculate outputs by running images through the network
                outputs = self.forward(images)
                # the class with the highest energy is what we choose as prediction
                _, predicted = torch.max(outputs.data, 1)
                total += labels.size(0)
                correct += (predicted == labels).sum().item()
            return correct // total

my_CNN_GPU = Lenet5_extended_GPU(dimx=32,nlabels=10,epochs=5,lr=1e-3)

my_CNN_GPU.trainloop(trainloader,valloader)

Epoch 0. Training loss: 1.722676, Validation loss: 1.503800, Time per epoch: 19.181138 seconds
Epoch 1. Training loss: 1.420213, Validation loss: 1.351454, Time per epoch: 17.826654 seconds
Epoch 2. Training loss: 1.283031, Validation loss: 1.255768, Time per epoch: 17.389950 seconds
Epoch 3. Training loss: 1.186067, Validation loss: 1.179107, Time per epoch: 17.145956 seconds
Epoch 4. Training loss: 1.114503, Validation loss: 1.149109, Time per epoch: 18.186628 seconds

With a GPU, it can be seen that the time per epoch roughly decreases significantly. Without GPU, the average execution time per epoch was found 21 seconds while with a GPU it was found more or less 17-18 seconds. As the model is trained deeper and much more complex networks, this difference grows exponentially fast. Using GPUs is a must for lage-scale deployment.

Regularizing the network and comparing with a MLP

1- Checking that the CNN is able to overfit

---

In order to make the model overfit, the model must be trained with more epochs.

my_CNN_GPU = Lenet5_extended_GPU(dimx=32,nlabels=10,epochs=20,lr=1e-3)

my_CNN_GPU.trainloop(trainloader,valloader)

Epoch 0. Training loss: 1.694066, Validation loss: 1.479521, Time per epoch: 17.405385 seconds
Epoch 1. Training loss: 1.417526, Validation loss: 1.378347, Time per epoch: 17.478302 seconds
Epoch 2. Training loss: 1.306848, Validation loss: 1.299959, Time per epoch: 17.472908 seconds
Epoch 3. Training loss: 1.207773, Validation loss: 1.240853, Time per epoch: 17.315703 seconds
Epoch 4. Training loss: 1.134177, Validation loss: 1.187412, Time per epoch: 17.762075 seconds
Epoch 5. Training loss: 1.073532, Validation loss: 1.170375, Time per epoch: 17.267591 seconds
Epoch 6. Training loss: 1.018931, Validation loss: 1.115519, Time per epoch: 17.452738 seconds
Epoch 7. Training loss: 0.969898, Validation loss: 1.127267, Time per epoch: 17.248549 seconds
Epoch 8. Training loss: 0.931270, Validation loss: 1.094133, Time per epoch: 17.425753 seconds
Epoch 9. Training loss: 0.885153, Validation loss: 1.100516, Time per epoch: 17.563225 seconds
Epoch 10. Training loss: 0.853715, Validation loss: 1.093692, Time per epoch: 17.341029 seconds
Epoch 11. Training loss: 0.820350, Validation loss: 1.112883, Time per epoch: 17.395027 seconds
Epoch 12. Training loss: 0.787324, Validation loss: 1.089389, Time per epoch: 17.458902 seconds
Epoch 13. Training loss: 0.751139, Validation loss: 1.110531, Time per epoch: 17.359501 seconds
Epoch 14. Training loss: 0.721949, Validation loss: 1.139330, Time per epoch: 17.456936 seconds
Epoch 15. Training loss: 0.699629, Validation loss: 1.149347, Time per epoch: 17.539114 seconds
Epoch 16. Training loss: 0.668349, Validation loss: 1.182674, Time per epoch: 17.260889 seconds
Epoch 17. Training loss: 0.645473, Validation loss: 1.191940, Time per epoch: 17.501725 seconds
Epoch 18. Training loss: 0.615676, Validation loss: 1.220180, Time per epoch: 17.418172 seconds
Epoch 19. Training loss: 0.600311, Validation loss: 1.268973, Time per epoch: 17.375661 seconds

plt.plot(my_CNN_GPU.loss_during_training,label='Training Loss')
plt.plot(my_CNN_GPU.valid_loss_during_training,label='Validation Loss')

plt.legend()

print(my_CNN_GPU.eval_performance(trainloader))
print(my_CNN_GPU.eval_performance(valloader))
print(my_CNN_GPU.test_performance(testloader))

tensor(0.8014)
tensor(0.6261)
0

The model has been trained for longer than required to ensure the model overfits. From the above visual it is clear that the training loss decreases throughout the epochs. On the other hand, the validaation loss decreases until the 7th epoch, but then it starts increasing with the number of epochs. This fact indicates that CNN is able to overfit.

2- Regularizing the network with both early stopping and dropout.

• Dropout implementation in the network:

class Lenet5_improved(nn.Module):
    def __init__(self,dimx,nlabels): #Nlabels will be 10 in our case
        super().__init__()

        # convolutional layer (sees 28x28x1 image tensor)
        self.conv1 = nn.Conv2d(in_channels=3, out_channels=6, 
                               kernel_size=5, stride=1, padding=0)
        
        # convolutional layer (sees 12x12x16 tensor)
        self.conv2 = nn.Conv2d(6, 16, 5, padding=0)

        
        # Max pool layer
        self.pool = nn.MaxPool2d(2, 2)

        # Linear layers

        self.linear1 = nn.Linear(16*5*5,120)
        
        self.linear2 = nn.Linear(120,84)
        
        self.linear3 = nn.Linear(84, nlabels)
    
        self.relu = nn.ReLU()
        
        self.logsoftmax = nn.LogSoftmax(dim=1) 
        
        # dropout layer (p=0.25)
        self.dropout = nn.Dropout(0.25)

        # Spatial dimension of the Tensor at the output of the 2nd CNN
        self.final_dim =int(((dimx-4)/2-4)/2)
        
    def forward(self, x):
        # Pass the input tensor through the CNN operations
        x = self.conv1(x)
        x = self.relu(x) 
        x = self.pool(x)
        x = self.conv2(x)
        x = self.relu(x)
        x = self.pool(x)
        # Flatten the tensor into a vector of appropiate dimension using self.final_dim
        x = x.view(-1, 16*self.final_dim**2)
        # add dropout layer
        x = self.dropout(x)
        # Pass the tensor through the Dense Layers

        x = self.linear1(x)
        x = self.relu(x)
        # add dropout layer
        x = self.dropout(x)

        x = self.linear2(x)
        x = self.relu(x)
        # add dropout layer
        x = self.dropout(x)
        
        x = self.linear3(x)
        x = self.logsoftmax(x) 
        return x

• Early Stopping Implementation:

As it has been proved before, the Lenet5 model could overfit when training the model during more than 7 epochs. In this step, the early stopping will be implemented in order to solve this problem. Early stopping is a method that allows us to stop training once the model performance stops improving on the validation set. In that way, it prevents overfitting in the model.

For early stopping method, 10 epoches are defined as the number of epoch limitation. In other words, after 10 epoches in which the best found validation loss could not be improved, the training will be stopped early. In each epoch iteration, the minimum loss value will be kept in the best_loss parameter and will be compared with the following epoch validation loss value. To decide whether there is an improvement or not, the difference parameter will be used. If best loss minus the current epoch's loss is larger than 0, that means the epoch could improve the validation loss. Therefore, the count will be 0. If the best loss minus the calculated current epoch's loss is smaller than 0, it means that the current epoch's loss is larger than the best loss. Therefore, there is no improvement in the loss. In that case, the count parameter will be iterated by 1. When the count parameter reaches to 10, the training will be stopped.

class EarlyStopping():

    def __init__(self, limit_epoch=10, difference=0):
        # Parameters
        self.limit_epoch = limit_epoch # to define after how many non-improved loss the algorithm should stop
        self.difference = difference # difference between the best loss and the epoch validation loss to track the improvement
        self.count = 0 # to count the not improved epoches
        self.best_loss = None # the best validation loss in all epoches
        self.early_stop = False # binary early_stop, later will be used in the if loop

    def __call__(self, validation_loss):
        if self.best_loss == None:
            self.best_loss = validation_loss
        elif self.best_loss - validation_loss > self.difference:
            self.best_loss = validation_loss
            # reset the count parameter, if validation loss improves
            self.count = 0
        elif self.best_loss - validation_loss < self.difference:
            # increase the count parameter, if validation loss improves
            self.count += 1
            print(f"Early stopping counter {self.count} of {self.limit_epoch}")
            if self.count >= self.limit_epoch:
                print('Early stopping!')
                self.early_stop = True

# Initializing the Early Stopping class:
early_stopping = EarlyStopping()

• Extending the new Lenet5 network with dropout:

class Lenet5_extended_GPU(Lenet5_improved):
    
    def __init__(self,dimx,nlabels,epochs,lr):
        
        super().__init__(dimx,nlabels)  
        
        self.lr = lr #Learning Rate
        
        self.optim = optim.Adam(self.parameters(), self.lr)
        
        self.epochs = epochs
        
        self.criterion = nn.NLLLoss()             
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        
        self.valid_loss_during_training = []
        
        self.device = torch.device("cuda" if torch.cuda.is_available() else "cpu")

        self.to(self.device)
        
    def trainloop(self,trainloader,validloader):
        
        # Optimization Loop
        valid_loss_min = np.Inf # track change in validation loss
        self.train()
        for e in range(int(self.epochs)):
              
            start_time = time.time()
            
            # keep track of training and validation loss
            train_loss = 0.
            valid_loss = 0.
              
            for images, labels in trainloader:
                  
                # Move input and label tensors to the default device
                images, labels = images.to(self.device), labels.to(self.device)  
          
                #Reset Gradients!
                self.optim.zero_grad() # zero the parameter gradients - # clear the gradients of all optimized variables
              
                out = self.forward(images)
                loss = self.criterion(out,labels) # calculate the loss 
                  
                #Compute gradients
                loss.backward()  # backward pass: compute gradient of the loss with respect to model parameters
                  
                #SGD stem
                self.optim.step() # perform a single optimization step (parameter update)
                  
                train_loss += loss.item()
                  
                  
            self.loss_during_training.append(train_loss/len(trainloader))
              
            # Validation Loss
              
            # Turn off gradients for validation, saves memory and computations
            with torch.no_grad():            
                
                # setting model to evaluation mode
                self.eval() 
                  
                for images,labels in validloader:
                      
                    # Move input and label tensors to the default device
                    images, labels = images.to(self.device), labels.to(self.device)                    
                      
                    # Compute output for input minibatch
                    out = self.forward(images)
                  
                    loss = self.criterion(out,labels) # calculate the loss

                    valid_loss += loss.item()
   
            self.valid_loss_during_training.append(valid_loss/len(validloader))

            early_stopping(valid_loss)
            if early_stopping.early_stop:
              break  


            print("Epoch %d. Training loss: %f, Validation loss: %f, Time per epoch: %f seconds" 
                        %(e,self.loss_during_training[-1],self.valid_loss_during_training[-1],
                        (time.time() - start_time)))
              

    def eval_performance (self, trainloader, testloader, validloader):
        
      with torch.no_grad():
            
          accuracy_train = 0
          accuracy_test = 0
          accuracy_val = 0

          self.eval()    # we are going to calculate the accuracies, so set the model to evaluation mode

          for images,labels in trainloader:
            # Move input and label tensors to the default device
            images, labels = images.to(self.device), labels.to(self.device) 
            logprobs_tr = self.forward(images)

            top_p, top_class = logprobs_tr.topk(1, dim=1)
            equals_tr = (top_class == labels.view(images.shape[0], 1))
            accuracy_train += torch.mean(equals_tr.type(torch.FloatTensor))

          
          for images,labels in validloader:

            # Move input and label tensors to the default device
            images, labels = images.to(self.device), labels.to(self.device) 
            
            logprobs_vl = self.forward(images)
            top_p_vl, top_class_vl = logprobs_vl.topk(1, dim=1)
            equals_vl = (top_class_vl == labels.view(images.shape[0], 1))
            accuracy_val += torch.mean(equals_vl.type(torch.FloatTensor))
            
          for images,labels in testloader:
            # Move input and label tensors to the default device
            images, labels = images.to(self.device), labels.to(self.device) 
            
            logprobs_ts = self.forward(images)
            top_p_ts, top_class_ts = logprobs_ts.topk(1, dim=1)
            equals_ts = (top_class_ts == labels.view(images.shape[0], 1))
            accuracy_test += torch.mean(equals_ts.type(torch.FloatTensor))
          
      return accuracy_train/len(trainloader), accuracy_val/len(validloader), accuracy_test/len(testloader)

In the code below, the model my_CNN_imp_GPU will be run with both dropout and early stopping regularization to eliminate the overfitting in the model. In the model as an epoch parameter, 100 is defined to converge.

my_CNN_imp_GPU = Lenet5_extended_GPU(dimx=32,nlabels=10,epochs=100,lr=1e-3)

my_CNN_imp_GPU.trainloop(trainloader,valloader)

Epoch 0. Training loss: 1.769636, Validation loss: 1.518724, Time per epoch: 15.772582 seconds
Epoch 1. Training loss: 1.516030, Validation loss: 1.380858, Time per epoch: 15.543652 seconds
Epoch 2. Training loss: 1.412666, Validation loss: 1.322238, Time per epoch: 15.703054 seconds
Epoch 3. Training loss: 1.346566, Validation loss: 1.236936, Time per epoch: 15.690132 seconds
Epoch 4. Training loss: 1.286451, Validation loss: 1.218293, Time per epoch: 15.597046 seconds
Epoch 5. Training loss: 1.248385, Validation loss: 1.176797, Time per epoch: 15.740614 seconds
Epoch 6. Training loss: 1.215839, Validation loss: 1.150784, Time per epoch: 15.675888 seconds
Epoch 7. Training loss: 1.182885, Validation loss: 1.109721, Time per epoch: 15.747235 seconds
Epoch 8. Training loss: 1.158381, Validation loss: 1.096875, Time per epoch: 16.472414 seconds
Early stopping counter 1 of 10
Epoch 9. Training loss: 1.143563, Validation loss: 1.110016, Time per epoch: 15.844998 seconds
Epoch 10. Training loss: 1.126278, Validation loss: 1.067368, Time per epoch: 15.870653 seconds
Epoch 11. Training loss: 1.110502, Validation loss: 1.048268, Time per epoch: 15.915680 seconds
Early stopping counter 1 of 10
Epoch 12. Training loss: 1.095606, Validation loss: 1.060708, Time per epoch: 15.728544 seconds
Epoch 13. Training loss: 1.076341, Validation loss: 1.036644, Time per epoch: 15.889160 seconds
Epoch 14. Training loss: 1.066478, Validation loss: 1.029816, Time per epoch: 15.874175 seconds
Epoch 15. Training loss: 1.058884, Validation loss: 1.016098, Time per epoch: 15.671093 seconds
Epoch 16. Training loss: 1.049241, Validation loss: 1.015457, Time per epoch: 15.815647 seconds
Epoch 17. Training loss: 1.041617, Validation loss: 1.009131, Time per epoch: 15.793412 seconds
Epoch 18. Training loss: 1.031145, Validation loss: 1.007883, Time per epoch: 15.882436 seconds
Epoch 19. Training loss: 1.015049, Validation loss: 1.002274, Time per epoch: 15.746040 seconds
Epoch 20. Training loss: 1.006010, Validation loss: 0.997033, Time per epoch: 15.739585 seconds
Epoch 21. Training loss: 1.006268, Validation loss: 0.979797, Time per epoch: 15.718642 seconds
Early stopping counter 1 of 10
Epoch 22. Training loss: 1.005344, Validation loss: 0.983090, Time per epoch: 15.699047 seconds
Early stopping counter 2 of 10
Epoch 23. Training loss: 1.004836, Validation loss: 0.980730, Time per epoch: 15.772775 seconds
Early stopping counter 3 of 10
Epoch 24. Training loss: 0.986797, Validation loss: 1.010714, Time per epoch: 15.684481 seconds
Epoch 25. Training loss: 0.982671, Validation loss: 0.972486, Time per epoch: 15.756565 seconds
Early stopping counter 1 of 10
Epoch 26. Training loss: 0.981342, Validation loss: 0.991195, Time per epoch: 15.688357 seconds
Epoch 27. Training loss: 0.970409, Validation loss: 0.971556, Time per epoch: 15.661725 seconds
Early stopping counter 1 of 10
Epoch 28. Training loss: 0.972671, Validation loss: 0.989703, Time per epoch: 15.723439 seconds
Early stopping counter 2 of 10
Epoch 29. Training loss: 0.962357, Validation loss: 0.982857, Time per epoch: 15.651598 seconds
Early stopping counter 3 of 10
Epoch 30. Training loss: 0.961194, Validation loss: 0.985967, Time per epoch: 15.572409 seconds
Epoch 31. Training loss: 0.963233, Validation loss: 0.969183, Time per epoch: 15.570376 seconds
Early stopping counter 1 of 10
Epoch 32. Training loss: 0.955439, Validation loss: 0.994489, Time per epoch: 15.516584 seconds
Epoch 33. Training loss: 0.942865, Validation loss: 0.956927, Time per epoch: 15.432597 seconds
Early stopping counter 1 of 10
Epoch 34. Training loss: 0.954593, Validation loss: 0.970753, Time per epoch: 15.598616 seconds
Early stopping counter 2 of 10
Epoch 35. Training loss: 0.942412, Validation loss: 0.964552, Time per epoch: 15.742693 seconds
Early stopping counter 3 of 10
Epoch 36. Training loss: 0.937879, Validation loss: 0.959243, Time per epoch: 15.608022 seconds
Epoch 37. Training loss: 0.933762, Validation loss: 0.955628, Time per epoch: 15.689507 seconds
Epoch 38. Training loss: 0.934677, Validation loss: 0.950597, Time per epoch: 15.688710 seconds
Epoch 39. Training loss: 0.928082, Validation loss: 0.943600, Time per epoch: 15.506280 seconds
Epoch 40. Training loss: 0.924632, Validation loss: 0.943329, Time per epoch: 15.661728 seconds
Early stopping counter 1 of 10
Epoch 41. Training loss: 0.921570, Validation loss: 0.975525, Time per epoch: 15.488001 seconds
Early stopping counter 2 of 10
Epoch 42. Training loss: 0.927202, Validation loss: 0.947416, Time per epoch: 15.416771 seconds
Epoch 43. Training loss: 0.921364, Validation loss: 0.937608, Time per epoch: 15.643391 seconds
Early stopping counter 1 of 10
Epoch 44. Training loss: 0.917405, Validation loss: 0.959538, Time per epoch: 15.548599 seconds
Early stopping counter 2 of 10
Epoch 45. Training loss: 0.917612, Validation loss: 0.955936, Time per epoch: 15.559103 seconds
Early stopping counter 3 of 10
Epoch 46. Training loss: 0.910418, Validation loss: 0.947744, Time per epoch: 15.351575 seconds
Early stopping counter 4 of 10
Epoch 47. Training loss: 0.909194, Validation loss: 0.991392, Time per epoch: 15.341860 seconds
Early stopping counter 5 of 10
Epoch 48. Training loss: 0.907219, Validation loss: 0.950951, Time per epoch: 15.418730 seconds
Early stopping counter 6 of 10
Epoch 49. Training loss: 0.906887, Validation loss: 0.942656, Time per epoch: 15.367084 seconds
Early stopping counter 7 of 10
Epoch 50. Training loss: 0.901775, Validation loss: 0.939970, Time per epoch: 15.498346 seconds
Early stopping counter 8 of 10
Epoch 51. Training loss: 0.902719, Validation loss: 0.966898, Time per epoch: 15.471646 seconds
Epoch 52. Training loss: 0.900131, Validation loss: 0.932391, Time per epoch: 15.328622 seconds
Early stopping counter 1 of 10
Epoch 53. Training loss: 0.892072, Validation loss: 0.939515, Time per epoch: 15.384465 seconds
Early stopping counter 2 of 10
Epoch 54. Training loss: 0.899635, Validation loss: 0.943041, Time per epoch: 15.257745 seconds
Early stopping counter 3 of 10
Epoch 55. Training loss: 0.896255, Validation loss: 0.937917, Time per epoch: 15.376848 seconds
Epoch 56. Training loss: 0.890247, Validation loss: 0.921407, Time per epoch: 15.400478 seconds
Early stopping counter 1 of 10
Epoch 57. Training loss: 0.883513, Validation loss: 0.939919, Time per epoch: 15.328808 seconds
Early stopping counter 2 of 10
Epoch 58. Training loss: 0.888784, Validation loss: 0.965999, Time per epoch: 15.413940 seconds
Early stopping counter 3 of 10
Epoch 59. Training loss: 0.884334, Validation loss: 0.950786, Time per epoch: 15.302099 seconds
Early stopping counter 4 of 10
Epoch 60. Training loss: 0.890814, Validation loss: 0.943779, Time per epoch: 15.405481 seconds
Early stopping counter 5 of 10
Epoch 61. Training loss: 0.880582, Validation loss: 0.944423, Time per epoch: 15.290680 seconds
Epoch 62. Training loss: 0.878316, Validation loss: 0.916767, Time per epoch: 15.319338 seconds
Early stopping counter 1 of 10
Epoch 63. Training loss: 0.876394, Validation loss: 0.947484, Time per epoch: 15.395790 seconds
Early stopping counter 2 of 10
Epoch 64. Training loss: 0.878223, Validation loss: 0.929745, Time per epoch: 15.272203 seconds
Early stopping counter 3 of 10
Epoch 65. Training loss: 0.879450, Validation loss: 0.936510, Time per epoch: 15.424749 seconds
Early stopping counter 4 of 10
Epoch 66. Training loss: 0.874268, Validation loss: 0.957469, Time per epoch: 15.354620 seconds
Early stopping counter 5 of 10
Epoch 67. Training loss: 0.874980, Validation loss: 0.939508, Time per epoch: 15.344408 seconds
Early stopping counter 6 of 10
Epoch 68. Training loss: 0.873062, Validation loss: 0.928563, Time per epoch: 15.115002 seconds
Early stopping counter 7 of 10
Epoch 69. Training loss: 0.874047, Validation loss: 0.939844, Time per epoch: 15.256528 seconds
Epoch 70. Training loss: 0.872894, Validation loss: 0.915981, Time per epoch: 15.398196 seconds
Early stopping counter 1 of 10
Epoch 71. Training loss: 0.867477, Validation loss: 0.924133, Time per epoch: 15.269915 seconds
Early stopping counter 2 of 10
Epoch 72. Training loss: 0.868941, Validation loss: 0.920570, Time per epoch: 15.092766 seconds
Early stopping counter 3 of 10
Epoch 73. Training loss: 0.863827, Validation loss: 0.934505, Time per epoch: 15.275906 seconds
Early stopping counter 4 of 10
Epoch 74. Training loss: 0.868805, Validation loss: 0.917821, Time per epoch: 15.249027 seconds
Early stopping counter 5 of 10
Epoch 75. Training loss: 0.857499, Validation loss: 0.927968, Time per epoch: 14.825669 seconds
Early stopping counter 6 of 10
Epoch 76. Training loss: 0.872411, Validation loss: 0.934008, Time per epoch: 15.289083 seconds
Early stopping counter 7 of 10
Epoch 77. Training loss: 0.860906, Validation loss: 0.995314, Time per epoch: 15.275891 seconds
Early stopping counter 8 of 10
Epoch 78. Training loss: 0.859075, Validation loss: 0.930024, Time per epoch: 15.561079 seconds
Early stopping counter 9 of 10
Epoch 79. Training loss: 0.859557, Validation loss: 0.924611, Time per epoch: 15.455220 seconds
Early stopping counter 10 of 10
Early stopping!

early_stopping.best_loss/len(valloader) # epoch 70

0.9159806978171039

The minimum validation loss has been found 0.9159 at 70th epoch. This best loss could not be decreased in the last 10 consecutive epoches. Therefore, the training has been stopped before the epoch value reaches 100.

3- Checking the train/validation/test performance, plot the train and validation losses

plt.plot(my_CNN_imp_GPU.loss_during_training,label='Training Loss')
plt.plot(my_CNN_imp_GPU.valid_loss_during_training,label='Validation Loss')
plt.legend()

train_acc, test_acc, val_acc = my_CNN_imp_GPU.eval_performance(trainloader,valloader,testloader)
print("Training accuracy: %f, Validation accuracy: %f, Test accuracy: %f" 
                        %(train_acc,val_acc,test_acc))

Training accuracy: 0.786700, Validation accuracy: 0.667695, Test accuracy: 0.665307

As can be seen in the above results, the performance of the model has significantly increased. Looking at the validation loss evaluation curve in each epoch,it can be seen that initially the validation loss decreases sharply, then the loss curve progresses almost stably. Of course, there are some fluctuations in the loss line, but these ups and downs are not sharp. This means that after these regulations, the overfitting situation has been fixed and an improvement in the performance of the model has been achieved.

To conclude, the best validation loss has been found at 70th epoch with 0.91 value. Training, validation and test accuracies have been calculated %78, %67 and %66, respectively. The difference between the accuracy o the train set and on the validation and test set is not very small. However, now the validation loss curve does not increase throughout epochs.

Training an MLP to compare the performance with LeNet 5 CNN Network

In this part, an MLP with four layers with hidden dimmensions 256, 128 and 64 neurons will be trained. As activation functions, ReLU will be used with a log-Softmax output layer. As mentioned, first the input size (3x32x32) is defined. To prevent overfitting in the model, dropout and early stopping regularizations are used.

input_size = 3*32*32

# define the NN architecture
class MLP(nn.Module):
    def __init__(self,hidden1,hidden2,hidden3,nlabels,p): #Nlabels will be 10 in our case
        super().__init__()
   
        self.output1 = nn.Linear(input_size, hidden1)
        self.output2 = nn.Linear(hidden1, hidden2)
        self.output3 = nn.Linear(hidden2, hidden3)
        self.output4 = nn.Linear(hidden3,nlabels)
        self.relu = nn.ReLU()
        
        self.logsoftmax = nn.LogSoftmax(dim=1)
        # dropout prevents overfitting of data
        self.dropout = nn.Dropout(p)       # We add the dropout

    def forward(self, x):
        # flatten image input
        x = x.view(-1, input_size)
        # add hidden layer, with relu activation function and droupout
        x = self.output1(x)
        x = self.relu(x)
        x = self.dropout(x)
        # add second hidden layer, with relu activation function and dropout
        x = self.output2(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.output3(x)
        x = self.relu(x)
        x = self.dropout(x)
        x = self.output4(x)
        x = self.logsoftmax(x)
        return x

# Initializing the Early Stopping class:
early_stopping = EarlyStopping()

class MLP_extended(MLP):
    
    def __init__(self,hidden1,hidden2,hidden3,nlabels,epochs,lr,p):
        
        super().__init__(hidden1,hidden2,hidden3,nlabels,p)  
        
        self.lr = lr 
        
        self.optim = optim.Adam(self.parameters(), self.lr) 
        
        self.epochs = epochs
        
        self.criterion = nn.NLLLoss()               
        
        # A list to store the loss evolution along training
        
        self.loss_during_training = [] 
        self.valid_loss_during_training = []
        
    def trainloop(self,trainloader, validloader):
        
        # Optimization Loop
        
        for e in range(int(self.epochs)):
            start_time = time.time()
            
            running_loss = 0.
            self.train()
            for images, labels in trainloader:       
        
                self.optim.zero_grad()  #TO RESET GRADIENTS!
            
                out = self.forward(images)

                loss = self.criterion(out, labels)  

                running_loss += loss.item()

                loss.backward() #The optimizer can be called one the gradients have been computed

                self.optim.step() #update the parameters
        
            self.loss_during_training.append(running_loss/len(trainloader))
                  
            with torch.no_grad():
              self.eval()
              running_loss_valid = 0

              for images, labels in validloader:
            
                out = self.forward(images)

                loss = self.criterion(out, labels)  

                running_loss_valid += loss.item()

            self.valid_loss_during_training.append(running_loss_valid/len(validloader))
            
            # early stopping check
            early_stopping(running_loss_valid)
            
            if early_stopping.early_stop:
              break

            print("Epoch %d. Training loss: %f, Validation loss: %f, Time per epoch: %f seconds" 
                        %(e,self.loss_during_training[-1],self.valid_loss_during_training[-1],
                        (time.time() - start_time)))


    def accuracy (self, trainloader, testloader, validloader):
        
      with torch.no_grad():
            
          accuracy_train = 0
          accuracy_test = 0
          accuracy_val = 0

          self.eval() ## setting to the evaluation mode to calculate the accuracies

          for images,labels in trainloader:

            logprobs_tr = self.forward(images)
            top_p, top_class = logprobs_tr.topk(1, dim=1)
            equals_tr = (top_class == labels.view(images.shape[0], 1))
            accuracy_train += torch.mean(equals_tr.type(torch.FloatTensor))

          for images,labels in testloader:

            logprobs_ts = self.forward(images)
            top_p_ts, top_class_ts = logprobs_ts.topk(1, dim=1)
            equals_ts = (top_class_ts == labels.view(images.shape[0], 1))
            accuracy_test += torch.mean(equals_ts.type(torch.FloatTensor))


          for images,labels in validloader:

            logprobs_vl = self.forward(images)
            top_p_vl, top_class_vl = logprobs_vl.topk(1, dim=1)
            equals_vl = (top_class_vl == labels.view(images.shape[0], 1))
            accuracy_val += torch.mean(equals_vl.type(torch.FloatTensor))
  
      return accuracy_train/len(trainloader), accuracy_test/len(testloader), accuracy_val/len(validloader)

my_MLP = MLP_extended(hidden1=256,hidden2=128,hidden3=64,nlabels=10,epochs=60,lr=0.001,p=0.25)
my_MLP.trainloop(trainloader,valloader)
train_acc, test_acc, val_acc = my_MLP.accuracy(trainloader,testloader,valloader)
print("Train Accuracy: %f" %(train_acc))
print("Test Accuracy: %f" %(test_acc))
print("Validation Accuracy: %f" %(val_acc))

Epoch 0. Training loss: 1.825299, Validation loss: 1.627108, Time per epoch: 19.953560 seconds
Epoch 1. Training loss: 1.662846, Validation loss: 1.558880, Time per epoch: 19.295268 seconds
Epoch 2. Training loss: 1.584658, Validation loss: 1.527750, Time per epoch: 18.919059 seconds
Epoch 3. Training loss: 1.535951, Validation loss: 1.471910, Time per epoch: 18.990492 seconds
Epoch 4. Training loss: 1.489256, Validation loss: 1.462670, Time per epoch: 19.015769 seconds
Epoch 5. Training loss: 1.460978, Validation loss: 1.428617, Time per epoch: 20.157516 seconds
Epoch 6. Training loss: 1.427745, Validation loss: 1.419624, Time per epoch: 19.085084 seconds
Epoch 7. Training loss: 1.402178, Validation loss: 1.413159, Time per epoch: 19.004879 seconds
Epoch 8. Training loss: 1.373958, Validation loss: 1.412606, Time per epoch: 19.269918 seconds
Epoch 9. Training loss: 1.350086, Validation loss: 1.400453, Time per epoch: 19.125472 seconds
Epoch 10. Training loss: 1.324644, Validation loss: 1.389628, Time per epoch: 19.167208 seconds
Epoch 11. Training loss: 1.305595, Validation loss: 1.385776, Time per epoch: 19.374792 seconds
Epoch 12. Training loss: 1.290861, Validation loss: 1.374217, Time per epoch: 19.071593 seconds
Epoch 13. Training loss: 1.271333, Validation loss: 1.370063, Time per epoch: 19.000846 seconds
Epoch 14. Training loss: 1.257472, Validation loss: 1.359224, Time per epoch: 19.214409 seconds
Early stopping counter 1 of 10
Epoch 15. Training loss: 1.234881, Validation loss: 1.361940, Time per epoch: 19.251578 seconds
Early stopping counter 2 of 10
Epoch 16. Training loss: 1.221904, Validation loss: 1.365007, Time per epoch: 19.188013 seconds
Epoch 17. Training loss: 1.208807, Validation loss: 1.344973, Time per epoch: 19.012641 seconds
Early stopping counter 1 of 10
Epoch 18. Training loss: 1.196489, Validation loss: 1.364623, Time per epoch: 18.934098 seconds
Early stopping counter 2 of 10
Epoch 19. Training loss: 1.186302, Validation loss: 1.348498, Time per epoch: 18.749934 seconds
Early stopping counter 3 of 10
Epoch 20. Training loss: 1.169538, Validation loss: 1.360516, Time per epoch: 18.809340 seconds
Early stopping counter 4 of 10
Epoch 21. Training loss: 1.162044, Validation loss: 1.354674, Time per epoch: 18.613539 seconds
Early stopping counter 5 of 10
Epoch 22. Training loss: 1.147266, Validation loss: 1.347783, Time per epoch: 18.680613 seconds
Early stopping counter 6 of 10
Epoch 23. Training loss: 1.138873, Validation loss: 1.362894, Time per epoch: 18.584562 seconds
Early stopping counter 7 of 10
Epoch 24. Training loss: 1.119814, Validation loss: 1.366091, Time per epoch: 18.601686 seconds
Early stopping counter 8 of 10
Epoch 25. Training loss: 1.109611, Validation loss: 1.367726, Time per epoch: 18.734206 seconds
Early stopping counter 9 of 10
Epoch 26. Training loss: 1.104065, Validation loss: 1.363505, Time per epoch: 18.476501 seconds
Early stopping counter 10 of 10
Early stopping!
Train Accuracy: 0.627550
Test Accuracy: 0.538416
Validation Accuracy: 0.538117

early_stopping.best_loss/len(valloader) # epoch 17

1.34497289331096

The best loss has been found 1.34 at the 17th epoch, which is quite worse than regularized CNN model. Moreover, even though the model trained with 60 epoch, the training stopped at 26th epoch as there were no improvements in the validation loss. According to the accuracy scores in training, validation and test, it can be said clearly that this MLP model has performed worse than the CNN model.

plt.plot(my_MLP.loss_during_training,label='Training Loss')
plt.plot(my_MLP.valid_loss_during_training,label='Validation Loss')
plt.legend()

<matplotlib.legend.Legend at 0x7ff64b01aa90>

Although the performance of the MLP model is lower than that of the CNN model, there is no overfitting in the model as seen in the validation loss curve in the graph above. However, both training and validation final losses are higher than the CNN one. For this data, the regularized CNN model has performed very well.