Why Are Fully Connected FC Layers Always Placed at the End of CNN?

It is often observed that fully connected (FC) layers are positioned at the end of convolutional neural networks (CNNs). However, this arrangement is not always the best or the only option. Understanding why and when fully connected layers are placed at the end, and exploring alternatives, can help in optimizing deep learning models. This article delves into the reasons behind the placement of fully connected layers in CNNs and discusses experimental evidence, relevant network architectures, and future directions in this area.

The Role of FC Layers in CNNs

The typical architecture of a CNN, like the one used in the ImageNet LSVRC-2010 contest, features five max-pooled convolution layers followed by three fully connected layers and a softmax layer, incorporating a total of 60 million parameters and 650,000 neurons. The main reason for placing the FC layers at the end is their role in classification. The last layer usually has the same number of output nodes as there are classes, and a softmax activation is applied to interpret the model's confidence in each class. However, placing FC layers at the beginning or somewhere else within the model can introduce computational inefficiencies and hinder the model's performance.

Theoretical and Practical Considerations

The point of convolutional layers is to exploit existing correlations between local pixels to reduce the number of parameters. If FC layers were placed earlier in the network, the model would lose these local relationships, making it harder to learn and generalize. Experimentally, placing FC layers at the beginning of a CNN leads to lower accuracy compared to when they are placed at the end. For instance, using TensorFlow and the MNIST dataset, fully connected layers at the beginning yielded an accuracy of 97% compared to 99% when placed at the end.

Why Not Use FC Layers Earlier?

Placing FC layers at the beginning would be impractical due to the dimensionality of the input space. For example, a fully connected layer with 100,000 neurons would result in 15 billion connections with the input space of 150,528 dimensions, making it infeasible. Before the convolutional era, high-dimensional input spaces were typically handled through manual feature extraction, but this method reduces dimensionality and memory usage while maintaining useful features for classification. Convolutional layers, on the other hand, automatically extract useful features through multiple levels of abstraction using an end-to-end gradient descent learning technique.

Alternatives to Fully Connected Layers

While fully connected layers are effective, they can be computationally expensive. An alternative approach is to replace fully connected layers with convolutional layers, which offer two key properties: local connections and shared weights. This shift can reduce the number of parameters required for the network. However, attaching a softmax layer directly above a convolutional layer can be challenging for efficient classification. If such an arrangement were possible, it would drastically reduce the number of parameters used in convolutional networks for image classification tasks.

Despite the advantages, fully connected layers are still widely used near the end or within CNNs because they effectively capture the higher-level features that convolutional layers often fail to fully exploit. Future research and advancements in deep learning may lead to more optimal layer configurations, balancing the computational efficiency and classification performance.