CSE5519 Advances in Computer Vision (Topic A: 2021 and before: Semantic Segmentation)

SETR

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers

link to the paper 

Treating semantic segmentation as a sequence-to-sequence prediction task.

Novelty in SETR

FCN-based semantic segmentation

An FCN encoder consists of a stack of sequentially connected convolutional layers.

Having limited receptive fields for context modeling is thus an intrinsic limitation of the vanilla FCN architecture.

Segmentation transformers (SETR)

Image to sequence.

By further mapping each vectorized patch $p$ into a latent $C$-dimensional embedding space using a linear projection function $f: p \rightarrow e \in \mathbb{R}^C$, we obtain a 1D sequence of patch embeddings for an image $x$. To encode the patch spacial information, we learn a specific embedding $p_i$ for every location $i$ which is added to $e_i$ to form the final sequence input $E = \{e_1 + p_1, e_2 + p_2, \cdots, e_L + p_L\}$. This way, spatial information is kept despite the orderless self-attention nature of transformers.

Decoder for segmentation

Three choices:

Naive Upsampling:

• Upsample the sequence to the original image size.
• Then use a 1x1 convolution to get the final segmentation map.

Use Progressive Upsampling to get the final segmentation map.

Use Multi-level feature aggregation to get the final segmentation map.

On lecture new takes

DeepLabv3+

Atrous convolutions (large receptive field)

Separable convolutions (depthwise convolutions)

SETR

Learned positional embeddings.