Deep network design is a fundamental challenge. A right trade-off between depth and complexity of convolutional neural networks is of significant importance to applications in low-level vision tasks. Wider feature maps could be beneficial to performance and generality but would increase computational complexity. In this paper, we rethink the balance between width of the feature maps and depth of the network especially for image restoration tasks including deblurring, dehazing, super-resolution and denoising. We explore a new approach to network structure by encouraging more depth to deal with restoration requirements while decreasing the width of some feature maps. Such a slimmer and deeper approach can enhance the performance while maintaining the same level of computational costs. We have experimentally evaluated the performances of the proposed approach on four image restoration tasks and obtained state-of-the-art results on quantitative measures and qualitative assessments, demonstrating the effectiveness of the approach.