The desire to produce high-quality single photons for applications in quantum information science has lead to renewed interest in exploring solid-state emitters in the weak excitation regime. Under these conditions it is expected that photons are coherently scattered, and so benefit from a substantial suppression of detrimental interactions between the source and its surrounding environment. Nevertheless, we demonstrate here that this reasoning is incomplete, as phonon interactions continue to play a crucial role in determining solid-state emission characteristics even for very weak excitation. We find that the sideband resulting from non-Markovian relaxation of the phonon environment is excitation strength independent. It thus leads to an intrinsic limit to the fraction of coherently scattered light and to the visibility of two-photon coalescence at weak driving, both of which are absent for atomic systems or within simpler Markovian treatments.