The modeling of evolutionary game dynamics in finite populations requires microscopic processes that determine how strategies spread. The exact details of these processes are often chosen without much further consideration. Different types of microscopic models, including in particular fitness-based selection rules and pairwise comparison dynamics, are often used as if they were interchangeable. We challenge this view and investigate how robust these choices on the micro-level really are. We focus on a key macroscopic quantity, the probability for a single mutant to take over a population of wild-type individuals. We show that even in unstructured populations there is only one pair of a fitness-based process and a pairwise comparison process leading to identical outcomes for arbitrary games and for all intensities of selection. This strong restriction is not relaxed even when the class of pairwise comparison processes is broadened. This highlights the perils of making arbitrary choices at the micro-level without regard of the consequences at the macro-level.