Previous neuroimaging studies support the assumption of a strong link between perception and action, demonstrating that the motor system is involved when others’ actions are observed. One question that is still open to debate is which aspects of observed actions engage the motor system. The present study tested whether motor activation corresponds to the difficulty of the observed action, using Fitts’s law. This law postulates that the difficulty of any movement (ID) is a function of the distance to the target (A) and the target width (W). In an observation task, the ID of the observed action was manipulated orthogonally to W (by using five different As). The results revealed activity in the primary motor cortex, the supplementary motor area, and the basal ganglia in response to increasing ID levels, but not in response to different levels of A or W. Thus, activation in the motor system during action observation is not driven by perceptual parameters but by the motor difficulty of the observed action.