In the literature on visual perception there are many examples of how what has previously been perceived affects subsequent perception. Repetition of the same target stimulus in a visual search task speeds task performance, and a stimulus that must be ignored in a particular setting is processed more slowly than otherwise, if it must subsequently be acted upon in another context. We set out to obtain a thorough characterization of such history effects in visual search requiring speeded judgments of target presence or absence among a set of distractors. Our results show that such priming effects have a major influence on visual search performance. Large priming effects are seen when a target is repeated between trials, and our results also show that priming does not only occur for the target but that there is considerable priming due to repeated distractor sets, even between successive trials where no target was presented on either trial. The search also proceeds faster if the same distractor sets are repeated, even when the current target is different from the last target. This suggests that priming can operate on the context of the search, rather than just the target in each case. Furthermore, we investigated the effects of role-reversals of particular display items, from being a target on one trial to being a distractor on the next, and vice versa, showing that such role reversals also affect search performance, over and above the priming effects themselves. We discuss how temporary representations based on previous history may be crucial for visual scene analysis, and how the results provide some clues about how the stability of the visual world is maintained. Finally we discuss the importance of priming of perceptual groups, and of the repetition of context for visual perception.