When we selectively attend to one set of objects and ignore another, we often fail to notice unexpected events. The likelihood of noticing varies depending on the similarity of an unexpected object to other items in the display, a process thought to be controlled by the attention set that we create for the attended and ignored objects. It remains unclear, though, how attention sets are formed and structured. Do they enhance features of attended objects (“white”) and suppress features of ignored objects (“black”), or do they distinguish objects based on relations or categories (“darker” versus “lighter,” or “dark objects” versus “light objects”)? In previous work, these explanations are confounded; the objects would be partitioned into the same groups regardless the structure of the attention set. In the present three experiments, the attended or ignored set of objects was a constant color while the other set was variable. When people attended white and ignored a multicolored set of objects (Experiment 1), novel colors were suppressed just as much as display colors, suggesting nonselective filtering of nonwhite objects. When the color of one set of objects varied across displays but was constant within them (Experiments 2 and 3), we again found as much suppression for task-irrelevant and novel colors as for actively ignored ones. Whenever people ignored a set of objects that varied in color, they suppressed unexpected objects that matched the ignored colors and that differed from the actively ignored items on the critical trial. In contrast, when people attended a varying set, noticing was enhanced only for unexpected objects that matched the currently attended color. In this task, attentional filtering is category-based and did not depend on the features of the individual objects.