Here is a definition of HashSet vs HashMap:
A HashMap provides fast access to a mapping from a unique key to a value. Keys are unordered, which makes it faster than the TreeMap, where the keys are ordered. A HashSet is fast access to unique values only (there are no keys because it is not a mapping, it's just the values). HashSet values are unordered, which makes it faster than the TreeSet, where the values are ordered. (Core Java - What is the difference between hashmap and hashset?
I guess in my case, I could just use HashSet because they're not in order, and the existance of those unique values already mean that the flag is set. Thanks!