Python set for analysts

Why set matters for analysts

You are dumping a list of user IDs into a notebook and need to know how many of them are unique, which ones overlap with last week's cohort, and which ones churned. You could write a triple-nested loop. Or you could spend two lines using a Python set and move on. The set is the data structure that quietly carries half of the ad-hoc analytics work on a Monday morning — deduplication, audience overlap, churn deltas, fast in checks — and almost nobody talks about it because it feels too basic to teach.

The reason it earns its keep is the hash table underneath. A set stores unique, hashable elements and lets you ask "is X in here?" in O(1) average time. A list answers the same question in O(n). On a million-row audience file, that gap is the difference between a query that returns instantly and one that locks up your kernel for thirty seconds. And once you internalize the trick, you stop reaching for for loops to check membership.

Creating a set

Three idiomatic constructors cover ninety percent of real usage. Curly braces with values, the set() factory on any iterable, and set comprehensions for transformed inputs.

# Literal — curly braces
colors = {'red', 'green', 'blue'}

# From any iterable — dedupes automatically
numbers = set([1, 2, 3, 2, 1])
print(numbers)  # {1, 2, 3}

# Empty set — only via set(), NOT {}
empty = set()           # correct
not_a_set = {}          # this is a dict, not a set

# From a string — splits by character
chars = set('hello')
print(chars)  # {'h', 'e', 'l', 'o'}

# Set comprehension
squares = {x**2 for x in range(10)}
print(squares)  # {0, 1, 4, 9, 16, 25, 36, 49, 64, 81}

The {} versus set() confusion bites every beginner exactly once. Empty braces are a dict literal because dict came first historically and gets the shorter syntax. Burn set() into muscle memory for empty sets.

Add, remove, and the methods that bite

Mutation is straightforward, but two methods look similar and behave differently when the element is missing. That subtle difference is a favorite warm-up question in a phone screen.

s = {1, 2, 3}

s.add(4)         # {1, 2, 3, 4}
s.add(2)         # {1, 2, 3, 4} — duplicate silently ignored

s.remove(3)      # {1, 2, 4} — KeyError if missing
s.discard(99)    # no-op, no exception
s.pop()          # removes an arbitrary element, returns it
s.clear()        # set()

remove() raises KeyError when the element does not exist. discard() quietly does nothing. The choice depends on whether a missing element is a bug or an expected state — and getting this wrong is how production scripts crash at 3 a.m. on an edge case.

Set operations: union, intersection, difference

This is the meat of why analysts reach for sets. Mathematical set operations map cleanly onto questions about audiences, cohorts, and segments.

a = {1, 2, 3, 4, 5}
b = {4, 5, 6, 7, 8}

# Union — everything in either
a | b              # {1, 2, 3, 4, 5, 6, 7, 8}
a.union(b)

# Intersection — common to both
a & b              # {4, 5}
a.intersection(b)

# Difference — in a but not in b
a - b              # {1, 2, 3}
a.difference(b)

# Symmetric difference — in one or the other, not both
a ^ b              # {1, 2, 3, 6, 7, 8}
a.symmetric_difference(b)

The operator form is shorter, but the method form accepts any iterable — not just a set. a.union([4, 5, 6]) works; a | [4, 5, 6] raises TypeError. Useful when one side is a list you don't want to copy into a set just to compute the union.

Membership and subset checks

Membership is the single most common reason to use a set. Subset and disjoint checks come up less often but are exactly the right tool for "is every premium user already in this cohort?" types of questions.

s = {1, 2, 3, 4, 5}

# Membership — O(1)
3 in s       # True
99 in s      # False
99 not in s  # True

# Subset
{1, 2}.issubset(s)         # True
{1, 2} <= s                # True (operator form)
{1, 2} < s                 # True — strict subset (not equal)

# Superset
s.issuperset({1, 2})       # True
s >= {1, 2}                # True

# Disjoint — no common elements
{1, 2}.isdisjoint({3, 4})  # True
{1, 2}.isdisjoint({2, 3})  # False (shares 2)

A membership check in a set is roughly 50 to 1000 times faster than the same check in a list once you cross a few hundred elements. The gap widens linearly as the collection grows.

Analytical use cases

These three patterns cover most of what you will actually do with sets in a notebook.

Audience overlap between two campaigns

campaign_a_users = {101, 202, 303, 404, 505}
campaign_b_users = {303, 404, 606, 707}

overlap = campaign_a_users & campaign_b_users
print(overlap)            # {303, 404}

only_a = campaign_a_users - campaign_b_users
print(only_a)             # {101, 202, 505}

total_reach = campaign_a_users | campaign_b_users
print(len(total_reach))   # 7

Churn, new, and retained users between two snapshots

users_march = {1, 2, 3, 4, 5}
users_april = {3, 4, 5, 6, 7}

new_users = users_april - users_march       # {6, 7}
churned   = users_march - users_april       # {1, 2}
retained  = users_march & users_april       # {3, 4, 5}

retention_rate = len(retained) / len(users_march)
print(f'Retention: {retention_rate:.0%}')   # Retention: 60%

Fast filter against a whitelist of IDs

all_orders = [
    {'user_id': 1, 'amount': 500},
    {'user_id': 2, 'amount': 300},
    {'user_id': 3, 'amount': 800},
    {'user_id': 4, 'amount': 150},
]

premium_ids = {1, 3}  # set, not list, for the membership check

premium_orders = [
    o for o in all_orders if o['user_id'] in premium_ids
]

Looping over all_orders and checking o['user_id'] in premium_ids against a set is O(n). Doing the same check against a list of premium IDs is O(n × m). On a 10-million-row orders table and 50 thousand premium users, that is the difference between a one-second job and one that never finishes.

Frozenset and immutability

frozenset is the immutable cousin of set. You cannot add or remove from it after creation, which is exactly what makes it hashable and therefore usable as a dict key or as an element of another set.

fs = frozenset([1, 2, 3])
# fs.add(4)  # AttributeError

# Frozenset as a dict key
d = {frozenset({1, 2}): 'group_a',
     frozenset({3, 4}): 'group_b'}

# Set of frozensets
groups = {frozenset({1, 2}), frozenset({3, 4})}

Use it when you need an immutable, hashable collection — for example, modeling unordered pairs of users in a graph, or memoizing function outputs keyed by a combination of inputs.

Set vs list — when to pick which

Pick a set when you need uniqueness, frequent membership checks, or mathematical operations between collections. Pick a list when order matters, you need indexing, or duplicates carry meaning. Tuples sit in a third bucket — ordered and immutable — and pair well with frozenset for hashable composite keys.

Common pitfalls

When teams first reach for sets, the most common mistake is creating an empty set with {}. That literal is a dict, not a set, and the bug only surfaces when you try to add the first element and Python complains about a missing key or stores it as a key with None value. Always use set() for an empty container, and reach for {1, 2} only when you have at least one element to give it.

Another trap is putting mutable objects into a set. Lists and dicts are unhashable, so s.add([1, 2]) raises TypeError: unhashable type: 'list'. The fix is to convert to a tuple (for lists) or frozenset (for sets) before adding. This bites people who try to dedupe a list of records represented as dicts — the right move is usually to dedupe by a key like user_id rather than by the whole dict.

The third pitfall is assuming order. Python sometimes appears to preserve insertion order for small integer sets because of how CPython hashes ints, but that is an implementation detail, not a guarantee. If you need both uniqueness and order, use list(dict.fromkeys(data)) — dict preserves insertion order since Python 3.7, and fromkeys collapses duplicates while keeping the first occurrence.

Performance notes

The headline number is that membership in a set is O(1) average, O(n) worst case when hash collisions degenerate. In practice, you will never see the worst case on standard data types.

A few numbers to anchor intuition. Creating a set from a list of one million unique ints takes roughly 30–60 ms on a modern laptop. Checking membership against that set is on the order of 100 nanoseconds. The same check against a list of one million elements averages 5 milliseconds — fifty thousand times slower. Multiply that by however many checks your script does and the choice becomes obvious.

Set operations like a & b and a - b run in O(min(len(a), len(b))) for intersection and O(len(a)) for difference. They are usually faster than the equivalent comprehension-based filter because they execute in C. If you find yourself writing [x for x in a if x in b], replace it with a & b and your code gets shorter and faster at the same time.

Related reading

• Python interview questions for data analysts
• Python dict cheatsheet for analysts
• Python cheatsheet for analysts
• Lambda functions in Python for analysts
• Pandas vs NumPy when to use which

If you want to drill Python interview questions like the ones below every day, NAILDD is launching with hundreds of analyst problems organized by exactly these patterns.

FAQ

When is a set noticeably faster than a list?

Any time you check membership in a loop. The crossover where set beats list is around 100 elements for a single check, and it kicks in immediately if you do more than one check. For deduplication of large iterables and for computing intersections or differences between collections, set wins by orders of magnitude. The only situations where a list wins are when order matters, when you need indexing, or when the collection is tiny enough that the constant overhead of hashing outweighs the linear scan.

Can I sort a set?

Not in place — set has no sort() method because it has no concept of order. What you can do is sorted(my_set), which returns a new sorted list. The common pattern when you need both uniqueness and order is sorted(set(data)) for ascending order, or list(dict.fromkeys(data)) if you want to preserve insertion order rather than sort.

How do I use sets with pandas?

Two main spots. First, df[df['user_id'].isin(my_set)] — isin accepts any iterable and internally hashes for O(1) lookups, so a set is slightly faster than a list for large collections. Second, df['col'].nunique() and df['col'].unique() give you the count and the distinct values. For chunked dedupe across multiple DataFrames, a plain Python set over df['col'].values is often simplest.

Why is {} a dict and not a set?

Historical accident. Dicts existed in Python before sets did, so {} was already taken when sets were added. The designers picked set() for the empty constructor and reused curly braces for non-empty sets like {1, 2}.

What can I put inside a set?

Only hashable objects: ints, floats, strings, tuples of hashables, frozensets, bools, and None. Lists, dicts, and regular sets are unhashable. The error is TypeError: unhashable type and the fix is to convert to the immutable equivalent (tuple, frozenset) first.

How do I find common elements between two lists in O(n)?

Convert both to sets and intersect: set(a) & set(b). The naive nested-loop version is O(n × m), painful past a few thousand elements per side. The set version pays a one-time O(n + m) hashing cost, then intersects in linear time — and is almost always the expected interview answer.