The slice has repeats. The map doesn't.
You pull a list of tags from a database query. The join brought back the same tag three times. You need a clean list to render a filter menu. Or you're parsing a config file where a user accidentally repeated a flag. The data is messy. You need the unique set.
Go slices are dynamic arrays. They allow duplicates. Go maps are hash tables. They enforce unique keys. To remove duplicates, you move data from a slice to a map and back. The map filters the repeats. This is the canonical pattern. Go doesn't provide a Set type in the standard library. The language designers prefer maps. A map is flexible. You can store values. You can iterate. A set is just a map with a void value.
Map as a set
The pattern is simple. Create a map. Iterate the slice. For each item, check if it exists in the map. If it does, skip it. If it doesn't, add it to the map and append it to a result slice. The map acts as a tracker. The result slice collects the unique items.
This approach preserves the order of first appearance. If the input is ["beta", "alpha", "beta", "gamma"], the output is ["beta", "alpha", "gamma"]. The second "beta" is ignored. Order preservation comes from the loop, not the map. Maps do not guarantee iteration order. If you range over the map later, the order is random. The slice append keeps the sequence stable.
Here's the standard pattern for strings: iterate the slice, check the map, append if new.
func removeDuplicates(items []string) []string {
// Map keys act as the set. The bool value is a placeholder.
seen := make(map[string]bool)
// Pre-allocate capacity to avoid reallocations during append.
result := make([]string, 0, len(items))
for _, item := range items {
// Check existence. If false, this is the first time we've seen it.
if !seen[item] {
seen[item] = true
result = append(result, item)
}
}
return result
}
The map lookup is O(1) on average. The loop runs N times. Total time is O(N). Space is O(N) for the map and the result slice. This is the fastest general-purpose approach. It trades memory for speed. The map stores every unique key. The result slice stores the output. If the input has many duplicates, the map is smaller than the input. If the input has no duplicates, the map matches the input size.
Maps are the set. Order is the loop. struct{} is the signal.
The zero-size value convention
The code above uses map[string]bool. That works. It's not idiomatic. The value true takes one byte per entry. The map stores that byte alongside the key. For a map with millions of entries, those bytes add up. The value is irrelevant. You only care about the key.
The Go community uses struct{} for set values. struct{} is an empty struct. It has size zero. The map entry stores only the key and the hash metadata. No value bytes. This saves memory. It also signals intent. When a reader sees map[T]struct{}, they know this is a set. The value is a placeholder.
Here's the idiomatic version using the zero-size struct.
func removeDuplicatesIdiomatic(items []string) []string {
// struct{} takes zero bytes. It's the idiomatic value for sets.
seen := make(map[string]struct{})
result := make([]string, 0, len(items))
for _, item := range items {
// The blank discards the zero value. We only care about existence.
if _, exists := seen[item]; !exists {
seen[item] = struct{}{}
result = append(result, item)
}
}
return result
}
The _, exists := seen[item] idiom is standard. The map lookup returns two values: the value and a boolean indicating existence. The underscore discards the value. The boolean tells you if the key is present. This is clearer than checking the value. It works even if the value could be a "falsey" zero value. With struct{}, the value is always the same, so checking existence is the only meaningful operation.
Use struct{} for sets. It saves memory and communicates purpose.
Realistic example: struct method
Real code often deals with structs, not raw slices. You might have a config struct with a list of allowed hosts. The list might contain duplicates from multiple config sources. You need a method to clean it up.
Here's a method on a config struct that deduplicates hosts while preserving order.
type Config struct {
// AllowedHosts might contain duplicates from multiple config sources.
AllowedHosts []string
}
// DedupHosts returns a new slice with unique hosts, preserving order.
func (c *Config) DedupHosts() []string {
seen := make(map[string]struct{})
// Capacity matches the input to minimize allocations.
result := make([]string, 0, len(c.AllowedHosts))
for _, host := range c.AllowedHosts {
if _, exists := seen[host]; !exists {
seen[host] = struct{}{}
result = append(result, host)
}
}
return result
}
The receiver name c matches the type Config. That's the convention. Receiver names are usually one or two letters. The method returns a new slice. It doesn't modify the struct in place. Go favors returning new values over mutating state. The caller decides whether to assign the result back. This keeps the function pure and predictable.
Don't mutate the receiver unless the name implies it. Return a new slice.
Generics: write it once
Go 1.18 introduced generics. You can write the deduplication function once and use it for any comparable type. The constraint is comparable. Map keys must be comparable. Types like int, string, bool, and structs with comparable fields satisfy this constraint. Slices and maps are not comparable. You can't use them as map keys.
Here's the generic version.
// RemoveDuplicates returns a slice with unique items, preserving order.
func RemoveDuplicates[T comparable](items []T) []T {
seen := make(map[T]struct{})
result := make([]T, 0, len(items))
for _, item := range items {
if _, exists := seen[item]; !exists {
seen[item] = struct{}{}
result = append(result, item)
}
}
return result
}
The [T comparable] clause tells the compiler that T must support equality checks. The function works for []int, []string, []MyStruct, and so on. You call it with type inference or explicit type parameters.
Generics reduce repetition. Comparable is the gate.
Pitfalls and compiler errors
Map keys have strict rules. If you try to use a slice as a map key, the compiler rejects the program. The error is invalid map key type []string. Slices are not comparable. Two slices with the same content are not equal in Go. The == operator doesn't work on slices. Maps require keys that support ==.
Use strings or arrays instead. Arrays are comparable. If the length is fixed, an array works. If the length varies, convert to a string or use a hash.
Memory is another concern. The map stores every unique key. If the input slice is huge and has few duplicates, the map might use more memory than the slice. Map entries have overhead. Each entry stores the key, the hash, and pointers. For small types like int, the overhead is significant. If memory is tight, consider alternatives.
Order is a choice. If you don't care about order, you can skip the result slice. Just return the map keys. Ranging over the map gives you the unique keys. The order is random. This saves the allocation of the result slice. It also saves the append logic. It's faster and uses less memory. But the output order changes every run.
If you need order, the slice append is mandatory.
Maps are fast. Order is a choice. Pick the tool that matches your constraint.
Alternatives: sort and scan
When memory is scarce, the map approach might be too heavy. You can sort the slice and scan for adjacent duplicates. Sorting takes O(N log N) time. The scan is O(N). Total time is O(N log N). This is slower than the map approach for large N. But it uses O(1) extra space if you sort in place. No map allocation.
This works well for sorted data. If the input is already sorted, you skip the sort step. The scan is O(N). It's very efficient.
Here's the sort-and-scan pattern for strings.
import "sort"
func removeDuplicatesSort(items []string) []string {
// Sort brings duplicates together.
sort.Strings(items)
if len(items) == 0 {
return items
}
// Write index tracks the position of the last unique item.
writeIdx := 1
for i := 1; i < len(items); i++ {
// Skip if equal to the previous unique item.
if items[i] != items[writeIdx-1] {
items[writeIdx] = items[i]
writeIdx++
}
}
// Slice to the new length.
return items[:writeIdx]
}
The function modifies the input slice in place. It returns a subslice. The caller gets a shorter slice with unique items. The order is sorted, not original. This is a trade-off. You save memory and avoid map overhead. You lose original order. You pay O(N log N) time.
Use this when memory is constrained or the data is already sorted.
Sort when memory is scarce. Map when time is scarce.
Decision matrix
Use a map-based loop when you need to preserve the order of first appearance. Use a map range when order doesn't matter and you want the shortest code. Use a generic function when you need to deduplicate multiple types across your codebase. Use a sort-and-scan approach when memory is tight or the data is already sorted. Use a bitset when the values are small integers and you need maximum speed and minimal memory. Use a database DISTINCT query when the duplicates live in the database and you can filter before fetching.