cswork / libraryOpen practice workspace

Coding questions · Applied Intuition

Group duplicate files by content

This question evaluates understanding of file deduplication, efficient I/O and hashing strategies, streaming large-file processing, and scalable algorithm design for identifying files with identical byte-for-byte contents.

Problem

You are given a list of files in a filesystem. Each file has:

  • a full path (string)
  • a file size in bytes (integer)
  • file contents (conceptually; assume you can stream/read the file when needed)

Two files are duplicates if their contents are identical (byte-for-byte), regardless of file name or directory.

Task: Return groups of duplicate files. Each group should contain the full paths of files that have identical content, and groups of size 1 should be omitted.

Constraints / Expectations

  • There can be millions of files.
  • Reading file contents is expensive.
  • Aim to minimize unnecessary content reads / hashing.

Follow-ups

  1. Describe an approach that avoids hashing every file if most files are unique.
  2. How would you handle extremely large files (e.g., >GB) without loading them fully into memory?
  3. If hash collisions are a concern, how do you confirm duplicates?

Example

Input files (path, size, content):

  • /a/x.txt, 4, "ABCD"
  • /b/y.txt, 4, "ABCD"
  • /c/z.txt, 4, "WXYZ"
  • /d/w.txt, 7, "1234567"

Output:

  • ["/a/x.txt", "/b/y.txt"]

Group Duplicate Files By Content

Given file records [path, size, content], return duplicate path groups for identical contents, omitting singleton groups.

Hints

  1. Group by size before checking contents.
  2. Return groups in first-seen order for deterministic grading.