The os should do file deduplication and compression and decompression faster than npm. But I guess the issue is npm cannot give the os hints to compress a folder or not?

Even if the file system does it that’s still per-file. You still need all the bookkeeping. And you’re not gonna get great compression if a lot of the files are tiny.

The tar contains a whole bunch of related files that probably compress a lot better together, and it’s only one file on disk for the file system to keep track of. It’s gonna be a lot more efficient.

When compiling you’re probably gonna touch all the source files anyway right? So if you load the zip into memory once and decompress it there it’s probably faster than loading each individual file and decompressing each of them, or just loading each individual file raw.

> And you’re not gonna get great compression if a lot of the files are tiny.

The future of compression is likely shared dictionaries, basically recognize the file type and then use a dictionary that is optimized for that file type. E.g. JavaScript/TypeScript or HTML or Rust or C++, etc. That can offset the problems with small files to a large degree.

But again, this is a complex problem and it should be dealt with orthogonally to npm in my opinion.

I also checked and it should be possible for npm to enable file system compression at the OS level on both Windows and MacOS.

Also if it was dealt with orthogonally to npm, then it could be used by pip, and other package systems.