AWS compression algorithm switched from gzip to zstd: Saving 30% storage space
- An American company made 0.7nm chips: EUV lithography machines can’t do it
- CVE-2007-4559 Python vulnerability ignored for 15 years puts 350,000 projects at risk of code execution
- RISC-V only takes 12 years to achieve the milestone of 10 billion cores
- 14000 cores + 450W: RTX 4080 graphics card perfectly replaces the RTX 3080
- Big upgrade: The difference between Bluetooth 5.0 and 5.2
- Geeks Disappointed that RTX 4080/4090 doesn’t come with PCIe 5.0
- What are advantages and disadvantages of different load balancing?
AWS compression algorithm switched from gzip to zstd: saving 30% storage space.
Adrian cockcroft, the former vice president of Amazon, broke the news on Twitter , saying that after the AWS compression algorithm was switched from gzip to zstd, a huge amount of memory was saved, the compressed S3 storage was reduced by about 30%, and the space saved can reach the scale of EB (1 EB). = 1024 PB = 1024 * 1024 TB).
A lot was saved at AWS switching from gzip to zstd – about 30% reduction in compressed S3 storage, exabyte scale.— adrian cockcroft (@adrianco) August 20, 2022
An Amazon employee added to the statement, saying that what Amazon changed was not the way of compressing data stored by customers, but the way S3 itself stores service data (mainly logs) – S3 itself switched from gzip logs to ztsd logs, making Storage costs are reduced by 30%.
However, Amazon did not issue an announcement related to changing the compression technology. Careful Holmes netizens found that Amazon S3 had a price reduction of 31% at the end of November 2021, which just corresponds to the 30% reduction in storage costs.
Zstd (Zstandard) is a fast lossless compression algorithm open sourced by Facebook. It is mainly used in real-time compression scenarios at the zlib level and has a better compression ratio.
Zstd can also provide stronger compression ratios at the expense of compression speed, with the ratio of speed to compression ratio configurable incrementally.
Zstd is a compression technology with excellent performance. Different from compression algorithms such as zlib , lz4, and xz, Zstd seeks a solution that combines compression performance and compression rate.
Zstd also provides a special compression mode “dictionary compression” for small data, which supports generating dictionary files in training mode to improve the compression ratio for small data packets.
The following is a performance comparison of common compression algorithms:
- DIY a PBX (Phone System) on Raspberry Pi
- How to host multiple websites on Raspberry Pi 3/4?
- A Free Intercom/Paging system with Raspberry pi and old Android phones
- DIY project: How to use Raspberry Pi to build DNS server?
- Raspberry Pi project : How to use Raspberry Pi to build git server?