Data Compression Software allows setting of compression levels.September 3, 2010 -
Providing intelligent file compression, kdb+ v2.7 enables user to set desired compression levels by specifying which data to compress and how heavily to compress it. Clients can use 3rd-party algorithms, or use Kx's built-in compression algorithms. Software also includes memory management as well as protection against invalid incoming data, with data structure verification mechanism that detects invalid user data structures.
(Archive News Story - Products mentioned in this Archive News Story may or may not be available from the manufacturer.)
Original Press release
Kx Systems, Inc.
555 Bryant St. No.75
Palo Alto, CA, 94301-1704
USA
Kx's New Release Allows Intelligent Compression, Reduces Latency, and Is Faster Than Ever
PALO ALTO, California - Kx Systems, the leader in high-performance database and timeseries analysis, has announced today the release of kdb+ v2.7. In response to growing market demand, this version comes with several enhancements, of which intelligent file compression, enhanced memory management and a performance increase of up to 20% are the most significant.
With constantly growing data volumes, the costs of storage are an increasingly important consideration for global financial institutions.
Regulatory requirements are an additional driver for storing greater quantities of data.
Kx chief strategist, Simon Garland, comments: "As far as our clients are concerned this is a straight latency to dollars equation. Compression only makes sense on fast, multi-core hardware, for example Intel's latest processors, because using a few cores for compression - out of a total of say 32 cores - gives very efficient compression and is a very good use of multi-core machines."
While most vendors only give their clients an 'on/off' option for compression, Kx allows intelligent compression through a choice of algorithms and a broad range of settings, driving down storage requirements and latency, and optimising CPU usage. The user is able to set the desired compression levels, by specifying which data to compress and how heavily to compress it. For example, data that's a day old might not be compressed at all, data that is up to a week old could have only selected, customer-specified fields compressed, while data that is older than three months might be fully compressed. kdb+ allows clients to use third-party algorithms, or use Kx's proprietary, built-in compression algorithms. Kx's default, fast, proprietary algorithm compresses data very efficiently, while the storage requirements for TAQ-like (trade & quote) data, for example, are significantly reduced.
Garland continued: "We have been considering whether to add file compression to kdb+ for a while, but have decided to do it now because of customer demand. Our customers wanted an efficient way of storing their data - large institutions have to store hundreds of terabytes of data, so how it's stored, and whether it's easily and quickly accessible - with minimal latency - is hugely important to them. Because there is usually a trade-off between the amount of compression vs. speed of access, we have given our clients a choice of which data to compress and how aggressively to compress it."
"Intel has a long track record working closely with leading software firms like Kx systems to take advantage of our technology. The new release of kdb+ represents an innovative use of the capabilities of the newest Intel® Xeon® processors, taking advantage of cores to enable compression," said Boyd Davis, vice president and general manager of Intel's Data Center Group Marketing. "The solution is a great example of how leading edge software running on advanced Xeon processors can deliver significant customer value."
Other noteworthy benefits of v2.7 include further enhanced memory management. This translates directly into improved speed: each new version of kdb+ is significantly faster than the previous, and this is no exception, being up to 20% faster than v2.6. The new version also introduces protection against invalid incoming data, with a data structure verification mechanism which detects invalid user data structures and issues an alert, allowing appropriate action to be taken. This provides more robust protection, something which is especially important in an environment of increasing data volumes.
Kx's compression algorithms are platform-independent and available on all operating systems. Third party compression hardware can also be used, if available. Clients can use operating system specific or third party algorithms.
About Kx
Kx offers a unified approach to real-time and historical data analysis with its high-performance kdb+ database platform. kdb+ provides a portable 64-bit implementation of our expressive query and programming language to analyze and store streaming events, manipulate in-memory and on-disk databases, and build distributed low-latency applications.
Our focus is on delivering the best performance and flexibility for high-volume, data-intensive analytics and applications. We built the technology. You build your business. For more information visit http://kx.com.
Source: Kx Systems Inc
|
|
|
|
|
|
|
| 
