Deephaven Community Core Release 0.27 brings exciting new features, quality-of-life improvements, and bug fixes. Of note are our new real-time data ingestion tool, improved Parquet support, and multi-joins.
Full release notes for 0.27.0 and 0.27.1 can be found on GitHub.
Deephaven’s Table Publisher brings a new way to write real-time data to Deephaven tables. Data is programmatically added to a blink table, which only retains rows from only the current update cycle. Blink tables can easily be converted to other table types to suit any use case. Anyone who uses DynamicTableWriter should get excited, since this is more efficient, scalable, and compatible with asyncio workflows.
Watch our developer demo to see this in action.
A multi-join allows users to join 3 or more tables together in an efficient and simple manner. The join happens using a single hash table regardless of the number of tables given. It’s currently available in the Groovy server-side API, and will be available in the Python server-side API in the next release.
See a demonstration of multi-join on our YouTube channel.
Changes to Deephaven’s Parquet library include:
- Users can specify a schema when reading hierarchical Parquet tables, rather than having to let the engine infer it from directory structure.
- Better support for read-modify-write workflows.
- Better support for reading and writing columns of arrays of data.
- Bug fixes for reading and writing nulls.
This release improves existing plugin support for custom object types. In addition to providing a snapshot of an object’s contents, an ObjectType plugin can continue sending updates and receiving requests from clients without re-fetching the object.
The first demo in the video below shows an example of using bidirectional plugins to create a real-time chat application with a JavaScript front end:
Popular Deephaven plugins for pandas, matplotlib, plotly, and Plotly Express have been updated: simply run pip to get the latest version. User documentation on how to create streaming plugins is coming soon.
Support for streaming messages required small breaking changes in existing Deephaven plugins, summarized below.
Updating existing Java `ObjectType“ plugins
Existing Java plugins not being rewritten to support bidirectional streaming should make the following changes:
Updating existing Python ObjectType plugins
Python time library
lower_bin and upper_bin have been removed from the Python time library. The methods were occasionally used instead of the preferred built-in query language methods. Users who want to bin times in tables should always use the built-in query language methods lowerBin and upperBin, as they’re much faster.
We look forward to hearing from you in Slack or GitHub discussions!
Source link
lol
