Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for storing numpy datetime64 type #1022

Closed
jverswijver opened this issue May 12, 2022 · 1 comment · Fixed by #1036
Closed

Support for storing numpy datetime64 type #1022

jverswijver opened this issue May 12, 2022 · 1 comment · Fixed by #1036

Comments

@jverswijver
Copy link
Contributor

Feature Request

Problem

Unpacking Python native type Datetime is slow, I currently am working on a PR to hopefully improve the performance of unpacking regular Datetime objects. In the process of reviewing this PR @dimitri-yatsenko has shown that the unpacking of np.datetime64 is much faster than native Datetime. We should look into adding support for np.datetime objects.

Requirements

Update Datajoint.Blob Unpack/Pack methods to support np.datetime64

Justification

Adding support for this type would greatly improve performance of retrieving datetime objects

Alternative Considerations

Current workaround is found here

@dimitri-yatsenko
Copy link
Member

The problem is that datatime.datetime objects are stored inside numpy.array as type object and must be serialized by explicit python code. The type numpy.datetime64 can be serialized by numpy's compiled code, a x500 speedup.

If we add support for numpy.datetime64, the change will not be backward compatible. We will continue to support datetime.datetime and numpy.datetime64 with very different deserialization speeds.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants