You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the only supported ways to use IDs/integers in a control column are as a timestamp directly representing seconds, ms, us or ns since the UNIX epoch; or as a generic, non-timestamp ID range. #683 adds functionality to use ranges of UUID and text values via conversion functions. I want to add support for integers as well.
The primary example is "snowflake" IDs, used by X, Discord, Instagram, Mastodon, and others. I brought them up in #388 and the resulting changes let me use snowflake IDs as ordinary integers, with offsets for interval and retention. This works well enough, but leads to gaps when no data is being added for a while since premake only looks at the current maximum value.
For background, snowflake values fit in 63 bits (for signed int64) and encode a timestamp plus a combination of worker and sequence information. Since each implementation uses a different bit range and epoch for the timestamp, partman won't be able to cover all possibilities with a single setting for p_epoch like 'snowflake'.
I propose a new option for p_epoch of 'func', which indicates that partman should use the encoder/decoder functions for IDs the same way it does for UUIDs and text. This should be a straightforward change, since most of the work has already been done in #683. Also, the parameter documentation for encoder/decoder mentions snowflakes, but those are only allowed if they are stored as a string rather than a bigint.
Example Discord ID encoder and decoder:
-- Discord snowflakes use a 41 bit timestamp and 22 bit worker/process/sequence.-- The timestamp is the number of milliseconds since the first second of 2015 UTC (1420070400s since epoch)CREATEFUNCTIONdiscord_snowflake_to_timestamp(id bigint)
RETURNS TIMESTAMPTZ LANGUAGE SQL IMMUTABLE PARALLEL SAFE AS
$$SELECT TO_TIMESTAMP((id >>22)/1000.0+1420070400)$$;
CREATEFUNCTIONtimestamp_to_discord_snowflake(ts TIMESTAMPTZ)
RETURNS BIGINT LANGUAGE SQL IMMUTABLE PARALLEL SAFE AS
$$SELECT CAST((EXTRACT(epoch FROM ts) -1420070400)*1000ASBIGINT) <<22$$;
The text was updated successfully, but these errors were encountered:
@akulapid
Just tagging you here on this one in case you might want to take a pass at it. Otherwise, I'll try and take a look at this in the future when I have some time.
This PR allows users to specify the special value of 'func' for p_epoch
to use custom functions to encode/decode time-ordered integers other than
the classic seconds, ms, us or ns since the UNIX epoch.
Resolvespgpartman#729.
This PR allows users to specify the special value of 'func' for p_epoch
to use custom functions to encode/decode time-ordered integers other than
the classic seconds, ms, us or ns since the UNIX epoch.
Resolvespgpartman#729.
Creating a new issue from #528 (comment).
Currently the only supported ways to use IDs/integers in a control column are as a timestamp directly representing seconds, ms, us or ns since the UNIX epoch; or as a generic, non-timestamp ID range. #683 adds functionality to use ranges of UUID and text values via conversion functions. I want to add support for integers as well.
The primary example is "snowflake" IDs, used by X, Discord, Instagram, Mastodon, and others. I brought them up in #388 and the resulting changes let me use snowflake IDs as ordinary integers, with offsets for interval and retention. This works well enough, but leads to gaps when no data is being added for a while since premake only looks at the current maximum value.
For background, snowflake values fit in 63 bits (for signed int64) and encode a timestamp plus a combination of worker and sequence information. Since each implementation uses a different bit range and epoch for the timestamp, partman won't be able to cover all possibilities with a single setting for
p_epoch
like 'snowflake'.I propose a new option for p_epoch of 'func', which indicates that partman should use the encoder/decoder functions for IDs the same way it does for UUIDs and text. This should be a straightforward change, since most of the work has already been done in #683. Also, the parameter documentation for encoder/decoder mentions snowflakes, but those are only allowed if they are stored as a string rather than a bigint.
Example Discord ID encoder and decoder:
The text was updated successfully, but these errors were encountered: