datetime and time equality and hashing problematic

[cjw296]

Bug report

Bug description:

I'm sure folks will pull out reasoning behind this from the inscrutable section in PEP 495 but this behaviour is honestly extremely surprising at best and just flat out wrong on first glance:

>>> from datetime import time, datetime, date
>>> time(fold=1) == time(fold=0)
True
>>> hash(time(fold=1)) == hash(time(fold=0))
True
>>> datetime(2024, 10, 27, fold=1) == datetime(2024, 10, 27, fold=0)
True
>>> hash(datetime(2024, 10, 27, fold=1)) == hash(datetime(2024, 10, 27, fold=0))
True

Even when deliberately specifying as explicitly as possible two points in time which are absolutely not the same (these are two points during the UK DST backwards transition later this year):

>>> from zoneinfo import ZoneInfo
>>> tz = ZoneInfo('Europe/London')
>>> dt1 = datetime(2024, 10, 27, 1, 30, fold=0, tzinfo=tz)
>>> dt2 = datetime(2024, 10, 27, 1, 30, fold=1, tzinfo=tz)
>>> dt1 == dt2 
True
>>> hash(dt1) == hash(dt2)
True

This has really unpleasant implications for things like storing points in time in a dictionary:

>>> d = {}
>>> d[dt1] = 1
>>> d[dt2] = 2
>>> d[dt1]
2
>>> d[dt2]
2

The inverse of this issue is reported in #115845.

Yes, I know the timestamp can be used:

>>> dt1.timestamp() == dt2.timestamp()
False

...but not on time objects, and timestamp also brings the local machine's timezone into play:

>>> def fold_equal(*args, **kw):
...   return datetime(*args, **kw, fold=0).timestamp() == datetime(*args, **kw, fold=1).timestamp()
... 
>>> fold_equal(2024, 10, 27, 0, 30)
True
>>> fold_equal(2024, 10, 27, 1, 30)
False
>>> fold_equal(2024, 10, 27, 2, 30)
True
>>> fold_equal(2024, 10, 27, 0, 30, tzinfo=ZoneInfo('America/Chicago'))
True
>>> fold_equal(2024, 10, 27, 1, 30, tzinfo=ZoneInfo('America/Chicago'))
True
>>> fold_equal(2024, 10, 27, 2, 30, tzinfo=ZoneInfo('America/Chicago'))
True

Concretely, it would be a lot less confusing if:

• time objects, and datetime objects where tzinfo is None are equal and hashed the same only if all of their attributes including fold are the same.

• datetime objects where tzinfo is not None are equal and hashed the same only if they represent the exact same point in time.

CPython versions tested on:

3.12

Operating systems tested on:

No response

Linked PRs

• gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same #116187
• [3.12] gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same (GH-116187) #116216
• [3.11] gh-116035: Document that both tzinfo and fold are ignored in comparisons if tzinfo is the same (GH-116187) #116217

[cjw296]

A quick scan of @pganssle 's https://blog.ganssle.io/articles/2018/02/a-curious-case-datetimes.html documents that this happens but I didn't spot any justification for it...

[pganssle]

I don't know if I've ever actually gotten to the bottom of why this requirement exists, but I've been meaning to figure it out for years. I don't have time to track it down at the moment, but it was the result of a discussion between @abalkin and @tim-one, with the result documented here.

This is the message where Tim explains that you can't make two dates differing only by their fold have the hash and equality guarantees required, but it's in the context of a suggestion to calculate the hash as if fold=0, which presumably they were doing to solve some other problem.

I don't really understand the concern with time, though, since frankly it's a bit weird that time has a fold attribute at all, since there's no way for time objects to have ambiguous times (I mean, you could deliberately write a time zone library that does this, but in the real world the only time zones where you can get an offset without an associated date are ones with fixed offsets).

[pganssle]

Also I don't think this should be labeled as a "bug" — I don't know what the labels are used for, but if we were to change this behavior it would be breaking a thoroughly documented feature with a decently long public discussion history behind it. This is not a situation where no one thought about this, it's more that they thought about it and picked a specific set of trade-offs. If we want to change it, it would be because we decided that we would prefer a different set of trade-offs (and prefer them so much that it is worth making breaking changes to datetime).

Not saying that we definitely can't do it or anything, just that it's not really a "bug" (and it's certainly not getting backported to feature-frozen branches).

[tim-one]

Removed the "bug" label because it's working as designed and documented. Plus the chance that it will change is approximately 0 😉.

It's not PEP 495 particularly at work here, but a basic rule throughout datetime: objects overwhelmingly work in "naïve time", the details of which are primarily spelled out across various footnotes on tables of operations. It means that timezone info is overwhelmingly ignored, unless two aware datetime objects have two different tzinfo members. Then things like equality comparison effectively convert to UTC under the covers before the comparison occurs.

But if two datetime objects have the same tzinfo member (as in your second example),, comparison treats them as naïve datetimes. Everything related to time zone is ignored then. That was intended from the start, and has worked that way from the start (long before PEP 495 came along, although that PEP had to try to preserve backward compatibility too).

The intent has always been that if you want timezone-aware arithmetic (including comparison), you should convert to UTC first (a zone where "naïve arithmetic" is always the right thing to do), do the arithmetic, and convert back when you're done.

A consequence is that, ya, you "shouldn't" be using aware datetimes as dict keys - unless naïve comparison is what you want (which, BTW, usually is what I want).

@pganssle, as I recall, time objects have a fold attribute for a marginal reason: so you can use

datetime.combine(dt.date(), dt.time())

to get back an object identical in all respects to dt. Adding a fold attribute to date objects would have been even sillier 😉.

I don't believe any other use was envisioned for giving time a fold.

[cjw296]

Good to see you're alive and well @tim-one, it's been a while!

Removed the "bug" label because it's working as designed and documented

Just because something is working as designed and documented doesn't mean there isn't a bug in either, in this case it would appear to be the design.

Plus the chance that it will change is approximately 0 😉

I don't doubt it, but that also doesn't mean it's not a bug ;-)

It means that timezone info is overwhelmingly ignored, unless two aware datetime objects have two different tzinfo members

Yeah, I mean this appears to be the crux of the behaviour problem. I think it's somewhat defensible, although I still view it as a design failure, that "folds are completely ignored when there's no tzinfo".

But I honestly can't see how this can be seen as anything but a bug:

>>> t1 = datetime(2024, 10, 27, 1, fold=0, tzinfo=ZoneInfo('Europe/London'))
>>> str(t1)
'2024-10-27 01:00:00+01:00'
>>> t2 = datetime(2024, 10, 27, 1, fold=1, tzinfo=ZoneInfo('Europe/London'))
>>> str(t2)
'2024-10-27 01:00:00+00:00'
>>> t1 == t2
True

These are fundamentally two different points in time, in what way can they be defended as being equal? :-)

The intent has always been that if you want timezone-aware arithmetic (including comparison), you should convert to UTC first (a zone where "naïve arithmetic" is always the right thing to do), do the arithmetic, and convert back when you're done.

A consequence is that, ya, you "shouldn't" be using aware datetimes as dict keys - unless naïve comparison is what you want (which, BTW, usually is what I want).

Where are these two things documented? They honestly feel like they should be in a .. warning:: at the top of https://docs.python.org/3/library/datetime.html

[pganssle]

These are fundamentally two different points in time, in what way can they be defended as being equal? :-)

It's because a lot of stuff around datetime handling is actually more context-dependent than you would think. They represent different absolute times, but the same civil time, because civil times with backwards offset changes have inherent ambiguities.

If you look at how the temporal library in Javascript has drawn their abstraction boundaries, you'll see why it is more complicated here — the datetime class collapses "exact times" and "civil times", and as Tim mentions, tends to actually be optimized for working with civil times. When you internalize this concept, everything about datetime makes a lot more sense.

If we had the option (and maybe we will one day), I'd probably build a new datetime module more along the lines of what temporal has done: with much cleaner separation between "instant in time" and "civil time".

So it's easily defensible why those things are equal: they both represent 01:00 on 2024-10-27 in London. The fact that 01:00 on 2024-10-27 happened twice and these represent the two different instances only matters if you are using a different concept of equality (specifically, the one where two datetimes are equal if they represent the same offset from epoch).

I also realized that yes, this particular fold "problem" is actually one that I understood already 😅 The one that I didn't fully grok yet is this one:

>>> from datetime import datetime, timedelta
>>> from zoneinfo import ZoneInfo
>>> dt1 = datetime(2024, 10, 27, 1, tzinfo=ZoneInfo("Europe/London"))
>>> dt1 == dt1.astimezone(ZoneInfo("America/New_York")
False
>>> dt2 = dt1 - timedelta(seconds=1)
>>> dt2 == dt2.astimezone(ZoneInfo("America/New_York"))
True

Any time changing fold would change the utcoffset() on either side of an equality sign, the datetimes are not considered equal, even if they represent the same time in UTC, despite the fact that that's normally how intra-zone comparisons work.

In that context, reading Tim's message in 2015 makes more sense, because if you are preserving hash/equality equivalence, either you need a situation where the hash depends on the fold, in which case equality must also depend on the fold for intra-zone comparisons or you need to do something like what they did in PEP 495, where the ambiguous datetimes don't compare equal to datetimes in other zones, and thus it's fine for them to have different hash values. Presumably reading through the original thread you can see why this particular set of trade-offs was made rather than another set. I would be surprised if it wasn't carefully considered, since years and years into my work with datetime I've still found stuff like this where I realize that datetime is actually very thoughtfully-designed, given the backwards compatibility constraints that exist and the original design intentions.

[tim-one]

But I honestly can't see how this can be seen as anything but a bug:

Your t1 and t2 have the same tzinfo member. Therefore every form of arithmetic (including comparisons) involving just them uses "naïve time". For example, not only do they compare equal, t1 - t2 returns timedelta(0). These things are all related.

It's what Guido intended, and is how it's always worked. You're not required to agree with the design, but fighting against such old and established decisions is almost certainly futile.

The docs could be clearer, and I've long thought they could be materially improved by adding a less formal introductory section pointing out the consequences of this design decision. They can be surprising. But then the very concept of "time zones" with changing offsets is an unprincipled political mess.

As a general thing, while the docs are pretty good about pointing out where tzinfo is ignored, those locations seemingly weren't updated to explicitly add that the later-added fold is also ignored in such cases.

For datetime __eq__, a footnote on the relevant table reads:

If both comparands are aware and have different tzinfo attributes, the comparison acts as comparands were first converted to UTC datetimes except that the implementation never overflows.

I could have sworn that at one time it also explicitly said that if both comparands are aware and have the same tzinfo attributes, comparison acts as if both comparands were naïve (as is done in the footnote for datetime.__sub__). It's weak to leave that implicit.

Paul wrote that datetime was "optimized" for "civil time", but it's more that Guido fundamentally only cared about "civil time". If you needed any form of time zone aware arithmetic, fine, convert to UTC, do naïve arithmetic there, and convert back.

Indeed, he cared so little about time zones that the original design deliberately omitted any way to disambiguate the "problem times" around DST transitions. I recall, at the time, his astonishment upon discovering that a C struct tm wasted an entire bit to store a tm_isdst flag, needed only for such disambiguation. He didn't want Python to do the same.

Which is the one original decision which eventually got reworked, to add fold.

I expect I'd agree with Paul about what would constitute a better design (stronger wall between naïve and aware). But in real life I don't have problems, because I don't fight the design: when I need aware arithmetic, I convert to UTC first.

>>> utc = ZoneInfo("utc")
>>> t2.astimezone(utc) == t1.astimezone(utc)
False
>>> print(t2.astimezone(utc) - t1.astimezone(utc))
1:00:00

[erlend-aasland]

Thanks for the explanations @tim-one and @pganssle. ISTM this should be retargeted as a docs issue.

[tim-one]

I could have sworn that at one time it also explicitly said that if both comparands are aware and have the same tzinfo attributes, comparison acts as if both comparands were naïve

Ya, it's seemingly random doc rot. I don't know when that info got lost, but at least the 3.8 docs were (to my eyes) quite clear about this:

If both comparands are aware, and have the same tzinfo attribute, the common tzinfo attribute is ignored and the base datetimes are compared. If both comparands are aware and have different tzinfo attributes, the comparands are first adjusted by subtracting their UTC offsets (obtained from self.utcoffset()).

Although everywhere the docs say (or said) tzinfo is ignored, they should add that fold is also ignored - or state that as a general rule.

EDIT: looks like the docs here dropped this important info starting with version 3.11. No idea why. Worse, they seem now to imply that equality does make time zone adjustments in all cases:

datetime objects are equal if they represent the same date and time, taking into account the time zone.

So I'll at least agree that the docs have become infested with bugs 😦.

[tim-one]

@serhiy-storchaka, "git blame" says you're the source of the "doc rot" 😉:

c12240e (Serhiy Storchaka 2024-02-02 20:53:24 +0200

Could you please skim this issue and restore former text appropriately? The major thing is that all text has vanished explicitly saying that comparison of two datetime objects with a common tzinfo attribute ignores the tzinfo attribute entirely. That's extremely important to know, but the docs no longer say so. This applies to all 6 comparison operators. UTC adjustments are done only when comparing two aware datetime objects with distinct tzinfo members.

For extra credit, it looks like the docs were never updated to add that fold attributes are also ignored in such cases.

[serhiy-storchaka]

Ignoring the tzinfo attribute if it is identical in both comparands looked like a trivial optimization to me, so I omitted it. I did not notice that it ignores also the fold attribute. Now it looks like a bug to me. See also #72787.

[tim-one]

Thanks for taking a look!

Ignoring the tzinfo attribute if it is identical in both comparands looked like a trivial optimization to me

Then please read this entire page. It's the very opposite of "trivial", and isn't an "optimization" at all: it can make profound difference to semantics. Chris gave an example in his initial complaint here:

Even when deliberately specifying as explicitly as possible two points in time which are absolutely not the same (these are two points during the UK DST backwards transition later this year):

>>> from zoneinfo import ZoneInfo
>>> tz = ZoneInfo('Europe/London')
>>> dt1 = datetime(2024, 10, 27, 1, 30, fold=0, tzinfo=tz)
>>> dt2 = datetime(2024, 10, 27, 1, 30, fold=1, tzinfo=tz)
>>> dt1 == dt2 
True
>>> hash(dt1) == hash(dt2)
True

They are different when converted to UTC, but because they have the same tzinfo attribute, the common tiznfo attribute (and fold) are ignored. The rest of this issue discusses that at length, and I'm not going to repeat it all again here.

[serhiy-storchaka]

It looked like optimization to me because I forgot about fold. It was my mistake. But with fold taken into account this of course affects semantic.

[tim-one]

Not quite. Ambiguous times are a fact of life, and time zone conversion always had to "pick one". The only difference adding fold made to this is that it's now possible to always pick the correct result.

Since Python didn't ship any concrete tzinfo classes at first, even at the first release of datetime we had to consider what user-written tzinfo classes might do. There was nothing to preclude that they might, e.g., implement their own notion of (what later came to be called in Python) fold themselves.

So, to cover all bases, it was documented from the start that comparison, like __sub__(), ignored everything about time zones when given two operands with the same tzinfo attribute - they worked in "naïve time". Not for speed, but as a consequence of the fundamental design decision that computation within a time zone would work in naïve time.

[tim-one]

[@cjw296]

Yes, I know the timestamp can be used:

I expect that most apps would use a datetime instead, but first converted to UTC. While that's roughly equivalent to a timestamp, local system time has nothing to do with it, and sticking to datetimes avoids all the annoying cases converting to a float can stumble into. Converting to UTC is precisely defined and lossless, and will remain so for thousands of years to come (eventually failing around datetime.datetime.max, in the year 9999).

[erlend-aasland]

With the docs amended, are there more actionable items left here?

[tim-one]

None left to my eyes, but Chris may yet disagree. In the meantime, I added the "docs" label.

[tim-one]

BTW, here's a specific example where the meaning of == changes despite that fold is irrelevant:

from datetime import datetime, timedelta
from zoneinfo import ZoneInfo

east = ZoneInfo("US/Eastern")
dt1 = datetime(2016, 3, 13, 2, 30, tzinfo=east)
dt2 = dt1 + timedelta(hours=1)
print(dt1) # 2016-03-13 02:30:00-05:00
print(dt2) # 2016-03-13 03:30:00-04:00
print(dt1 == dt2)  # False
utc = ZoneInfo("utc")
print(dt1.astimezone(utc) == dt2.astimezone(utc)) # True

This is shortly after the start of DST in that zone. The time "2:30" doesn't exist on a local wall clock (although it does if you forget to move the clock forward at 2am - which is how conversion to UTC acts in this case). It's not a time in "a fold", but one in "a gap".

But in "naïve time", they both exist, and are an hour apart - and both subtraction and == agree about that. Convert to UTC, and both subtraction and == agree they're the same.

EDIT - In case it wasn't apparent, I'll point out explicitly that nothing about this example changed when fold was introduced. It always worked this way. fold only gets set "by magic" in the outputs of time conversion functions. dt1 and dt2 both get the default value of fold=0. PEP 495 did, however, define what a time "in the gap" should return when converted to UTC. For the default fold=0 case, that was defined to be what pre-495 Python happened to do (which in turn was based on the observation that the natural way to get suckered into creating a missing time to begin with was to forget to move the wall clock ahead exactly at the transition-to-DST instant, so that treating it as still being in "standard time" seemed the more defensible choice).

[cjw296]

but the same civil time

@pganssle - can you point me to where your definition of "civil time" is coming from? While we'd hopefully all agree that the best response to "let's meet at 1:30am on the 27th Oct 2024" would be "what? no! I'll be asleep in bed", I can certainly think of situations where it would be "well, which 1:30am on the 27th Oct 2024?". My point being that even in "civil time", more information is used and folks don't just go "well, they're the same time, there's no difference" - if you're being payed to play a 1hr set at 1:30am on that date, I'm very sure you'll find out which one it is...

If you look at how the temporal library in Javascript has drawn their abstraction boundaries

Yeah, it looks like they've done a more thorough job than Python 😞

@tim-one - you're throwing Guido under the bus here, but sounds like he probably deserves it in this case! "fighting against such old and established decisions is almost certainly futile" - in terms of changing the implementation, I guess, which is definitely saddening, but dates, times and zones are always saddening. However, an overhaul of the docs with way bigger warnings about this "surprising" behaviour might help folks in the future.

Since Python didn't ship any concrete tzinfo classes at first

Again, with hindsight, this feels like a shame - opening an even bigger pandoras box than just getting a standardized ZoneInfo class into Python and making the API surface you highlight as problematic to be smaller from the start.

sticking to datetimes avoids all the annoying cases converting to a float can stumble into

Can you give an example for this in relation to datetimes? Is it just rounding at the sub-second level or is there more?

The time "2:30" doesn't exist on a local wall clock

Yeah, I separately moaned about this on #116111 and #116038.

[serhiy-storchaka]

Here is an opposite example:

from datetime import datetime, UTC
from zoneinfo import ZoneInfo
east = ZoneInfo("US/Eastern")
dt1 = datetime(2016, 11, 6, 1, 30, tzinfo=east, fold=0)
dt2 = datetime(2016, 11, 6, 1, 30, tzinfo=east, fold=1)
print(dt1)  # 2016-11-06 01:30:00-04:00
print(dt2)  # 2016-11-06 01:30:00-05:00
print(dt1 == dt2)  # True
print(dt2 - dt1)  # 0:00:00
print(dt2.timestamp() - dt1.timestamp())  # 3600.0
print(dt1.astimezone(UTC) == dt2.astimezone(UTC))  # False
print(dt2.astimezone(UTC) - dt1.astimezone(UTC))  # 1:00:00

There are perhaps good reasons for this, but the fact that datetime objects which refer to different points of time are considered equal, is confusing. It means that only datetime objects in UTC and raw timestamps can be used to identify the time. Other time zones can only be used to format and parse the local time.

[tim-one]

@serhiy-storchaka, yes, the introduction of fold did change how that one worked. Times "in a fold" (end of DST) were impossible to distinguish before fold was introduced. The example I gave was "in a gap" (start of DST) instead, and those were always distinguishable (the introduction of fold was irrelevant to this).

DST is confusing, period. In your example, show a person the local wall clock at dt1 and dt2. They're identical. That's "naïve time".

Or, in most of the US, just watch random Facebook feeds this Sunday to enjoy a variety of annoyed posts from people who got burned by the switch to DST 😉.

[tim-one]

However, an overhaul of the docs with way bigger warnings about this "surprising" behaviour might help folks in the future.

Docs patches can be contributed by anyone 😉. I was too close to this from the start to really grok the intensity of the complaints. Life is easy if you play along with the design: if you care about time zone effects, convert to UTC and work in that. There are no surprises in UTC (or, for that matter, in any other fixed-offset zone). That advice is crucial, but I don't see any value in laboriously documenting all the things that can go wrong if someone is determined to fight the design.

sticking to datetimes avoids all the annoying cases converting to a float can stumble into

Can you give an example for this in relation to datetimes? Is it just rounding at the sub-second level or is there more?

Not inclined to spend time on it. It was again never intended that lossless conversion to a float be possible. If you try, you're on your own. The docs are strong enough already (although, ya, don't even mention possible rounding effects):

fromtimestamp() may raise OverflowError, if the timestamp is out of the range of values supported by the platform C localtime() or gmtime() functions, and OSError on localtime() or gmtime() failure. It’s common for this to be restricted to years in 1970 through 2038. Note that on non-POSIX systems that include leap seconds in their notion of a timestamp, leap seconds are ignored by fromtimestamp(), and then it’s possible to have two timestamps differing by a second that yield identical datetime objects.

Yeah, I separately moaned about this on #116111 and #116038.

Same thing: if you want the time zone to affect intrazone computation, convert to UTC first.