I think the breaks_width() and label_time() function from {scales} are already flexible enough to handle such cases, or am I missing something here?

library(hms)
library(ggplot2
library(scales)

df <- data.frame(diff = as_hms(c(1850000, 800)), time = rep(Sys.time(), 2))
p <- ggplot(df) +
  geom_point(aes(time, diff))
p + scale_y_time(
    name   = "Week",
    breaks = breaks_width("1 week"),
    labels = label_time("%W")
  )

p + scale_y_time(
  name = "Days",
  breaks = breaks_width("4 days"),
  labels = label_time("%j")
)

^{Created on 2022-12-03 by the reprex package (v2.0.1)}

I was not aware of this functions when I opened the issue, so this might solve the inmediat problem. However, it wouldn't be consistent with other time related scales_*, which simply use date_breaks = "1 week" and the rest is done by ggplot2 itself.