Cannot Import from https://wakapi.dev/

[solonovamax]

Describe the bug

I'm attempting to import the data from https://wakapi.dev/ to my own self-hosted instance, and for some reason it keeps failing.
I configured wakapi to import from https://wakapi.dev/api/compat/wakatime/v1, with my wakapi token, (tested both the normal importer & legacy importer) and

In the logs, it says the following:

wakapi-1  | 2024-02-18T04:22:20.072738807Z [INFO ] [request] status=200, method=GET, uri=/summary?interval=today, duration=4.198602ms, bytes=17266, addr=174.91.204.250, user=solonovamax
wakapi-1  | 2024-02-18T04:22:37.351212285Z [INFO ] scheduling wakatime import for user 'solonovamax' (interval [0001-01-01 00:00:00 +0000 UTC, 2024-02-18 04:22:37.351202115 +0000 UTC m=+66.577188448])
wakapi-1  | 2024-02-18T04:22:37.351345669Z [INFO ] running wakatime import for user 'solonovamax'
wakapi-1  | 2024-02-18T04:22:37.354936161Z [INFO ] [request] status=202, method=POST, uri=/settings, duration=7.358299ms, bytes=73176, addr=174.91.204.250, user=solonovamax
wakapi-1  | 2024-02-18T04:22:38.247704114Z [ERROR] failed to fetch heartbeats for day '0001-01-01' and user 'solonovamax' - got status 500 from wakatime api

I'm not sure why it keeps failing.

I can send my config.yml if needed.

To re-attempt the imports, I'm connecting to my database and running

DELETE FROM key_string_values WHERE key LIKE 'last%import%solonovamax';

to remove the key that says when I last imported.

System information

Please provide information on:

• Wakapi version: 2.10.5
• Operating system: Debian Linux 6.1.67-1, Docker 25.0.3, docker image: latest
• Database: PostgreSQL (not running in docker, running on host machine)

[muety]

Hi @solonovamax, thanks for reporting this. I checked the server logs, here's an excerpt:

...
Feb 18 05:22:38 wakapi.dev wakapi[537064]: 2024-02-18T05:22:38.208600967+01:00 [ERROR] failed to retrieve heartbeats - year is not in the range [1, 9999]: 0
Feb 18 05:22:38 wakapi.dev wakapi[537064]: 2024-02-18T05:22:38.208834952+01:00 [INFO ] [request] status=500, method=GET, uri=/api/compat/wakatime/v1/users/current/heartbeats?date=0001-01-01, duration=1.4894ms, bytes=25, addr=86.48.27.55, user=solonovamax
Feb 18 05:22:38 wakapi.dev wakapi[537064]: 2024-02-18T05:22:38.208948648+01:00 [INFO ] [request] status=200, method=GET, uri=/api/compat/wakatime/v1/users/current/heartbeats?date=0001-01-03, duration=1.450698ms, bytes=101, addr=86.48.27.55, user=solonovamax
...

Looks like a time zone issue to me (whatelse). You're located in Toronto time zone (UTC-5) and request 0001-01-01 (without TZ information). The endpoint will try to interpret the date according to your local timezone (see

wakapi/routes/compat/wakatime/v1/heartbeat.go

Line 68 in 7dda996

rangeFrom, rangeTo := datetime.BeginOfDay(date.In(timezone)), datetime.EndOfDay(date.In(timezone))

), which would be five hours before the very "beginning" of time, which is why the request fails.

I'll fix this (hopefully still today). Actually, while investigating this, I realized the importer is super inefficient, by requesting heartbeats starting at year 1. We could save a couple ten thousand requests by starting in year 2000 or in 2013 rather (which is when the first commit to WakaTime was made).

[muety]

Also, until the bug is fixed, you can probably work around it by inserting a heartbeat to your database with a recent date and its origin field set to "wakatime", see

wakapi/routes/settings.go

Lines 536 to 545 in 7dda996

	if latest, err := h.heartbeatSrvc.GetLatestByOriginAndUser(imports.OriginWakatime, user); latest == nil || err != nil {
	stream, importError = importer.ImportAll(user)
	} else {
	// if an import has happened before, only import heartbeats newer than the latest of the last import
	stream, importError = importer.Import(user, latest.Time.T(), time.Now())
	}
	if importError != nil {
	conf.Log().Error("wakatime import for user '%s' failed - %v", user.ID, importError)
	return
	}

.

[muety]

Should work now. I updated wakapi.dev, it should return a safe import range now. Please give it a try (you won't have to recompile your Wakapi).

[solonovamax]

I'll fix this (hopefully still today). Actually, while investigating this, I realized the importer is super inefficient, by requesting heartbeats starting at year 1. We could save a couple ten thousand requests by starting in year 2000 or in 2013 rather (which is when the first commit to WakaTime was made).

rather than hardcoding to not fetch prior to a certain date, it might be a good idea to have some way to query "what is the oldest hearbeat you have" and then use that

[muety]

That's what's already being done. The hard-coded date is only used if no heartbeats exist or if something goes wrong.

[solonovamax]

That's what's already being done. The hard-coded date is only used if no heartbeats exist or if something goes wrong.

ah, I see