GeoLite2 db access, with license, fails at "error retrieving updates: error creating database writer: could not acquire lock " #70

pgnd · 2020-01-09T19:44:39Z

after reading notice

https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/

i'm updating my local installs.

building clean, as usual, from git sources, with no errors,

	-rw-r--r-- 1 root root  507 Jan  9 10:28 /usr/local/include/maxminddb_config.h
	-rw-r--r-- 1 root root 8.4K Jan  9 10:28 /usr/local/include/maxminddb.h
	-rwxr-xr-x 1 root root  943 Jan  9 10:28 /usr/local/lib64/libmaxminddb.la*
	lrwxrwxrwx 1 root root   21 Jan  9 10:28 /usr/local/lib64/libmaxminddb.so -> libmaxminddb.so.0.0.7*
	lrwxrwxrwx 1 root root   21 Jan  9 10:28 /usr/local/lib64/libmaxminddb.so.0 -> libmaxminddb.so.0.0.7*
	-rwxr-xr-x 1 root root 121K Jan  9 10:28 /usr/local/lib64/libmaxminddb.so.0.0.7*
	-rw-r--r-- 1 root root  281 Jan  9 10:28 /usr/local/lib64/pkgconfig/libmaxminddb.pc

and

GOPATH=/usr/local/ go get -u github.com/maxmind/geoipupdate/cmd/geoipupdate
ls -al `which geoipupdate`
	-rwxr-xr-x 1 root root 7.4M Jan  9 10:31 /usr/local/bin/geoipupdate*

I've created an account, and gotten my acctID + license key; config file is

	cat /usr/local/etc/geoip/GeoIP.conf
		AccountID ######
		LicenseKey ################
		EditionIDs GeoLite2-City GeoLite2-Country GeoLite2-ASN

on exec of goeipupdate

	geoipupdate -v \
	 -f /usr/local/etc/geoip/GeoIP.conf \
	 -d /usr/local/share/GeoIP2

I get an error,

	Using config file /usr/local/etc/geoip/GeoIP.conf
	Using database directory /usr/local/share/GeoIP2
	Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-City
	Acquired lock file lock (/usr/local/share/GeoIP2/.geoipupdate.lock)
	Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-City/update?db_md5=00000000000000000000000000000000
	error closing temporary file: close /usr/local/share/GeoIP2/GeoLite2-City.mmdb.temporary: file already closed
	Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-Country
	error retrieving updates: error creating database writer: could not acquire lock on /usr/local/share/GeoIP2/.geoipupdate.lock

Note, the first lock is acquired

here's with --stack-trace

Using config file /usr/local/etc/geoip/GeoIP.conf
Using database directory /usr/local/share/GeoIP2
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-ASN
Acquired lock file lock (/usr/local/share/GeoIP2/.geoipupdate.lock)
Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=00000000000000000000000000000000
error closing temporary file: close /usr/local/share/GeoIP2/GeoLite2-ASN.mmdb.temporary: file already closed
Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-City
error retrieving updates: could not acquire lock on /usr/local/share/GeoIP2/.geoipupdate.lock
github.com/maxmind/geoipupdate/pkg/geoipupdate/database.CreateLockFile
        /usr/local/src/github.com/maxmind/geoipupdate/pkg/geoipupdate/database/writer.go:41
github.com/maxmind/geoipupdate/pkg/geoipupdate/database.NewLocalFileDatabaseWriter
        /usr/local/src/github.com/maxmind/geoipupdate/pkg/geoipupdate/database/local_file_writer.go:41
main.run
        /usr/local/src/github.com/maxmind/geoipupdate/cmd/geoipupdate/main.go:65
main.main
        /usr/local/src/github.com/maxmind/geoipupdate/cmd/geoipupdate/main.go:48
runtime.main
        /usr/lib64/go/1.13/src/runtime/proc.go:203
runtime.goexit
        /usr/lib64/go/1.13/src/runtime/asm_amd64.s:1357
error creating database writer
main.run
        /usr/local/src/github.com/maxmind/geoipupdate/cmd/geoipupdate/main.go:67
main.main
        /usr/local/src/github.com/maxmind/geoipupdate/cmd/geoipupdate/main.go:48
runtime.main
        /usr/lib64/go/1.13/src/runtime/proc.go:203
runtime.goexit
        /usr/lib64/go/1.13/src/runtime/asm_amd64.s:1357

The text was updated successfully, but these errors were encountered:

horgh · 2020-01-09T20:09:56Z

Thanks for the report! What OS are you using? I think this is caused by what #69 is fixing - the error check when closing is not quite right, causing us to not unlock the file. It continues despite the problematic check in at least some environments it seems.

pgnd · 2020-01-09T20:27:24Z

@horgh

What OS are you using

I'm seeing this on a variety of my linux boxen

e.g., `this one',

lsb_release -rd
	Description:    openSUSE Leap 15.1
	Release:        15.1

uname -rm
	5.4.6-24.ge5f8301-default x86_64

perl -v
	This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux-thread-multi

go version
	go version go1.13.5 linux/amd64

horgh · 2020-01-09T20:39:32Z

Interesting! I wonder what causes the behaviour difference since it works on other Linux hosts I try.

I don't have a good workaround - we'll need a fix like #69 I think. Essentially I believe Unlock() here is not getting called because of the first error in your output causes us to return before doing so - error closing temporary file: close /usr/local/share/GeoIP2/GeoLite2-ASN.mmdb.temporary: file already closed.

pgnd · 2020-01-09T20:55:52Z

@horgh

fwiw, here's my usual procedure, unchanged except for the acct/license, that's (now) consistently reproducing these^^ errors


git clone https://github.com/maxmind/libmaxminddb
git clone https://github.com/maxmind/GeoIP2-python
git clone https://github.com/maxmind/GeoIP2-perl
git clone https://github.com/maxmind/GeoIP2-php

cd libmaxminddb
git checkout master
git submodule update --init --recursive

git log | head -n 10
	commit fc183662e85cb1b252c2f0272dc41f79e52e50fa
	Author: Gregory Oschwald <[email protected]>
	Date:   Sat Nov 2 09:00:08 2019 -0700

	    Bumped version to 1.4.2

	commit 05b7083b85c86f94893be319b7f13eaeb6198313
	Author: Gregory Oschwald <[email protected]>
	Date:   Sat Nov 2 08:59:56 2019 -0700


./bootstrap
./configure \
 --prefix=/usr/local \
 --enable-shared --disable-static \
 --libdir=/usr/local/lib64 \
 --with-pic \
 --with-gnu-ld
make V=1 -j
make install

ls -al $( find /usr/local/include /usr/local/lib* | egrep -i "geoip|maxmind" )
	-rw-r--r-- 1 root root  507 Jan  9 10:28 /usr/local/include/maxminddb_config.h
	-rw-r--r-- 1 root root 8.4K Jan  9 10:28 /usr/local/include/maxminddb.h
	-rwxr-xr-x 1 root root  943 Jan  9 10:28 /usr/local/lib64/libmaxminddb.la*
	lrwxrwxrwx 1 root root   21 Jan  9 10:28 /usr/local/lib64/libmaxminddb.so -> libmaxminddb.so.0.0.7*
	lrwxrwxrwx 1 root root   21 Jan  9 10:28 /usr/local/lib64/libmaxminddb.so.0 -> libmaxminddb.so.0.0.7*
	-rwxr-xr-x 1 root root 121K Jan  9 10:28 /usr/local/lib64/libmaxminddb.so.0.0.7*
	-rw-r--r-- 1 root root  281 Jan  9 10:28 /usr/local/lib64/pkgconfig/libmaxminddb.pc

ldconfig

cd /usr/local/src/GeoIP2-perl
make clean
perl Makefile.PL LIBS='-L/usr/local/lib64' INC='-I/usr/local/include'
make
make install

ls -al /usr/lib/perl5/site_perl/5.26.1/GeoIP2
module_info GeoIP2
	Name:        GeoIP2
	Version:     undef
	Directory:   /usr/lib/perl5/site_perl/5.26.1
	File:        /usr/lib/perl5/site_perl/5.26.1/GeoIP2.pm
	Core module: no

cpanp -i Math::Int128 Net::Works::Network
cpanp -z MaxMind::DB::Reader::XS

perl Build.PL
./Build
./Build test
./Build install
exit

module_info MaxMind::DB::Reader::XS
	Name:        MaxMind::DB::Reader::XS
	Version:     undef
	Directory:   /usr/lib/perl5/site_perl/5.26.1/x86_64-linux-thread-multi
	File:        /usr/lib/perl5/site_perl/5.26.1/x86_64-linux-thread-multi/MaxMind/DB/Reader/XS.pm
	Core module: no

cd /usr/local/src/GeoIP2-php
git checkout master
composer global require geoip2/geoip2
composer global show | egrep -i "geoip|maxmind"
	geoip2/geoip2              v2.10.0 MaxMind GeoIP2 PHP API
	maxmind-db/reader          v1.6.0  MaxMind DB Reader API
	maxmind/web-service-common v0.6.0  Internal MaxMind Web Service API

GOPATH=/usr/local/ go get -u github.com/maxmind/geoipupdate/cmd/geoipupdate
ls -al `which geoipupdate`
	-rwxr-xr-x 1 root root 7.4M Jan  9 11:25 /usr/local/bin/geoipupdate*

cat /usr/local/etc/geoip/GeoIP.conf
	AccountID ######
	LicenseKey ###...
	EditionIDs GeoLite2-ASN GeoLite2-City GeoLite2-Country

geoipupdate -v --stack-trace \
 -f /usr/local/etc/geoip/GeoIP.conf \
 -d /usr/local/share/GeoIP2

pgnd · 2020-01-10T18:42:14Z

atm, this^ issue is 100% reproducible on the 20-or-so boxes I've tested.

of course, COULD be something common across my env, &/or #69 et al.

in the interim, any clever workarounds come to mind?

oschwald · 2020-01-13T23:30:08Z

@pgnd, are you compiling geoipupdate from the source? Does it still happen if you use one of the binaries we provide?

I believe this is caused by pkg/errors#219. We currently are pinned against 0.8.1, which should not have this issue.

pgnd · 2020-01-14T00:05:18Z

@oschwald

are you compiling geoipupdate from the source?

yup, tho with go, rather than a direct pull/build from git. namely,

GOPATH=/usr/local/ go get -u github.com/maxmind/geoipupdate/cmd/geoipupdate

which I assume grabs latest head.

Does it still happen if you use one of the binaries we provide?

dunno, haven't tried -- always used the above method -- but can explore.

AaronReboot · 2020-01-14T02:49:54Z

Confirming we too see the same bugs will latest code from master. Running on mac OS or fresh debian install

pgnd · 2020-01-14T02:57:47Z

doing an explicit clean+pull (NOT something I've typically bothered with b4 ...) seems to do the trick, even with master

cd ~
rm -rf /usr/local/src/github.com/maxmind/geoipupdate
GOPATH=/usr/local/ go get -u github.com/maxmind/geoipupdate/cmd/geoipupdate

cd /usr/local/src/github.com/maxmind/geoipupdate

git tag -l | /usr/bin/tail -n 5
	v4.0.6
	v4.1.0
	v4.1.3
	v4.1.4
	v4.1.5

git checkout master

rm -f /usr/local/bin/geoipupdate
GOPATH=/usr/local/ go build -o /usr/local/bin/geoipupdate github.com/maxmind/geoipupdate/cmd/geoipupdate

ls -al `which geoipupdate`
	-rwxr-xr-x 1 root root 7.4M Jan 13 17:21 /usr/local/bin/geoipupdate*

geoipupdate -V
	geoipupdate unknown

geoipupdate -v \
 -f /usr/local/etc/geoip/GeoIP.conf \
 -d /usr/local/share/GeoIP2

	Using config file /usr/local/etc/geoip/GeoIP.conf
	Using database directory /usr/local/share/GeoIP2
	Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-ASN
	Acquired lock file lock (/usr/local/share/GeoIP2/.geoipupdate.lock)
	Calculated MD5 sum for /usr/local/share/GeoIP2/GeoLite2-ASN.mmdb: 255c01c9399729ab1acd5f4a6db9ae44
	Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-ASN/update?db_md5=255c01c9399729ab1acd5f4a6db9ae44
	No new updates available for GeoLite2-ASN
	Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-City
	Acquired lock file lock (/usr/local/share/GeoIP2/.geoipupdate.lock)
	Calculated MD5 sum for /usr/local/share/GeoIP2/GeoLite2-City.mmdb: ef353d80737938317ed4f9a1b4369518
	Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-City/update?db_md5=ef353d80737938317ed4f9a1b4369518
	No new updates available for GeoLite2-City
	Performing get filename request to https://updates.maxmind.com/app/update_getfilename?product_id=GeoLite2-Country
	Acquired lock file lock (/usr/local/share/GeoIP2/.geoipupdate.lock)
	Calculated MD5 sum for /usr/local/share/GeoIP2/GeoLite2-Country.mmdb: c1455ec3f83bfba8b86b8db68c2fea3d
	Performing update request to https://updates.maxmind.com/geoip/databases/GeoLite2-Country/update?db_md5=c1455ec3f83bfba8b86b8db68c2fea3d
	No new updates available for GeoLite2-Country

an errant remnant from prior pull/build causing a problem? I'll have to try on a couple of machines that I've not yet cleaned, and pay closer attention.

This^^ work for anyone else here?

Closes #70 and #69.

4.2.0 * The major version of the module is now included at the end of the module path. Previously, it was not possible to import the module in projects that were using Go modules. Reported by Roman Glushko. GitHub #81. * The minimum Go version is now 1.13. * A valid account ID and license key combination is now required for database downloads, so those configuration options are now required. * The error handling when closing a local database file would previously ignore errors and, upon upgrading to `github.com/pkg/errors` 0.9.0, would fail to ignore expected errors. Reported by Ilya Skrypitsa and pgnd. GitHub #69 and #70. * The RPM release was previously lacking the correct owner and group on files and directories. Among other things, this caused the package to conflict with the `GeoIP` package in CentOS 7 and `GeoIP-GeoLite-data` in CentOS 8. The files are now owned by `root`. Reported by neonknight. GitHub #76.

horgh added the bug Something isn't working label Jan 9, 2020

oschwald added a commit that referenced this issue Jan 14, 2020

Update pkg/errors and fix Close error handling

911ee6b

Closes #70 and #69.

oschwald mentioned this issue Jan 14, 2020

Update pkg/errors and fix Close error handling #73

Merged

oschwald added a commit that referenced this issue Jan 14, 2020

Update pkg/errors and fix Close error handling

e5a70ba

Closes #70 and #69.

oschwald added a commit that referenced this issue Jan 14, 2020

Update pkg/errors and fix Close error handling

20dd948

Closes #70 and #69.

oschwald added a commit that referenced this issue Jan 14, 2020

Update pkg/errors and fix Close error handling

b1d1c1b

Closes #70 and #69.

horgh closed this as completed in #73 Jan 15, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GeoLite2 db access, with license, fails at "error retrieving updates: error creating database writer: could not acquire lock " #70

GeoLite2 db access, with license, fails at "error retrieving updates: error creating database writer: could not acquire lock " #70

pgnd commented Jan 9, 2020 •

edited

Loading

horgh commented Jan 9, 2020

pgnd commented Jan 9, 2020

horgh commented Jan 9, 2020

pgnd commented Jan 9, 2020

pgnd commented Jan 10, 2020

oschwald commented Jan 13, 2020

pgnd commented Jan 14, 2020

AaronReboot commented Jan 14, 2020 •

edited

Loading

pgnd commented Jan 14, 2020

GeoLite2 db access, with license, fails at "error retrieving updates: error creating database writer: could not acquire lock " #70

GeoLite2 db access, with license, fails at "error retrieving updates: error creating database writer: could not acquire lock " #70

Comments

pgnd commented Jan 9, 2020 • edited Loading

horgh commented Jan 9, 2020

pgnd commented Jan 9, 2020

horgh commented Jan 9, 2020

pgnd commented Jan 9, 2020

pgnd commented Jan 10, 2020

oschwald commented Jan 13, 2020

pgnd commented Jan 14, 2020

AaronReboot commented Jan 14, 2020 • edited Loading

pgnd commented Jan 14, 2020

pgnd commented Jan 9, 2020 •

edited

Loading

AaronReboot commented Jan 14, 2020 •

edited

Loading