add first round of pgtools #642

bgruening · 2017-09-28T15:19:42Z

This will use a in-user-space postgresql version and can load schemas into it and query the db. Many more tools will be added in the future. Open for comments.

hexylena

LOVE IT.

hexylena · 2017-09-28T15:22:46Z

tools/pg_tools/macros.xml

+    </token>
+
+    <token name="@ARCHIVE_DATABASE@"><![CDATA[
+        tar -cvJf postgresql_out.tar.xz ./postgresql


use the html extra files path thing to avoid the overhead of tarring / un-tarring.

Retracted: out of band discussion concluded that disk space was more important than the computational overhead.

hexylena · 2017-09-28T15:23:08Z

tools/pg_tools/macros.xml

+
+    <token name="@PG_START@"><![CDATA[
+        '$__tool_directory__/pglite' start  -d ./postgresql &&
+        sleep 10


slightly worried about this. Wonder if we could switch a while loop?

We could put the sleep in a while look and ask if the server is available? Not sure what the best way is here. Maybe we could make this start command blocking?

I'd love a sleep in a while. Maybe something like the following would be good so we'd be safe from sleeping too much or too little, and we can be sure we won't sleep forever if something goes wrong.

exec_count=0 '$__tool_directory__/pglite' status | grep -F -q 'server is running' while [ $? -ne 0 ]; do sleep 2 exec_count=$((exec_count+1)) if [[ "$exec_count" -gt 10 ]]; then exit 1; fi '$__tool_directory__/pglite' status | grep -F -q 'server is running' done

(Completely untested.)

I used unless ... status still does not work reliably, but checking for the socket file in addition does the trick.

hexylena · 2017-09-28T15:24:23Z

tools/pg_tools/pg-query.xml

@@ -0,0 +1,48 @@
+<tool id="pg_query" name="Postgresql" version="@PG_VERSION@">
+    <description>import sql dump</description>


description

? I don't see it :(

change description from "import sql dump" to something like "query postgres database"

hexylena · 2017-09-28T15:24:46Z

tools/pg_tools/pg-query.xml

+        <param format="postgresql" name="infile" type="data" label="Input database" />
+        <param name="query" type="text" area="True" size="5x50" label="The query to be issued to the database">
+            <sanitizer>
+                <valid initial="string.letters,string.digits">


this won't be enough, we'll need ' at least.

fixed, other can be at any time.

hexylena · 2017-09-28T15:27:13Z

tools/pg_tools/pg-query.xml

+            @PG_START@ &&
+            ls -l ./postgresql
+            &&
+            @PSQL@ -c '$query' -L logfile.log > '$outfile' &&


could we wrap this in a copy ($query) to stdout with csv delimiter ' '; (i.e. raw tab char)? Then you'd get automatic tabular files. (or at least make it an option and make tab output default)

bgruening · 2017-10-09T09:11:16Z

I added a new test and addressed your other concerns, besides the sleep thing.

hexylena · 2017-10-10T19:15:52Z

tools/pg_tools/pglite

+
+######################### Delegates to subcommands or runs main, as appropriate
+
+main "$@"


We need to document the license / source+license for this script somewhere, i.e. https://github.com/solidsnack/pglite/blob/master/LICENSE

I don't think we should ship this tool at all, we should rather depend on it.

works for me! (But we still need to either remove the script or add the license header)

removed this file and we now depend on a conda package

bgruening · 2017-10-15T11:25:17Z

So tests are passing. Its a start I guess ...

hexylena · 2017-10-16T10:26:15Z

tools/pg_tools/macros.xml

+    </token>
+
+    <token name="@STATUS_RUNNING@"><![CDATA[
+        timeout 30 bash -c 'until pglite status -d ./postgresql | grep -F -q "server is running"; do sleep 1; done' &&


just fyi, the reason for my horribly long while loop was that timeout is unavailable on OSX. Not sure if we care about those guys (I don't really that much.)

I don't care that much either and doubt that anyone one is running a server under OSX.
I think this is ready to go for a first version.

bgruening · 2017-11-17T12:27:33Z

Like the additional tool!

yvanlebras · 2018-01-10T22:41:57Z

Ready to start things on a postgis version with @ValentinChCloud and @Alanamosse for Galaxy-E!

bgruening · 2018-01-10T22:58:09Z

@yvanlebras awesome. Let me know if these tools work for you and what I can do to improve it.
One other requirement would be a postgresql-gis conda package :)

yvanlebras · 2018-01-11T11:43:49Z

Have you a postgresql datatype definition already ?

hexylena · 2018-01-11T23:00:30Z

@yvanlebras galaxyproject/galaxy#4730

yvanlebras · 2018-01-11T23:16:07Z

Thanks @erasche ! I missed it ! Nice end of week.

bgruening · 2019-04-24T09:52:03Z

Ok, let's get this in. I'm currently trying to ship with a GIS and rdkit enabled PG version, but this can also be an update.

Thanks everyone!

ping @abretaud

bgruening · 2020-07-19T09:28:47Z

ping @annefou, this might be interesting for you as well. It can use GIS-enabled postgresql databases in user-space. You can treat a PG database like any other Galaxy database.

yvanlebras · 2020-07-19T09:47:08Z

This is something we have to work on. Maybe it's time to go further and try to test creating a qgis 3 interactive took!

annefou · 2020-07-19T10:10:44Z

This is something we have to work on. Maybe it's time to go further and try to test creating a qgis 3 interactive took!

Yes. We can discuss it during the co-fest?!

bgruening · 2020-07-19T10:12:07Z

Sure :)

yvanlebras · 2020-07-19T12:45:29Z

Yes! Very good idea Anne. I will try to have some GIS colleagues, maybe you have this ?

annefou · 2020-07-19T13:55:25Z

Yes we use arcGIS a lot but more and more QGIS too. We have dockerized versions of QGIS desktop that could be added to Galaxy. I am not sure what is best so we can discuss it during the co-fest!

yvanlebras · 2020-07-19T16:25:56Z

I was notably thinking about something like thishttps://github.com/jancelin/geo-poppy

bgruening mentioned this pull request Sep 28, 2017

Get postgresql running in userspace #370

Open

hexylena reviewed Sep 28, 2017

View reviewed changes

hexylena reviewed Oct 10, 2017

View reviewed changes

bgruening force-pushed the pgtools branch from a13e0f9 to 0f5170c Compare October 15, 2017 01:36

bgruening and others added 7 commits October 15, 2017 12:25

add first round of pgtools

7839e8b

tabular separated query outputs

a228cc0

add new test data

fccf1e8

remove pglite

ed54204

some changes

8c11e57

update description

e2f86ae

add test file

84b316d

bgruening force-pushed the pgtools branch from f9d6430 to 84b316d Compare October 15, 2017 10:26

bgruening added 2 commits October 15, 2017 12:35

pin to the pg datatype commit

4e3581d

Update .travis.yml

d534fa7

hexylena reviewed Oct 16, 2017

View reviewed changes

hexylena added 6 commits November 17, 2017 09:18

Add command to dump db

e5cd63f

fix help text

f4af097

should be safer

c17ce7d

do not permit usage of semicolon

abf1cea

fix linting errors

88c6459

forgot line terminator

068ba52

abretaud mentioned this pull request Jun 11, 2018

Add python-chado tools galaxy-genome-annotation/galaxy-tools#17

Merged

bgruening and others added 2 commits April 22, 2019 00:44

Merge branch 'master' into pgtools

e95d9cb

update tools and sanatizer

7a20e38

bgruening merged commit ab93c4c into master Apr 24, 2019

bgruening deleted the pgtools branch April 24, 2019 10:04

abretaud mentioned this pull request Jul 6, 2019

[WIP] Add support for pgtools/pgutil for chado tools galaxy-genome-annotation/galaxy-tools#40

Merged

bgruening mentioned this pull request Sep 16, 2020

fixes for pgchem #1028

Merged

		@@ -0,0 +1,48 @@
		<tool id="pg_query" name="Postgresql" version="@PG_VERSION@">
		<description>import sql dump</description>


		######################### Delegates to subcommands or runs main, as appropriate

		main "$@"

add first round of pgtools #642

add first round of pgtools #642

Conversation

bgruening commented Sep 28, 2017

hexylena left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexylena Sep 28, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening Oct 15, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening Oct 9, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening commented Oct 9, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hexylena Oct 11, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening commented Oct 15, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bgruening commented Nov 17, 2017

yvanlebras commented Jan 10, 2018

bgruening commented Jan 10, 2018

yvanlebras commented Jan 11, 2018

hexylena commented Jan 11, 2018

yvanlebras commented Jan 11, 2018

bgruening commented Apr 24, 2019

bgruening commented Jul 19, 2020

yvanlebras commented Jul 19, 2020

annefou commented Jul 19, 2020

bgruening commented Jul 19, 2020

yvanlebras commented Jul 19, 2020

annefou commented Jul 19, 2020

yvanlebras commented Jul 19, 2020

hexylena Sep 28, 2017 •

edited

Loading

bgruening Oct 15, 2017 •

edited

Loading

bgruening Oct 9, 2017 •

edited

Loading

hexylena Oct 11, 2017 •

edited

Loading