Skip to content
This repository has been archived by the owner on Dec 16, 2019. It is now read-only.

Ditching GnuPG #261

Merged
merged 99 commits into from
Mar 9, 2018
Merged
Show file tree
Hide file tree
Changes from 89 commits
Commits
Show all changes
99 commits
Select commit Hold shift + click to select a range
cdcdf3c
Parsing a few packets
Feb 11, 2018
0dc32f4
This can now decrypt the test file, given a test key and a passphrase.
Feb 17, 2018
af2ce2a
Progress on recursion and streaming
Feb 21, 2018
b241324
Adding a script entrypoint and moving code around
Feb 22, 2018
e6d1b49
Adding cryptography in requirement
Feb 22, 2018
ad71776
Fixing the issues with the cleardata buffer
Feb 22, 2018
5a39a3c
Using generators. First step towards streaming
Feb 23, 2018
625368f
Using generators to process data stream
Feb 26, 2018
d96a7b6
Streaming solution for PGP
Feb 28, 2018
cf3d017
Updating unlock for private key to only return the material
Feb 28, 2018
2349fe1
Updating the consumer generator and changing the loglevel to CRITICAL
Feb 28, 2018
b986534
Unlock return 2 bytes object, for the public/private key material
Feb 28, 2018
f7e3405
NBISweden/LocalEGA#257 basic keyserver with caching
blankdots Feb 28, 2018
c024710
new cache mechanism
blankdots Feb 28, 2018
393f363
NBISweden/LocalEGA#259 caching and unlock request
blankdots Mar 1, 2018
cf502e5
NBISweden/LocalEGA#259 check ttl and docker image
blankdots Mar 2, 2018
754049a
NBISweden/LocalEGA#259 fix for ttl
blankdots Mar 2, 2018
214b821
Parsing a few packets
Feb 11, 2018
26b89bb
This can now decrypt the test file, given a test key and a passphrase.
Feb 17, 2018
7871a73
Progress on recursion and streaming
Feb 21, 2018
2ee0337
Adding a script entrypoint and moving code around
Feb 22, 2018
cf2fa1a
Adding cryptography in requirement
Feb 22, 2018
7454465
Fixing the issues with the cleardata buffer
Feb 22, 2018
6b6275a
Using generators. First step towards streaming
Feb 23, 2018
c2b6f55
Using generators to process data stream
Feb 26, 2018
b7d7eb3
Streaming solution for PGP
Feb 28, 2018
c727986
Updating unlock for private key to only return the material
Feb 28, 2018
00a014f
Updating the consumer generator and changing the loglevel to CRITICAL
Feb 28, 2018
5ca8c89
Unlock return 2 bytes object, for the public/private key material
Feb 28, 2018
538fb2c
Adding docstrings documentation
Feb 28, 2018
647c445
Updating the gpg_cmd with our python pgp tool
Mar 2, 2018
6bb67ee
Merge remote-tracking branch 'origin/feature/pgp-keyserver' into feat…
Mar 2, 2018
965b61e
keys back in bootstrap
Mar 2, 2018
97dd968
Ditching GnuPG.
Mar 2, 2018
bf2fdc5
NBISweden/LocalEGA#259 new keyserver
blankdots Mar 2, 2018
9568df4
NBISweden/LocalEGA#259 fallback, fixed syntax
blankdots Mar 2, 2018
bf3ca12
Merge remote-tracking branch 'origin/feature/pgp-ks' into feature/pgp
Mar 2, 2018
0ec224f
NBISweden/LocalEGA#259 fix typos and addressing comments
blankdots Mar 2, 2018
e6fae2d
Merge remote-tracking branch 'origin/feature/pgp-ks' into feature/pgp
Mar 2, 2018
39bc946
Adjusted the ingestion workers with the keyserver routes.
Mar 2, 2018
fab8922
Making the ingestion worker ask the PGP private key to the keyserver
Mar 2, 2018
9f1994c
Removed the unnecessary entrypoint script for the keyserver.
Mar 2, 2018
b013481
Removing the template for frontend and socket proxy/forwarder.
Mar 2, 2018
78624a0
Updating the hard-coded value in some tests.
Mar 2, 2018
abd182d
Chaning permissions on the PGP public key so that Travis can access it
Mar 2, 2018
8640745
No gnupg bin files anymore
Mar 2, 2018
1e366fa
cleanup
Mar 2, 2018
e848b8d
Making setup.py ALSO (!!!!!) install lega.openpgp
Mar 2, 2018
54629ba
Adding more debug output to be logged on the keyserver
Mar 2, 2018
e943ad5
Fixing the keyserver port to 443 (https)
Mar 2, 2018
b4f1cb0
Catching when the key is not found and making the PGP keyID uppercase…
Mar 2, 2018
7cf0736
Reshaping a bit the decrypt code
Mar 2, 2018
06e66d1
Removed gnupg folder from the config.properties
Mar 2, 2018
e6b1b40
Moving the generate_pgp_key.py away to extras. Normally, we won't nee…
Mar 5, 2018
87b1152
Removed a message from the log
Mar 5, 2018
36652b3
Moving the generate py.... Fo'Real
Mar 5, 2018
a1050f2
No bootstrap image is needed
Mar 5, 2018
4202e5e
Updating the inbox with the cache system
Mar 5, 2018
e8f9b08
aiohttp is updated so server needed it too
Mar 5, 2018
de0cd72
Ditching GnuPG
Mar 5, 2018
fe28acb
No fake Cega-MQ. Using the real CentralEGA one.
Mar 5, 2018
44107aa
Added a function to generate PGP keys, and generate just calls it
Mar 5, 2018
f14ff96
pip install with requirements
Mar 5, 2018
49b244e
Removing socket dependency from ega-ingest systemd unit
Mar 5, 2018
9588014
psycopg2 version typo
Mar 5, 2018
9951881
NBISweden/LocalEGA#259 keyserver adapted to support multiple keys and…
blankdots Mar 5, 2018
7f61510
Misc updates
Mar 5, 2018
947b046
NBISweden/LocalEGA#259 fixed typos and refactored to new .conf file; …
blankdots Mar 5, 2018
3b4efef
NBISweden/LocalEGA#259 addressing comments
blankdots Mar 6, 2018
a5e0beb
NBISweden/LocalEGA#259 parse Defaults properly
blankdots Mar 6, 2018
12ba3b6
Update on the parser to handle compression packet of given size
Mar 6, 2018
cb50971
Merge remote-tracking branch 'origin/feature/pgp-keyserver' into feat…
Mar 6, 2018
4ba633e
Adjusting the bootstrap script to match the keyserver endpoints
Mar 6, 2018
2837b45
Not using the install_package for pip install git+https://LocalEGA
Mar 6, 2018
c3c5389
Moving nc to common
Mar 6, 2018
2e02ece
Encrypting with PGPy (instead of GnuPG) a quite small file.
Mar 6, 2018
a405e1a
Moving to alpine based image
blankdots Mar 7, 2018
12db24e
Lazy logging
Mar 7, 2018
7ba20f3
travis dry run, with debug
blankdots Mar 7, 2018
82dc071
travis dry run install lega module
blankdots Mar 7, 2018
314280c
travis dry run, install package before bootstrap
blankdots Mar 7, 2018
e921928
clean up Dockerfiles
blankdots Mar 7, 2018
4504bf9
python3.6 -> python
Mar 7, 2018
7eaba81
Merge remote-tracking branch 'origin/feature/pgp-alpine' into feature…
Mar 7, 2018
a91c8b5
Making one common image instead of multiple ones
Mar 7, 2018
c655000
Timing delay?
Mar 7, 2018
3d68a9a
Removing debug code and unbuffered flag
Mar 7, 2018
117b4cf
Fixing typos
Mar 8, 2018
256a455
Making the keyserver port configurable
Mar 8, 2018
f2db3f2
Including review comments from Johan.
Mar 8, 2018
cc737ae
Adding a general description for how a packet is structured
Mar 8, 2018
d593f3c
Renaming the read_X functions
Mar 8, 2018
bd31d95
Bootstrapping back in a container, and not on the host
Mar 8, 2018
6307220
Updating some permissions.
Mar 8, 2018
240f025
Ingestigating travis issue
Mar 8, 2018
a6d6d9a
Making travis user own the private directory
Mar 8, 2018
9ad747b
Adding back the fake Message Broker for Central EGA
Mar 8, 2018
fd0d6bf
Trying 2 big images (base and inbox) containing everything.
Mar 8, 2018
5226e0b
switch back to common
blankdots Mar 9, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 7 additions & 2 deletions .travis.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,6 @@
language: common
language: python
python:
- 3.6

services:
- docker
Expand All @@ -7,10 +9,12 @@ before_install:
# https://elk-docker.readthedocs.io/#es-not-starting-max-map-count
# mostly used by ELK stack; Solving issue #252
# - sudo sysctl -w vm.max_map_count=262144
- pip3.6 install pgpy
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need Python on the "host" machine? Is it for replacing PGP keys generation in bootstrapping from GPG to Python implementation? Do we really need this replacement? I feel like it's redundant because GPG is pre-installed on Ubuntu and we don't need to install anything else.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No we don't, but here is the issue we fixed with it.
I explain.

We can use GnuPG to generate a key (as we used to do) and export the public key. However, to export the secret key, we need to be at the prompt to type the passphrase. So, that doesn't work for us (and tell me how we could make it work if you know), so we generate the public/private keys in another way (using the PGPy python module).

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it was exactly what we used to generate the key.
That generates a keyring, and you can export the public from it.
However, to export the secrete key, the passphrase is needed (and prompted).

- pip3.6 install -e .
- |
cd deployments/docker
# make -C images pull # Not used at the moment, cuz we don't manage to build from cache
make -C images images
make -C images -j 4
make bootstrap

install:
Expand All @@ -24,6 +28,7 @@ script:
# comment out sleep if no ELK stack is used; Solving issue #252
# - sleep 20
- cd ../../tests
- /bin/sleep 10
- mvn test -B

after_success:
Expand Down
2 changes: 1 addition & 1 deletion deployments/docker/Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ help:
@echo "where <target> is: 'bootstrap', 'up', 'all-up', 'ps', 'down', 'network' or 'clean'\n"

private bootstrap:
@docker run --rm -it -v ${PWD}:/ega -v ${PWD}/../../extras/db.sql:/tmp/db.sql nbisweden/ega-bootstrap ${ARGS}
@./bootstrap/boot.sh ${ARGS}

network:
@docker network inspect cega &>/dev/null || docker network create cega &>/dev/null
Expand Down
7 changes: 1 addition & 6 deletions deployments/docker/bootstrap/boot.sh
Original file line number Diff line number Diff line change
Expand Up @@ -7,20 +7,17 @@ HERE=$(dirname ${BASH_SOURCE[0]})
PRIVATE=${HERE}/../private
DOT_ENV=${HERE}/../.env
SETTINGS=${HERE}/settings
EXTRAS=${HERE}/../../../extras

# Defaults
VERBOSE=no
FORCE=yes
OPENSSL=openssl
GPG=gpg2
GPG_CONF=gpgconf

function usage {
echo "Usage: $0 [options]"
echo -e "\nOptions are:"
echo -e "\t--openssl <value> \tPath to the Openssl executable [Default: ${OPENSSL}]"
echo -e "\t--gpg <value> \tPath to the GnuPG executable [Default: ${GPG}]"
echo -e "\t--gpgconf <value> \tPath to the GnuPG conf executable [Default: ${GPG_CONF}]"
echo ""
echo -e "\t--verbose, -v \tShow verbose output"
echo -e "\t--polite, -p \tDo not force the re-creation of the subfolders. Ask instead"
Expand All @@ -35,8 +32,6 @@ while [[ $# -gt 0 ]]; do
--help|-h) usage; exit 0;;
--verbose|-v) VERBOSE=yes;;
--polite|-p) FORCE=no;;
--gpg) GPG=$2; shift;;
--gpgconf) GPG_CONF=$2; shift;;
--openssl) OPENSSL=$2; shift;;
--) shift; break;;
*) echo "$0: error - unrecognized option $1" 1>&2; usage; exit 1;; esac
Expand Down
9 changes: 8 additions & 1 deletion deployments/docker/bootstrap/cega_users.sh
Original file line number Diff line number Diff line change
Expand Up @@ -100,17 +100,24 @@ services:

cega-users:
env_file: cega/env
image: nbisweden/ega-cega-users
image: nbisweden/ega-common
hostname: cega-users
container_name: cega-users
ports:
- "9100:80"
expose:
- "80"
volumes:
- ./cega/users:/cega/users:rw
- ../images/cega-users/users.html:/cega/users.html
- ../images/cega-users/server.py:/cega/server.py
# - ../..:/root/.local/lib/python3.6/site-packages:ro
restart: on-failure:3
networks:
- cega
command: ["python", "/cega/server.py"]


EOF

# For the compose file
Expand Down
Loading