Changes needed for the integration of ABRT in QE workflows #388

jfilak · 2015-10-20T09:09:37Z

We need to be able to attach the URL of the uploaded problem data to the relevant micro-report. Here are the problems addressed by this pull request.

reporter-upload does not save its report results in reported_to file
reporter-ureport does not know 'url' micro-report attachment type [URL attachment saving faf#467]

How should be this pull request used in the integration efforts:

EVENT=notify
  reporter-ureport
  EC=$?

  # See  EXIT_STOP_EVENT_RUN in /usr/include/libreport/internal_libreport.h
  if [ $EC -eq 70 ]; then
   echo "Already reported problem:"
    cat reported_to
    exit 0
  fi

  if [ $EC -ne 0 ]; then
    exit $EC
  fi

  reporter-upload -u ftp://qe.example.org/abrt-reports || exit 1
  reporter-ureport -r "upload" -L "URL" -T "url" || exit 1

Introduce dd_create_archive() function. I added the argument 'flags' in order to avoid the need to introduce dd_create_archive_ext() function in the future. I am sure will use this flag in the future (e.g. Encrypted). Signed-off-by: Jakub Filak <[email protected]>

Use the test function instead of own untested one. Signed-off-by: Jakub Filak <[email protected]>

Make sure we can convert data back and forth without losing information. The introduced function complements iso_date_string(). Signed-off-by: Jakub Filak <[email protected]>

Use this function to ensure consistent reported_to formatting. Signed-off-by: Jakub Filak <[email protected]>

This commit should make the code less fragile. There is no need for this commit, however I plan to make the '.label' member configurable through environment variables. Signed-off-by: Jakub Filak <[email protected]>

jfilak · 2015-10-20T11:40:24Z

ABRT integration test case (not pushed yet):
abrt/abrt@3c3544b

jfilak · 2015-10-20T11:56:59Z

Do not merge this pull request. The lines added to reported_to file can contain usernames and passwords:

upload: URL=scp://root:redhat@localhost/tmp/tmp.RaYnVplNNv/target/problem_dir.tar.gz

I need to either ask curl to return URL without credentials or remove everything between :// and @.

jfilak · 2015-10-20T13:23:37Z

I've fixed the problem with credentials and added the following commit to abrt:
abrt/abrt@4b307cc

The function expects a valid URL. Signed-off-by: Jakub Filak <[email protected]>

All clients should work with URLs without userinfo. This commit will ensure that. This commit also changes log messages from: Sending /var/tmp/problem_dir.tar.gz to scp://localhost/tmp/tmp.x5WVgpgUsY/target/ Error while uploading: 'curl_easy_perform: Login denied' Please enter user name for 'scp://localhost/tmp/tmp.x5WVgpgUsY/target/': root Please enter password for 'root': Sending /var/tmp/problem_dir.tar.gz to scp://localhost/tmp/tmp.x5WVgpgUsY/target/ Successfully sent /var/tmp/problem_dir.tar.gz to scp://localhost/tmp/tmp.x5WVgpgUsY/target/ to: Sending /var/tmp/problem_dir.tar.gz to scp://localhost Error while uploading: 'curl_easy_perform: Login denied' Please enter user name for 'scp://localhost': root Please enter password for 'scp://root@localhost': Sending /var/tmp/problem_dir.tar.gz to scp://localhost Successfully created scp://localhost/tmp/tmp.x5WVgpgUsY/target/problem_dir.tar.gz Signed-off-by: Jakub Filak <[email protected]>

All other plugins do so. We also need this commit to be able to attach URLs to uploaded dump directories to uReport on FAF servers. Signed-off-by: Jakub Filak <[email protected]>

Signed-off-by: Jakub Filak <[email protected]>

jfilak · 2015-10-21T14:52:49Z

This pull request can be reviewed and merged.

sorki · 2015-10-21T15:11:29Z

src/plugins/reporter-upload.c

+    {
+        /* Load Password only if Username is configured, it doesn't make */
+        /* much sense to load Password without Username. */
+        state->password = getenv("Upload_Password");


What if user configures scp to use keys? There might be username set but no password is needed. Also using CURL to upload stuff via SCP is not good at all, would be much better to just use SCP so user can configure it as usual and not fiddle with CURL/abrt specific configuration.

Good point! I will try to rework this part later on: #389

jfilak · 2015-10-22T06:37:35Z

I corrected the documentation issues found by @sorki

sorki · 2015-10-22T11:52:29Z

Good to go from my side, thanks.

Whenever we introduce a new report type we need to teach reporter-ureport to attach its report results to micro-reports. This commit adds support for attaching arbitrary values, so we will not need to modify reporter-ureport when a new reporter is added. Two operating modes are being introduced. The mode for attaching random values: $ reporter-ureport -l "foo" -T "blah" -A and the mode for attaching the report results (data from reported_to file): $ reporter-ureport -L "URL" -T "url" -r "upload" -A '-v', '--value' is taken by '--verbose'. '-d', '--data' is taken by '--directory'. Hence I used '-l', '--value'. '-t', '--type' is taken by '--auth'. Hence I used '-T', '--type'. Signed-off-by: Jakub Filak <[email protected]>

mhabrnal · 2015-10-27T12:36:38Z

Looks good to me, thanks.

Changes needed for the integration of ABRT in QE workflows

Jakub Filak added 2 commits October 19, 2015 15:02

uploader: use shared dd_create_archive function

5617cfe

Use the test function instead of own untested one. Signed-off-by: Jakub Filak <[email protected]>

jfilak assigned jfilak and mhabrnal and unassigned jfilak Oct 20, 2015

Jakub Filak added 3 commits October 20, 2015 12:44

lib: introduce parser of ISO date strings

10a871d

Make sure we can convert data back and forth without losing information. The introduced function complements iso_date_string(). Signed-off-by: Jakub Filak <[email protected]>

reported_to: add a function formatting reported_to lines

b54128a

Use this function to ensure consistent reported_to formatting. Signed-off-by: Jakub Filak <[email protected]>

plugins: port reporters to add_reported_to_entry

95a8406

This commit should make the code less fragile. There is no need for this commit, however I plan to make the '.label' member configurable through environment variables. Signed-off-by: Jakub Filak <[email protected]>

jfilak force-pushed the wip_jfilak_base_os_qa branch 2 times, most recently from 151cc52 to e75c504 Compare October 20, 2015 11:33

jfilak force-pushed the wip_jfilak_base_os_qa branch from e75c504 to 17d9b75 Compare October 20, 2015 13:22

jfilak mentioned this pull request Oct 20, 2015

Test cases for QE workflow libreport changes abrt/abrt#1074

Merged

Jakub Filak added 4 commits October 21, 2015 16:45

lib: add function for removing userinfo from URIs

120b80c

The function expects a valid URL. Signed-off-by: Jakub Filak <[email protected]>

uploader: save remote name in reported_to

18a88d7

All other plugins do so. We also need this commit to be able to attach URLs to uploaded dump directories to uReport on FAF servers. Signed-off-by: Jakub Filak <[email protected]>

update .gitignore

3f47cdc

Signed-off-by: Jakub Filak <[email protected]>

jfilak force-pushed the wip_jfilak_base_os_qa branch from 17d9b75 to 5b8a29f Compare October 21, 2015 14:49

sorki reviewed Oct 21, 2015
View reviewed changes

jfilak force-pushed the wip_jfilak_base_os_qa branch from 5b8a29f to 99d45ca Compare October 22, 2015 06:35

jfilak force-pushed the wip_jfilak_base_os_qa branch 2 times, most recently from c5b0dcc to 5b9d8bf Compare October 27, 2015 12:03

jfilak force-pushed the wip_jfilak_base_os_qa branch from 5b9d8bf to ca9e85e Compare October 27, 2015 12:06

mhabrnal pushed a commit that referenced this pull request Oct 27, 2015

Merge pull request #388 from abrt/wip_jfilak_base_os_qa

2d34318

Changes needed for the integration of ABRT in QE workflows

mhabrnal merged commit 2d34318 into master Oct 27, 2015

jfilak deleted the wip_jfilak_base_os_qa branch October 30, 2015 08:13

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Changes needed for the integration of ABRT in QE workflows #388

Changes needed for the integration of ABRT in QE workflows #388

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 21, 2015

sorki Oct 21, 2015

jfilak Oct 22, 2015

jfilak commented Oct 22, 2015

sorki commented Oct 22, 2015

mhabrnal commented Oct 27, 2015

Changes needed for the integration of ABRT in QE workflows #388

Changes needed for the integration of ABRT in QE workflows #388

Conversation

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 20, 2015

jfilak commented Oct 21, 2015

sorki Oct 21, 2015

Choose a reason for hiding this comment

jfilak Oct 22, 2015

Choose a reason for hiding this comment

jfilak commented Oct 22, 2015

sorki commented Oct 22, 2015

mhabrnal commented Oct 27, 2015