Environment Variable for artifactsLocationTemplate()

[johncmckim]

Currently all artifacts are uploaded to a bucket with the prefix pipeline/stage/job/pipeline_counter.stage_counter.

We would like to be able to upload artifacts to a bucket with a different prefix specified in a pipeline parameter i.e. s3://bucket-name/application-name/client-name/version/.

Can GoEnvironment.java be changed to first check if a specific pipeline parameter exists and use that in the place of the standard template?

[brewkode]

@johncmckim We need to take care of how getLatestRevision would work too. Right now, getLatestRevision is based on this convention - both in terms of the dir structure and the way the version is maintained - pipeline_counter.stage_counter. So, any changes to the path will affect the fetch-s3-plugin too.

OTOH, if you want to use publish s3 plugin for just artifact storage, which wont be used for dependency management via fetch-s3 plugin, then it might work. Thoughts?

[johncmckim]

In our use case, we are only interested in the publish component. Our goal is to publish deployable packages to S3 that is then distributed and installed via our internal systems. The current structure does not work well for this.

However, I can see the issue for those using for both publish and fetch. If I was using this purely for artifact storage I would not to change it away from the default.

Considering this, perhaps another option is the setting this on the Publish task configuration. An option to specify your own prefix (i.e. / or /application-name ) would give the user the greatest control over the destination of the artifacts.

[brewkode]

Exactly. The intention was to use them in tandem. I've had (many) situations where I wanted to publish to a specific prefix in S3. The right solution for that would be to keep it as task level configuration(which would override default settings). It gives best of both worlds.

@manojlds @ashwanthkumar Thoughts?

[ashwanthkumar]

+1

[manojlds]

+1 something we should have.

[johncmckim]

Thanks everyone. This is something that we are going to need in the near term. However, I understand our timeline might not fit in with yours.

If you think this might be a while away from being implemented, I am happy to contribute with a pull request.

[ashwanthkumar]

It would be great if you could send in a PR, we'll be happy to review and merge the changes.

[johncmckim]

Awesome. I'll put something together soon. Looks like a simple change.

[kitplummer]

Trying to make sure I understand the context here, before I submit another GH issue. Is the objective here to be able to create a 'static' location/bucket in S3 as the destination...to avoid the 'build_number' bucket? This is what I need...so I can deploy artifacts to package repositories (e.g. Yum et al).

[johncmckim]

@kitplummer that's the idea. This pull request add's a prefix field that can be used to replace the default prefix, i.e. normally artifacts are uploaded to pipeline/stage/job/pipelineCounter.stageCounter, but you can change that to something different (supports parameters and environment variables). I've been using my fork for ages now and it's working well for me.

[kitplummer]

@johncmckim - thanks for the clarification. @ashwanthkumar @manojlds - any ideas when the PR will/could be merged in to the project?

[timanrebel]

@johncmckim is it possible to use the root of an S3 bucket as the destination? When I enter / I get a root folder with an empty name.

[johncmckim]

@timanrebel I can see that happening too. I'm guessing here, but I think the issue is that a / is appended to the destination key before being sent to Amazon. So when you input / as the destination prefix, you end up with a key that is //, causing the empty folder. It should be easy to fix. I'll set a reminder to have a look at it soon.

[johncmckim]

@timanrebel I have resolved this issue and updated the pull request. Apparently keys that start with / cause this.

@ashwanthkumar @manojlds any idea when this could be looked at?

[ashwanthkumar]

Fixed via #11