Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

AWS-SDK: Fixing broken exports #2410

Merged
merged 22 commits into from
Aug 19, 2024
Merged
Changes from 1 commit
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
dac4adf
new means of creating a bucket
engelhartrueben Jul 30, 2024
98b8563
remove now unused S3 import
engelhartrueben Jul 30, 2024
8338ac1
remove double await that was doing nothing
engelhartrueben Jul 30, 2024
8cc52b8
remove expires in parameter in getSignedURL that is taken care of in …
engelhartrueben Jul 30, 2024
43fd02d
change var name s3bucket to client + comment
engelhartrueben Jul 30, 2024
5e31c04
Merge branch 'main' into re/fix-eports
engelhartrueben Jul 30, 2024
3fcab79
call region from env
engelhartrueben Jul 30, 2024
af2bddd
move bucket name to func call. add location var found in example docs
engelhartrueben Jul 30, 2024
6818d2a
implement PutObjectCommand w/ supporting parameters.
engelhartrueben Jul 30, 2024
0a52da7
make pretty
engelhartrueben Jul 30, 2024
78c2aaa
remove logging
engelhartrueben Jul 30, 2024
2c101dc
add front end language for when export will end up in local Spoke dir…
engelhartrueben Jul 31, 2024
2e623b6
add hard check on email set-up. Next step is to (re-add) fix front e…
engelhartrueben Aug 2, 2024
87538cf
Add error handling when exporting to an S3 bucket
mau11 Aug 14, 2024
fb0f56d
misspointed check on emailEnabled
engelhartrueben Aug 14, 2024
48f095e
typos
engelhartrueben Aug 14, 2024
243b610
change Snackbar logic to better fit outcome
engelhartrueben Aug 15, 2024
052685b
change exportCacheKey expiration to match AWS expiration of 1 day
engelhartrueben Aug 15, 2024
7417b63
cache error
engelhartrueben Aug 15, 2024
0d1055a
reduce timout to 15 sec
engelhartrueben Aug 15, 2024
ee6a29c
adjust export UI logic to only show each respective methods output, a…
engelhartrueben Aug 15, 2024
be0bd30
revert back to emailEnabled check
engelhartrueben Aug 15, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 44 additions & 5 deletions src/workers/jobs.js
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,14 @@ import { rawIngestMethod } from "../extensions/contact-loaders";

import { Lambda } from "@aws-sdk/client-lambda";
import { getSignedUrl } from "@aws-sdk/s3-request-presigner";
import { CreateBucketCommand, GetObjectCommand, waitUntilBucketExists, S3Client, PutObjectCommand } from "@aws-sdk/client-s3";
import {
CreateBucketCommand,
HeadBucketCommand,
GetObjectCommand,
waitUntilBucketExists,
S3Client,
PutObjectCommand
} from "@aws-sdk/client-s3";
import { SQS } from "@aws-sdk/client-sqs";
import Papa from "papaparse";
import moment from "moment";
Expand Down Expand Up @@ -865,14 +872,46 @@ export async function exportCampaign(job) {
region: process.env.AWS_REGION
});
const bucketName = process.env.AWS_S3_BUCKET_NAME;
const command = new CreateBucketCommand({ Bucket : bucketName });

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we want to create a new bucket each time we export data, it's likely someone will already have bucket created for the data and want to upload directly to it

we can add a check, if the bucket exists, save the exported data, if it doesn't, create a new one and save

// this can return Location and $metadata, we just don't need that info
await client.send(command);
try {
// Check if the S3 bucket already exists
const verifyBucketCommand = new HeadBucketCommand({
Bucket: bucketName
});
await client.send(verifyBucketCommand);

console.log(`S3 bucket "${bucketName}" already exists.`);
} catch (error) {
if (error.name === "NotFound") {
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if (error.code === "NoSuchBucket")

Source

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When the bucket is not found, it doesn't look like we get this error code, only the error name

console.log(
`S3 bucket "${bucketName}" not found. Creating a new bucket.`
);

try {
// Create the S3 bucket
const createBucketCommand = new CreateBucketCommand({
Bucket: bucketName
});
await client.send(createBucketCommand);

console.log(`S3 bucket "${bucketName}" created successfully.`);
} catch (createError) {
console.error(
`Error creating bucket "${bucketName}":`,
createError
);
}
} else {
console.error("Error checking bucket existence:", error);
}
}

// verifies that the bucket exists before moving forward
// if for some reason this fails, Spoke defensively deletes the job
await waitUntilBucketExists({ client, maxWaitTime: 60 }, { Bucket : bucketName });
await waitUntilBucketExists(
{ client, maxWaitTime: 60 },
{ Bucket: bucketName }
);

const campaignTitle = campaign.title
.replace(/ /g, "_")
Expand Down
Loading