[EVENT] PaleoHackWeek 3

[choldgraf]

Summary

The PaleoHackWeek community is running another hackathon in about 10 days! This is similar to the previous hackathon's that they've run (though it will be 3 days instead of 2).

Note that this hub is already running at https://paleohack2021.hackathon.2i2c.cloud/, but we'll probably need to ready the hub for another scale-up event, so creating this issue so that we can track it.

In addition, @CommonClimate and @khider were hoping to do a trial run on March 9th where they'd ask the attendees to try logging in to make sure the scaling worked (even though they plan to start on March 10th).

Event Info

• Community Representative: @CommonClimate and @khider
• Event begin: March 9th
• Event end: March 11th
• Active times: Roughly working hours on US/Pacific
• Number of attendees: ~60
• Hub Events Calendar

Hub info

• Hub URL: https://paleohack2021.hackathon.2i2c.cloud/hub/login?next=%2Fhub%2F
• Hub decommisioned after event?: No

Task List

Before the event

• Dates confirmed with the community representative and added to Hub Events Calendar
• One week before event Hub is running.
• Confirm with Community Representative that their workflows function as expected.
  • 👉Template message to send to community representative

    ``` Hey {{ COMMUNITY REPRESENTATIVE }}, the date of your event is getting close!

    Could you please confirm that your hub environment is ready-to-go, and matches your hub's infrastructure setup, by ensuring the following things:

    • Confirm that the "Event Info" above is correct
    • On your hub: log-in and authentication works as-expected
    • nbgitpuller links you intend to use resolve properly
    • Your notebooks and content run as-expected

    </details>  
    
    

• 1 day before event, either a separate nodegroup is provisioned for the event or the cluster is scaled up.

During and after event

• Confirm event is finished.
• Nodegroup created for the hub is decommissioned / cluster is scaled down.
• Debrief with community representative.
  • 👉Template debrief to send to community representative

    Hey {{ COMMUNITY REPRESENTATIVE }}, your event appears to be over 🎉
    
    We hope that your hub worked out well for you! We are trying to understand where we can improve our hub infrastructure and setup around events, and would love any feedback that you're willing to give. Would you mind answering the following questions? If not, just let us know and that is no problem!
    
    - Did the infrastructure behave as expected?
    - Anything that was confusing or could be improved?
    - Any extra functionality you wish you would have had?
    - Could you share a story about how you used the hub?
    
    - Any other feedback that you'd like to share?
    
    

[choldgraf]

cc @khider and @CommonClimate - it would be helpful if you could run a representative notebook from top to bottom a day or two before the event, just to confirm that the environment is correct and such!

for the @2i2c-org/tech-team - can somebody either:

1. Run a "scale up" script to stress-test the hub this week (@yuvipanda I know we did this at Berkeley but not sure if we have this built for our hubs in 2i2c yet?
2. Be around on March 9th when they test out the hub scale-up before they start using it the next day.

[khider]

Thanks! Yes, I'm hoping to release the latest version of the code today or tomorrow so I can rebuild the environment and see if everything works.

They will be busy with another platform on Wednesday March 9th (some may make it to the hub during the night between 9th and 10th).

[choldgraf]

Sounds good @khider !

Also just in case it'll take a bit of time to figure out stress-testing stuff, I opened an issue so that we can track it: #1041

[khider]

@choldgraf We updated all the notebooks and environment. The container built automatically and I grabbed the image from quay for the Configurator but this is the error I get when restarting:

[choldgraf]

Hmmm - that might be because you're just using the tag and not the fully-resolved name. I just tried setting it to:

2i2c/paleohack-2021:cb5757e7c9f8

try logging in and see if that works?

[choldgraf]

ah it needed to be the fully-resolved name:

quay.io/2i2c/paleohack-2021:cb5757e7c9f8

it should be working now

[choldgraf]

I opened up a PR to clarify the image tag prompt here: yuvipanda/jupyterhub-configurator#11

[khider]

Thanks! I got it to run but now I can't get it to sync with GitHub. It tells me that it did but actually didn't for mine. When opening a new user server, I don't get the notebooks at all.

[sgibson91]

What's your nbgitpuller link @khider ?

[khider]

https://paleohack2021.hackathon.2i2c.cloud/hub/login?next=%2Fhub%2Fuser-redirect%2Fgit-pull%3Frepo%3Dhttps%253A%252F%252Fgithub.com%252FLinkedEarth%252FpaleoHackathon%26urlpath%3Dtree%252FpaleoHackathon%252F%26branch%3Dmain

[choldgraf]

@khider I just tried clicking that link and it worked for me ). Can you share a specific notebook that you expect to be there, but that is not there?

Here are the notebooks in my folder:

[sgibson91]

I just tried clicking that link and it worked for me

same here

[khider]

For me, it seems to merge old/new versions of the notebooks.

This is the hub version:

But this is the version on GitHub

If this is just me (because I have been using the same account for 3 hackathons), then not a problem.

[khider]

And the merge decision is somewhat random. I changed the preamble from the version we had and it got updated but the working with datasets didn't.

[choldgraf]

Hmmm - the language around the merging behavior is defined here: https://jupyterhub.github.io/nbgitpuller/topic/automatic-merging.html

Does that help disambiguate at all? If we can clarify that language let me know and we can update it.

Also just a note that the easiest way to understand what an attendee will experience is to rename the folder that nbgitpuller downloaded, so that it will trigger a fresh clone when you click the link next

[khider]

Yes, that makes more sense now. I'll delete my files and retry.

[khider]

Perfect!

[choldgraf]

wohoo! I am curious - where could put the link to these docs in our own 2i2c docs so it is discoverable?

[khider]

I have been building my own Google Doc with all the links.

[choldgraf]

Ah cool - well, maybe once this round is over you'd be willing to tell us all of the things that were confusing, and that we could improve in the docs? :-)

[khider]

We are ready on the notebook side. Should I try to launch all the servers at the same time?

[choldgraf]

I believe that @yuvipanda has provisioned a dedicated set of nodes for the event (see #1058) so there's no problem in giving it a shot if you like

[khider]

Thank you! And we're all ready to go.

[yuvipanda]

@khider sorry, it's not done yet, I'm doing that by the next hour - sorry.

[yuvipanda]

@khider ok I've scaled up a lot of nodes (20), and enabled an additional performance optimization (pre-pulling the existing image).

[khider]

BTW, everything is working great!

[choldgraf]

[choldgraf]

Hey @khider - just want to confirm that this event is over, and that you no longer need the extra nodes for the event!

[yuvipanda]

I just set the minimum node size to 0, so it can scale down.

[khider]

Yes, we're done! Thank you; everything worked great!

[choldgraf]

Wohoo, thanks @khider ! If you like, it'd be helpful for any thoughts that you might have about how the event went:

If you like, could you answer any of the following questions?

• Did the infrastructure behave as expected?
• Anything that was confusing or could be improved?
• Any extra functionality you wish you would have had?
• Could you share a story about how you used the hub?
• Any other feedback that you'd like to share?

I'll close this one since I don't think there are any more actions to take!

[yuvipanda]

@choldgraf I need to put the hub back into the general pool (rather than to its own), will close when done.

[yuvipanda]

Done in #1123