Modify first h1/p just like npm does

[chadwhitacre]

Part of #4148.

[chadwhitacre]

I'm going to stop the scheduled job and reprocess already-processed readmes before starting it up again.

[chadwhitacre]

Let's get a sense of how long it will take to reprocess readmes ...

[chadwhitacre]

Looks like scheduler tasks are not in heroku ps. I'm not sure how to kill it.

[chadwhitacre]

Maybe if we remove and then re-add the addon?

[chadwhitacre]

Hmm ... heroku logs --tail | grep scheduler isn't turning up anything. Is the process already dead?

[chadwhitacre]

watch -n5 "heroku pg:psql -c 'select count(*) from packages where readme_raw is not null'"

/me watches for a minute to see if the number changes ...

[chadwhitacre]

It's not changing. I'm going to run a short scheduled job to see if it shows up in heroku ps normally.

[chadwhitacre]

[chadwhitacre]

Yes, scheduler processes show up in heroku ps. 👍

[chadwhitacre]

Alright, so I'm thinking we should split readme fetching and processing into two separate processes. We can use readme=null to signal a need to process, just as we are using readme_raw=null to signal the need to refetch. We just need to make sure that readme=null shows up in the UI in a sane way. Then we can run both processes as scheduled jobs.

[chadwhitacre]

Actually, let's keep readme as the one we read from and display to the user, and add a readme_processed that is set to NULL after fetching.

[chadwhitacre]

Alright, ready for review. cc: @rohitpaulk @aandis @nobodxbodon @kaguillera et al.

[chadwhitacre]

What's up with this marky-markdown failure at Travis? Looks like its installing 9.0.1, which is also what I have locally. The tests pass for me locally.

[chadwhitacre]

Drat. Does 9edf5b8 fail in the same way?

[chadwhitacre]

No!

[chadwhitacre]

Hmm ... now marky-markdown is spitting out markup that I think I saw before, but then switched away from because I thought I saw marky-markdown spitting out different markup ...

[chadwhitacre]

Hmm ... might actually be a type discrepancy, Markup vs unicode?

[chadwhitacre]

Ah, yeah, at some point this weird inline SVG snuck. O.o Now it's gone again?

[chadwhitacre]

Oh! This is over in test_markdown, not in test_npm_sync ...

[chadwhitacre]

Here we go, let's try 9ec8a8b ...

[chadwhitacre]

Yesssssss. 👍

Gonna clean up a few more things here ...

[chadwhitacre]

Okay! I think this is ready to go. What are the next steps once this is merged and deployed? What's our current state and how do we get to a database full of properly-rendered readmes to back #4151?

[chadwhitacre]

gratipay::MAROON=> select count(*) from packages where readme != '';
┌────────┐
│ count  │
├────────┤
│ 133331 │
└────────┘
(1 row)

Heh. :)

I started update packages set readme='' but it takes a while and locks the db(? site was slow to load), so I killed it. I guess it's fine to leave the crap that's in there in there and let it be overwritten by the new processor.

Okay! So I think the thing to do is to:

• Merge and deploy this.
• Schedule a fetcher process.
• Schedule a processing process.

[chadwhitacre]

branch.sql sets readme_needs_to_be_processed to true for all records, so even though some readmes are already processed, all should be processed again.

What about refetching? How do we schedule that?

[chadwhitacre]

Oh right! upsert is supposed to reset readme_raw to null. Hmm ... maybe another PR? Let's get the first round of readmes in there.

[chadwhitacre]

Here we go!