Import Ficly stories (and associated historical user data)

[ethel-t-frog]

My current thinking on this is to import the raw story HTML and just present that from the database. I have a copy of the entire Ficly site on my disc at home, so the data is easily available.

For Ficly uids, I thought that it should be possible to create user records using Ficlatté_uid = -Ficly_uid. If a Ficlatté user wishes to claim the stories of a Ficly uid, there are two possible ways forward: 1. make a link between the two uids; 2. change the uids on the Ficly stories to match the Ficlatté user.

1. is easily reversible
2. doesn't need any more coding to display correct authorship

Another thing that might work is to add a "ficly_uid" field to the story table, which allows the old stories to retain their historical authorship but also allow them to be claimed by the Ficlatté user.

In any case, making the link between old uids and new uids should be an admin task, we shouldn't allow any random new user to claim authorship of a whole canon of Ficly stories.

Thoughts?

[HSAR]

My gut instinct is to go for suggestion number 2, since it has the least impact in terms of code required.

However, I like suggestion number 1 a lot now that I read it. It would, for instance, allow us to present Ficly stories differently (for instance: "Author (Ficly): Old Tom (Young Tom)", since some users don't have the same display name as they did).

I agree that linking should be the task of the administration. The exact process will probably need to be refined as we go along, though it's likely that the number of cases will be rather small.

[ethel-t-frog]

Using OneToOneField is an easy way to link records in Django, and it provides a really easy way to reference the data at the other end of the link.

I'll have a think about that.

[HSAR]

An implementation detail: I would suggest simply special-casing stories below ID 40000 rather than using minus numbers, since the ID is visible to the user in the URL.

[ethel-t-frog]

How about adding "ficly_uid" to stories, which can link to a separate Ficly user table that preserves the original Ficly uids and optionally links back to the main user table with a OneToOneField? We should keep Ficly user data separate from Ficlatté user data. My original idea to use negative uids is probably wrong, though. A separate table feels more sensible to me now I think about it.

[HSAR]

I don't see the necessity of separating the Ficly data, myself. Can you explain your reasoning?

[ethel-t-frog]

Ficly user data is all historical and does not need any ongoing maintenance (eg. pen-name changes) or a record of user's e-mail address. Crucially, Ficly uids have the same range as Ficlatté uids, which means that there will be collisions between the two datasets, which we avoid by storing the sets of user data in separate tables.

[HSAR]

I don't necessarily agree that Ficly data is historical. What if a user claims the story and wants to edit it? That seems like potentially a valid scenario to me, though it would be equally valid to flat-out declare all Ficly imports as read-only.

The uid problem I thought was solved? The last Ficly ID is 38980, while the first Ficlatte ID is 40006. As far as I recall, this was a deliberate design decision intended to make importing the archive easy.

[ethel-t-frog]

You're confusing story IDs with User ids.

The Ficlatté SIDs begin at 40000 in order to make room for the Ficly import, that is true. The user ids were not kept separate. When we import Ficly stories, they will have a specific author ID that refers to the Ficly user table. I am proposing to separate the user table for Ficly imports from the main Ficlatté user table.

After import, Ficlatté story 38936 will be imported from Ficly story 38936 and linked to Ficly author 269 (ethelthefrog). Site admin can link Ficly user 269 to Ficlatté user 1 so I can be credited appropriately.

[HSAR]

Oh, I see. My mistake. Yes, the user tables will have to be seperate.

Side note: could we rename this ticket a little, since it implies it covers the stories only?

[ethel-t-frog]

Note to self: remember that some people had multiple Ficly accounts and that any associations between Ficly uids and Ficlatté uids must allow for multiple Ficly uids to point to the same Ficlatté uid. The OneToOneField type is unsuitable for this.

[stitzelj]

You'll notice I haven't volunteered to take this one on. I know that integration is going to be.... messy, at best.

[ethel-t-frog]

No problems. It's firmly on my desk. Sadly, lots of other projects are as well. ;)

[stitzelj]

Heh, if I get enough of these other issues knocked out of the way, I might circle back around to this if you haven't gotten to it by then. I figure it's probably one of the lower items on the priority list at the moment.

[ethel-t-frog]

You're a star, Jim.

[stitzelj]

Heh, mostly I just have the time these days. Being unemployed for almost a year has its downsides, but one of the nice perks is I finally have time to pick up new coding languages.