HT-2898: Format and Overlap determination documentation #77

jsteverman · 2021-04-13T12:43:52Z

No description provided.

format_and_overlap_determination.md

nfulkers · 2021-05-10T16:44:51Z

format_and_overlap_determination.md

+: Volume ID and associated metadata taken from the Hathifiles. 
+
+**Print Serials List**
+: List of Record Ids determined to be serials by some unknown UMich process. List is limited to those held by UMich.


How do we get this list? How often?

Margaret Kelly ([email protected]) emails this as a .txt file to Martin & Josh at the end of each month. We could request instead that it be uploaded to a certain dropbox location.

I think we should say:

we get the file from AIM (do not specify individuals)

how the file is generated is not in scope for this document

We were told that the latest file (end of May 2021) would be the last file Margaret would be able to produce, as Umich is switching from Aleph to Alma (and presumably this report was based on Aleph).

format_and_overlap_determination.md

nfulkers · 2021-05-11T16:29:14Z

format_and_overlap_determination.md

+**Cost Per Volume**
+: Target Cost / Total Number of HT Items
+
+**Member Weight**


Is this the same as the "tier"? If not we should add what a tier is.

It's the same thing.

Well. The Member Tier is a label that exists outside the system.

Melissa will tell me "Please make an estimate for member xyz, they are a US institution and tier 1". She determines tier based on the IPEDS. From her boilerplate that goes out to prospective members: "Each member is assigned to one of three tiers based on the member's Total Library Expenditures as reported in public sources."

Each tier (1,2,3) has a weight associated. This association is not codified, other than perhaps in Confluence, but as far as I'm concerned lives in our minds.
Tier 1: 0.67
Tier 2: 1.00
Tier 3: 1.33

Then there are other members that fall outside the tier system, with a weight of 0.00 or above 1.33. These are:

+------------+--------+
| member_id | weight |
+------------+--------+
| ucmerced | 0.00 |
| ucsf | 0.00 |
| hathitrust | 0.00 |
| utexas | 3.00 |
| flbog | 5.00 |
| usg | 3.00 |
+------------+--------+

Those non-tier weights were, I believe, rather arbitrarily cobbled together, probably by Mike.

ooookay, this is where the system thing from my comment above gets worked out. As in, this is where the "special processing" for USG and FLBOG (and utexas, I guess these are the only 3 "systems") happens.

Is this really the right way to do this??

My impression was that the weights (including for the system members, which I think is based on the number of R1 research schools in the system) are encoded in the billing model which has been approved by the Board, so I would not describe them as "arbitrary". I'm checking on this with Mike, but yes, I think that given our current billing model this is the right way to do it.

The system weights come from this: https://www.hathitrust.org/sites/www.hathitrust.org/files/member-criteria-formalize.pdf which Natalie quoted from below

The tier to weight mapping for Tier 1, 2, 3 is in a policy here:
https://www.hathitrust.org/sites/www.hathitrust.org/files/fee-model-change-proposal.pdf

@nfulkers will add info about systems & specific R1 schools in Confluence

format_and_overlap_determination.md

nfulkers · 2021-05-11T17:53:09Z

format_and_overlap_determination.md

+**SPM**
+: Is not MPM, SER/SPM, or SER.
+
+## Cost Allocation ##


I'm not sure how or where to address it, but isn't there something funky about how costs are allocated for university Systems like FLBOG and the Georgia State system?

There is no unique processing for FLBOG or the Georgia State system in this repository.

does that mean we need the system to be doing something that it currently doesn't do?

I don't think so. I think the system handles those appropriately. If there are some separate requirements documents for those systems we can walk through the logic and check.

The old system does not do anything different for these.
From its point of view they are 2 normal members.

So how does this get worked out in the legacy system? (from https://www.hathitrust.org/eligibility_agreements):
Systems with more than one institution classified as Carnegie R1 will pay a public domain fee for each R1 institution, tiered according to its total library expenditures. For systems without an R1 institution, the member will pay a public domain fee based on the system’s assignment to a tier that corresponds with the sum total expenditures of all institutions in the system. In-copyright fees are assessed for the entire system.

They have a weight in the old system, the same as in the new system. There is no special handling for system members in that the fee is always scaled by the weight; the thing that differs for system members (in both old and new systems) is the weight.

See above discussion about systems

nfulkers · 2021-05-11T19:59:41Z

format_and_overlap_determination.md

+
+For items in **clusters** that are not MPM, all organizations with a holding in the cluster and the billing entity for the item are allocated a share. 
+
+For items in **clusters** that are MPM, the process is more complicated. Organizations with holdings:


I don't get this section at all, which is a bummer since it seems like this is the crux of the document ¯_(ツ)_/¯

A topic of conversation for our weekly meeting. @mwarin @billdueber @aelkiss

What about:

For items in clusters that are not MPM (i.e. none of the items in the cluster have the item format multi-part monograph), an institutions is considered to hold the item and is allocated a share for it if:

It is the billing entity for the item (i.e. the depositor of an item is always assumed to hold it)

It submitted a holding that is in the cluster

For items in clusters that are MPM, the process is more complicated. An institution is considered to hold the item and is allocated a share if:

It is the billing entity for the item (i.e. the depositor of an item is always assumed to hold it)

It has a holding in the cluster with an n_enum matching the n_enum of an item.

It has a holding in the cluster that has an empty n_enum. The same is not true of items with empty n_enums: holdings with empty n_enums match all items in an MPM cluster; but items with empty n_enums match only holdings with n_enums.

It has holdings in the cluster, but none of the reported n_enums match any of the Item n_enums: that is, if no holdings n_enums match, then this is assumed to be a data problem, and the institution is considered to hold all the items in the cluster. [^2]

@nfulkers Does that address any of the confusion? I think it is important this section be as clear as possible. I don't know whether my option is any better. It may be helpful for you to try stating the behavior in your words as you understand it, and then seeing if there's any spots we need to correct or clarify.

I edited the above to replace 'organization' with 'institution'.

Might need some more introductory text to make it clear that we are not actually matching holdings -- we are figuring out who holds an item, which includes (but is not limited to) matching holdings to HT items

@nfulkers believes these revisions address the confusion

nfulkers · 2021-05-11T20:06:36Z

format_and_overlap_determination.md

+**NB**: Billing entities apply only to the Item they are on. For example, a billing entity on an HT Item with an empty n_enum will **not** match other items in the cluster.
+
+## Frequency Table ##
+Item shares are held in a frequency table separated by member and **item** format.[^3] This gets compiled into per format per member cost allocations. 


shares as in Hscore? If yes might be good to use the standard term, though I admit the sentence sounds weirder that way. If not, we should clarify what "share" means in this context.

"HScore" is terminology taken from the existing system, and it's a little awkward when "share" seems to be perfectly adequate. Changed to HScore all the same.

format_and_overlap_determination.md

nfulkers · 2021-05-20T19:52:57Z

format_and_overlap_determination.md

+
+## HT Item Overlap ##
+Determining who holds a particular in-copyright HT Item.
+


In the legacy system, I think, we somehow allocate a share for each copy of an item, if a member reports holding more than one copy?

Is that right and if so does the new system also do this?

@mwarin I know I've been confused on this point in the past. I don't think the old system does this, though. As far as I can tell this test addresses this in the new system: https://github.com/hathitrust/holdings-backend/blob/master/spec/cost_report_spec.rb#L222-L233 -- multiple reported holdings of the same OCN lead to only one allocated share of each matching HTItem

For fees it doesn't matter how many copies you hold (>0).

This makes sense considering that
the fee for each member associated with a volume
is multiplied by 1/H, where H is the number of members associated by that volume,
not the total number of copies (of the title) reported held.

Number of copies (of a title) reported held only matter for access (e.g. ETAS).

nfulkers · 2021-05-20T19:59:06Z

format_and_overlap_determination.md

+For HT Items in **clusters** that are not MPM, all organizations with a holding in the cluster and the billing entity for the HT Item are allocated a share. 
+
+For HT Items in **clusters** that are MPM, the process is more complicated. Organizations with holdings:
+- Organizations with holdings with empty **n_enum**. The reverse is **not** true; i.e. Items with empty n_enum don't match everything.


"Organizations with holdings with empty n_enum" ... what? What do they match to? Is there some kind of logic implied by the syntax or notation of this sentence that I'm missing?

In natural language line 108 feels like it should be written "If a member reports a holding that the system determines to be MPM AND the reported holding lacks enumeration, that holding will match to all other holdings in the same cluster that also lack n_enum"" (or whatever the true behavior actually is).

Or something like

"For HT items in clusters that are MPM, if the HT item lacks enumchron, each organization that submitted a record also lacking enumchron is allocated a share"

???

Does my proposed revision above #77 (comment) make this point any clearer?

@nfulkers believes my revisions above are sufficient

nfulkers · 2021-05-20T19:59:47Z

format_and_overlap_determination.md

+
+For HT Items in **clusters** that are not MPM, all organizations with a holding in the cluster and the billing entity for the HT Item are allocated a share. 
+
+For HT Items in **clusters** that are MPM, the process is more complicated. Organizations with holdings:


What is the "Organizations with holdings:" phrase meant to designate in line 107? I think I'm not understanding how the word "organizations" applies here.

Probably should be written as "members with holdings" or perhaps "institutions with holdings"?

aelkiss · 2021-05-28T14:08:48Z

re: the section on member weight, this google sheet is public-facing and describes how the PD weight is determined for system members (see the "2. System Members tab, Column H). So it is technically the ED who assigns the weight, but that determination is made based on a formula approved by the HT membership in 2019. Can we revise the section to reflect that (and include the link to the worksheet)?

I have edited the corresponding Confluence page to include these details as well.

We will add these links to the documentation

jsteverman force-pushed the HT-2898_format_overlap_documentation branch from 7726d6f to 0df1261 Compare April 30, 2021 19:10

jsteverman force-pushed the HT-2898_format_overlap_documentation branch from 0df1261 to c03517c Compare May 7, 2021 21:39

HT-2898: Format and Overlap determination documentation

3918acc

jsteverman force-pushed the HT-2898_format_overlap_documentation branch from c03517c to 3918acc Compare May 7, 2021 21:41

nfulkers reviewed May 10, 2021

View reviewed changes

format_and_overlap_determination.md Outdated Show resolved Hide resolved

nfulkers reviewed May 10, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 10, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 10, 2021

View reviewed changes

format_and_overlap_determination.md Outdated Show resolved Hide resolved

nfulkers reviewed May 10, 2021

View reviewed changes

format_and_overlap_determination.md Outdated Show resolved Hide resolved

nfulkers reviewed May 11, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 11, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 11, 2021

View reviewed changes

format_and_overlap_determination.md Outdated Show resolved Hide resolved

nfulkers reviewed May 11, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 11, 2021

View reviewed changes

Clarifications of format and overlap documentation

cb54c07

mwarin reviewed May 14, 2021

View reviewed changes

format_and_overlap_determination.md Show resolved Hide resolved

nfulkers reviewed May 20, 2021

View reviewed changes

Clarification of overlap determination process

1ffc207

jsteverman added 2 commits May 28, 2021 13:28

HT-2898 Add links to "weight" documentation

b8b4aa5

clarify MPM non-matching behavior

6f57544

aelkiss merged commit 8e375a9 into master Jun 11, 2021

aelkiss deleted the HT-2898_format_overlap_documentation branch June 11, 2021 14:07


		For items in clusters that are not MPM, all organizations with a holding in the cluster and the billing entity for the item are allocated a share.

		For items in clusters that are MPM, the process is more complicated. Organizations with holdings:


		## HT Item Overlap ##
		Determining who holds a particular in-copyright HT Item.


		For HT Items in clusters that are not MPM, all organizations with a holding in the cluster and the billing entity for the HT Item are allocated a share.

		For HT Items in clusters that are MPM, the process is more complicated. Organizations with holdings:

HT-2898: Format and Overlap determination documentation #77

HT-2898: Format and Overlap determination documentation #77

Conversation

jsteverman commented Apr 13, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfulkers May 20, 2021 • edited Loading

Choose a reason for hiding this comment

aelkiss May 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfulkers May 11, 2021 • edited Loading

Choose a reason for hiding this comment

jsteverman May 12, 2021 • edited Loading

Choose a reason for hiding this comment

aelkiss May 14, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nfulkers May 20, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aelkiss commented May 28, 2021

nfulkers May 20, 2021 •

edited

Loading

aelkiss May 20, 2021 •

edited

Loading

nfulkers May 11, 2021 •

edited

Loading

jsteverman May 12, 2021 •

edited

Loading

aelkiss May 14, 2021 •

edited

Loading

nfulkers May 20, 2021 •

edited

Loading