feat: exclude observation join id from standard and author categories

[joyceyan]

https://app.zenhub.com/workspaces/single-cell-5e2a191dad828d52cc78b028/issues/gh/chanzuckerberg/single-cell/597

I was able to produce a CXG with observation_joinid in it: liver_observation_joinid.cxg. The first step was to process liver.h5ad (this is just the smallest dataset we have in prod, downloaded as h5ad) with the schema 4 CLI, to annotate it with observation_joinid:

import anndata
import numpy as np
import pandas as pd
from backend.layers.processing.h5ad_data_file import H5ADDataFile

# Pull h5ad into AnnData
h5ad = H5ADDataFile("liver.h5ad")
adata = anndata.read_h5ad('liver.h5ad')

# Manipulate adata. We need to remove all of the labels we add as part of CXG conversion
del adata.uns["schema_version"]
del adata.var["feature_name"]
del adata.var["feature_reference"]
del adata.var["feature_biotype"]
del adata.raw.var["feature_name"]
del adata.raw.var["feature_reference"]
del adata.raw.var["feature_biotype"]
del adata.obs["cell_type"]
del adata.obs["assay"]
del adata.obs["disease"]
del adata.obs["organism"]
del adata.obs["sex"]
del adata.obs["tissue"]
del adata.obs["self_reported_ethnicity"]
del adata.obs["development_stage"]
adata.obs["CNA_value"] = "test-string"
adata.obs["tissue_type"] = pd.Categorical(["tissue"] * len(adata.obs))
adata.obs["tissue_type"] = adata.obs["tissue_type"].astype("category")

# Write to new h5ad
adata.write_h5ad("liver_fixed.h5ad")

Then, I validate it with the main version of cellxgene-schema, which will add the schema 4 labels.

cellxgene-schema validate liver_fixed.h5ad -a liver_output.h5ad

Next, we just need to convert it to a CXG:

# Pull h5ad into AnnData
h5ad = H5ADDataFile("liver_output.h5ad")
adata = anndata.read_h5ad('liver_output.h5ad')
# Verify that this has observation_joinid on it
print(adata.obs["observation_joinid"])
h5ad.to_cxg(output_cxg_directory="liver_observation_joinid.cxg", sparse_threshold=100)

Then, I just added liver_observation_joinid.cxg to single-cell-explorer/example-dataset.

This is what Explorer shows in main, which has observation_joinid as the 5th author category:

And after it's filtered on this branch:

[codecov]

Codecov Report

Merging #674 (9b161b9) into main (64316e0) will not change coverage.
The diff coverage is n/a.

❗ Current head 9b161b9 differs from pull request most recent head 904adcf. Consider uploading reports for the commit 904adcf to get more accurate results

@@           Coverage Diff           @@
##             main     #674   +/-   ##
=======================================
  Coverage   77.67%   77.67%           
=======================================
  Files          88       88           
  Lines        6772     6772           
=======================================
  Hits         5260     5260           
  Misses       1512     1512           

Flag	Coverage Δ
frontend	77.67% <ø> (ø)
javascript	77.67% <ø> (ø)
smokeTest	?
unitTest	77.67% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[prathapsridharan]

lgtm