December 22 update (#466)

* Update NEWS.md * Changed typo in substance misuse cohort assignment * Add change to cohorts to NEWS * Fixes for GPOOH smrtype (#436) * Bring 2014/15 code up to date Missed at the last update * Update SMRtype for GP-OOH Missed at the last update. I categorised the COVID types but didn't include these in smrtype. * Carry across some git/GitHub changes from `main-R` * Fix sds option4 bug - SPSS (December 2022 Update) (#439) * Fix sds_end_date is earlier than sds_start_date by setting end_date to be the end of fyear (#424) Co-authored-by: Zihao Li <[email protected]> * Fix sds_end_date is earlier than sds_start_date by setting end_date to be the end of fyear (#424) Co-authored-by: Zihao Li <[email protected]> * Fix option 4 cases/styling Where all 3 options are missing/0 then this will be coded as option4 * Simplify code for fixing sds_option_4 * Deal with NA's and update logic * Convert to logical and deal with NA in one go Co-authored-by: Zihao Li <[email protected]> Co-authored-by: Zihao Li <[email protected]> Co-authored-by: James McMahon <[email protected]> * Update Macros for Dec22 update SPD and SIMD also updated, however, there is no zsav file available * Update geography macros * Amend HC year specific code Some of this code is now done in the All SC HC code and not needed here * Update IT reference * Add some extra tests to PIS Also improve formatting and lower case lots of variable names * Read new variables for PIS Keep and use the Paid items, and the Gross paid cost. * Update prescribing tests to compare new vs old * Rename PIS items variable in episode file * Update the individual file to use the new PIS variables * Add NEWS item for prescribing changes * Update variable names For running code successfully, these variables have changed in the SPD * Update GP OoH cost lookup I have updated the GP OoH costs with the latest numbers from the publication. In the spreadsheet I have uplifted the costs for those years (20/21, 21/22). The code needed updating to take into account that there are more years in the spreasheet now, and to generate costs for 22/23. * Update SPD var `UR2_2020` * Add Alter Type and update tidy`UR_2020` vars * Remove some redundant test code * Update to `urx_2020` in the individual file * Fix for datazone population lookup In the December 2022 update the datazone population lookup SPSS version had the wrong length variable for `datazone2011`. This seemed like the simplest fix but it will hopefully be redundant in future updates * Tweak tests for including `pis_paid_items` Small change for comparing the old file vs new file as this picks up `pis_dispensed_items` in the old file. This will need changing back at the next update * Fix bug - GP OoH cases being overcounted SPSS was counting `sysmis` as a case, so everyone was getting at least one case, and everyone with some GP OoHs was getting an extra case, unless they happened to **only** have GP OoH cases. * Add new variables to indiv for GP OoH COVID cases Added `ooh_covid_advice`, `ooh_covid_assesment` and `ooh_covid_other` * Temporarily add new OoH covid vars to indiv tests Can be removed at the next update * Remove `health_net_costincIncomplete` from indiv file I'm not sure how useful this ever was but now it only applies to DN which is only for limited years. * Remove the `HRI_LCA*_incDN` vars from indiv * Only include people with health activity in the HRI calculation * minor code tidy * Add additional indiv file tests for looking at the HRI distribution * Copy changes to HRI from 1920 to other years * Fix for prescribing tests being missed from zip * Make compression the maximum * Update anon chi add files * Reorder NEWS under a new GP OoH section. * Add updates regarding lookup changes to NEWS * Update NEWS with the release date Co-authored-by: shintoLamp <[email protected]> Co-authored-by: Jennit07 <[email protected]> Co-authored-by: Zihao Li <[email protected]> Co-authored-by: Zihao Li <[email protected]> Co-authored-by: Jennifer Thom <[email protected]>
Public-Health-Scotland · Dec 9, 2022 · 0035b82 · 0035b82
1 parent 5d420f4
commit 0035b82
Show file tree

Hide file tree

Showing 238 changed files with 2,112 additions and 1,522 deletions.
diff --git a/.github/workflows/R-CMD-check.yaml b/.github/workflows/R-CMD-check.yaml
@@ -0,0 +1,46 @@
+# Workflow derived from https://github.com/r-lib/actions/tree/v2/examples
+# Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help
+on:
+  push:
+    branches: [main-R]
+  pull_request:
+    branches: [main-R]
+
+name: R-CMD-check
+
+jobs:
+  R-CMD-check:
+    runs-on: ${{ matrix.config.os }}
+
+    name: ${{ matrix.config.os }} (${{ matrix.config.r }})
+
+    strategy:
+      fail-fast: false
+      matrix:
+        config:
+          - {os: ubuntu-latest,   r: '3.6'}
+          - {os: ubuntu-latest,   r: 'release'}
+
+    env:
+      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
+      R_KEEP_PKG_SOURCE: yes
+
+    steps:
+      - uses: actions/checkout@v3
+
+      - uses: r-lib/actions/setup-pandoc@v2
+
+      - uses: r-lib/actions/setup-r@v2
+        with:
+          r-version: ${{ matrix.config.r }}
+          http-user-agent: ${{ matrix.config.http-user-agent }}
+          use-public-rspm: true
+
+      - uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::rcmdcheck
+          needs: check
+
+      - uses: r-lib/actions/check-r-package@v2
+        with:
+          upload-snapshots: true
diff --git a/.github/workflows/document.yaml b/.github/workflows/document.yaml
@@ -15,7 +15,7 @@ jobs:
       GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
     steps:
       - name: Checkout repo
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           fetch-depth: 0
 

diff --git a/.github/workflows/lint.yaml b/.github/workflows/lint.yaml
@@ -13,17 +13,24 @@ jobs:
     env:
       GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
     steps:
-      - uses: actions/checkout@v2
+      - name: Checkout repo
+        uses: actions/checkout@v3
 
-      - uses: r-lib/actions/setup-r@v2
+      - name: Setup R
+        uses: r-lib/actions/setup-r@v2
         with:
           use-public-rspm: true
 
-      - uses: r-lib/actions/setup-r-dependencies@v2
+      - name: Install dependencies
+        uses: r-lib/actions/setup-r-dependencies@v2
         with:
           extra-packages: any::lintr
           needs: lint
 
-      - name: Lint
+      - name: Lint package functions and tests
         run: lintr::lint_package()
         shell: Rscript {0}
+
+      - name: Lint production scripts
+        run: lintr::lint_dir("Production_scripts/")
+        shell: Rscript {0}
diff --git a/.github/workflows/pkgdown.yaml b/.github/workflows/pkgdown.yaml
@@ -22,18 +22,27 @@ jobs:
       NETLIFY_AUTH_TOKEN: ${{ secrets.NETLIFY_AUTH_TOKEN }}
       NETLIFY_SITE_ID: ${{ secrets.NETLIFY_SITE_ID }}
       isPush: ${{ github.event_name == 'push' }}
-      isPR: ${{ github.event.issue.pull_request && (github.event.comment.author_association == 'MEMBER' || github.event.comment.author_association == 'OWNER') && startsWith(github.event.comment.body, '/pkgdown') }}
+      isPR: ${{ github.event.issue.pull_request && github.event.comment.body, '/pkgdown' }}
 
     steps:
-      - uses: actions/checkout@v2
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Checkout Pull Request
+        if: contains(env.isPR, 'true')
+        run: hub pr checkout ${{ github.event.issue.number }}
+        env:
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
 
       - uses: r-lib/actions/setup-pandoc@v2
 
-      - uses: r-lib/actions/setup-r@v2
+      - name: Setup R
+        uses: r-lib/actions/setup-r@v2
         with:
           use-public-rspm: true
 
-      - uses: r-lib/actions/setup-r-dependencies@v2
+      - name: Install dependencies
+        uses: r-lib/actions/setup-r-dependencies@v2
         with:
           extra-packages: any::pkgdown, local::.
           needs: website
@@ -47,26 +56,25 @@ jobs:
           git config --local user.name "$GITHUB_ACTOR"
           git config --local user.email "[email protected]"
           Rscript -e 'pkgdown::deploy_to_branch(new_process = FALSE)'
-          
+
       - name: Create website
         if: contains(env.isPR, 'true')
         run: |
           pkgdown::build_site()
         shell: Rscript {0}
-        
+
       - name: Create index file
         if: contains(env.isPR, 'true')
         run: |
           echo '<!DOCTYPE html><html><head><meta http-equiv="refresh" content="0;URL=/dev/index.html" />  <script language="javascript"> window.location.replace('/dev/index.html')</script></head></html>' > ./docs/index.html
-      
+
       - name: Deploy to Netlify
         if: contains(env.isPR, 'true')
         id: netlify-deploy
-        uses: nwtgck/actions-netlify@v1.2      
+        uses: nwtgck/actions-netlify@v1
         with:
           publish-dir: './docs'
-          production-branch: main-R          
-          github-token: ${{ secrets.GITHUB_TOKEN }}
+          production-branch: main-R          github-token: ${{ secrets.GITHUB_TOKEN }}
           deploy-message:
             'Deploy from GHA: ${{ github.event.pull_request.title || github.event.head_commit.message }} (${{ github.sha }})'
-          timeout-minutes: 1
+          timeout-minutes: 12
diff --git a/.github/workflows/style.yaml b/.github/workflows/style.yaml
@@ -15,7 +15,7 @@ jobs:
       GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
     steps:
       - name: Checkout repo
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           fetch-depth: 0
 
@@ -54,19 +54,22 @@ jobs:
             ${{ runner.os }}-styler-
             ${{ runner.os }}-
 
-      - name: Style standard pacakge files
+      - name: Style standard package functions and tests
         run: styler::style_pkg(filetype = c(".R", ".Rmd", ".Rmarkdown", ".Rnw"))
         shell: Rscript {0}
 
       - name: Style production scripts
         run: styler::style_dir("Production_scripts/")
         shell: Rscript {0}
 
-
       - name: Style other scripts
         run: styler::style_dir("Make_R_files/")
         shell: Rscript {0}
 
+      - name: Style older scripts
+        run: styler::style_dir("All_years/")
+        shell: Rscript {0}
+
       - name: Commit and push changes
         uses: stefanzweifel/git-auto-commit-action@v4
         with:

diff --git a/.github/workflows/test-coverage.yaml b/.github/workflows/test-coverage.yaml
@@ -0,0 +1,34 @@
+# Workflow derived from https://github.com/r-lib/actions/tree/v2/examples
+# Need help debugging build failures? Start at https://github.com/r-lib/actions#where-to-find-help
+on:
+  push:
+    branches: [main-R]
+  pull_request:
+    branches: [main-R]
+
+name: test-coverage
+
+jobs:
+  test-coverage:
+    runs-on: ubuntu-latest
+    env:
+      GITHUB_PAT: ${{ secrets.GITHUB_TOKEN }}
+
+    steps:
+      - name: Checkout repo
+        uses: actions/checkout@v3
+
+      - name: Setup R
+        uses: r-lib/actions/setup-r@v2
+        with:
+          use-public-rspm: true
+
+      - name: Install dependencies
+        uses: r-lib/actions/setup-r-dependencies@v2
+        with:
+          extra-packages: any::covr
+          needs: coverage
+
+      - name: Test coverage
+        run: covr::codecov(quiet = FALSE)
+        shell: Rscript {0}
diff --git a/.gitignore b/.gitignore
@@ -8,13 +8,60 @@ Make\ SLFs/.Rproj.user/*
 *.docx
 #Ignore SPSS job files
 *.spj
+
 #Ignore MS Office files
 *.xls
 *.xlsx
 *.doc
 *.docx
+
+# SPSS output
 *.spv
-.Rproj.user
-*.Rhistory
 Make_R_files/log.txt
+
+# History files
+.Rhistory
+.Rapp.history
+
+# Session Data files
+.RData
+
+# User-specific files
+.Ruserdata
+
+# Example code in package build process
+*-Ex.R
+
+# Output files from R CMD build
+/*.tar.gz
+
+# Output files from R CMD check
+/*.Rcheck/
+
+# RStudio files
+.Rproj.user/
+
+# produced vignettes
+vignettes/*.html
+vignettes/*.pdf
+
+# OAuth2 token, see https://github.com/hadley/httr/releases/tag/v0.3
+.httr-oauth
+
+# knitr and R markdown default cache directories
+*_cache/
+/cache/
+
+# Temporary files created by R markdown
+*.utf8.md
+*.knit.md
+
+# R Environment Variables
+.Renviron
+
+# html files / ouputs
 *.html
+
+Make SLFs/log.txt
+.Rproj.user
+docs
diff --git a/2014-15/B01 Process Acute (SMR01 and GLS) extract.sps b/2014-15/B01 Process Acute (SMR01 and GLS) extract.sps
@@ -373,4 +373,4 @@ erase file = !Year_dir + 'acute_temp.zsav'.
 erase file = !Year_dir + 'acute_monthly_costs_and_beddays_by_uri.sav'.
 
  * zip up the raw data.
-Host Command = ["gzip " + !Year_Extracts_dir + "Acute-episode-level-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "Acute-episode-level-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B02 Process Maternity (SMR02) extract.sps b/2014-15/B02 Process Maternity (SMR02) extract.sps
@@ -295,4 +295,4 @@ save outfile = !Year_dir + 'maternity_for_source-20' + !FY + '.zsav'
 get file = !Year_dir + 'maternity_for_source-20' + !FY + '.zsav'.
 
 * zip up the raw data.
-Host Command = ["gzip " + !Year_Extracts_dir + "Maternity-episode-level-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "Maternity-episode-level-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B03 Process Outpatients (SMR00) extract.sps b/2014-15/B03 Process Outpatients (SMR00) extract.sps
@@ -186,4 +186,4 @@ save outfile = !Year_dir + 'outpatients_for_source-20'+!FY+'.zsav'
 get file = !Year_dir + 'outpatients_for_source-20' + !FY + '.zsav'.
 
 * zip up the raw data.
-Host Command = ["gzip " + !Year_Extracts_dir + "Outpatients-episode-level-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "Outpatients-episode-level-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B04 Process Mental Health (SMR04) extract.sps b/2014-15/B04 Process Mental Health (SMR04) extract.sps
@@ -322,4 +322,4 @@ get file = !Year_dir + 'mental_health_for_source-20' + !FY + '.zsav'.
 erase file = !Year_dir + 'mh_temp.zsav'.
 erase file = !Year_dir + "MH_monthly_costs_and_beddays_by_uri.sav".
 
-Host Command = ["gzip " + !Year_Extracts_dir + "Mental-Health-episode-level-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "Mental-Health-episode-level-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B05a Process A&E (AE2) extract.sps b/2014-15/B05a Process A&E (AE2) extract.sps
@@ -586,4 +586,4 @@ save outfile = !Year_dir + 'a&e_data-20' + !FY + '.zsav'
 get file = !Year_dir + 'a&e_data-20' + !FY + '.zsav'.
 
 * Zip up raw data.
-Host Command = ["gzip " + !Year_Extracts_dir + "A\&E-episode-level-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "A\&E-episode-level-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B05b Link CUP marker to A&E.sps b/2014-15/B05b Link CUP marker to A&E.sps
@@ -103,4 +103,4 @@ get file = !Year_dir + 'a&e_for_source-20' + !FY + '.zsav'.
 Erase file = !Year_dir + 'a&e_data-20' + !FY + '.zsav'.
 
 * Zip up raw data.
-Host Command = ["gzip " + !Year_Extracts_dir + "A\&E-UCD-CUP-extract-20" + !FY + ".csv"].
+Host Command = ["gzip -v9 " + !Year_Extracts_dir + "A\&E-UCD-CUP-extract-20" + !FY + ".csv"].
diff --git a/2014-15/B06 Process PIS extract.sps b/2014-15/B06 Process PIS extract.sps
@@ -1,14 +1,14 @@
-* Encoding: UTF-8.
+* Encoding: UTF-8.
 
 ********************************************************************************************************.
 * Run 01-Set up Macros first!.
 ********************************************************************************************************.
 *This should unzip the PIS extract in the IT extract directory.
 * !PIS_extract_file is dependant on the macro !Extract_Number which can be found in A01b 'Year' Macro and the number should be specific to FY. 
-Host Command = ["gunzip " + !PIS_extract_file_OLD].
+Host Command = ["gunzip " + !PIS_extract_file].
 
 GET DATA  /TYPE=TXT
-    /FILE=!PIS_extract_file_OLD
+    /FILE=!PIS_extract_file
     /ENCODING='UTF8'
     /DELIMITERS=","
     /QUALIFIER='"'
@@ -20,9 +20,12 @@ GET DATA  /TYPE=TXT
     PatGender F1.0
     PatPostcode A8
     PracticeCode A6
-    NumberofDispensedItems F2.0
+    NumberofDispensedItems F4.0
     DIPaidNICexcl.BB F6.2
     DIPaidGICexcl.BB F6.2
+    NumberofPaidItems F4.0
+    PDPaidNICexcl.BB F6.2
+    PDPaidGICexcl.BB F6.2
     /MAP.
 CACHE.
 
@@ -32,8 +35,9 @@ rename variables
     PatGender = gender
     PatPostcode = postcode
     PracticeCode = gpprac
+    NumberofPaidItems = no_paid_items
     NumberofDispensedItems = no_dispensed_items
-    DIPaidNICexcl.BB = cost_total_net.
+    PDPaidGICexcl.BB = cost_total_net.
 
 select if chi ne "".
 
@@ -56,10 +60,10 @@ compute year = !FY.
 sort cases by chi.
 
 save outfile = !Year_dir + "prescribing_file_for_source-20" + !FY + ".zsav"
-    /Drop DIPaidGICexcl.BB
+    /Drop DIPaidGICexcl.BB DIPaidNICexcl.BB PDPaidNICexcl.BB
     /zcompressed.
 
 get file = !Year_dir + "prescribing_file_for_source-20" + !FY + ".zsav".
 
 * zip raw data back up.
-Host Command = ["gzip " + !PIS_extract_file_OLD].
+Host Command = ["gzip -v9 " + !PIS_extract_file].