Skip to content

Commit

Permalink
Finalizes jlu-ilr-hydro#16
Browse files Browse the repository at this point in the history
  • Loading branch information
@cpwnd
cpwnd committed Mar 12, 2019
1 parent f6f262e commit 6cdede5
Show file tree
Hide file tree
Showing 11 changed files with 133 additions and 223 deletions.
12 changes: 6 additions & 6 deletions docs/build/html/_sources/odmf/about.md.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,12 @@ TBD by Philipp

## Features of ODMF

* Upload measurement data for future use
* Data is persisted in an Postgres database in special schema structure
* Analyze data via the platform or export in csv or plot format
* Upload measurement data for future use e.g. analyze and download
* Data is persisted in an Postgres database in specific schema structure
* Analyze data via the platform, export it in CSV format and plot an image
* Organization of different user roles within the online system
* Augment measurement data gathered from the work in the field via meta data, such as logs and pictures
* Integration of Google Maps UI panel
* Integration of data into Google Maps interactive UI

Also it can be integrated into the [CUAHSI](https://www.cuahsi.org) network and used with the tools provided by
the CUAHSI organization. For example to inspect or download data via the [Data.CUAHSI](https://data.cuahsi.org) page.
Expand All @@ -28,9 +28,9 @@ Mentales Gerüst (optional)

## CUAHSI WaterOneFlow Interface

To access the Schwingbach valley data with other software than the ODMF server, an interface is used.
To access the Schwingbach valley data with other software than the ODMF server, a special interface is used.
This interface is called WaterOneFlow interface. It makes public datasets accessible in the so called HIS Central network. All endpoints in this network can be accessed and utilized by tools, provided by the CUAHSI organziation.
The integration of the WOF interface into the ODMF server system is described in the further chapter.
The integration of the WOF interface into the ODMF server system is described in the following chapter.

### CUAHSI Tools

Expand Down
24 changes: 10 additions & 14 deletions docs/build/html/_sources/odmf/development.md.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -49,16 +49,6 @@ The code base is divided in three main parts. **Server**, **automated import** a

Utility code for calibration, conf and markdown parsing are saved in the `tools` module.

## Core thoughts behind design decisions
To connect more of the ideas under the hood of the server there are diagrams describing the user interaction for the most important functions of the platform.

* Schwingbach FMC (Detail)
- User interaction (WOF)
- User interaction (data import)
- ...
* CUAHSI WOF transactions diagram
* Views a. CV-Tables extending ODMF data schema

## Data

This chapter describes the database schema, the differences and limitations of the ORM mapping and finally extending
Expand Down Expand Up @@ -245,7 +235,7 @@ TBD

This is the most special way of uploading data. There are no real time series of data, rather
there are much more single data rows that are identified either with tuple of `dataset.valuetype`,
`dataset.site` and so on (Further informations :ref:`see dataset <schema-dataset>`), or with
`dataset.site` and so on (Further information :ref:`see dataset <schema-dataset>`), or with
an explicit `dataset.id`, to determine one matching dataset where the data row is appended to.

### XlsImport
Expand All @@ -262,15 +252,15 @@ is returned to the user, as part of an error message.

## Migration

Differences of the database schemas of ODMF (Observatory Data Management Framework) and ODM (Observation Data Model).
* odmf.dataset attributes start and end, can be identical in the rare case of a size of just one record.
Differences of the database schemata of ODMF (Observatory Data Management Framework) and ODM (Observation Data Model).

### WaterOneFlow

To understand the details of the *How* on the concrete implementation of the WaterOneFlow interface for ODMF software system, using a dedicated server.

* What parts communicate how
* SQL Views as mapping from ODMF schema to ODM schema
* How to configure a HydroServerLite instance to fit your needs

#### Architecture overview

Expand Down Expand Up @@ -329,7 +319,7 @@ schema constraints.

Items of `ODMF.dataset` are omitted when the following rules apply on them:

* `dataset.start = dataset.end`
* `dataset.start = dataset.end` (`odmf.dataset` attributes start and end, can be identical in the rare case of a size of just one record.)
* `dataset.access = 0`, data is only for internal use
* `dataset.project is NULL`

Expand Down Expand Up @@ -359,3 +349,9 @@ Different phases described:

The realized records cannot be queried by the ODMF software system. They are only visible to the WOF
API, because the ORM layer only accesses records with `dataset.type` that are non-`transformed_timeseries`.

### Configuration

An easy way of configuration is not possible at the moment, because the deployment of the HydroServerLite
is done via static SQL scripts. To configure the system, this scripts can be altered, but this is somehow
impractical, when done more than once.
81 changes: 28 additions & 53 deletions docs/build/html/_sources/odmf/installation.md.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,62 +4,28 @@

How to install the ODMF server software to maintain measurement data.

1. Clone github repository and install python dependencies
```
$ git clone https://github.com/jlu-ilr-hydro/odmf.git
$ cd odmf;
$ pip install -r requirements.txt # if you want a dev install
$ python setup.py install
```
or install via pip
```
$ pip install odmf
```
1. Clone the github repository and install python dependencies

After this create a directory, for later use we name it `odmf-server` and place a `conf.py` into it.
Your directory looks like this:
```
$ tree .
odmf-server/
└── conf.py
$ git clone https://github.com/jlu-ilr-hydro/odmf.git
$ cd odmf
$ pip install -r requirements.txt
$ cd odmf
$ cp conf-template.py conf.py
```
2. Install Postgres database

We recommend the latest version but at least 9.5
2. If you havn't already, install a postgres database.
We recommend the latest version but at least 9.5

```
$ sudo apt-get install postgresql-dev-9.5
```
3. Optional you can set up the HydroServerLite too, if you want to publish your data into the CUAHSI network.
See the HydroServerLite tutorial for this.

When all prerequesites are met, you navigate to the `odmf-server` folder and type:
```
$ odmf-install .
```
Note the `.` at the end.
If you set everything up correctly, this script should set up (1) you initial database contents and (2) the folder
structure for your server-files.

Your directory should now look like this:
```
$ tree .
odmf-server/
├── conf.py
└── webpage
├── datafiles
├── preferences
└── sessions

```

We have now finished the installation process and are able to start the server with:
```
$ odm-start
```

More details on the usage of the platform can be found in the [usage chapter](usage.html).
3. Change the config with the instructions from [conf.py](https://jlu-ilr-hydro.github.io/odmf/source/conf.py) wiki page. When the configuration is changed to meet your requirements, start the server and browse to https://localhost:8080

```
$ python3 start.py
```
### Configuration

Describes more sophisticated server configurations via one file.
Expand All @@ -86,21 +52,30 @@ parts of the ODMF installation.
* **CFG_DATETIME_DEFAULT_TIMEZONE**: Timezone in pytz compatible format
* **CFG_MEDIA_IMAGE_PATH**: Path relative to server script. See [usage](usage.html#media-folder)
* **CFG_MANUAL_MEASUREMENTS_PATTERN**: Regular expression pattern, where in the relative path of datafiles
([see usage](usage.html#datafiles-folder)) manual measurements files are stored. This folder and the subfolders get
a special treatment from dataimport/mm.py
([see usage](usage.html#datafiles-folder)) manual measurements files are stored. This folder and the subfolders
get a special treatment from dataimport/mm.py
* **CFG_MAP_DEFAULT**: Default location where the map is pointing when hitting the landing page
* **CFG_UPLOAD_MAX_SIZE**: Maximum file size for files uploaded into the [download view](views.html#download).

Now your server is configured and you can start [using](usage.html) it.

## WaterOneFlow

Optional you can set up the HydroServerLite too, if you want to publish your data via the CUAHSI network using
the WaterOneFlow interface.

This covers the installation of the PHP server [HydroServerLite](https://github.com/CUAHSI/HydroServerLite)
and the deployment of SQL Views. This views are used on the side of the ODMF schema to map it to the relations which
are used with the HydroServer.
and the deployment of SQL Views. This views are used on the side of the ODMF schema to map it to the relations
which are used with the HydroServer.

### HydroServerLite

The code is downloaded from Github and exposed via an Apache instance.
If you set everything up correctly, this script should set up (1) you initial database contents and (2) the folder
structure for your server-files.

1. Migrate ODMF schema to ODM
2. Create SQL views, materialized views and helper tables
3. Register cron jobs for daily update of transformed timeseries

Possible alternative to PHP based server is this python solution [WOFpy](https://github.com/ODM2/WOFpy). It relies on
the ODM schema 2.0.
Possible alternative to PHP based server is this python solution [WOFpy](https://github.com/ODM2/WOFpy).
It relies on the ODM schema 2.0.
1 change: 0 additions & 1 deletion docs/build/html/index.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -205,7 +205,6 @@ <h1>ODMF Docs<a class="headerlink" href="#odmf-docs" title="Permalink to this he
</li>
<li class="toctree-l1"><a class="reference internal" href="odmf/development.html">Development</a><ul>
<li class="toctree-l2"><a class="reference internal" href="odmf/development.html#structure">Structure</a></li>
<li class="toctree-l2"><a class="reference internal" href="odmf/development.html#core-thoughts-behind-design-decisions">Core thoughts behind design decisions</a></li>
<li class="toctree-l2"><a class="reference internal" href="odmf/development.html#data">Data</a></li>
<li class="toctree-l2"><a class="reference internal" href="odmf/development.html#upload-or-dataimport">Upload or Dataimport</a></li>
<li class="toctree-l2"><a class="reference internal" href="odmf/development.html#migration">Migration</a></li>
Expand Down
12 changes: 6 additions & 6 deletions docs/build/html/odmf/about.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -172,12 +172,12 @@
<div class="section" id="features-of-odmf">
<span id="features-of-odmf"></span><h2>Features of ODMF<a class="headerlink" href="#features-of-odmf" title="Permalink to this headline"></a></h2>
<ul class="simple">
<li>Upload measurement data for future use</li>
<li>Data is persisted in an Postgres database in special schema structure</li>
<li>Analyze data via the platform or export in csv or plot format</li>
<li>Upload measurement data for future use e.g. analyze and download</li>
<li>Data is persisted in an Postgres database in specific schema structure</li>
<li>Analyze data via the platform, export it in CSV format and plot an image</li>
<li>Organization of different user roles within the online system</li>
<li>Augment measurement data gathered from the work in the field via meta data, such as logs and pictures</li>
<li>Integration of Google Maps UI panel</li>
<li>Integration of data into Google Maps interactive UI</li>
</ul>
<p>Also it can be integrated into the <a class="reference external" href="https://www.cuahsi.org">CUAHSI</a> network and used with the tools provided by
the CUAHSI organization. For example to inspect or download data via the <a class="reference external" href="https://data.cuahsi.org">Data.CUAHSI</a> page.</p>
Expand All @@ -194,9 +194,9 @@
</div>
<div class="section" id="cuahsi-wateroneflow-interface">
<span id="cuahsi-wateroneflow-interface"></span><h2>CUAHSI WaterOneFlow Interface<a class="headerlink" href="#cuahsi-wateroneflow-interface" title="Permalink to this headline"></a></h2>
<p>To access the Schwingbach valley data with other software than the ODMF server, an interface is used.
<p>To access the Schwingbach valley data with other software than the ODMF server, a special interface is used.
This interface is called WaterOneFlow interface. It makes public datasets accessible in the so called HIS Central network. All endpoints in this network can be accessed and utilized by tools, provided by the CUAHSI organziation.
The integration of the WOF interface into the ODMF server system is described in the further chapter.</p>
The integration of the WOF interface into the ODMF server system is described in the following chapter.</p>
<div class="section" id="cuahsi-tools">
<span id="cuahsi-tools"></span><h3>CUAHSI Tools<a class="headerlink" href="#cuahsi-tools" title="Permalink to this headline"></a></h3>
<p>Two official tools are provided, to request data from a HydroServer. So these tools can also be used to debug the server behaviour.</p>
Expand Down
32 changes: 11 additions & 21 deletions docs/build/html/odmf/development.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -93,7 +93,6 @@
<li class="toctree-l3"><a class="reference internal" href="#components">Components</a></li>
</ul>
</li>
<li class="toctree-l2"><a class="reference internal" href="#core-thoughts-behind-design-decisions">Core thoughts behind design decisions</a></li>
<li class="toctree-l2"><a class="reference internal" href="#data">Data</a><ul>
<li class="toctree-l3"><a class="reference internal" href="#odmf-schema">ODMF schema</a></li>
<li class="toctree-l3"><a class="reference internal" href="#orm-mapping">ORM mapping</a><ul>
Expand Down Expand Up @@ -142,6 +141,7 @@
</li>
<li class="toctree-l3"><a class="reference internal" href="#schema-mapping-validity">Schema mapping validity</a></li>
<li class="toctree-l3"><a class="reference internal" href="#daily-jobs">Daily jobs</a></li>
<li class="toctree-l3"><a class="reference internal" href="#configuration">Configuration</a></li>
</ul>
</li>
</ul>
Expand Down Expand Up @@ -256,20 +256,6 @@
<p>Utility code for calibration, conf and markdown parsing are saved in the <code class="docutils literal notranslate"><span class="pre">tools</span></code> module.</p>
</div>
</div>
<div class="section" id="core-thoughts-behind-design-decisions">
<span id="core-thoughts-behind-design-decisions"></span><h2>Core thoughts behind design decisions<a class="headerlink" href="#core-thoughts-behind-design-decisions" title="Permalink to this headline"></a></h2>
<p>To connect more of the ideas under the hood of the server there are diagrams describing the user interaction for the most important functions of the platform.</p>
<ul class="simple">
<li>Schwingbach FMC (Detail)<ul>
<li>User interaction (WOF)</li>
<li>User interaction (data import)</li>
<li>…</li>
</ul>
</li>
<li>CUAHSI WOF transactions diagram</li>
<li>Views a. CV-Tables extending ODMF data schema</li>
</ul>
</div>
<div class="section" id="data">
<span id="data"></span><h2>Data<a class="headerlink" href="#data" title="Permalink to this headline"></a></h2>
<p>This chapter describes the database schema, the differences and limitations of the ORM mapping and finally extending
Expand Down Expand Up @@ -443,7 +429,7 @@
<span id="upload-type-3-log-data"></span><h5>Upload type 3: Log data<a class="headerlink" href="#upload-type-3-log-data" title="Permalink to this headline"></a></h5>
<p>This is the most special way of uploading data. There are no real time series of data, rather
there are much more single data rows that are identified either with tuple of <code class="docutils literal notranslate"><span class="pre">dataset.valuetype</span></code>,
<code class="docutils literal notranslate"><span class="pre">dataset.site</span></code> and so on (Further informations :ref:<code class="docutils literal notranslate"><span class="pre">see</span> <span class="pre">dataset</span> <span class="pre">&lt;schema-dataset&gt;</span></code>), or with
<code class="docutils literal notranslate"><span class="pre">dataset.site</span></code> and so on (Further information :ref:<code class="docutils literal notranslate"><span class="pre">see</span> <span class="pre">dataset</span> <span class="pre">&lt;schema-dataset&gt;</span></code>), or with
an explicit <code class="docutils literal notranslate"><span class="pre">dataset.id</span></code>, to determine one matching dataset where the data row is appended to.</p>
</div>
</div>
Expand All @@ -462,16 +448,14 @@
</div>
<div class="section" id="migration">
<span id="migration"></span><h2>Migration<a class="headerlink" href="#migration" title="Permalink to this headline"></a></h2>
<p>Differences of the database schemas of ODMF (Observatory Data Management Framework) and ODM (Observation Data Model).</p>
<ul class="simple">
<li>odmf.dataset attributes start and end, can be identical in the rare case of a size of just one record.</li>
</ul>
<p>Differences of the database schemata of ODMF (Observatory Data Management Framework) and ODM (Observation Data Model).</p>
<div class="section" id="wateroneflow">
<span id="wateroneflow"></span><h3>WaterOneFlow<a class="headerlink" href="#wateroneflow" title="Permalink to this headline"></a></h3>
<p>To understand the details of the <em>How</em> on the concrete implementation of the WaterOneFlow interface for ODMF software system, using a dedicated server.</p>
<ul class="simple">
<li>What parts communicate how</li>
<li>SQL Views as mapping from ODMF schema to ODM schema</li>
<li>How to configure a HydroServerLite instance to fit your needs</li>
</ul>
<div class="section" id="architecture-overview">
<span id="architecture-overview"></span><h4>Architecture overview<a class="headerlink" href="#architecture-overview" title="Permalink to this headline"></a></h4>
Expand Down Expand Up @@ -526,7 +510,7 @@
schema constraints.</p>
<p>Items of <code class="docutils literal notranslate"><span class="pre">ODMF.dataset</span></code> are omitted when the following rules apply on them:</p>
<ul class="simple">
<li><code class="docutils literal notranslate"><span class="pre">dataset.start</span> <span class="pre">=</span> <span class="pre">dataset.end</span></code></li>
<li><code class="docutils literal notranslate"><span class="pre">dataset.start</span> <span class="pre">=</span> <span class="pre">dataset.end</span></code> (<code class="docutils literal notranslate"><span class="pre">odmf.dataset</span></code> attributes start and end, can be identical in the rare case of a size of just one record.)</li>
<li><code class="docutils literal notranslate"><span class="pre">dataset.access</span> <span class="pre">=</span> <span class="pre">0</span></code>, data is only for internal use</li>
<li><code class="docutils literal notranslate"><span class="pre">dataset.project</span> <span class="pre">is</span> <span class="pre">NULL</span></code></li>
</ul>
Expand Down Expand Up @@ -554,6 +538,12 @@
<p>The realized records cannot be queried by the ODMF software system. They are only visible to the WOF
API, because the ORM layer only accesses records with <code class="docutils literal notranslate"><span class="pre">dataset.type</span></code> that are non-<code class="docutils literal notranslate"><span class="pre">transformed_timeseries</span></code>.</p>
</div>
<div class="section" id="configuration">
<span id="configuration"></span><h3>Configuration<a class="headerlink" href="#configuration" title="Permalink to this headline"></a></h3>
<p>An easy way of configuration is not possible at the moment, because the deployment of the HydroServerLite
is done via static SQL scripts. To configure the system, this scripts can be altered, but this is somehow
impractical, when done more than once.</p>
</div>
</div>
</div>

Expand Down
Loading

0 comments on commit 6cdede5

Please sign in to comment.