Choose a station:
+ + + + + + +Python example +-------------- + +In this example we will use several standard Python libraries and the +popular plotting library `matplotlib' + + '' + + _("Hide Search Matches") + + "
" + ) + ); + }, + + /** + * helper function to hide the search marks again + */ + hideSearchWords: () => { + document + .querySelectorAll("#searchbox .highlight-link") + .forEach((el) => el.remove()); + document + .querySelectorAll("span.highlighted") + .forEach((el) => el.classList.remove("highlighted")); + localStorage.removeItem("sphinx_highlight_terms") + }, + + initEscapeListener: () => { + // only install a listener if it is really needed + if (!DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS) return; + + document.addEventListener("keydown", (event) => { + // bail for input elements + if (BLACKLISTED_KEY_CONTROL_ELEMENTS.has(document.activeElement.tagName)) return; + // bail with special keys + if (event.shiftKey || event.altKey || event.ctrlKey || event.metaKey) return; + if (DOCUMENTATION_OPTIONS.ENABLE_SEARCH_SHORTCUTS && (event.key === "Escape")) { + SphinxHighlight.hideSearchWords(); + event.preventDefault(); + } + }); + }, +}; + +_ready(() => { + /* Do not call highlightSearchWords() when we are on the search page. + * It will highlight words from the *previous* search query. + */ + if (typeof Search === "undefined") SphinxHighlight.highlightSearchWords(); + SphinxHighlight.initEscapeListener(); +}); diff --git a/api.html b/api.html new file mode 100644 index 000000000..2c9a1d125 --- /dev/null +++ b/api.html @@ -0,0 +1,145 @@ + + + + + + + +Application Programming Interface for HiSPARC Public Database
+The API simplifies data access for data contained in the +HiSPARC Public Database. It was born out of the +need for easy access to up-to-date information about stations.
+Contents:
+The HiSPARC API (Application Programming Interface) simplifies +metadata access from other applications. In this tutorial, we’ll give +some examples of how this data can be accessed and used with Javascript +(jQuery) and Python. We’ll show you how to do some neat things. +How can you get a list of all HiSPARC stations in Denmark? What is the +position of station 201? Which stations had data on 20 October 2010? How +does the number of measured events change during a few weeks? This +tutorial will give you an overview of some of possibilities with the +HiSPARC API. For details on all available classes and methods, +please see the API Reference.
+Note
+We’ll require you to know some basic programming, i.e. to understand
+what an if statement is and for loop does. If you
+are new to coding you can try a tutorial online, for instance
+Codeacademy, we recommend learning
+Python or jQuery.
First we will just look at what this API is. The API can be accessed +via the internet by opening urls. Instead of a website you get data as a +response. This data is formatted as a JSON (JavaScript Object Notation), +this format can be understood by many programming languages.
+To see what options the API has we will look at it in a browser. Open +the following link in your browser (this will not work in Internet +Explorer): https://data.hisparc.nl/api/.
+You should now see some text, like this:
+
+ {"base_url": "https://data.hisparc.nl/api/",
+ "clusters": "clusters/",
+ "clusters_in_country": "countries/{country_id}/",
+ "configuration": "station/{station_id}/config/{year}/{month}/{day}/",
+ "countries": "countries/",
+ "has_data": "station/{station_id}/data/{year}/{month}/{day}/",
+ ...
+ "subclusters_in_cluster": "clusters/{cluster_id}/"}
+This is the JSON, it is a dictionary (indicated by the { and
+} enclosing brackets): an object which has keys and values. Each
+key ("clusters", "has_data") refers to a value
+("clusters/",
+"station/{station_id}/data/{year}/{month}/{day}/").
This tells us that if we want a list of all clusters we need to use the
+clusters option by appending "clusters/" to the base url,
+resulting in the following:
+https://data.hisparc.nl/api/clusters/.
With this result:
+[{"name": "Amsterdam",
+ "number": 0},
+ {"name": "Utrecht",
+ "number": 1000},
+ ...
+ {"name": "Karlsruhe",
+ "number": 70000}]
+This JSON is a list (indicated by the [ and ] enclosing
+brackets) of dictionaries, one for each cluster. Each dictionary
+contains the name and number of a cluster. This way information about
+the network of stations can be retrieved.
The following code example will generate a webpage which will use the
+API to get an up-to-date list of stations. It will then show a
+drop-down menu from which a station can be selected, once you have
+chosen a station you can click the Get info button to make
+Javascript get the station information. To try this you can either use
+this example page: jsFiddle or create
+your own HTML file with this code:
<html>
+<head>
+<script src="http://code.jquery.com/jquery-1.11.1.min.js"></script>
+<script>
+ $(function() {
+ // Get an up-to-date list of HiSPARC stations
+ $.getJSON(
+ 'https://data.hisparc.nl/api/stations/',
+ function(data) {
+ // Create the drop-down menu
+ var select = $('<select>');
+ var id, name;
+ for (var i in data) {
+ id = data[i].number;
+ name = data[i].name;
+ select.append($('<option>').attr('value', id).text(id + ' - ' + name));}
+ $('#station_list').append(select);});
+
+ // Attach a function to the Get info button
+ $('#get_station').on('click', function() {
+ var id = $('#station_list').find('select').val();
+ // Get info for selected station and display it in a nice way
+ $.getJSON('https://data.hisparc.nl/api/station/' + id + '/',
+ function(data) {
+ $('#station_info').text(JSON.stringify(data, undefined, 4));
+ });
+ });
+ });
+</script>
+</head>
+<body style="font-family: sans-serif;">
+ <h2>Station info</h2>
+ <p id="station_list">Choose a station: </p>
+ <input type="submit" id="get_station" value="Get info">
+ <pre id="station_info"></pre>
+</body>
+</html>
+In this example we will use several standard Python libraries and the +popular plotting library matplotlib (pylab). +We start by importing the required libraries, one to get data from the +urls, one to make working with dates easy and the plotting library. Then +define the start values and perpare two empty lists in which the data +can be placed. Then a while loop is used to loop over all days between +datum and end_datum, reading each corresponding url. Finally a plot is +made, setting the dates against their values.
+Start Python and type (or copy/paste without the >>>) the
+following lines of code:
>>> from urllib.request import urlopen
+>>> from datetime import date, timedelta
+>>> from pylab import plot, show
+>>> id = 501
+>>> datum = date(2010, 10, 1)
+>>> end_datum = date(2011, 2, 1)
+>>> base = 'https://data.hisparc.nl/api/station/%d/num_events/%d/%d/%d'
+>>> events = []
+>>> dates = []
+>>> while datum < end_datum:
+... url = urlopen(base % (id, datum.year, datum.month, datum.day))
+... events.append(url.read())
+... dates.append(datum)
+... datum += timedelta(days=1)
+...
+>>> step(dates, events)
+>>> show()
+The HiSPARC Python framework SAPPHiRE includes an API module. This module +simplifies access to the API. See the SAPPHiRE documentation for more +information: +https://docs.hisparc.nl/sapphire/.
+Get cluster list
+Retrieve a list of all clusters or only the clusters in a specific country. +By cluster we here mean the main clusters, which contain subclusters.
+country_number – a country number identifier, give this to only get +clusters from a specific country.
+list of dictionaries containing the name and number of all +clusters that matched the given parameters.
+Get station config settings
+Retrieve the entire configuration of a station. If no date if given the +latest config will be sent, otherwise the latest on or before the given +date.
+station_number – a station number identifier.
date – the date for which to get a configuration.
dictionary containing the entire configuration from +the HiSPARC DAQ.
+Get country list
+Retrieve a list of all countries.
+list of dictionaries containing the name and number of +all countries.
+Get the traces for an event
+station_number – a station number identifier.
ext_timestamp – extended timestamp (nanoseconds since UNIX epoch).
raw – (optional, GET) if present get the raw trace, i.e. without +subtracted baseline.
two or four traces.
+Return list of station numbers and names
+For all non-test stations in the given subcluster
+Check for presence of cosmic ray data
+Find out if the given station has measured shower data, either on a +specific date, or at all.
+station_number – a stationn number identifier.
type – the data type: events, singles or weather.
year,month,date – the date, this has to be within the time +HiSPARC has been operational and can be as specific as +you desire.
boolean, True if the given station has data, False otherwise.
+Create a json HTTPResponse
+Give overview of the possible urls
+Get status of the network
+dictionary containing status info for each station.
+Get number of events for a station
+Retrieve the number of events that a station has measured during its +entire operation or during a specific period, which can be a year, +month, day or an hour.
+station_number – a stationn number identifier.
year,month,date,hour – the date, this has to be within the time +HiSPARC has been operational and can be as specific as you desire.
integer containing the total number of events ever recorded by +the given station.
+Get station info
+Retrieve general information about a station. If no date if given +the latest valid info will be sent, otherwise the latest on or +before the given date.
+station_number – a station number identifier.
date – the date for which to get station info.
dictionary containing info about the station. Most importantly, +this contains information about the location of the station GPS +and the relative locations of the individual scintillators.
+Get station list
+Retrieve a list of all stations or all stations in a subcluster.
+subcluster_number – a subcluster number identifier. If given, only +stations belonging to that subcluster will be included in the list.
+list containing dictionaries which consist of the name and number +of each station (matching the subcluster).
+Get stations with event or weather data
+Retrieve a list of all stations which have recorded events, singles or +weather data in the given year, month, day or at all.
+type – data type to check for: events, singles or weather.
year,month,date – the date, this has to be within the time +HiSPARC has been operational and can be as specific as you desire.
list of dictionaries containing the name and number of each +station that has measured weather data in the given year.
+Get subcluster list
+Retrieve a list of all subclusters or all subclusters in a specific +cluster.
+cluster_number – a cluster number identifier, give this to only get +subclusters from this cluster.
+list of dictionaries containing the name and number of all +subclusters that matched the given parameters.
+Check if date is outside HiSPARC project range
+If not valid, a 404 (Not Found) should be returned to the user.
+boolean, True if the date is in the range, False otherwise.
+Contents:
+There are several ways in which the HiSPARC data can be accessed:
+Via the Public database download forms.
Via Python using the SAPPHiRE framework (see SAPPHiRE Tutorial).
Via the jSparc web applications.
To access metadata, like a list of all stations or information about a +specific station see the API Tutorial.
+When looking at the data page for a station (i.e. Kaj Munk College), you will see a ‘Download +event summary data’ link on the right. When this link is clicked you +will be taken to the Data download form. On this page you can select +the station, the start and end date for which you want to download the +data. There is also an option to download weather data instead of events, +however, not all stations gather weather data.
+When you click the Submit button without checking the checkbox to +Download, the data should (Firefox always downloads the data) show up in +your browser window. If you check the Download box the data will be +downloaded to your PC as a csv file.
+This csv file can be read by many programs, including Excel. Use the +Import option in Excel to make it recognize tabs as delimiter between +columns.
+Note
+An easy to use module has been added to SAPPHiRE to download +data from the ESD. The following is an old (but working) example.
+This is an example of how to download the data from the Public database +in Python. In this example we will download 1 hour of data for station +202 on July 2, 2013.
+First load the required libraries (requires the numpy package).
+>>> from datetime import datetime
+>>> from urllib import urlencode
+>>> from urllib.request import urlopen
+>>> from StringIO import StringIO
+>>> from numpy import genfromtxt
+Then define the url and place the start and end datetime in the query. +To download weather data instead, replace ‘events’ by ‘weather’ in +the url (and choose a station that has weather data, e.g. 3 or 501).
+Note
+Do not pass the query to the urlopen ‘data’ argument because that +changes the request into a POST request, but you need a GET +request.
+>>> url = 'https://data.hisparc.nl/data/202/events'
+>>> start = str(datetime(2013, 7, 2, 11, 0))
+>>> end = str(datetime(2013, 7, 2, 12, 0))
+>>> query = urlencode({'download': False, 'start': start,'end': end})
+>>> full_url = url + '?' + query
+Download the data and store it in a variable
+>>> data = urlopen(full_url).read()
+Now use numpy to convert the data from csv to a numpy array.
+>>> format = [('date', 'datetime64[D]'), ('time', '|S8'),
+ ('timestamp', 'uint32'), ('nanoseconds', 'uint32'),
+ ('pulseheights', '4int16'), ('integrals', '4int32'),
+ ('n1', 'float32'), ('n2', 'float32'),
+ ('n3', 'float32'), ('n4', 'float32'),
+ ('t1', 'float32'), ('t2', 'float32'),
+ ('t3', 'float32'), ('t4', 'float32'),
+ ('t_trigger', 'float32'),
+ ('zenith', 'int16'), ('azimuth', 'int16')]
+>>> a = genfromtxt(StringIO(data), delimiter="\t", dtype=format)
+>>> print(a[0])
+(datetime.date(2013, 7, 2), '11:00:02', 1372762802L, 466307811L,
+ [78, 798, -1, -1], [535, 10882, -1, -1],
+ 0.14720000326633453, 3.854599952697754, -1.0, -1.0,
+ 345.0, 12.5, -1.0, -1.0, 345.0, -999, -999)
+| + | + |
| + | + |
| + |
| + | + |
|
+
| + | + |
| + | + |
| + |
| + |
Each HiSPARC station is equipped with a GPS antenne. This GPS is used +to for time synchronization between stations and to determine the exact +location of each station. All these locations are stored in our +database. The GPS positions can be found on the data pages of stations, +via the API (API Reference) and in the raw data.
+Using the Leafet library with CARTO maps tiles (based on OpenStreetMap data) we are able to visualize the +detector network by showing the locations of the stations on a map.
+Here is an overview of the network: Stations on map
+When you click on a station marker a popup will show information about +that station, its name and cluster. If the station has data the name of +the station will be a link to its data page.
+The stations on the map can have one of 4 colors to indicate the current +status of that station.
+Green: when a station is operating properly
Yellow: it is responding to the server but has a problem
Red: it is completely unresponsive (offline)
Gray: it is is no longer active or the status can’t be determined.
On several public database pages a map is embedded to show the station +location or provide an overview of multiple stations. Moreover, maps are +also used on our main website to give an overview of the station +organization and clustering (e.g. Bristol, Science +Park +). This is accomplished by placing an iframe on those pages that +shows another page which only has the map as its content. Example code:
+<iframe src="https://data.hisparc.nl/maps/Netherlands/Amsterdam/Science%20Park/"
+ scrolling="no" frameborder="0" width="600" height="300"></iframe>
+Result:
+To show a map of a specific region or location, use the syntax explained +here. First start with the base url:
+https://data.hisparc.nl/maps/
+When no extra options are given the page zooms and positions the map +such that all stations fit in the window. But you can also focus on a +specific region or station. Several levels of regions are possible:
+https://data.hisparc.nl/maps/[Country]/[Cluster]/[Subcluster]/
+https://data.hisparc.nl/maps/[Station number]/
+An overview of countries, clusters and subclusters can be found on +https://data.hisparc.nl/ . First you can choose to focus on a Country:
+ +Then focus more closely on a cluster, note that you also need to give +the country:
+And to focus on a subcluster, also specifying the country and cluster:
+ +Finally you can also focus on one specific station by simply giving its +station number:
+ +The HiSPARC Public Database is the interface through which everyone can access +the data and our station administration is done.
+Contents:
+| + p | ||
| + |
+ publicdb | + |
| + |
+ publicdb.api | + |
| + |
+ publicdb.api.views | + |
| + |
+ publicdb.status_display | + |
| + |
+ publicdb.status_display.views | + |
+ Searching for multiple words only shows matches that contain + all words. +
+ + + + + + + + + +Each HiSPARC station has a GPS antenne and either 2 or 4 scintillator +detectors. This GPS is used to determine the exact location of the +station. However, the GPS is often not at the center of the station, +moreover, not all stations are oriented the same way, and the distances +between detector may differ. For reconstruction of the measured data it +is important to know the location of each detector, this section +explains how the location of the detectors can be measured and +communicated to us.
+The coordinate system we have chosen for describing the position of +detectors is illustrated in the following figure.
+
+
+For each detector 3 (or 4) coordinates need to be determined:
+First the distance (r) from the GPS to the center of the
+scintillator (in meters).
Next the alpha angle (α) which is the clock-wise turning angle
+between North and the detector as seen from the GPS (in degrees).
A height (z) coordinate can be measured if there is a
+significant altitude difference between the GPS antenna and the
+detectors, or if the detectors are not at the same height. Up is
+positive (in meters).
The rotation of the detector is described by the beta angle
+(β), which is the clock-wise turning rotation of the long side
+of the detector relative to North (in degrees).
For more information about the coordinate systems used in HiSPARC see: +Coordinate systems and units in HiSPARC.
+For Dutch schools we have an assignment sheet in the infopakket which +walks students through the process of measuring the station layout: +De stationsplattegrond.
+New layouts can be submitted via the layout submit form. A verification e-mail will be sent +to the submitter. The submitted measurements will be reviewed before they are +made available on the website and via the API.
+The detector coordinates can be accessed via the API. +These can, for example, be used when analysing data or when making a schematic +drawing of the station layout.
+Station Status Display
+The Status Display provides webpages that display summaries of data measured +by stations.
+Contents:
+
LatestNetworkSummaryRedirectView
+LatestSummaryRedirectView
+NetworkSummaryDetailViewNetworkSummaryDetailView.get_context_data()NetworkSummaryDetailView.get_object()NetworkSummaryDetailView.get_queryset()NetworkSummaryDetailView.http_method_namesNetworkSummaryDetailView.nav_calendar()NetworkSummaryDetailView.nav_months()NetworkSummaryDetailView.nav_years()NetworkSummaryDetailView.template_nameSummaryDetailView
+create_plot_object()get_config_source()get_dataset_source()get_detector_timing_offsets()get_detector_timing_offsets_source()get_eventtime_histogram_sources()get_eventtime_source()get_focus()get_gpslocations()get_histogram_source()get_specific_config_source()get_specific_dataset_source()get_specific_histogram_source()get_specific_network_histogram_source()get_station_layout_source()get_station_timing_offsets()get_station_timing_offsets_source()help()none_to_nan()plot_config()plot_dataset()plot_histogram()plot_timing_offsets()station_config()station_has_config()station_has_data()station_latest()station_status()stations()stations_by_country()stations_by_name()stations_by_number()stations_by_status()stations_on_map()stations_with_data()Show most recent coincidence data page
+Constructor. Called in the URLconf; can contain helpful extra +keyword arguments, and other things.
+Return the URL redirect to. Keyword arguments from the URL pattern +match generating the redirect request are provided as kwargs to this +method.
+Show most recent data for a particular station
+Constructor. Called in the URLconf; can contain helpful extra +keyword arguments, and other things.
+Return the URL redirect to. Keyword arguments from the URL pattern +match generating the redirect request are provided as kwargs to this +method.
+Constructor. Called in the URLconf; can contain helpful extra +keyword arguments, and other things.
+Insert the single object into the context dict.
+Get the object this request displays.
+Return the QuerySet that will be used to look up the object.
+This method is called by the default implementation of get_object() and +may not be called if get_object() is overridden.
+Create a month calendar with links
+Create list of months with links
+Create list of previous years
+Constructor. Called in the URLconf; can contain helpful extra +keyword arguments, and other things.
+Insert the single object into the context dict.
+Get the object this request displays.
+Return the QuerySet that will be used to look up the object.
+This method is called by the default implementation of get_object() and +may not be called if get_object() is overridden.
+Create a month calendar with links
+Create list of months with links
+Create list of previous years
+Get configuration data for a specific station
+station_number – station for which to get the configuration data.
type – the type of configuration data to get. The following +are supported: voltage, current, gps, trigger.
list of lists containing the configuration history.
+Get a dataset for a specific date and station
+date – the date for which to get the dataset.
type – the type of dataset to retrieve.
station_number – the station to which the data belongs.
list of tuples containing (x, y) pairs.
+Get all eventtime data from start to end
+Get all valid GPS locations from the configs
+Get histogram data for a specific date
+date – the date for which to get the histogram data.
type – the type of histogram to retrieve.
station_number – if None a NetworkHistogram is looked for, otherwise +a DailyHistogram for a specific station is looked for.
list of tuples containing (bin, value) pairs.
+Get a station histogram for a specific date
+Get all station timing offsets for a station pair
+ref_station_number,station_number – station numbers.
+list of tuples with date, offset, and error.
+Show the static help page
+Create a plot object from station configs
+Create a dataset plot object
+Create a histogram object
+Create a plot object from station configs
+Show configuration history for a particular station
+Check if there is a valid configuration for the given station
+station – Station object for which to check.
+boolean indicating if the station has a configuration available.
+Check if there is valid event or weather data for the given station
+station – Station object for which to check.
+boolean indicating if the station has recorded data, either +weather or shower, between 2002 and now.
+Show daily histograms for a particular station
+Show data status for a particular station
+Show the default station list
+Show a list of stations, ordered by country, cluster and subcluster
+Show a list of stations, ordered by station name
+Show a list of stations, ordered by number
+Show a list of stations, ordered by status
+Show all stations from a subcluster on a map
+Get list of station numbers with valid event or weather data
+list with station numbers for stations that recorded data, either +weather or shower, between 2004 and now.
+Contents:
+LatestNetworkSummaryRedirectView
+LatestSummaryRedirectView
+NetworkSummaryDetailViewNetworkSummaryDetailView.get_context_data()NetworkSummaryDetailView.get_object()NetworkSummaryDetailView.get_queryset()NetworkSummaryDetailView.http_method_namesNetworkSummaryDetailView.nav_calendar()NetworkSummaryDetailView.nav_months()NetworkSummaryDetailView.nav_years()NetworkSummaryDetailView.template_nameSummaryDetailView
+create_plot_object()get_config_source()get_dataset_source()get_detector_timing_offsets()get_detector_timing_offsets_source()get_eventtime_histogram_sources()get_eventtime_source()get_focus()get_gpslocations()get_histogram_source()get_specific_config_source()get_specific_dataset_source()get_specific_histogram_source()get_specific_network_histogram_source()get_station_layout_source()get_station_timing_offsets()get_station_timing_offsets_source()help()none_to_nan()plot_config()plot_dataset()plot_histogram()plot_timing_offsets()station_config()station_has_config()station_has_data()station_latest()station_status()stations()stations_by_country()stations_by_name()stations_by_number()stations_by_status()stations_on_map()stations_with_data()