Strong Towns Langley's DevelopmentActivity project seeks to use real-time data processing tools such as Kafka and the ELK Stack to track and analyze the Development Activity in the Township of Langley, allowing the tracking of new developments, the rate of new developments and applications, approval times, and so on, which will be derived from the static data snapshots available on the Township of Langley's open data portal.
In particular this will be used to track the uptake and success of the pending local implementation of British Columbia's Bill 44 SSMUH Multiplex legislation as data is being tracked beginning in April 2024, with implementation scheduled for June 2024.
This module is written in C# .NET 6.0 and uses the Confluent.Kafka library to send Development Activity data to a Kafka instance and the Elastic.Clients.Elasticsearch library to send data to an ElasticSearch instance. It downloads the latest Development Activity data in JSON format, compares it with the last data in the Kafka topic and/or ElasticSearch index, and if it has changed, updates them with the latest data.
- Install the .NET 6.0 runtime for linux
- Upload the source code to a folder on the server and cd to the folder
- Run dotnet build
- Place config.xml in binary folder (bin/debug/net6.0)
- Run dotnet run (you may wish to do this in screen to allow disconnecting/reconnecting)
This project is written by James Hansen.
The Township of Langley releases this data with no license requirements or restrictions on their OpenData Portal. They also provide an API to read the data in JSON format as detailed on the About Page for the Development Activity Status Table.
Currently this API URL is https://services5.arcgis.com/frpHL0Fv8koQRVWY/arcgis/rest/services/Development_Activity_Status_Table/FeatureServer/1/query?outFields=*&where=1%3D1&f=json The "f" parameter is shown on the about page as geojson, but as their is no geojson data such as polygon information, we can change this to json and also receive all the column specifications as well as the data.
The producer is complete and operational, running on our StrongTownsLangley.org VPS server alongside the Apache Kafka and ElasticSearch / Kibana instance.
Demo access to the Kibana for live viewing of the data coming soon.