Fiona's new command line interface is a program named "fio".
Usage: fio [OPTIONS] COMMAND [ARGS]...
Fiona command line interface.
Options:
-v, --verbose Increase verbosity.
-q, --quiet Decrease verbosity.
--aws-profile TEXT Select a profile from the AWS credentials file
--aws-no-sign-requests Make requests anonymously
--aws-requester-pays Requester pays data transfer costs
--version Show the version and exit.
--gdal-version Show the version and exit.
--python-version Show the version and exit.
--help Show this message and exit.
Commands:
bounds Print the extent of GeoJSON objects
calc Calculate GeoJSON property by Python expression
cat Concatenate and print the features of datasets
collect Collect a sequence of features.
distrib Distribute features from a collection.
dump Dump a dataset to GeoJSON.
env Print information about the fio environment.
filter Evaluate pipeline expressions to filter GeoJSON features.
info Print information about a dataset.
insp Open a dataset and start an interpreter.
load Load GeoJSON to a dataset in another format.
ls List layers in a datasource.
map Map a pipeline expression over GeoJSON features.
reduce Reduce a stream of GeoJSON features to one value.
rm Remove a datasource or an individual layer.
It is developed using the click
package.
The bounds command reads LF or RS-delimited GeoJSON texts, either features or collections, from stdin and prints their bounds with or without other data to stdout.
With no options, it works like this:
$ fio cat docs/data/test_uk.shp | head -n 1 \
> | fio bounds
[0.735, 51.357216, 0.947778, 51.444717]
Using --with-id
gives you
$ fio cat docs/data/test_uk.shp | head -n 1 \
> | fio bounds --with-id
{"id": "0", "bbox": [0.735, 51.357216, 0.947778, 51.444717]}
The calc command creates a new property on GeoJSON features using the specified expression.
The expression is evaluated in a restricted namespace containing 4 functions (sum, pow, min, max), the math module, the shapely shape function, type conversions (bool, int, str, len, float), and an object f representing the feature to be evaluated. This f object allows access in javascript-style dot notation for convenience.
The expression will be evaluated for each feature and its return value will be added to the properties as the specified property_name. Existing properties will not be overwritten by default (an Exception is raised).
$ fio cat data.shp | fio calc sumAB "f.properties.A + f.properties.B"
Note
fio calc
requires installation of the "calc" set of extra requirements
that will be installed by pip install fiona[calc]
.
The cat command concatenates the features of one or more datasets and prints them as a JSON text sequence of features. In other words: GeoJSON feature objects, possibly pretty printed, optionally separated by ASCII RS (x1e) chars using --rs.
The output of fio cat
can be piped to fio load
to create new
concatenated datasets.
$ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \
> | fio load /tmp/double.shp --driver Shapefile
$ fio info /tmp/double.shp --count
96
$ fio info docs/data/test_uk.shp --count
48
The cat command provides optional methods to filter data, which are
different to the fio filter
tool.
A bounding box --bbox w,s,e,n
tests for a spatial intersection with
the geometries. An attribute filter --where TEXT
can use
an SQL WHERE clause.
If more than one datasets is passed to fio cat
, the attributes used
in the WHERE clause must be valid for each dataset.
The collect command takes a JSON text sequence of GeoJSON feature objects, such
as the output of fio cat
and writes a GeoJSON feature collection.
$ fio cat docs/data/test_uk.shp docs/data/test_uk.shp \
> | fio collect > /tmp/collected.json
$ fio info /tmp/collected.json --count
96
The inverse of fio-collect, fio-distrib takes a GeoJSON feature collection and writes a JSON text sequence of GeoJSON feature objects.
$ fio info --count tests/data/coutwildrnp.shp
67
$ fio cat tests/data/coutwildrnp.shp | fio collect | fio distrib | wc -l
67
The dump command reads a vector dataset and writes a GeoJSON feature collection
to stdout. Its output can be piped to fio load
(see below).
$ fio dump docs/data/test_uk.shp --indent 2 --precision 2 | head
{
"features": [
{
"geometry": {
"coordinates": [
[
[
0.9,
51.36
],
You can optionally dump out JSON text sequences using --x-json-seq
. Since
version 1.4.0, fio cat
is the better tool for generating sequences.
$ fio dump docs/data/test_uk.shp --precision 2 --x-json-seq | head -n 2
{"geometry": {"coordinates": [[[0.9, 51.36], [0.89, 51.36], [0.79, 51.37], [0.78, 51.37], [0.77, 51.38], [0.76, 51.38], [0.75, 51.39], [0.74, 51.4], [0.73, 51.41], [0.74, 51.43], [0.75, 51.44], [0.76, 51.44], [0.79, 51.44], [0.89, 51.42], [0.9, 51.42], [0.91, 51.42], [0.93, 51.4], [0.94, 51.39], [0.94, 51.38], [0.95, 51.38], [0.95, 51.37], [0.95, 51.37], [0.94, 51.37], [0.9, 51.36], [0.9, 51.36]]], "type": "Polygon"}, "id": "0", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}
{"geometry": {"coordinates": [[[-4.66, 51.16], [-4.67, 51.16], [-4.67, 51.16], [-4.67, 51.17], [-4.67, 51.19], [-4.67, 51.19], [-4.67, 51.2], [-4.66, 51.2], [-4.66, 51.19], [-4.65, 51.16], [-4.65, 51.16], [-4.65, 51.16], [-4.66, 51.16]]], "type": "Polygon"}, "id": "1", "properties": {"AREA": 244820.0, "CAT": 232.0, "CNTRY_NAME": "United Kingdom", "FIPS_CNTRY": "UK", "POP_CNTRY": 60270708.0}, "type": "Feature"}
The info command prints information about a dataset as a JSON object.
$ fio info docs/data/test_uk.shp --indent 2
{
"count": 48,
"crs": "+datum=WGS84 +no_defs +proj=longlat",
"driver": "ESRI Shapefile",
"bounds": [
-8.621389,
49.911659,
1.749444,
60.844444
],
"schema": {
"geometry": "Polygon",
"properties": {
"CAT": "float:16",
"FIPS_CNTRY": "str:80",
"CNTRY_NAME": "str:80",
"AREA": "float:15.2",
"POP_CNTRY": "float:15.2"
}
}
}
You can process this JSON using, e.g., underscore-cli.
$ fio info docs/data/test_uk.shp | underscore extract count
48
You can also optionally get single info items as plain text (not JSON) strings
$ fio info docs/data/test_uk.shp --count
48
$ fio info docs/data/test_uk.shp --bounds
-8.621389 49.911659 1.749444 60.844444
The load command reads GeoJSON features from stdin and writes them to a vector dataset using another format.
$ fio dump docs/data/test_uk.shp \
> | fio load /tmp/test.shp --driver Shapefile
This command also supports GeoJSON text sequences. RS-separated sequences will
be detected. If you want to load LF-separated sequences, you must specify
--x-json-seq
.
$ fio cat docs/data/test_uk.shp | fio load /tmp/foo.shp --driver Shapefile
$ fio info /tmp/foo.shp --indent 2
{
"count": 48,
"crs": "+datum=WGS84 +no_defs +proj=longlat",
"driver": "ESRI Shapefile",
"bounds": [
-8.621389,
49.911659,
1.749444,
60.844444
],
"schema": {
"geometry": "Polygon",
"properties": {
"AREA": "float:24.15",
"CNTRY_NAME": "str:80",
"POP_CNTRY": "float:24.15",
"FIPS_CNTRY": "str:80",
"CAT": "float:24.15"
}
}
}
The underscore-cli process command is another way of turning a GeoJSON feature collection into a feature sequence.
$ fio dump docs/data/test_uk.shp \
> | underscore process \
> 'each(data.features,function(o){console.log(JSON.stringify(o))})' \
> | fio load /tmp/test-seq.shp --x-json-seq --driver Shapefile
For each feature read from stdin, filter evaluates a pipeline of one or more steps described using methods from the Shapely library in Lisp-like expressions. If the pipeline expression evaluates to True, the feature passes through the filter. Otherwise the feature does not pass.
For example, this pipeline expression
$ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
| fio filter '< (distance g (Point -109.0 38.5)) 100'
lets through all features that are less than 100 meters from the given point and filters out all other features.
New in version 1.10: these parenthesized list expressions.
The older style Python expressions like
'f.properties.area > 1000.0'
are deprecated and will not be supported in version 2.0.
Note this tool is different from fio cat --where TEXT ...
, which provides
SQL WHERE clause filtering of feature attributes.
Note
fio filter
requires installation of the "calc" set of extra requirements
that will be installed by pip install fiona[calc]
.
For each feature read from stdin, fio map
applies a transformation pipeline and
writes a copy of the feature, containing the modified geometry, to stdout. For
example, polygonal features can be roughly "cleaned" by using a buffer g 0
pipeline.
$ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
| fio map 'buffer g 0'
New in version 1.10.
Note
fio map
requires installation of the "calc" set of extra requirements
that will be installed by pip install fiona[calc]
.
Given a sequence of GeoJSON features (RS-delimited or not) on stdin this prints
a single value using a provided transformation pipeline. The set of geometries
of the input features in the context of these expressions is named c
.
For example, the pipeline expression
$ fio cat zip+https://s3.amazonaws.com/fiona-testing/coutwildrnp.zip \
| fio reduce 'unary_union c'
dissolves the geometries of input features.
New in version 1.10.
Note
fio reduce
requires installation of the "calc" set of extra requirements
that will be installed by pip install fiona[calc]
.
The rm command deletes an entire datasource or a single layer in a multi-layer datasource. If the datasource is composed of multiple files (e.g. an ESRI Shapefile) all of the files will be removed.
$ fio rm countries.shp
$ fio rm --layer forests land_cover.gpkg
fio filter
, fio map
, and fio reduce
expressions take the form of
parenthesized lists that may contain other expressions. The first item in a
list is the name of a function or method, or an expression that evaluates to a
function. The second item is the function's first argument or the object to
which the method is bound. The remaining list items are the positional and
keyword arguments for the named function or method. The list of functions and
callables available in an expression includes:
- Python operators such as
+
,/
, and<=
- Python builtins such as
dict
,list
, andmap
- All public functions from itertools, e.g.
islice
, andrepeat
- All functions importable from Shapely 2.0, e.g.
Point
, andunary_union
- All methods of Shapely geometry classes
- Functions specific to Fiona
Expressions are evaluated by fiona.features.snuggs.eval()
. Let's look at
some examples using that function.
Note
The outer parentheses are not optional within snuggs.eval()
.
Note
snuggs.eval()
does not use Python's builtin eval()
but isn't intended
to be a secure computing environment. Expressions which access the
computer's filesystem and create new processes are possible.
bool()
>>> snuggs.eval('(bool 0)')
False
range()
>>> snuggs.eval('(range 1 4)')
range(1, 4)
list()
>>> snuggs.eval('(list (range 1 4))')
[1, 2, 3]
Values can be bound to names for use in expressions.
>>> snuggs.eval('(list (range start stop))', start=0, stop=5)
[0, 1, 2, 3, 4]
Here's an example of using itertools.repeat()
.
>>> snuggs.eval('(list (repeat "*" times))', times=6)
['*', '*', '*', '*', '*', '*']
Here's an expression that evaluates to a Shapely Point instance.
>>> snuggs.eval('(Point 0 0)')
<POINT (0 0)>
The expression below evaluates to a MultiPoint instance.
>>> snuggs.eval('(union (Point 0 0) (Point 1 1))')
<MULTIPOINT (0 0, 1 1)>
The fio CLI introduces four new functions not available in Python's standard
library, or Shapely: collect()
, dump()
, identity()
, and
vertex_count()
.
The collect()
function turns a list of geometries into a geometry
collection and dump()
does the inverse, turning a geometry collection into
a sequence of geometries.
>>> snuggs.eval('(collect (Point 0 0) (Point 1 1))')
<GEOMETRYCOLLECTION (POINT (0 0), POINT (1 1))>
>>> snuggs.eval('(list (dump (collect (Point 0 0) (Point 1 1))))')
[<POINT (0 0)>, <POINT (1 1)>]
The identity()
function returns its single argument.
>>> snuggs.eval('(identity 42)')
42
To count the number of vertices in a geometry, use vertex_count()
.
>>> snuggs.eval('(vertex_count (Point 0 0))')
1
The area()
, buffer()
, distance()
, length()
, simplify()
, and
set_precision()
functions shadow, or override, functions from the shapely
module. They automatically reproject geometry objects from their natural
coordinate reference system (CRS) of OGC:CRS84
to EPSG:6933
so that the
shapes can be measured or modified using meters as units.
buffer()
dilates (or erodes) a given geometry, with coordinates in decimal
longitude and latitude degrees, by a given distance in meters.
>>> snuggs.eval('(buffer (Point 0 0) :distance 100)')
<POLYGON ((0.001 0, 0.001 0, 0.001 0, 0.001 0, 0.001 -0.001, 0.001 -0.001, 0...>
The area()
and length()
of this polygon have units of square meter and
meter.
>>> snuggs.eval('(area (buffer (Point 0 0) :distance 100))')
31214.451487413342
>>> snuggs.eval('(length (buffer (Point 0 0) :distance 100))')
627.3096977558143
The distance()
between two geometries is in meters.
>>> snuggs.eval('(distance (Point 0 0) (Point 0.1 0.1))')
15995.164946207413
A geometry can be simplified to a tolerance value in meters using
simplify()
. There are more examples of this function later in this
document.
>>> snuggs.eval('(simplify (buffer (Point 0 0) :distance 100) :tolerance 100)')
<POLYGON ((0.001 0, 0 -0.001, -0.001 0, 0 0.001, 0.001 0))>
The set_precision()
function snaps a geometry to a fixed precision grid
with a size in meters.
>>> snuggs.eval('(set_precision (Point 0.001 0.001) :grid_size 500)')
<POINT (0 0)>
fio filter
and fio map
evaluate expressions in the context of a GeoJSON
feature and its geometry attribute. These are named f
and g
. For example,
here is an expression that tests whether the input feature is within 62.5
kilometers of the given point.
< (distance g (Point 4 43)) 62.5E3
fio reduce
evaluates expressions in the context of the sequence of all input
geometries, named c
. For example, this expression dissolves input
geometries using Shapely's unary_union
.
unary_union c
The fio cat
command can optionally transform feature geometries to a new
coordinate reference system specified with --dst_crs
. The fio collect
command can optionally transform from a coordinate reference system specified
with --src_crs
to the default WGS84 GeoJSON CRS. Like collect, fio load
can accept non-WGS84 features, but as it can write files in formats other than
GeoJSON, you can optionally specify a --dst_crs
. For example, the WGS84
features read from docs/data/test_uk.shp,
$ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \
> | fio collect --src_crs EPSG:3857 > /tmp/foo.json
make a detour through EPSG:3857 (Web Mercator) and are transformed back to WGS84 by fio cat. The following,
$ fio cat docs/data/test_uk.shp --dst_crs EPSG:3857 \
> | fio load --src_crs EPSG:3857 --dst_crs EPSG:4326 --driver Shapefile \
> /tmp/foo.shp
does the same thing, but for ESRI Shapefile output.
The following examples use the program jq
and a 25-feature shapefile. You
can get the data from from rmnp.zip or access it in
a streaming fashion as shown in the examples below.
The builtin vertex_count()
function, in conjunction with fio map
's
--raw
option, prints out the number of vertices in each feature. The
default for fio-map is to wrap the result of every evaluated expression in
a GeoJSON feature; --raw
disables this. The program jq
provides a nice
way of summing the sequence of numbers.
fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
| fio map 'vertex_count g' --raw \
| jq -s 'add'
28915
Here's what the RMNP wilderness patrol zones features look like in QGIS.
One traditional way of simplifying an area of interest is to buffer and
simplify. There's no need to use jq
here because fio reduce
prints out
a sequence of exactly one feature. The effectiveness of this method depends
a bit on the nature of the data, especially the distance between vertices. The
total length of the perimeters of all zones is 889 kilometers.
fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
| fio map 'length g' --raw \
| jq -s 'add'
889332.0900809917
The mean distance between vertices on the edges of zones is 889332 / 28915, or 30.7 meters. You need to buffer and simplify by this value or more to get a significant reduction in the number of vertices. Choosing 40 as a buffer distance and simplification tolerance results in a shape with 469 vertices. It's a suitable area of interest for applications that require this number to be less than 500.
fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
| fio reduce 'unary_union c' \
| fio map 'simplify (buffer g 40) 40' \
| fio map 'vertex_count g' --raw
469
Convex hulls are an easy means of simplification as there are no distance
parameters to tweak. The --dump-parts
option of fio map
turns the parts of
multi-part features into separate single-part features. This is one of the ways
in which fio-map can multiply its inputs, printing out more features than it
receives.
fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
| fio map 'convex_hull g' --dump-parts \
| fio reduce 'unary_union c' \
| fio map 'vertex_count g' --raw
157
Convex hulls simplify, but also dilate concave areas of interest. They fill the "bays", so to speak, and this can be undesirable. Concave hulls do a better job at preserving the concave nature of a shape and result in a smaller increase of area.
fio cat zip+https://github.com/Toblerity/Fiona/files/14749922/rmnp.zip \
| fio map 'concave_hull g :ratio 0.4' --dump-parts \
| fio reduce 'unary_union c' \
| fio map 'vertex_count g' --raw
301