Import / Export of Dataset or site lists

[philippkraft]

Sometimes datasets need to be created in bulk, eg. distributed sampling at many places with multiple value types. Using a spreadsheet can make this simpler to repeat similar information.

[philippkraft]

• Export Sites (no filter)
• Export datasets (with filter)
• Import sites
• Import datasets

[philippkraft]

Idea for Import:

class DatasetImportAction(FileAction):
    name = 'import-dataset'
    icon = 'clipboard'
    title = 'dataset'
    tooltip = 'Import datasets'

    def href(self, path: Path):
        return conf.url('/download/to_db/dataset', filename=path.name)

    def check(self, path: Path):
        return False
        try:
            df = pd.read_excel(path.absolute, nrows=1)
            return all(colname in df.columns for colname in ['start', 'end', 'site', 'valuetype', 'project'])    
        
        except (OSError, ValueError):
            return False

[philippkraft]

Another idea:

• Add to datasetlist.html an upload form like in download.html directly under the new dataset button
• Add to datasetpage.py a file receiving function that does read_excel and checks the columns (id optional)

Same for sites

download.html

odmf/odmf/static/templates/download.html

Lines 300 to 322 in 647f114

	<form id="upload-form" method="post" enctype="multipart/form-data" action="${conf.root_url}/download/upload" class="form-group">
	<div class="input-group mb-3">
	<div class="input-group-prepend">
	<button type="submit" class="btn btn-secondary">
	<i class="fas fa-file-upload fa-lg" />
	</button>
	</div>
	<div class="custom-file">
	<input type="file" class="custom-file-input form-control"
	name="datafiles" id="datafile" multiple="multiple"
	/>
	<label id="upload-label" class="custom-file-label" for="datafile">upload...</label>
	</div>
	<input name="dir" type="hidden" value="${curdir}" />
	<div class="input-group-append">
	<div class="input-group-text">
	<input id="overwrite" type="checkbox" value="overwrite" name="overwrite"
	data-toggle="tooltip" title="Allow to overwrite existing file with the same name"
	/>
	</div>
	</div>
	</div>
	</form>

Receiving function

odmf/odmf/webpage/filemanager/upload.py

Lines 170 to 209 in 647f114

	@expose_for(Level.editor)
	@web.method.post_or_put
	def upload(self, dir, datafiles, **kwargs):
	"""
	Uploads a list of files. Make sure the upload element is like
	<input type="file" multiple="multiple"/>
	Parameters
	----------
	dir
	The target directory
	datafiles
	The datafiles to upload
	kwargs
	"""
	error = []
	msg = []
	# Loop over incoming datafiles. Single files need some special treatment
	for datafile in (list(datafiles) or [datafiles]):
	path = Path(dir)
	if not path:
	path.make()
	fn = path + datafile.filename
	if not fn.islegal():
	error.append(f'{fn.name} is not legal')
	elif fn and 'overwrite' not in kwargs:
	error.append(f"'{fn.name}' exists already, if you want to overwrite the old version, check allow overwrite")
	else:
	# Buffer file for first check encoding and secondly upload file
	with BytesIO(datafile.file.read()) as filebuffer:
	try:
	write_to_file(fn.absolute, filebuffer)
	msg.append(f'- {fn.href} uploaded')
	except Exception as e:
	error.append(f'- {fn.href} upload failed: {e}')
	raise goto(dir, '\n'.join(error), '\n'.join(msg))