Made all notes available from nested subdirectories inside /data

[Gedulis12]

Hi,

First of all amazing project, the moment i saw this i realized this is a perfect remote access solution for my current note system.

I understand that the goal of the project is for the application to be as simple as possible, however not being able to have deeper directory structure in /data is making this app unusable for users who are also managing their notes through git.

I've made a few simple changes which makes all of the markdown files available from any sub directory inside /data.
Please note that this does not clutter the front-end in any way, the titles are displayed as filename (without .md extension) in the web view, hence the simplicity aspect of the app is kept.

Fingers crossed for this to go through as i would finally have a note taking system that 100% fits my needs

[dullage]

Hi @Gedulis12, thanks for contributing. I appreciate that this change is aimed at keeping the simplicity in the front end. It does however add complexity to the back end so before I dig deeper into the changes, can I ask a couple of initial questions:

1. Could you please elaborate a little on how the subdirectory compatibility will help when using git.
2. If subdirectories are not exposed in the front end, how do the proposed changes deal with potential duplicate filenames (in different sub-directories)?

[Gedulis12]

Hi @dullage , thanks for the feedback.

To answer your questions:

Could you please elaborate a little on how the subdirectory compatibility will help when using git.

Say i already have a note taking system in place where i use markdown files and manage them through git. I might already have directory structure imposed for example:

.
├── ideas.md
├── school
│   ├── curriculum1.md
│   ├── curriculum2.md
│   └── curriculum3.md
├── shortcut-cheatsheet.md
├── useful-links.md
├── quick-gists
│   ├── 20230217.md
│   ├── 20230221.md
│   ├── 20230224.md
│   └── 20230322.md
└── somerandomenote.md

so if flatnotes would be compatible with subdirectories, a user would be able to simply git clone their repository in /data folder and have all notes accessible to read/edit through the flatnotes web UI and do git pull, git push when needed.

If, on the other hand compatability for subdirectories is not enabled, the user would have to manually move notes to the root directory which would break the structure in their git repository, which is not ideal if one wants to keep notes somewhat organized.

If subdirectories are not exposed in the front end, how do the proposed changes deal with potential duplicate filenames (in different sub-directories)?

I have checked and confirmed that when creating new note through web UI, if a note with the same name already exists anywhere under /data, the error is handled gracefully with a pop-up:

Duplicate ✘
A note with this title already exists. Please try again with a new title.

If, however i try to create a new .md file with an already existing name manually, web-ui does display both notes, however the content of the note gets duplicated through both of the notes.

I believe this can be addressed in a few ways:

• We could simply warn users about potential missbehaviour if multiple files exists with the same name;
• We could modify the titles of those notes that have matching filenames, for e.g. let's say we have two notes /data/school/quicknote.md and /data/personal/quicknote.md, they could both be displayed as school → quicknote and personal → quicknote in web-ui. (i could work on a PR if you agree on the approach);

---

I would also argue that complexity is not necessarily added for those users who simply want plug and play solution and don't care about the availability of nested directory structure, as the app does not impose it by default. The bit of complexity gets added only when the user decides that he needs such additional functionality.

Also another thing i could suggest is that nested subdirectories could be not compatible by default, but could be enabled in docker-compose through an optional environment variable. That way any additional complexity would be disabled by default.

Please let me know what you think and whether you would like me to work on further changes to address the points mentioned above.

[dullage]

Hi @Gedulis12. If I'm honest, I'm not keen on either of the proposed solutions. They both feel like a hack around the flat nature of the app.

That's not to say that I don't want to try and help here though. I can imagine there are others who are in a similar position and would like to use flatnotes and it would be great to have a solution.

What I'm thinking is, perhaps there is a way to achieve a flattened shadow copy of a multi-tiered directory structure outside of flatnotes. And then just back flatnotes onto that.

As an example, in theory, you could use inotify to create and destroy symlinks in a flat directory to point to files in a hierarchical directory.

Or perhaps this is possible with something like mergerfs or UnionFS?

[Gedulis12]

Hi @dullage , thanks for the feedback.

To be honest the proposed workarounds also seem very hacky in nature.

They would suggest that complexity which could be handled by the application would be shifted to be handled and maintained by the user himself. I personally would rather not use the application at all if i had dance around with symlinks, shadow copies or other additional layers.

On that note, do you think that there is any improvements that could be applied to address your concerns and make you accept the possibility to have notes accessible on nested directories?

In any case i'm prepared to maintain a forked version of the application, but i would rather have us reach a consensus :)

[dullage]

I did have another thought but I haven't yet worked out how it could be achieved or how much work would be involved.

What I'm thinking is, maybe flatnotes could support some kind of a bring-your-own-storage model where it would ship with a model to support the current functionality (a single flat directory in the local filesystem) but allow for other models to be used if wanted. Then, rather than maintaining a fork, you'd just need to maintain a plugin.

I'd have to separate out the logic a bit but the number of operations the plugin would need to support is minimal (list files, get a file, create a file etc).

The benefit of this is that it could potentially allow for other storage options such as S3 or GitHub.

[Gedulis12]

Wouldn't my previous suggestion work in order to achieve something like this?

Also another thing i could suggest is that nested subdirectories could be not compatible by default, but could be enabled in docker-compose through an optional environment variable. That way any additional complexity would be disabled by default.

The additional functionality could be enabled only if some environment variable is set to true, for example NESTED_DIRECTORIES_ENABLED: true in docker-compose.yml. This way any additional functionality could be easily separated from the main default version of an app.

[dullage]

Sorry for the late reply. The difference in the plugin approach I mentioned is that the nested directories code would be external to this repo. Although the solution works, it is still a bit of a workaround and as such, not something I'd really like to support as a 1st party feature.

[benooye]

Just to let you know, I'm also very interrested by the support of existing folder structure .
It would be very great to use the very nice and clean interface of flatnotes with existing MD files folders without the need to rearrange the files
Thanks a lot for the project!

[Gedulis12]

@benooye As a workaround i myself am currently using the forked version with support for subdirectories, you can use my version by changing the image in docker-compose.yml configuration to gedulis12/flatnotes:latest

[benooye]

@benooye As a workaround i myself am currently using the forked version with support for subdirectories, you can use my version by changing the image in docker-compose.yml configuration to gedulis12/flatnotes:latest

Thanks for the information I will try it

[ItsNoted]

Are there any thoughts about making this an official feature? It's a common practice in documentation apps like Docusaurus for instance. Organizing notes within the file structure using a folder system is really helpful. Thanks for all your efforts on the project!

[ItsNoted]

Hi @dullage , thanks for the feedback.

To answer your questions:

Could you please elaborate a little on how the subdirectory compatibility will help when using git.

Say i already have a note taking system in place where i use markdown files and manage them through git. I might already have directory structure imposed for example:

.
├── ideas.md
├── school
│   ├── curriculum1.md
│   ├── curriculum2.md
│   └── curriculum3.md
├── shortcut-cheatsheet.md
├── useful-links.md
├── quick-gists
│   ├── 20230217.md
│   ├── 20230221.md
│   ├── 20230224.md
│   └── 20230322.md
└── somerandomenote.md

so if flatnotes would be compatible with subdirectories, a user would be able to simply git clone their repository in /data folder and have all notes accessible to read/edit through the flatnotes web UI and do git pull, git push when needed.

If, on the other hand compatability for subdirectories is not enabled, the user would have to manually move notes to the root directory which would break the structure in their git repository, which is not ideal if one wants to keep notes somewhat organized.

If subdirectories are not exposed in the front end, how do the proposed changes deal with potential duplicate filenames (in different sub-directories)?

I have checked and confirmed that when creating new note through web UI, if a note with the same name already exists anywhere under /data, the error is handled gracefully with a pop-up:

Duplicate ✘
A note with this title already exists. Please try again with a new title.

If, however i try to create a new .md file with an already existing name manually, web-ui does display both notes, however the content of the note gets duplicated through both of the notes.

I believe this can be addressed in a few ways:

• We could simply warn users about potential missbehaviour if multiple files exists with the same name;
• We could modify the titles of those notes that have matching filenames, for e.g. let's say we have two notes /data/school/quicknote.md and /data/personal/quicknote.md, they could both be displayed as school → quicknote and personal → quicknote in web-ui. (i could work on a PR if you agree on the approach);

I would also argue that complexity is not necessarily added for those users who simply want plug and play solution and don't care about the availability of nested directory structure, as the app does not impose it by default. The bit of complexity gets added only when the user decides that he needs such additional functionality.

Also another thing i could suggest is that nested subdirectories could be not compatible by default, but could be enabled in docker-compose through an optional environment variable. That way any additional complexity would be disabled by default.

Please let me know what you think and whether you would like me to work on further changes to address the points mentioned above.

I gave your fork a try and it looks decent. However, there's still an issue with the organization on the flatnotes front end. It basically just lists all the .md files in all the folders without displaying the folders themselves or letting you navigate within them. Is there any way to better organize how this works on the front end? Maybe I am doing it wrong?

[dullage]

Are there any thoughts about making this an official feature? It's a common practice in documentation apps like Docusaurus for instance. Organizing notes within the file structure using a folder system is really helpful. Thanks for all your efforts on the project!

Unfortunately not. Really it's a different app entirely ("deepnotes"?). I understand that having just one big folder isn't everyone's cup of tea but it's one of the fundamental design decisions behind the app.

[Gedulis12]

Hi @dullage , thanks for the feedback.
To answer your questions:
Could you please elaborate a little on how the subdirectory compatibility will help when using git.
Say i already have a note taking system in place where i use markdown files and manage them through git. I might already have directory structure imposed for example:

.
├── ideas.md
├── school
│   ├── curriculum1.md
│   ├── curriculum2.md
│   └── curriculum3.md
├── shortcut-cheatsheet.md
├── useful-links.md
├── quick-gists
│   ├── 20230217.md
│   ├── 20230221.md
│   ├── 20230224.md
│   └── 20230322.md
└── somerandomenote.md

so if flatnotes would be compatible with subdirectories, a user would be able to simply git clone their repository in /data folder and have all notes accessible to read/edit through the flatnotes web UI and do git pull, git push when needed.
If, on the other hand compatability for subdirectories is not enabled, the user would have to manually move notes to the root directory which would break the structure in their git repository, which is not ideal if one wants to keep notes somewhat organized.
If subdirectories are not exposed in the front end, how do the proposed changes deal with potential duplicate filenames (in different sub-directories)?
I have checked and confirmed that when creating new note through web UI, if a note with the same name already exists anywhere under /data, the error is handled gracefully with a pop-up:

Duplicate ✘
A note with this title already exists. Please try again with a new title.

If, however i try to create a new .md file with an already existing name manually, web-ui does display both notes, however the content of the note gets duplicated through both of the notes.
I believe this can be addressed in a few ways:

• We could simply warn users about potential missbehaviour if multiple files exists with the same name;
• We could modify the titles of those notes that have matching filenames, for e.g. let's say we have two notes /data/school/quicknote.md and /data/personal/quicknote.md, they could both be displayed as school → quicknote and personal → quicknote in web-ui. (i could work on a PR if you agree on the approach);

I would also argue that complexity is not necessarily added for those users who simply want plug and play solution and don't care about the availability of nested directory structure, as the app does not impose it by default. The bit of complexity gets added only when the user decides that he needs such additional functionality.
Also another thing i could suggest is that nested subdirectories could be not compatible by default, but could be enabled in docker-compose through an optional environment variable. That way any additional complexity would be disabled by default.
Please let me know what you think and whether you would like me to work on further changes to address the points mentioned above.

I gave your fork a try and it looks decent. However, there's still an issue with the organization on the flatnotes front end. It basically just lists all the .md files in all the folders without displaying the folders themselves or letting you navigate within them. Is there any way to better organize how this works on the front end? Maybe I am doing it wrong?

This was intentional, i was trying to implement the support for nested directory structure on the backend, without compromising the "flat" nature of the app on the frontend.

[rroller]

I'd also love to have directory support added to flatnotes. It would make this app nearly perfect. Those who don't want a directory structure could just no make directories?

[imaegoo]

Love this feature!

[adoyle-h]

This feature is necessary for me. Hoping it merged soon!

[danielmcmillan]

I think this would be great as well, so I can use flatnotes to search and edit existing notes and wikis.

It might make sense to keep the relative path to the notes in the title. That's what you'd get if you just list the files from a flat filesystem like S3.
This has the advantage that existing relative links work as expected, I can search on parts of a file path, and it's possible to create new notes within the subdirectories.
A poc is here develop...danielmcmillan:flatnotes:notepaths

I did come across a few caveats when looking at this solution though..

• requires some small adjustment to api routes and links since "/" is special
• the rules for note titles are more complex as we need to check for things like empty path segments and "..", and (hopefully) prevent something like arbitrary filesystem access. Although there does anyway seem to be an existing issue with notes titled "." or "..".
• someone might inadvertently create subdirectories by naturally putting / in their note title (though it would behave as expected)
• deleting and renaming notes can leave behind empty directories
• it would prevent being able to allow "/" as a normal character in note titles in future
• references to attachments within nested note directories aren't going to work

If we replace the path separator with something else (like → as suggested earlier) then it avoids some of these issues, but also links wouldn't work.

Anyway I understand why there's reservations about such a change.

I also like the idea of being able to implement a plugin to provide the note storage.
Perhaps the plugin should also be able to customise the allowed note title characters to some extent.
And probably it should be able to provide attachment storage and retrieval.

[clach04]

... * the rules for note titles are more complex as we need to check for things like empty path segments and "..", and (hopefully) prevent something like arbitrary filesystem access. Although there does anyway seem to be an existing issue with notes titled "." or "..".

@danielmcmillan can you expand on the existing issue(s)? That maybe worth a new issue.

I was able to create a title in the public demo with the title

..%2F..%2Ftestdots

Without error but https://demo.flatnotes.io/note/..%252F..%252Ftestdots is not accessible nor https://demo.flatnotes.io/note/../../testdots

I've not setup an instance yet to see what the actual behavior on the filesystem is.

[danielmcmillan]

... * the rules for note titles are more complex as we need to check for things like empty path segments and "..", and (hopefully) prevent something like arbitrary filesystem access. Although there does anyway seem to be an existing issue with notes titled "." or "..".

@danielmcmillan can you expand on the existing issue(s)? That maybe worth a new issue.

@clach04 I didn't see any major issue, I don't think path traversal is possible. The only issue I found is the ability to create notes that cannot be accessed from the browser like what you have seen. Created #209