Local Rule packages path

[Digi92]

Hi,
why not support file paths on the server for rule packages as an alternative to a URL?

In the case of self created rule packages, I would currently have to place them in a publicly accessible directory and protect them with a password to prevent unauthorized access.
However, this carries the risk that in the event of a misconfiguration (e.g. accidental removal of password protection) the rule packages become publicly accessible.

Is there a particular reason why this option has not yet been provided?

Kind regards
Digi92

[zepich]

Hi @Digi92

Thank you very much for your idea!

Let's start with the last question because it's easy to answer:

Is there a particular reason why this option has not yet been provided?

We never had this amazing idea! :)

When we started the project, we only had rules. Then, we thought about sharing rules over the internet, so we added rule packages via URL. We had never considered importing rules from a file path.

Your suggestion makes absolutely sense. After reading your idea, I had two additional ideas, so I think we should enhance the rule packages with the following options:

Automatic synchronization

• Import from URL (if it is publicly available)
• Import from a file path (if it is stored on the server or a shared file system)

Manual synchronization

• Import from the CLI (with a command like php bin/console mosparo:import-rule-set <ID> <FILE-PATH>)
• Import via API endpoint (we add an API endpoint to fill in the content of a rule package.)

With the two manual synchronization options, you can save some resources since you can synchronize the rule package whenever it changes and not every ? hours.

You choose one of these types when you create the package in mosparo, as you do when you create a new rule.

What do you think about these four options?

Kind regards,

zepich

[Digi92]

Hi @zepich,

the suggestion for the two automatic options sounds good to me.
In addition, it would probably be helpful for local rulesets to deactivate the checksum check as an optional option.

Regarding the CLI command “mosparo:import-rule-set”:
Wouldn't the “Update Rulesets” command with an optional parameter for a specific rule package make more sense here?
I think a “import” endpoint might be more confusing than helpful.
How should this import command be displayed in the backend? Should a new “Automatic” entry be created, or does it check if the file path already exists and then update this entry accordingly?

To prevent unnecessary updates, we could also save the last JSON file checksum of a ruleset and check whether the checksum has changed before an update and only perform an update if there is a change.

I like the idea of an API endpoint, as this would significantly simplify the integration of rule packages, e.g. via lists such as from StopForumSpam, without having to create a huge JSON file.

Additional question now that I think about it.
Are there already API endpoints for adding, editing or removing rule packages?

Kind regards
Digi92

[zepich]

Hi @Digi92

Thank you very much for your feedback.

In addition, it would probably be helpful for local rulesets to deactivate the checksum check as an optional option.

From my perspective, for the local rulesets, we can generate the checksum directly from the file instead of using the additional file. The goal of the additional file is not to transfer the whole ruleset file over the internet for every update check. However, since the file is local, we can directly create the checksum of the file and do not need the additional checksum file.

Regarding the CLI command "mosparo:import-rule-set":
Wouldn't the "Update Rulesets" command with an optional parameter for a specific rule package make more sense here?
I think a "import" endpoint might be more confusing than helpful.
How should this import command be displayed in the backend? Should a new "Automatic" entry be created, or does it check if the file path already exists and then update this entry accordingly?

You always manually create a ruleset first before you can import it. When creating the ruleset, you choose between the four options (from a URL, from a file, manually by API, manually by CLI). This will then create the ruleset entry. If you open one of the manual rulesets (by API, by CLI) after creating it, you will not see any content because you have to feed in the content first. Instead, you will see some kind of introduction text and/or some text, like when creating a new repository on GitHub/GitLab, about what you must do next.

To import the ruleset by CLI, you must open the CLI and execute the command. Now, what is the idea behind this option? Option 1 (from URL automatic) and Option 2 (from file automatic) require you to have a web or file server running 24/7.

But you may not want to have such a server, and/or you do not want the risk of exposing it to the internet. So, how about manual options?

That's why I had the idea for these two additional options. The API option is the best because you can control it from everywhere, but it requires you to develop some script to access the API and import the data. What if you do not want to invest this time or if you can't develop at all? The answer is the new CLI command. With the CLI command, you can synchronize the file with rsync/SCP (or some other kind of tool) to your server(s) and then execute the import command when the file is synchronized, either manually or automatically, for example with inotify or something like that (could also be a cronjob).

This gives you an easy option to import the file without installing and maintaining a web/file server for the ruleset file and without developing a tool/script to import the ruleset via API.

By choosing the type when creating the ruleset, we can also limit which rulesets are available for the API import functionality, which are for the CLI command, and which are automatically updated (with the existing CLI command).

Additional question now that I think about it.
Are there already API endpoints for adding, editing or removing rule packages?

No, we have almost no API endpoints right now except the endpoints to validate and verify the form data, the API to get the statistical data, and, since v1.3.0, the health check API. Adding API endpoints for a lot of stuff (like managing projects, managing project members, and managing rules) is on our long-term to-do list.

Kind regards,

zepich

[Digi92]

Hi @zepich,
now I understand better why you want to add the CLI command "mosparo:import-rule-set" and I think you're absolutely right.

I'm with you on the setup.
My only concern is that it might be confusing for some users which of the six possible ways they should choose:

1. Maintain rules directly in the backend
2. Maintain rule packages in the backend
3. Import rules from a URL
4. Importing rules from a file
5. Add rules manually via API
6. Add rules manually via CLI

Unfortunately, I currently have no better solution than a detailed description or info-overlay to make it easier for users to understand which way would be right for them.

Kind regards
Digi92

[zepich]

Hi @Digi92

Thank you very much for your feedback.

Maybe we should add a better intro description for rule packages to explain what they are. Rule packages are intended for more advanced users or users with advanced requirements. An owner of a simple website with a contact form may never need rule packages.

I would like to focus on #36 in the long term for the simple use case since I think that would help new users to start with mosparo and managing rules the most. Additionally, with the suggestion in #303, we will add a simple function to import rules from a file when creating the rule (additionally to the Add one item and Add multiple items options).

For the four different types of rule packages, I think of two sections for the automatic and manual rule packages, each with a title and explanation of the difference.

We're also working on some how-to videos to explain how to create rules, set the rating, and use the different rule types. We're also planning to cover other topics, like rule packages.

I think that should work pretty well.

Kind regards,

zepich

[Digi92]

Hi @zepich ,
i also think that having a well-written intro description and visually structuring the sections would be the best approach.

I really like the functionality approach of #36 for smaller pages.

The suggested are nice and if any misunderstandings arise, there’s always the possibility to refine things further.

Kind regards,
Digi92