mosparo
Add pagination and import txt, csv files and search function #303
Replies: 4 comments 12 replies
-
|
Thank you very much for your great suggestions! To be honest, I never thought about these two things, but they would be very helpful and make a lot of sense! I plan to add these functions/enhancements with the next mosparo version (v1.4). Thank you very much for your help in making mosparo better! Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
|
Currently, i am using this as database https://www.stopforumspam.com/downloads . When i succeeded to importing all of the over 1 million records, then I will export and upload it here to github so everybody can use it. No idea how will be the import yet with large data but I will check later when I am done. |
Beta Was this translation helpful? Give feedback.
-
|
For this idea, I recommend creating a rulesets. Rulesets are exactly for this kind of sharing of rules between mosparo installations and projects. Why? Of course, the main problem with your method is managing a rule in mosparo with 1 million rule items. That will not work right now, so this will end in broken rules. Additionally, the list from stopforumspam.com will be updated (I guess) from time to time, so everybody has to reimport your export file from time to time — which costs them time to do that. Rulesets, on the other side, are exactly made for that kind of idea. The rulesets are specified in a JSON file hosted on a URL. If someone wants to use your ruleset, the user has to add the URL of your ruleset as a ruleset in mosparo. mosparo then automatically downloads the ruleset from your URL and caches it in mosparo. This step will be repeated automatically (as long as the cronjob is configured for mosparo), so if you update the list because the original list was updated, everybody receives the updated ruleset within 24 hours (normally). Additionally, the ruleset management is optimized for many rules and items. It is also much easier to create a ruleset JSON file than to manually import all the rules into mosparo and then export them. Additionally, I would like to point out that from my understanding, you cannot share the exported file or a ruleset without their permission (See https://www.stopforumspam.com/license under "No Derivative Works"). Please let me know if I can help you with the ruleset. Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
|
Hi @zepich , Manually would work too for sure but if it needs to be managed, then more work would be too manually. |
Beta Was this translation helpful? Give feedback.
-
|
Ruleset tool: Thank you for this idea! I will put that on the list of ideas for the future. In the meantime, I can share with you a simple script I wrote to generate a ruleset with random values (to test the whole functionality). I've merged your code above with my code to create the ruleset. This is the code: <?php
function generateUUID() {
return sprintf('%04x%04x-%04x-%04x-%04x-%04x%04x%04x',
mt_rand(0, 0xffff), mt_rand(0, 0xffff),
mt_rand(0, 0xffff),
mt_rand(0, 0x0fff) | 0x4000,
mt_rand(0, 0x3fff) | 0x8000,
mt_rand(0, 0xffff), mt_rand(0, 0xffff), mt_rand(0, 0xffff)
);
}
$file = fopen("emails.txt", "r");
if ($file) {
// Specify the rule
$rule = [
'name' => 'Blocked Email Addresses', // Name of the rule (as in mosparo directly)
'description' => '', // Description (as in mosparo directly)
'type' => 'email', // This type is different for other rule types
'items' => [],
'spamRatingFactor' => 1,
'uuid' => generateUUID(),
];
while (($line = fgets($file)) !== false) {
$email = trim($line);
// Validate email format and exclude emails with special characters
if (filter_var($email, FILTER_VALIDATE_EMAIL) && !preg_match('/[*%]/', $email)) {
// Add the rule item
$rule['items'][] = [
'uuid' => generateUUID(),
'type' => 'email', // This type is different for other rule types
'value' => $email, // This is the rule item value
'rating' => 10, // You can specify any rating value as in the rule editor
];
} else {
echo "Invalid email format skipped: $email\n"; // Log invalid email addresses
}
}
fclose($file);
} else {
echo "Unable to open the file.";
}
// Specify the ruleset
$ruleset = [
'lastUpdatedAt' => date('Y-m-d H:i:s'),
'refreshInterval' => 86400, // Specifies how long mosparo has to wait until it can update the list again
'rules' => [$rule],
];
// Store the file and create the additional checksum file
file_put_contents(__DIR__ . '/emails.json', json_encode($ruleset));
file_put_contents(__DIR__ . '/emails.json.sha256', hash_file('sha256', __DIR__ . '/emails.json'));This generates the following ruleset: {
"lastUpdatedAt": "2025-01-28 20:41:02",
"refreshInterval": 86400,
"rules": [
{
"name": "Blocked Email Addresses",
"description": "",
"type": "email",
"items": [
{
"uuid": "a5ec707f-6c95-43c4-bf9f-019d370eb3a9",
"type": "email",
"value": "****@********",
"rating": 10
},
{
"uuid": "203da67f-6a16-470b-9bbf-0d22d3e51f9d",
"type": "email",
"value": "****@********",
"rating": 10
},
{
"uuid": "4d1cb347-b54b-49c4-9842-e18c3b9c9b0a",
"type": "email",
"value": "****@********",
"rating": 10
}
],
"spamRatingFactor": 1,
"uuid": "fc29d9ef-9837-4b8b-a098-78baeb97bbef"
}
]
}The two generated files (emails.json and emails.json.sha256) now have to be hosted on a web server so that mosparo can access them via a URL. (Side note: my emails.txt had only three email addresses. If you have 1000 email addresses, you will have 1000 rule items - of course) Does this help? Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
|
After when I answered I made similar php code just with html so I can see what is imported correctly. When I check the log file then I get this because I get on the website error 500 when I add the ruleset: I understand this is for security purpose just I have this way a problem to test the ruleset. My version of the php code: <?php
function generateUuid() {
return sprintf(
'%04x%04x-%04x-%04x-%04x-%04x%04x%04x',
mt_rand(0, 0xffff), mt_rand(0, 0xffff),
mt_rand(0, 0xffff),
mt_rand(0, 0x0fff) | 0x4000,
mt_rand(0, 0x3fff) | 0x8000,
mt_rand(0, 0xffff), mt_rand(0, 0xffff), mt_rand(0, 0xffff)
);
}
header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');
header('Connection: keep-alive');
$txtFile = 'emails.txt';
$jsonFile = 'emails.json';
$txtData = file($txtFile, FILE_IGNORE_NEW_LINES);
$totalLines = count($txtData);
$errorCount = 0;
$skippedCount = 0;
$jsonData = [
"lastUpdatedAt" => date('Y-m-d H:i:s'),
"refreshInterval" => 3600,
"rules" => [
[
"name" => "Test-rule",
"description" => "",
"type" => "email",
"items" => [],
"spamRatingFactor" => 101,
"uuid" => generateUuid()
]
]
];
foreach ($txtData as $index => $line) {
$lineData = str_getcsv($line);
if (empty($lineData[0]) || empty($lineData[1])) {
$errorCount++;
echo "data: " . json_encode([
'message' => "Error processing line " . ($index + 1),
'isError' => true
]) . "\n\n";
} else {
try {
$jsonData['rules'][0]['items'][] = [
'uuid' => generateUuid(),
'type' => 'email',
'value' => $lineData[0], // 1st column from txt as value
'rating' => $lineData[1], // 2nd column from txt as rating
];
$jsonData['rules'][0]['spamRatingFactor'] = (int)$lineData[1];
$percentage = intval(($index + 1) / $totalLines * 100);
echo "data: " . json_encode([
'percentage' => $percentage,
'message' => "Processing line " . ($index + 1) . " of $totalLines",
]) . "\n\n";
} catch (Exception $e) {
$skippedCount++;
echo "data: " . json_encode([
'message' => "Skipped line " . ($index + 1),
'isSkipped' => true
]) . "\n\n";
}
}
ob_flush();
flush();
usleep(100000); // Delay for 100 milliseconds
}
file_put_contents($jsonFile, json_encode($jsonData, JSON_PRETTY_PRINT));
// Generate SHA256 hash of the emails.json file
$hash = hash_file('sha256', $jsonFile);
$hashFile = 'emails.json.sha256';
file_put_contents($hashFile, $hash);
echo "data: " . json_encode([
'percentage' => 100,
'message' => "Import complete. Errors: $errorCount, Skipped: $skippedCount.",
'complete' => true,
'hash' => $hash
]) . "\n\n";
ob_flush();
flush();
?>HTML for that: <!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Email Import Progress</title>
<style>
body {
background-color: #121212;
color: #e0e0e0;
font-family: Arial, sans-serif;
}
#progress-bar {
width: 100%;
background-color: #333;
}
#progress {
width: 0%;
height: 30px;
background-color: #4caf50;
text-align: center;
line-height: 30px;
color: white;
}
.button {
margin-top: 20px;
padding: 10px 20px;
background-color: #4caf50;
color: white;
border: none;
cursor: pointer;
}
.button:disabled {
background-color: #555;
cursor: not-allowed;
}
#console, #errorConsole, #skippedConsole {
margin-top: 20px;
padding: 10px;
background-color: #333;
border: 1px solid #555;
height: 200px;
overflow-y: scroll;
}
.error {
color: red;
}
.skipped {
color: #add8e6; /* light blue */
}
</style>
</head>
<body>
<h1>Email Import Progress</h1>
<div id="progress-bar">
<div id="progress">0%</</div>
</div>
<button id="importButton" class="button" onclick="startImport()">Start Import</button>
<button id="pauseButton" class="button" onclick="pauseImport()" disabled>Pause</button>
<div id="console"></div>
<h2>Error Console</h2>
<div id="errorConsole"></div>
<h2>Skipped Console</h2>
<div id="skippedConsole"></div>
<script>
let source;
let isPaused = false;
const maxLogs = 15;
function updateProgress(percentage) {
var progressBar = document.getElementById('progress');
progressBar.style.width = percentage + '%';
progressBar.textContent = percentage + '%';
}
function logMessage(message, isError = false, isSkipped = false) {
var consoleDiv = document.getElementById('console');
if (isError) {
message = '<span class="error">' + message + '</span>';
var errorConsoleDiv = document.getElementById('errorConsole');
errorConsoleDiv.innerHTML += message + '<br>';
errorConsoleDiv.scrollTop = errorConsoleDiv.scrollHeight;
} else if (isSkipped) {
message = '<span class="skipped">' + message + '</span>';
var skippedConsoleDiv = document.getElementById('skippedConsole');
skippedConsoleDiv.innerHTML += message + '<br>';
skippedConsoleDiv.scrollTop = skippedConsoleDiv.scrollHeight;
}
// Add new log entry
consoleDiv.innerHTML += message + '<br>';
// Keep only the last 15 entries
const logs = consoleDiv.innerHTML.split('<br>');
if (logs.length > maxLogs) {
consoleDiv.innerHTML = logs.slice(-maxLogs).join('<br>');
}
consoleDiv.scrollTop = consoleDiv.scrollHeight;
}
function startImport() {
if (!isPaused) {
document.getElementById('importButton').disabled = true;
document.getElementById('pauseButton').disabled = false;
source = new EventSource('import_email_to_json.php');
}
isPaused = false;
source.onmessage = function(event) {
var data = JSON.parse(event.data);
if (data.percentage) {
updateProgress(data.percentage);
}
if (data.message) {
logMessage(data.message, data.isError, data.isSkipped);
}
if (data.complete) {
source.close();
document.getElementById('importButton').disabled = false;
document.getElementById('pauseButton').disabled = true;
if (data.hash) {
logMessage("SHA256 Hash: " + data.hash, false, false);
}
}
};
}
function pauseImport() {
if (source) {
source.close();
isPaused = true;
document.getElementById('importButton').disabled = false;
document.getElementById('pauseButton').disabled = true;
}
}
</script>
</body>
</html>Update: Changed $hashFile name to emails.json.sha256 and the http works now however I get error "An error occurred while updating the ruleset: The ruleset content is not valid against the schema." However the json file looks like this when generated: {
"lastUpdatedAt": "2025-01-28 22:02:54",
"refreshInterval": 3600,
"rules": [
{
"name": "Test-rule",
"description": "",
"type": "email",
"items": [
{
"uuid": "4bb65f30-518d-4b80-994a-df14ddb67805",
"type": "email",
"value": "a*****@g*****.com",
"rating": "1"
},
{
"uuid": "78c5783c-9035-4bfb-9c5b-59039b87a77f",
"type": "email",
"value": "*****g@**********8.com",
"rating": "1"
},
{
"uuid": "c142105a-dd15-4aca-b4b2-9d836331c346",
"type": "email",
"value": "*****@f*****r.com",
"rating": "1"
},
],
"spamRatingFactor": 1,
"uuid": "299b8207-a156-46f8-912f-134667aa7df4"
}
]
} |
Beta Was this translation helpful? Give feedback.
-
|
Update 2: If someone wants to use this script then here is the PHP updated version, now it works: <?php
function generateUuid() {
return sprintf(
'%04x%04x-%04x-%04x-%04x-%04x%04x%04x',
mt_rand(0, 0xffff), mt_rand(0, 0xffff),
mt_rand(0, 0xffff),
mt_rand(0, 0x0fff) | 0x4000,
mt_rand(0, 0x3fff) | 0x8000,
mt_rand(0, 0xffff), mt_rand(0, 0xffff), mt_rand(0, 0xffff)
);
}
header('Content-Type: text/event-stream');
header('Cache-Control: no-cache');
header('Connection: keep-alive');
$txtFile = 'emails.txt';
$jsonFile = 'emails.json';
$txtData = file($txtFile, FILE_IGNORE_NEW_LINES);
$totalLines = count($txtData);
$errorCount = 0;
$skippedCount = 0;
$jsonData = [
"lastUpdatedAt" => date('Y-m-d H:i:s'),
"refreshInterval" => 3600,
"rules" => [
[
"name" => "Test-rule",
"description" => "",
"type" => "email",
"items" => [],
"spamRatingFactor" => 101,
"uuid" => generateUuid()
]
]
];
foreach ($txtData as $index => $line) {
$lineData = str_getcsv($line);
if (empty($lineData[0]) || empty($lineData[1])) {
$errorCount++;
echo "data: " . json_encode([
'message' => "Error processing line " . ($index + 1),
'isError' => true
]) . "\n\n";
} else {
try {
$jsonData['rules'][0]['items'][] = [
'uuid' => generateUuid(),
'type' => 'email',
'value' => $lineData[0], // 1st column from txt as value
'rating' => (int)$lineData[1], // 2nd column from txt as rating
];
$percentage = intval(($index + 1) / $totalLines * 100);
echo "data: " . json_encode([
'percentage' => $percentage,
'message' => "Processing line " . ($index + 1) . " of $totalLines",
]) . "\n\n";
} catch (Exception $e) {
$skippedCount++;
echo "data: " . json_encode([
'message' => "Skipped line " . ($index + 1),
'isSkipped' => true
]) . "\n\n";
}
}
ob_flush();
flush();
usleep(100000); // Delay for 100 milliseconds
}
file_put_contents($jsonFile, json_encode($jsonData, JSON_PRETTY_PRINT));
// Generate SHA256 hash of the emails.json file
$hash = hash_file('sha256', $jsonFile);
$hashFile = 'emails.json.sha256';
file_put_contents($hashFile, $hash);
echo "data: " . json_encode([
'percentage' => 100,
'message' => "Import complete. Errors: $errorCount, Skipped: $skippedCount.",
'complete' => true,
'hash' => $hash
]) . "\n\n";
ob_flush();
flush();
?> |
Beta Was this translation helpful? Give feedback.
-
|
Thank you very much for sharing this code and the updates! I'm happy that you were able to make it work! How did you solve the SSL issue? In general, You can add the self-signed certificate to the trusted certificates (but please don't ask me for a how-to for that ;) ). Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
Hi @zepich , About Stop Forum Spam topic, well no need to include by you however you can include the function as an option for the users and the users can decide if they want to use or not as you mentioned with the GeoIP. The license is not limited the development for that however just the database usage. Other option to have a database format something like stopforumspam has and the people can collect from everywhere the database where is free. For example people can use this too: Or paid version which can be downloaded too: The main question eventually is not that you need to provide ruleset but the tool to implement in the project. If you don't provide the blacklist as in the project ruleset then the license doesn't touching it. If a member wants to provide a ruleset which is under a license then that is on their shoulder the responsibility. You are just giving the option to make something better. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @StrangerGithuber and @zepich , I integrated the command into my Mosparo instance via Docker Volumes and then integrated the generated rule package as a ruleset. What exactly does the script do?
The execution commandThe command for creating a rule package could look like this: NoteThe command currently only supports rules for IP addresses and e-mail addresses. Symfony command<?php
namespace Mosparo\Command;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputArgument;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\Console\Output\OutputInterface;
/**
* Example:
* bin/console mosparo:generate-rule-set-mail-addresses "https://www.stopforumspam.com/downloads/listed_email_1.zip" "/var/www/html/public/rule-sets" "spam-emails.json" -t email -i 180 -s 1 -r 1 -N "List of spam emails" -d "Spam email list powered by www.stopforumspam.com" -v
* bin/console mosparo:generate-rule-set-mail-addresses "https://www.stopforumspam.com/downloads/listed_ip_1_ipv46.zip" "/var/www/html/public/rule-sets" "spam-ips.json" -t ipAddress -i 180 -s 1 -r 1 -N "List of spam IPs" -d "Spam IP list powered by www.stopforumspam.com" -v
*
* Class GenerateRuleSetMailAddressesCommand
* @package Mosparo\Command
*/
class GenerateRuleSetMailAddressesCommand extends Command
{
protected static $defaultName = 'mosparo:generate-rule-set-mail-addresses';
protected $allowedRuleTypes = [
'ipAddress',
'email',
// 'uuid'
];
public function __construct()
{
parent::__construct();
}
protected function configure(): void
{
$this
->setDescription('Generate a ruleset JSON with EMail addresses from the list "stopforumspam.com".')
->addArgument(
'sourceUrl',
InputArgument::REQUIRED,
'The URL to the packed text file with the one value per line'
)
->addArgument(
'saveDestination',
InputArgument::REQUIRED,
'The path where the ruleset file will be saved.'
)
->addArgument(
'rulesetFilename',
InputArgument::REQUIRED,
'The name of the ruleset file.'
)
->addOption(
'ruleType',
't',
InputOption::VALUE_OPTIONAL,
'Defines the rule type of the created rule list. (ipAddress or email)',
'ipAddress'
)
->addOption(
'refreshInterval',
'i',
InputOption::VALUE_OPTIONAL,
'Defines the time in seconds when mosparo is allowed to retrieve the rule package again.',
86400
)
->addOption(
'spamRatingFactor',
's',
InputOption::VALUE_OPTIONAL,
'Rating factor of the rule to strengthen or weaken the rule items.',
1
)
->addOption(
'ruleRatingFactor',
'r',
InputOption::VALUE_OPTIONAL,
'Defines the spam value of the rule item.',
1
)
->addOption(
'ruleName',
'N',
InputOption::VALUE_OPTIONAL,
'Defines the name of the rule.',
''
)
->addOption(
'ruleDescription',
'd',
InputOption::VALUE_OPTIONAL,
'Defines the description of the rule.',
''
);
}
protected function execute(InputInterface $input, OutputInterface $output): int
{
$sourceUrl = $input->getArgument('sourceUrl');
$saveDestination = $input->getArgument('saveDestination');
$rulesetFileName = $input->getArgument('rulesetFilename');
$rulesetFilePath = rtrim($saveDestination, '/') . "/" . $rulesetFileName;
$ruleType = $input->getOption('ruleType');
if (!in_array($ruleType, $this->allowedRuleTypes)) {
$output->writeln('Invalid rule type! ', OutputInterface::OUTPUT_NORMAL);
return Command::FAILURE;
}
$refreshInterval = $input->getOption('refreshInterval');
$spamRatingFactor = $input->getOption('spamRatingFactor');
$ruleRatingFactor = $input->getOption('ruleRatingFactor');
$ruleName = $input->getOption('ruleName');
$ruleDescription = $input->getOption('ruleDescription');
$output->writeln('Source URL: ' . $sourceUrl, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Save destination: ' . $saveDestination, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Ruleset file name: ' . $rulesetFileName, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Ruleset file path: ' . $rulesetFilePath, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Rule type: ' . $ruleType, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Refresh interval: ' . $refreshInterval, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Spam rating factor: ' . $spamRatingFactor, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Rule rating factor: ' . $ruleRatingFactor, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Rule name: ' . $ruleName, OutputInterface::OUTPUT_NORMAL);
$output->writeln('Rule description: ' . $ruleDescription, OutputInterface::OUTPUT_NORMAL);
$compressedFile = $this->downloadCompressedFile(
$output,
$sourceUrl,
rtrim($saveDestination, '/') . '/tmp.zip',
$sourceUrl . '.md5'
);
if ($compressedFile === false) {
return Command::FAILURE;
}
$existingItems = $this->loadUuidsFromRulesetFile(
$output,
$rulesetFilePath
);
if ($this->generateRulesetFromCompressedCSV(
$output,
$compressedFile,
$rulesetFilePath,
$existingItems,
$ruleType,
(int)$refreshInterval,
(int)$spamRatingFactor,
$ruleName,
$ruleDescription,
(int)$ruleRatingFactor
) === false
) {
return Command::FAILURE;
}
if ($this->generateChecksumFile(
$output,
$rulesetFilePath
) === false
) {
return Command::FAILURE;
}
$output->writeln('Done!', OutputInterface::OUTPUT_NORMAL);
return Command::SUCCESS;
}
/**
* This function tries to create a checksum file.
* The name of the checksum file is "$filename . '.' . $algo"
* If an old checksum file exists, it will be overwritten
*
* @param OutputInterface $output
* @param string $filename URL describing location of file to be hashed; Supports fopen wrappers
* @param string $algo Name of selected hashing algorithm (i.e. "md5", "sha256", "haval160,4", etc..). For a list of supported algorithms see hash_algos().
* @return bool
*/
private function generateChecksumFile(
OutputInterface $output,
string $filename,
string $algo = 'sha256'
): bool {
$checksumFilename = $filename . '.' . $algo;
$checksum = hash_file($algo, $filename);
if ($checksum !== false) {
if (file_exists($checksumFilename)) {
$output->writeln('Remove old checksum file. File: '. $checksumFilename, OutputInterface::OUTPUT_NORMAL);
unlink($checksumFilename);
}
$output->writeln('Save new checksum file. File: '. $checksumFilename, OutputInterface::OUTPUT_NORMAL);
if (file_put_contents($checksumFilename, $checksum) !== false) {
return true;
}
}
return false;
}
/**
* This function creates a new ruleset json with the specified compressed CSV file.
* This function overwrites existing ruleset files with the same name,
* but tries to keep the uuid with the same rules.
*
* @param OutputInterface $output
* @param string $compressedCSV The path to the compressed file.
* @param string $rulesetFilePath The path where the final ruleset file should be saved and the name of the file
* @param array $existingItems The array from the function loadUuidsFromRulesetFile
* @param string $ruleType The rule type of the created rule list
* @param int $refreshInterval This value will be set as refreshInterval in the JSON, if not found in $existingItems
* @param int $spamRatingFactor This value will be set as spamRatingFactor in the JSON, if not found in $existingItems
* @param string $ruleName This value will be set as rule name in the JSON, if not found in $existingItems
* @param string $ruleDescription This value will be set as rule description in the JSON, if not found in $existingItems
* @param int $ruleRatingFactor This value will be set as rating in the rule JSON
* @return bool
*/
private function generateRulesetFromCompressedCSV(
OutputInterface $output,
string $compressedCSV,
string $rulesetFilePath,
array &$existingItems, // & serves the performance to create a reference and not a copy of the variable
string $ruleType,
int $refreshInterval = 86400,
int $spamRatingFactor = 1,
string $ruleName = '',
string $ruleDescription = '',
int $ruleRatingFactor = 1
): bool {
$output->writeln('Begin to generate Ruleset from compressed CSV', OutputInterface::OUTPUT_NORMAL);
$zip = new \ZipArchive;
if (file_exists($compressedCSV) &&
$zip->open($compressedCSV) === TRUE &&
$zip->getNameIndex(0) !== false // We use only the first file
) {
// Use fopen and fgets to read the file line by line
$output->writeln('Open the first file contained in the compressed file', OutputInterface::OUTPUT_NORMAL);
$fp = $zip->getStream($zip->getNameIndex(0));
if (!$fp) {
$output->writeln('Error when opening the CSV file!', OutputInterface::OUTPUT_NORMAL);
return false;
}
// Use json_encode with the JSON_PARTIAL_OUTPUT_ON_ERROR flag,
// to write the rule package to a file step by step
$tempFile = $rulesetFilePath . '_tmp';
$outFp = fopen($tempFile, 'w');
$output->writeln('Write ruleset header in JSON', OutputInterface::OUTPUT_NORMAL);
fwrite(
$outFp,
'{
"lastUpdatedAt":"' . date('c') . '",
"refreshInterval":' . ($existingItems['refreshInterval'] ?? $refreshInterval) . ',
"rules":[{
"name":"' . ($existingItems['ruleName'] ?? $ruleName) . '",
"description":"' . ($existingItems['ruleDescription'] ?? $ruleDescription) . '",
"type":"' . $ruleType . '",
"uuid":"' . ($existingItems['uuid'] ?? uuid_create(UUID_TYPE_RANDOM)) . '",
"spamRatingFactor": ' . ($existingItems['spamRatingFactor'] ?? $spamRatingFactor) . ',
"items":['
);
$output->writeln('Start creating/writing the individual rules. This may take a while...', OutputInterface::OUTPUT_NORMAL);
$first = true;
while (!feof($fp)) {
$line = fgets($fp);
$email = trim($line);
if (filter_var($email, FILTER_VALIDATE_IP) ||
filter_var($email, FILTER_VALIDATE_EMAIL)
) {
// Use existing uuid if email exists
if (isset($existingItems[$email])) {
$uuid = $existingItems[$email];
} else {
$uuid = uuid_create(UUID_TYPE_RANDOM);
}
// Add rule
$itemJson = json_encode(
[
"uuid" => $uuid,
"type" => $ruleType,
"value" => $email,
"rating" => $ruleRatingFactor
],
JSON_PARTIAL_OUTPUT_ON_ERROR
);
if (!$first) {
fwrite($outFp, ',');
}
fwrite($outFp, "\n" . $itemJson);
$first = false;
} else {
$output->writeln('Failed validation! Skip value: ' . $email, OutputInterface::OUTPUT_NORMAL);
}
}
fclose($fp);
$zip->close();
$output->writeln('Write JSON closing brackets in JSON', OutputInterface::OUTPUT_NORMAL);
fwrite($outFp, ']}]}');
fclose($outFp);
$output->writeln('Remove compressed file. File: '. $compressedCSV, OutputInterface::OUTPUT_NORMAL);
unlink($compressedCSV);
if (file_exists($rulesetFilePath)) {
$output->writeln('Remove old ruleset file. File: '. $rulesetFilePath, OutputInterface::OUTPUT_NORMAL);
unlink($rulesetFilePath);
}
$output->writeln('Rename new ruleset file from temp name to final name. File: '. $rulesetFilePath, OutputInterface::OUTPUT_NORMAL);
rename($tempFile, $rulesetFilePath);
return true;
} else {
$output->writeln('No existing compressed CSV found.', OutputInterface::OUTPUT_NORMAL);
return false;
}
}
/**
* This function creates an array with all uuids and values from all rules within the ruleset json.
* This is needed to use the same uuid for already existing values
*
* @param OutputInterface $output
* @param string $existingRulesetPath The path with filename of the existing ruleset
* @return array
*/
private function loadUuidsFromRulesetFile(
OutputInterface $output,
string $existingRulesetPath
): array {
$existingItems = [];
$output->writeln('Read the existing ruleset file, to generate mapping of existing uuids', OutputInterface::OUTPUT_NORMAL);
if (file_exists($existingRulesetPath)) {
$json = file_get_contents($existingRulesetPath);
if ($json !== false) {
$ruleset = json_decode($json, true);
// Clear variable
unset($json);
if (json_last_error() !== JSON_ERROR_NONE &&
isset($ruleset['uuid']) &&
isset($ruleset['refreshInterval']) &&
isset($ruleset['spamRatingFactor']) &&
isset($ruleset['rules'][0]['name']) &&
isset($ruleset['rules'][0]['description']) &&
isset($ruleset['rules'][0]['items'])
) {
$errorMsg = json_last_error_msg();
$output->writeln('Error on reading JSON! Skip generation! Error: ' . $errorMsg, OutputInterface::OUTPUT_NORMAL);
return $existingItems;
}
// Save metadata
$existingItems['uuid'] = $ruleset['uuid'];
$existingItems['refreshInterval'] = $ruleset['refreshInterval'];
$existingItems['spamRatingFactor'] = $ruleset['spamRatingFactor'];
$existingItems['ruleName'] = $ruleset['rules'][0]['name'];
$existingItems['ruleDescription'] = $ruleset['rules'][0]['description'];
// Add existing uuid entries
foreach ($ruleset['rules'][0]['items'] as $item) {
$existingItems[$item['value']] = $item['uuid'];
}
// Clear variable
unset($ruleset);
} else {
$output->writeln('Existing rule set could not be opened! Skip generation!', OutputInterface::OUTPUT_NORMAL);
}
} else {
$output->writeln('No existing ruleset file found. Skip generation!', OutputInterface::OUTPUT_NORMAL);
}
return $existingItems;
}
/**
* The downloadCompressedFile function downloads a ZIP file from a specified URL and saves it to a specified
* destination.
* Optionally, the MD5 checksum is also checked.
*
* @param OutputInterface $output
* @param string $sourceUrl The URL to the compressed file.
* @param string $fileDestination The path where the downloaded file will be saved, with the filename
* @param string|null $sourceMd5Url Optional. URL to the MD5 hash of the downloaded file.
* If set, the file is compared with it
* @return false|string
*/
private function downloadCompressedFile(
OutputInterface $output,
string $sourceUrl,
string $fileDestination = 'file.zip',
string $sourceMd5Url = null
) {
// Download the compressed file
$output->writeln('Downloading compressed file', OutputInterface::OUTPUT_NORMAL);
if (!$this->checkForAllowMIMEType(['application/zip'], $sourceUrl)) {
$output->writeln('File MIME Type is not allowed!', OutputInterface::OUTPUT_NORMAL);
return false;
}
// Download file
if (!copy($sourceUrl, $fileDestination)) {
$output->writeln('Failed to download compressed file!', OutputInterface::OUTPUT_NORMAL);
return false;
}
$output->writeln('Downloading complete', OutputInterface::OUTPUT_NORMAL);
if ($sourceMd5Url !== null) {
// Download the MD5 checksum
$output->writeln('Download MD5 hash', OutputInterface::OUTPUT_NORMAL);
if (!$this->checkForAllowMIMEType(['text/plain','application/zip'], $sourceMd5Url)) {
$output->writeln('File MIME Type is not allowed!', OutputInterface::OUTPUT_NORMAL);
return false;
}
$md5Checksum = trim(file_get_contents($sourceMd5Url));
$output->writeln('Generate MD5 hash for compressed file', OutputInterface::OUTPUT_NORMAL);
$checksum = md5_file($fileDestination);
$output->writeln('Compare hash', OutputInterface::OUTPUT_NORMAL);
if ($checksum !== $md5Checksum) {
$output->writeln('Hash compare failed!', OutputInterface::OUTPUT_NORMAL);
// Remove downloaded file because the hash does not match.
unlink($fileDestination);
return false;
}
$output->writeln('Hash equal!', OutputInterface::OUTPUT_NORMAL);
}
return $fileDestination;
}
/**
* Check the MIME type of the file is allowed without downloading
*
* @param array $allowedTypes Array of allowed MIME types
* @param string $sourceUrl The file url
* @return bool
*/
private function checkForAllowMIMEType(array $allowedTypes, string $sourceUrl): bool {
$result = false;
// Check the MIME type of the file before downloading
$headers = get_headers($sourceUrl, 1);
if (isset($headers['Content-Type'])) {
$contentType = $headers['Content-Type'];
if (is_array($contentType)) {
$contentType = end($contentType);
}
// Check if the MIME type is allowed
if (in_array($contentType, $allowedTypes)) {
$result = true;
}
}
return $result;
}
}What do you think of my command and approach? Kind regards |
Beta Was this translation helpful? Give feedback.
-
|
Looks interesting. Unfortunately I can't answer. I am not that smart as you two. I will let to answer to your question the big boss. 😅 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Digi92 Wow! Thank you very much for sharing your awesome work! It looks like a lot of work! Just out of curiosity, did you build it for performance by default, or did you have some memory issue (because you're building the JSON string manually)? My first thought was to ask you about the license (to add this to mosparo). However, we do not need such functionality inside every mosparo installation because if there is a publicly shared ruleset, we can simply use it instead of generating new ones. My second thought was to add the StopForumSpam source as a third-party provider like GeoIP2, so mosparo could download the files automatically and scan the IP/email addresses/URLs. But here we have the problem of the number of downloads per IP being limited. So, if we add it to mosparo, and there are 2+ installations on a shared web hosting, some installations will not receive the files because the IP is already blocked. Additionally, their license does not allow automatic downloading of files. My third thought was to mirror the files to one of our own servers and download them from there. But the license explicitly does not allow that (except with permission). Additionally, it would be an automatic tool, which is again not allowed by their license. Using the API also has a limit of 20'000 requests per day per IP and (more importantly) is not compatible with GDPR in my eyes (because we would share personal information with a 3rd party without approval). This leads us back to field one: Without permission, we (mosparo) cannot do anything officially. Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Digi92 and @StrangerGithuber We could make an additional tool/project, outside of mosparo. For example, "mosparo Ruleset Generator". It could contain scripts/commands like the one that @Digi92 provided (depending on the license) and offer other conversion tools to convert existing lists into the mosparo ruleset format. For example, simple general ones like "convert this list of XYZ into a ruleset" could be doable (a list of email addresses, IP addresses, or any other stuff like country codes, unicode blocks and so on). With that, we would provide a tool to do that while not putting it into the main code. What do you think about that approach? Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
|
Sounds promising. 👍 |
Beta Was this translation helpful? Give feedback.
-
|
Hi @zepich and @StrangerGithuber, My code hasn't had a license yet, but feel free to do what you want with it :) I think that mosparo should at least offer the option of converting CSV lists into rule packages. I cannot say for sure whether a direct implementation, as exists in GeoIP, would be useful. It could lead to a high maintenance effort, as either the benefit would be limited to a single data source or additional sources would be added over time, which would also have to be maintained. I like the idea of an additional tool or project. Kind regards, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Digi92 and @StrangerGithuber Thank you very much for your feedback. I had a lot of ideas this afternoon, so let me explain what we should do with a separate tool. GeneralFirst, the tool will be CLI-only, so we will not have to bother with the UI. The tool will require a configuration file, most likely YAML-based. Technically, it would be possible to build a simple UI to generate these YAML configuration files, but that is a different project and/or something a third party could do. The tool will be started with the command The tool's functionality would be customizable. The configuration file would have two main sections: the InputOut of the box, we could add at least four different options as input sources:
2025-01-31/zepich: Edited the sources to make them more general All these inputs will return the required rules in a specified format. The main application will combine these rules into one rule package. OutputOut of the box, we could add at least three different options as output destinations:
Every configuration file can contain multiple input sources and output destinations at once, so it is possible to generate a rule package, store it in a file, and import it into a mosparo project. DistributionWe can distribute the tool via What do you think? Kind regards, zepich |
Beta Was this translation helpful? Give feedback.
-
Hi
I tried to add more than 1000 emails in the Rules from spam databases from other php file because I were lazy to add manually 1 by 1 and the "Add multiple items" doesn't add all of them, most of the time just the Chrome based webbrowsers says "out of memory" and firefox just stuck.
If someone has a database and want to import it then they could just upload the txt or csv files which contains the emails, ip, etc... It would be nice if there would be a function.
So far I am importing like this from txt file:
The other suggestion is to have pagination on the edit rule page because if I want to edit a rule which has more than 1000 emails or other records then the webbrowser says out of memory or just crash the page.
The third suggestion would be a search function within rules if the user added already the record or not. If the user wants to delete or edit a record.
Beta Was this translation helpful? Give feedback.
All reactions