Teal Blauwbok: Grub

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I hope that everything is going right for you and your loved one.
This release just provides an improvement of the way we open and close a connection to a MySQL/MariaDB database.

Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter 😄!

I wish you all the best.
Stay safe and healthy!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed ?

• An issue while saving user-inputted database credentials (if not found).

What was improved ?

• The way we open and close a connection to a MySQL/MariaDB database.

What is coming ?

• Fix of reported issues.
• Improvement of the documentation (help welcome!).
• Introduction of possible and interesting features.

Teal Blauwbok: Centipede

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I hope that everything is going right for you and your loved one.
This release just provides some improvements and a few new features.

There are a few important changes to notice. The first one is the switch from raw SQL to SQLAlchemy. The reason behind the switch is a change in the development process. I cannot endorse nor want to maintain raw SQL anymore.
To understand the matter, before the switch to SQLAlchemy I had code blocks that handled MySQL connections and statements and another one to handle MariaDB connections and statements. Most of the time it was 1 block for both but in other - not so peaceful - parts there were 2. Meaning that I was handling every use case by myself. With SQLAlchemy, we have 1 line/block for multiple database types. Of course we only support MariaDB and MySQL but it simplifies the integration of other database types in the future. The other reason which convinced me that it's the right way is that it forced me to review and simplify the table structure. Meaning that:

• We minimize the total size,
• We optimize the SQL flow,
• We minimize the number of SQL queries being made.

To simplify the switch and keep it as smooth as possible, a migrator is integrated into PyFunceble. That way, you don't have to take any special action, PyFunceble will handle everything.

The second important change is the sorting algorithms that were improved. Meaning that I simplified them to be efficient but before anything
correct. This change has been made because the sorting of digits was not human friendly. So I took advantage of the issue to completely improve the sorting algorithms.

The third important change is the introduction of new special rules which I let you discover through the documentation.

The fourth and last important change is the new --store-whois argument and logic. Before the introduction of this logic, we were storing the whois WHOIS record into the database. From now, it's only if it is needed by you. The idea behind this is to reduce the size of our database. Please note that the database migrator will follow this argument too.

There is a lot more but that's it for the most important changes!

Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter 😄!

I wish you all the best.
Stay safe and healthy!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed ?

• Typos.
• Badges URL.
• An issue under macOS while using the --multiprocess argument.

What was removed ?

• The db_types directory as it is now obsolete.

What was improved ?

• Some sections of the documentation.
• The standard sorting of file content.
• The hierarchical sorting of the file content.
• The cleanup logic behind the --clean and --clean-all arguments.
• The way we process the mining of information.
• The way we open and close a database (MariaDB/MySQL) connection.

What is new ?

• The usage of SQLAlchemy instead of raw SQL.
  • With an automatic migration routine if the old layout is detected.
• The *.wix.com special rule.
• The *.hpg.com.br special rule.
• The *.000webhostapp.com special rule.
• The *.angelfire.com special rule.
• Introduction of the --store-whois argument and logic.
• A file position tracker which improve the continuation from an interrupted test.
• The mining capabilities are now available through the --domain and --url arguments.
• The multiprocessing mode is automatically deactivated if less than 2 processes are allocated/allowed.
• The --all argument prints everything.
• We now retry 3 times with a sleep of 3 seconds between each while downloading vital files. If the URL is still not reachable after 3 retries, the system will stop.
• CI tests with Python 3.6.11, 3.7.8, and 3.8.5.
• The PYFUNCEBLE_DEBUG and PYFUNCEBLE_DEBUG_ONSCREEN environment variables.

What is coming ?

• Fix of reported issues.
• Improvement of the documentation (help welcome!).
• Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

• spirillen - @spirillen

Teal Blauwbok: Bombardier

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I hope that everything is going right for you and your loved one.
This release just provides a 🔥 issue which was provided by my last release.

Indeed, as I automate everything, I did not notice that the input file was being deleted.

The reason for the 3.2.2 instead of the 3.2.1 is because I did a mistake in the versioning a few days ago which resulted in me being forced to release 3.2.1 to PyPi and all other platforms.

Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter 😄!

I wish you all the best.
Stay safe and healthy!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed ?

• An issue with the deletion of the input source at the end of a test.

What is coming ?

• Fix of reported issues.
• Improvement of the documentation (help welcome!).
• Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

• spirillen - @spirillen

Teal Blauwbok: Moth

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I hope that everything is going right for you and your loved one. This release just provides some fixes and very few new features.

This release includes one important change. Indeed, in the previous version, I included the notion of "shadow file" and made it default.
I now consider that decision as a bad one because it may have created unexpected behaviors especially while talking about execution time for some huge lists.
Therefore, from now, if you want to use that feature, you should use or append the --shadow argument or its configuration equivalent to access it.

The other important thing which is not a change is the deployment and provision of docker images and conda packages to the grand-public.
That means that from now, you can install PyFunceble (or its -dev counterpart) from PyPi, AUR, Docker Hub, and Anaconda Cloud. I spend a huge amount of time to automate the process. So, let me know if something is wrong with the two new repositories.

Let me know what you think or how you use PyFunceble with #PyFunceble or @PyFunceble on Twitter 😄!

I wish you all the best.
Stay safe and healthy!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed ?

• An issue with the creation of files when the output/ directory is missing.
• An issue with some analytic files which were named NoneNone.
• An issue with the quiet mode which was not completely quiet while using the -m argument.

What was removed ?

• The direct dependency on urllib3.

What was improved ?

• The documentation.
• The tests of the AdBlock decoder.
• The way we handle subdomains which are behind a public suffix.
• The way we generate the "shadow file". It can now take advantage of multiprocessing.

What is new ?

• Deployment to Docker Hub.
• Deployment to Anaconda Cloud.
• While testing domains, we convert them to lowercase.
• CI tests with Python 3.6.10, 3.7.7, and 3.8.3.
• A way to define the location of the output/ directory. It can be done through the PYFUNCEBLE_OUTPUT_LOCATION environment variable.
• The "shadow file" is not generated, nor used by default. You should explicitly give us permission to generate and use it through the --shadow argument or its configuration counterpart.

What is coming ?

• Fix of reported issues.
• Improvement of the documentation (help welcome!).
• Introduction of possible and interesting features.

---

Contributors

Thanks to those awesome people who contributed to this release!

• NeolithEra - @NeolithEra
• spirillen - @spirillen
• ZeroDot1 - @jawz101

Teal Blauwbok: Ladybug

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I hope that everything is going right for you and your loved one. This release took a bit of time because it was hard to find some time to give it some love. Anyway, here we are!

This release includes 2 important changes. The first one, is the fix of the way we connect to your MariaDB or MySQL database. There was a bug because I assumed that PyMySQL will close the connection by itself while using the with-statement. It turns out that it was not the case.

The second one and probably the most important one, is the inclusion of the self-cleaning of the inactive database. After collecting data in your local storage, this feature was highly wanted and here it is. Indeed, before today, the system was collecting and saving information into the inactive database but it never cleaned the unneeded one. From now, and after 28 days sitting in there, a subject will be automatically removed.

I spend most of my time restructuring the documentation so that we could export it into PDF without having to lose the structure. I was a bit shocked by the generated PDF, that I restructured the whole documentation for it to be rendered correctly. As you know, using PyFunceble requires an internet connection so PDF won't be a big deal. But on my side, I would like to be able (and to start) to download the PDF, print it, and correct what I wrote. That's how I will try to work in the coming month in order to complete the documentation.

By the way, I would like to thank all of those who contributed to the documentation! A lot of work is still waiting but we will come that far one day.

Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter 😄!

I wish you all the best.
Stay safe and healthy!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed ?

• A MySQL/MariaDB connection and possibly communication issue.
• The multiprocessing logic.
• The length of the first line of all output files.
• The coloration of the PyFunceble ASCII-Art at the end which may not have been correct while using the MySQL or MariaDB database.

What was improved?

• The configuration loader (completely rewritten).
• The behavior behind the generation of the .pyfunceble-env file for those who use the MySQL or MariaDB database.
• The help messages.
• The documentation's grammar, spelling, and structure.
• README's grammar and spelling.
• The structure of the inactive database (JSON).

What is new?

• A section in the documentation about some limitation which we may meet while using the MySQL or MariaDB database.
• The --dots argument and logic.
• The self-cleaning of the inactive database after 28 days (configurable).

What is coming?

• Fix of reported issues.
• Improvement of the documentation (help welcome!).

---

Contributors

Thanks to those awesome people who contributed to this release!

• AnonymousPoster - @AnonymousPoster
• jawz101 - @jawz101
• Mitchell Krog - @mitchellkrogza
• spirillen - @spirillen

Teal Blauwbok

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I'm glad to announce this new major version of PyFunceble!

It took me some time, but it's time to release this new version. This version provides a complete different tool. Indeed, the structure and the components of the project are totally different.
The idea of this major version was mainly to review the structure, but it went beyond my expectation thanks to the awesome users and contributors who gave some amazing feedback about PyFunceble(-dev).

One of the most fun I had, among the fix and development of new features, is the implementation of a total automation behind the generation and distribution of the IANA and Public Suffix databases file and the User-Agents. Indeed, before this version, I had to generate them and your own instance of PyFunceble had to fetch it after each release. From now, they are totally automated, and your own instance of PyFunceble will fetch and get the latest one when it judge that it's necessary to update your local version (generally after 24 hours).

This version is also the beginning of a new adventure, some new features and ideas.

Between what you may discover by yourself in this version, there is the support of GitLab CI/CD, IPv6 and even DNS over TCP to just name them.
As a list maintainer, this version provides you a switch which will checks the reputation of the subjects in your list with the help of the AlienVault's public reputation data. I highly invite you to check that as I'm thinking about the inclusion or improvement of reputation data in the future.

As a lot has been fixed, therefore, I'm not going to list them here, but if there are questions about the changes in details or a behavior which changed, please let me know on Keybase!

Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter 😄!

I wish you all the best!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was improved?

• The project structure and source code.
• The documentation.
• The help menu/message.

What is new?

• The support of GitLab CI/CD.
• The support of Python 3.8.
• A way to check the reputation of a given subject (domain and IP).
• Then .onion TDL is not INVALID anymore.
• The support of the port when you give a custom DNS server.
• The possibility to give a MySQL/MariaDB Unix Socket file as hosts.
• The possibility to work from 2 branches (distribution and test branch) with the CI mode.
• *.github.io which return 404 are returned as INACTIVE.
• The possibility to fork the CLI behavior (file generation) from the API.
• A proper logging mechanism for better debugging.
• PyFunceble now initiate and configure the GIT repository for you under the CI mode.
• The tests where completely rewritten.
• You will get some warning and message from the CLI if your configuration are a bit extreme.
• The possibility to set a cooldown time between each test in order to reduce stress and pressure to DNS, WHOIS and target servers.
• We now look for CNAME, DNAME, A, and AAAA when the NS record is not given.
• Automation of a lot of administrative tasks that I had to do manually. Indeed, from now, the IANA, Public Suffix and even the User-Agent are automated in order for you to always get the latest one.
• The doubleclick SPECIAL rule is removed.
• The AdBlock decoder now decodes everything possible.
• ... and a lot more to discover!

What is coming?

• Fix of reported issues.
• Improvement of the behavior of the Inactive DB.
• Improvement of the documentation.

Migration from 2.x

The migration to 3.x is normally done automatically through the configuration checker.

In some rare case (and before submitting a new issue), you may delete your configuration files (.PyFunceble_production.yaml and .PyFunceble.yaml) first.

---

Contributors

Thanks to those awesome people who contributed to this release!

• Daniel - @dnmTX
• Imre Kristoffer Eilertsen - @DandelionSprout
• jawz101 - @jawz101
• Kowith Singkornkeeree - @kowith337
• Mitchell Krog - @mitchellkrogza
• opav - @opav
• ScriptTiger - @ScriptTiger
• sjhgvr - @sjhgvr
• spirillen - @spirillen
• ZeroDot1 - @ZeroDot1

Green Galago: Skitterbug

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

Here is a quick fix of the --clean-all argument.
Indeed before this release when --clean-all was used along with the MySQL, SQLite or MariaDB database type, everything was literally cleaned.

From now, everything except the WHOIS database table is deleted. Why not the WHOIS database?
It simple, it's actually a static file which saves information about domains and is used every time you retest something with PyFunceble to
reduce test time.

Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter 😄!

I wish you all the best!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was improved?

• The --clean-all logic while using a database type different from JSON.

What is coming?

• Fix of reported issues.
• Review of the tests.

---

Contributors

Thanks to those awesome people who contributed to this release!

• Mitchell Krog - @mitchellkrogza

Green Galago: Spider

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

Here is a quick fix of the --local argument (and its configurable equivalent).
Indeed before this release in some rare cases, it is possible that an AttributeError might be raised.

For more accuracy, I renamed the source IANA to Syntax. That's it for this new release!

Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter 😄!

I wish you all the best!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed?

• An AttributeError which appear in some rare cases while using the --local argument.

What was improved?

• The accuracy of the source for INVALID output. It is not IANA but SYNTAX.

What is coming?

• Fix of reported issues.
• Review of the tests.

Green Galago

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

Hello, World!

I'm glad to announce this new major version of PyFunceble!

Many of you (if not everybody) wrote, talked and/or mentioned that PyFunceble is a great idea but it takes some times if not days to test lists.
Let me invite you to review your input with this new version which offers the usage of multiprocessing (as an option) while testing for files.

This new version includes many improvements, review and sometimes complete rewrite of most (if not all) subsystems. As we are all human, I strongly invite you to report any issue which you might meet or get with this new version. A lot of things have been taken into consideration to minimize issue but I'm and you should still stay aware that even if we are stable enough, bugs can emerge from nowhere.

Let me know what you think or how you use PyFunceble with #PyFunceble on Twitter 😄!

I wish you all the best!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

What was fixed?

• The generation of the percentage.txt file which did not work when the quiet/silent mode was in use.
• The Git command output which was not showing up while using PyFunceble under Travis CI.

What was improved?

• The project structure.
• The documentation structure.
• The documentation.
• The way we do a DNS Lookup.
• The way we determine, manage and return the status of a subject.
• The way we imported some entities.
• The way we share states and information across classes/subsystems.
• The way we generate/update iana-domains-db.json
• The Python API (Mostly backend).
• The way we load the configuration file.
• The way we clean what needs to be cleaned.
• The way we clean everything we generated.
• The way we handle some exceptions.
• The way the databases/class/subsystems are structured.
• The inactive database subsystem.
• The whois database subsystem.
• The auto-continue subsystem.
• The default user-agent.
• iana-domains-db.json.
• public-suffix.json.
• The CLI dispatcher.

What is new?

• The support of some custom DNS Server for DNS Lookup.
• The environment variable PYFUNCEBLE_CONFIG_DIR which will replace PYFUNCEBLE_OUTPUT_DIR in the future.
• The support of dotenv files for environment variable declaration.
• The possibility to make a DNS lookup through our Python API.
• The possibility to make a WHOIS lookup through our Python API.
• The support of Python 3.7.3.
• The usage of Codacy code review.
• The test of a file using multiprocessing (optional).
• Logs sharing is now disabled by default.
• The possibility to generate and fill a normal output/ directory structure from the API.
• The possibility to use a different database type: SQLite, MySQL, MariaDB ou JSON.
• The possibility to test complements.
  • Complements are for example www.example.org if example.org is given and vice-versa.
• We now ignore the well-known list of reserved IPv4 address.
  • If you need to test such IPs please use the --local argument from the CLI or its configuration equivalent.
• A safety check if a domain is syntactical INVALID.
  • Indeed, we try to resolve to confirm it's invalidity.
• The percentage which shows up (if needed) in simple mode.
• The version number is now appended into all generated files.
• The execution time (when needed) is now appended to percentage.txt.
• The support of explicit port while testing for URL.
• The possibility to know the status of subject indexed into the inactive database.
• The separation of API, CLI Simple, CLI File, CLI Multiprocessing in the code for more clarity.

What is coming?

• Fix of reported issues.
• Review of the tests.

---

Contributors

Thanks to those awesome people who contributed to this release!

• Daniel - @dnmTX
• Mitchell Krog - @mitchellkrogza
• ZeroDot1 - @ZeroDot1

Blue Bontebok: Caterpillar

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟
Hello, World!

This release provides a minor but still important fix of the auto continue subsystem. Indeed, thanks to @dnmTX, I was able to find out that the auto continue subsystem was working wrongly. In fact, I was first filtering the whole list before trying to get the part we are going to test. Now I get the part we are going to test then process the filtering of all elements which are into the different databases.

Otherwise, because it was asked to me privately, we now write stdout with the output of the git push command when under Travis CI.

Finally, dear Arch Linux (and derivated) users, please note that from now, the package name changed. Indeed, it is now pyfunceble for the stable version and pyfunceble-dev for the development version.

I wish you all the best!

🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟🎉 🌟

---

What was fixed?

• The autocontinue subsystem.

What was deleted or moved somewhere else?

• The counter indexes from the configuration file are now internal.

What was improved?

• The way we initiate the autosave subsystem.
  • Indeed, we now ensure that the directory structure has been built before initiating this subsystem.

What is coming?

• Addition of features proposed by individuals or organizations via GitHub issue, GitHub comments, email, Keybase (Personal) or Keybase (PyFunceble Chat).
• Fix of reported issues.
• Improvement of the actual features.
• Review of the tests.

---

Contributors

Thanks to those awesome people who contributed to this release!

• dnmTX - @dnmTX