Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[FEATURE] RPZ Format #623

Open
alsyundawy opened this issue Aug 12, 2021 · 21 comments
Open

[FEATURE] RPZ Format #623

alsyundawy opened this issue Aug 12, 2021 · 21 comments
Assignees
@smed79 @mitchellkrogza @funilrys
Labels
enhancement

Comments

@alsyundawy
Copy link

Can You Make RPZ For My Own DNS Resolver Using bind9
@dnmTX
Copy link

dnmTX commented Aug 12, 2021 •
edited
Loading

Harry @alsyundawy see this issue.
We all left some good ideas there how to do it yourself,so try it,see how it goes 👍
Meanwhile ping @funilrys for assist 👍

@spirillen
Copy link
Contributor

There have been other request for this in the past latest by @Somebodyisnobody wrote a request for wildcarding the records where possible, but not many raised there hands to maintain hose lists 😏 In addition to this, I do maintain wildcard lists in the My Privacy DNS and they are all properly hosted over DNS, See bind script example below. While UHB contains a hole lot more records, as they are collected from various lists, then Mypdns is a handcrafted user contributed lists where all records are manually confirmed. Just never forget your private whitelists when using mypdns 😃

If you follow @dnmTX link then you get all these lines vs 2 lines and the code is NOT RPZ COMPLIANT...

  • zone-name is missing from all records.
  • SOA record is missing. Which WILL cause stalled cache

https://github.com/AdguardTeam/cname-trackers/blob/4e8f42456cb616adfc1f4a17faed1a4f0a4dc2b8/combined_disguised_trackers_rpz.txt#L679-L1027

The two right lines for actonservice.com would be looking like this + SOA line (en contra 348 lines).

tracking.mypdns.cloud.  86400   IN      SOA     ns1.mypdns.org. need.to.know.only. 2021042867 300 60 604800 3600
actonservice.com.tracking.mypdns.cloud	CNAME	.
*.actonservice.com.tracking.mypdns.cloud	CNAME	.

Bind setup example of https://github.com/DNS-RPZ/dns-rpz-integration (Outdated use example below)

        // RPZ
        response-policy { zone "adult.mypdns.cloud";
                zone "adware.mypdns.cloud";
                zone "coinblocker.mypdns.cloud";
                zone "gambling.mypdns.cloud";
                zone "malicious.mypdns.cloud";
                zone "phishing.mypdns.cloud";
                zone "pirated.mypdns.cloud";
                zone "redirector.mypdns.cloud";
                zone "rpz.mypdns.cloud";
                zone "safesearch.mypdns.cloud";
                zone "spam.mypdns.cloud";
                zone "spyware.mypdns.cloud";
                zone "tracking.mypdns.cloud";
                zone "typosquatting.mypdns.cloud";
                zone "whitelist.mypdns.cloud";
        };

zone "adult.mypdns.cloud" IN {
        type slave;
        file "slave/adult.mypdns.cloud";
        masters port 5303 {188.176.49.219;};
        };

zone "adware.mypdns.cloud" {type slave; file "slave/adware.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "coinblocker.mypdns.cloud" {type slave; file "slave/coinblocker.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "gambling.mypdns.cloud" {type slave; file "slave/gambling.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "malicious.mypdns.cloud" {type slave; file "slave/malicious.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "phishing.mypdns.cloud" {type slave; file "slave/phishing.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "pirated.mypdns.cloud" {type slave; file "slave/pirated.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "redirector.mypdns.cloud" {type slave; file "slave/redirector.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "rpz.mypdns.cloud" {type slave; file "slave/rpz.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "safesearch.mypdns.cloud" {type slave; file "slave/safesearch.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "spam.mypdns.cloud" {type slave; file "slave/spam.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "spyware.mypdns.cloud" {type slave; file "slave/spyware.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "tracking.mypdns.cloud" {type slave; file "slave/tracking.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "typosquatting.mypdns.cloud" {type slave; file "slave/typosquatting.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };
zone "whitelist.mypdns.cloud" {type slave; file "slave/whitelist.mypdns.cloud"; masters port 5303 {188.176.49.219;}; };

logging{
        channel normal-log{
                // alternatively use default_syslog above to log
                // everything apart from RPZ info to syslog and omit
                // the file statement below
                file "/var/log/named/named.log" versions 3 size 1m;
                severity info;
        };
        channel named-rpz {
                // change path as appropriate
                file "/var/log/named/rpz.log" versions 3 size 250k;
                severity error;
        };
        channel rpzlog {
                file "/var/log/named/rpz.log" versions unlimited size 100m;
                print-time yes;
                print-category yes;
                print-severity yes;
                severity info;
    };
    category rpz {
                rpzlog;
        };
        category rpz{
                named-rpz;
        };
        // everything else
        category default{
                normal-log;
        };

@ppaeps
Copy link

ppaeps commented Aug 13, 2021

@spirillen I wrote the patch to generate the cname-trackers list in RPZ format.

The zone-name is not missing from the records. The records do not have a trailing . which, by DNS convention, makes them part of the containing zone.

I omitted the SOA (and NS) record on purpose. The response policy zones live in the namespace of the administrative domain setting the policy. They will be different for every user of the data. To make the zone "RPZ compliant", all you need to do is prepend correct SOA and NS records for your administrative domain. My patch documented this requirement in the README.md file of the cname-trackers repo.

Distributing the data in "RPZ format" without SOA/NS records is the most flexible solution. Different administrative domains will feed the data to their DNS servers differently. The simplest setups only need to cat the data with a fixed preamble. More complicated setups can generate the records they need as part of their pipelines.

For what it's worth, this is (part of) how I integrate the cname-trackers list in my DNS:

# This script runs from cron and updates the zone if the source files change

next_serial() {
  d=$(date +%Y%m%d)
  awk "/; Serial/{
    if (\$0 ~ /${d}/) \$1++; else \$1=${d}00; print \$1
  }" ${1} 2>/dev/null || echo "${d}00"
}

write_header() {
  serial=$1
  url=$2
  srchash=$3

  echo "\$TTL 300"
  echo ""
  echo "; Generated from ${url}"
  echo "; srchash: ${srchash}"
  echo ""
  echo "@       SOA     ${mname}. ${rname}. ("
  echo "                ${serial}"
  echo "                43200"
  echo "                900"
  echo "                604800"
  echo "                3600 )"
  echo ""
  echo "@       NS      ${mname}."
  echo ""
}

update_from_plain_rpz() {
  url=$1
  outfile=$2

  srchash=$(fetch_source ${url} ${outfile}) || return $?
  serial=$(next_serial ${outfile})
  write_header ${serial} ${url} ${srchash} > ${tmpdir}/blacklist.rpz
  cat ${tmpdir}/blacklist.txt >> ${tmpdir}/blacklist.rpz
  cat ${tmpdir}/blacklist.rpz > ${outfile}

  echo ${serial}
}

@spirillen
Copy link
Contributor

spirillen commented Aug 13, 2021 •
edited
Loading

The simplest setups only need to cat the data with a fixed preamble.

No the simplest is to run on DNS port :53 where DNS servers are exchanging data based on a SOA record. CAT / wget / curl or what ever are for people who doesn't understand DNS and it's sub process RPZ and think you are turning a perfect DNS axfr funktion into a plain a hosts file that requirs to download all data to update just one record (kb vs MB).

The zone-name is not missing from the records.

Yes it is, while you are speaking of the a particular DNS program Bind9 then the real zone data exchange are performed in ABSOLUTE records. Yes you can cheat that in Bind9 by adding $zone-name in the top of a local zone record to have it applied to ALL zone records within that file. However that does NOT correct the failed zone record linked above as it is not a fully complaint record. and ONLY supported by bind9+

For what it's worth, this is (part of) how I integrate the cname-trackers list in my DNS:

RPZ + DNS are not to be run from a cron job... is it a dumb hosts file you are operating here?? or is it a DNS zone which holds RPZ records updated in accordance to the SOA serial?

The fact is, your script only partial support Bind9's DNS software. While if you IE. tried to setup one of the RPZ zone files the right way DNS software to DNS software exchange you'll see they are operating properly without any CRON based commend but simply uses the SOA record to determine if a new exchange is needed.

I omitted the SOA (and NS) record on purpose.

True the NS record isn't needed in RPZ zones, in my cases they are added as a simple workaround to satisfy pdnsutil (Bash tool for PowerDNS)

you can do a simple test on your degraded Bind9 installation a do a drill axfr rpz-zone-name.tld and see how the exchange for looks like!!

Here is a example of coinblocker.srv

drill axfr coinblocker.srv @35.156.219.71 | head
coinblocker.srv.        86400   IN      SOA     need.to.know.only. hostmaster.deteque.com. 1628812861 28800 60 432000 60
coinblocker.srv.        86400   IN      NS      localhost.
seed.ac.coinblocker.srv.        86400   IN      CNAME   .
bittrex.academy.coinblocker.srv.        86400   IN      CNAME   .
hostingcloud.accountant.coinblocker.srv.        86400   IN      CNAME   .
www.hostingcloud.accountant.coinblocker.srv.    86400   IN      CNAME   .
miner.ad.coinblocker.srv.       86400   IN      CNAME   .
www.miner.ad.coinblocker.srv.   86400   IN      CNAME   .
aba.ae.coinblocker.srv. 86400   IN      CNAME   .
www.aba.ae.coinblocker.srv.     86400   IN      CNAME   .

Notice the FULL zone record in the following syntax: rpz-record.$zone.name TTL RPZ-record-types VALUE

Full RPZ-record-types scheme can be found here

UPDATE:
You can of curse also read the documentation for Bind9+RPZ at http://www.zytrax.com/books/dns/ch9/rpz.html for how to actually operate the software and the a RPZ zone.

When I said all of this, then yes, an zone file can be exchanged in a flat-file style, but it have to be a fully valid zone file like https://urlhaus.abuse.ch/downloads/rpz/ or https://sslbl.abuse.ch/blacklist/sslbl.rpz with a fixed update frequents like 300 seconds

@ppaeps
Copy link

ppaeps commented Aug 13, 2021

@spirillen I'm sorry my example confused you. Those two functions are part of a script that runs from cron which generates RPZ zones for one of my networks. The RPZ zones are served by NSD and distributed to my resolvers (which run unbound) using AXFR.

As for the FQDN discussion: Every DNS server I have encountered which consumes zonefiles in a format like that described in RFC 1035, interprets the absence of an empty terminal label to mean the name belongs to the last start of authority rather than as an FQDN.

@spirillen
Copy link
Contributor

Those two functions are part of a script that runs from cron which generates RPZ zones for one of my networks

I know, and as said, I believe it is as wrong as it can be 😉

IF you are unable to retain the data with AXFR, then it should be possible to "just" download it and the reload the zone. I do this for the PowerDNS like this

sslbl.abuse.ch.sh 
#!/usr/bin/env bash

zoneName="sslbl.abuse.ch.rpz.zone"
dir="/var/lib/powerdns/zones.slave.d"
zoneFile="${dir}/${zoneName}"

cd ${dir}

lftp -c "set xfer:clobber on; set xfer:use-temp-file on; set xfer:timeout 15; get https://sslbl.abuse.ch/blacklist/sslbl.rpz -o ${zoneName}"

As for the FQDN discussion: Every DNS server I have encountered which consumes zonefiles in a format like that described in RFC 1035, interprets the absence of an empty terminal label to mean the name belongs to the last start of authority rather than as an FQDN.

That's true 👍 But if you are sticking to relay on the abuse of such rules, you will get a number of errors over time. I can and will not recommand anybody to relay on this versus generating a fully qualified zone file.

Interesting, you are using NSD to serve RPZ ??? hmm will have to re-read the docs before commenting on the syntax for that.

Non the less, I do believe the best would be to generate that flat-file to a fully compliant bind zone syntax with absolute records to avoid any issues.

The RPZ zones are served by NSD and distributed to my resolvers (which run unbound) using AXFR.

Is these public available servers for other to use, in that case I believe you should announce them.

This was some detour with how to import badly formatted records with a cron Versus just have a valid RPZvia AXFR from a primary server.

Any further discussion on this should be in another thread as it is OT to OP.

@alsyundawy
Copy link
Author

alsyundawy commented Aug 14, 2021 •
edited
Loading

i have custom domain name to block using rpz but i dont undertand about scripting using bash, sed aor awk, can anybody help me to make like

input domain.txt
abcd.com
aaskakas.net
bsbs.mkns.org
*.bsjlwlw.info

output domain.rpz

abcd.com IN CNAME dns-rpz
aaskakas.net IN CNAME dns-rpz
bsbs.mkns.org IN CNAME dns-rpz
*.bsjlwlw.info IN CNAME dns-rpz

@spirillen
Copy link
Contributor

You can not use CNAME dns-rpz to anything...

The valid syntax's and codes can be found here: https://mypdns.org/mypdns/support/-/wikis/RPZ-record-types

The short version for how to do this is:

for lines in domain.txt
do
    echo -e "$lines\tCNAME . ; UHB"
done

You can then prepend the script with the write_header suggestion from @ppaeps in #623 (comment)

But as mentioned earlier, without a list of records that should be handled by wildcard, you actually doesn't have any benefit by converting this into a RPZ, the server load would be the same as if you served it any other way

Alternatively, I have been building on a hosts2RPZ in the past: https://github.com/DNS-RPZ/mypdns-hosts2rpz Can't remember if I had it working, but give it a spin and let me now. Could be we could spend some time on fixing it if it isn't working

@alsyundawy
Copy link
Author

You can not use CNAME dns-rpz to anything...

The valid syntax's and codes can be found here: https://mypdns.org/mypdns/support/-/wikis/RPZ-record-types

The short version for how to do this is:

for lines in domain.txt
do
    echo -e "$lines\tCNAME . ; UHB"
done

You can then prepend the script with the write_header suggestion from @ppaeps in #623 (comment)

But as mentioned earlier, without a list of records that should be handled by wildcard, you actually doesn't have any benefit by converting this into a RPZ, the server load would be the same as if you served it any other way

Alternatively, I have been building on a hosts2RPZ in the past: https://github.com/DNS-RPZ/mypdns-hosts2rpz Can't remember if I had it working, but give it a spin and let me now. Could be we could spend some time on fixing it if it isn't working

found it using python3 script from @iffanux

import os

print('Merge the last database')

inputFile = "input.rpz.tmp"
outputFile = "output.rpz.zone"

# Change your subdomain
my_string = "IN CNAME dns-rpz"
file = open("input.rpz.tmp", "r")
with open(inputFile, 'r') as inFile:
    with open(outputFile, 'w+') as outFile:
     for line in inFile:
        line = line.rstrip("\n")
        outFile.write(line + my_string + "\n")
file.close()
# Delete a tmp file
os.remove("input.rpz.tmp")
print('Database merged successfully')

@spirillen
Copy link
Contributor

That most have been a non standard program he uses that for.

Because this one will not be working for anything standard RPZ

for line in inFile:
        line = line.rstrip("\n")
        outFile.write(line + my_string + "\n")
file.close()

@ghost
Copy link

ghost commented Sep 11, 2021

Check this code at https://github.com/markster101/Unbound

@spirillen
Copy link
Contributor

I can help you cheat on this one

https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L206

https://archive.mypdns.org/w/recursorserver/#rpz-urlhaus-abuse-ch just replace the DNS server ip with 188.176.49.219 😉

You might also run into some issues here, as when they are blocked you won't get the HTTP code 200 and the response could just as well be 301/302/307/308

https://github.com/markster101/Unbound/blob/be23f351b51289e6c995a982c719ee1731c1e962/zone-load#L139

Just a few quick notes as I'm hungry.

@ghost
Copy link

ghost commented Sep 11, 2021 via email

@funilrys
Copy link
Member

funilrys commented Nov 19, 2021 •
edited
Loading

Mhh, It's maybe time to publish our Bind9 RPZ zones.

@spirillen
Copy link
Contributor

Mhh, It's maybe time to publish our Bind9 RPZ zones.

Publish? do you mean releasing information for IXFR for the zone(s)?

@funilrys
Copy link
Member

Well, at the time we (@mitchellkrogza and I) developed and imagined our DNS infrastructure, I had to write a piece of tool that generate the RPZ zones - statically. So, the RPZ zone actually exists within our server ... It's just not public ... There is a tool that automatically fetches all data from here and parses /put them into multiple RPZ zones.

@spirillen
Copy link
Contributor

spirillen commented Nov 19, 2021 •
edited
Loading

There is a tool that automatically fetches all data from here and parses /put them into multiple RPZ zones.

I do not quit understand what you are trying to say here, but if you do have the RPZ zones inside your bind, you only have to setup IXFR for all and share the connection information to for IXFR transfer (not AXFR), that will make the zones works like ordinary zone exchange as any other DNS zone from master to slaves (zone transfer)

It not even resource expensive on a bind as it is flat files over the network

http://www.zytrax.com/books/dns/ch9/rpz.html#config & http://www.zytrax.com/books/dns/ch7/rpz.html

If you like we can setup a ixfrdist as proxy and I would help sharing it as I would append it to my own DNS if I knew the the zone names you have been using

@funilrys
Copy link
Member

It's never incremental. Everything is rebuilt each time.

@spirillen
Copy link
Contributor

It's never incremental.

There you have your next project 😉

PS: powerdns have a nice api for this.... 😏

@alsyundawy
Copy link
Author

when rpz files officially released

@spirillen
Copy link
Contributor

when rpz files officially released

https://infosec.exchange/@mypdns/110845441169549255

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@smed79 smed79

@mitchellkrogza mitchellkrogza

@funilrys funilrys

Labels
enhancement
Projects
Ultimate-Hosts-Blacklist Backlog (public)
Status: 🆕 New
Milestone
No milestone
Development

No branches or pull requests
7 participants
@ppaeps @smed79 @alsyundawy @mitchellkrogza @funilrys @dnmTX @spirillen