Gateway info is not valid

[rugal0123]

Greeting,
i tried to use your libraries to test on my esp32 and somehow end up with this error. I've tried with 2 different router, 1 is my ASUS TM- AC 1900 and the other is my Samsung S7 Mobile HotSpot and Tethering. I've double checked on the require "filled_in" info and tried with different listen port, also tried to change the LEASE_DURATION. Please help i'm getting lost...

[ofekp]

No device on your network responded to the M-SEARCH message.

Can you confirm that your internal network is 192.168.29.0? do the routers have the same address format (except for the last number)?
Can you please check that you router has UPnP enabled?

I do not know Samsung S7 Mobile HotSpot but I am guessing it is cellular network? I am not sure this can work since cellular networks to not give you much control, so I am not sure you will be able to check what I asked on this router.

[rugal0123]

Hi ofekp, thanks for the response. I'm able to setup port forwarding manually and verify it's working with my router by connecting to my esp32 from outside network. But i have no luck with trying to implement the libraries. I also set the UpnP feature enable at first.
Yes, the Samsung S7 Hotspot is just the cellular network as a 2nd test case for me. I was hoping it should work with my Asus router as first.

[ofekp]

This seems to be related to issue #35
I have merged the code few week ago, but it is not deployed to the Arduino Library Manager yet.

Can you take the code from master branch and try again?
Simply use the code as you see it GitHub rather than use the library from the Library Manager.

Let me know how it went please.

[rugal0123]

I just deleted the libraries and used the latest from GitHub as you recommended but still no luck. Same result again, my router completely ignored the M-SEARCH message...

I've checked the router log to verify the Upnp feature is enabled

[ofekp]

why is it listening on 5351?
Is that the UPnP? should be 1900...
Do you have access to 'upnpc' bash program? If so please show me the output for upnpc -l

Make sure it looks like this:

I did see a post about issues with UPnP specifically with your router:
https://community.t-mobile.com/thread/140179

Have you tried that firmware update?

[rugal0123]

I just bought a new asus router RT-AX58U to try but still got the same result after update the firmware to latest...

I'm not sure about the 'upnpc' bash program but this is the log report from the router web page after i disabled and re-enabled the UPnP service. No idea why all Asus router keep listening on port 5351 by default...


I guess Asus routers doesn't work at all..

[ofekp]

I am not yet sure it actually means they listen on that port, it might mean something else.

Can you please read this short post and perform the actions described in it?
https://zapek.com/blog/how-to-fix-upnp-igd-not-working-on-asus-rt-ac87u/

[rugal0123]

Yes i've tried that also but not working either. For the RT-AX58U router, look like they have the "IGMP snooping" feature enabled by default.

[ofekp]

Can you change this line to the 5351 port:
https://github.com/ofekp/TinyUPnP/blob/master/src/TinyUPnP.h#L16

If that does not work, please restore 1900 port.

Then, we'll need to set you up with upnpc tool.
Do you have a linux env somewhere? Old computer, Raspberry Pi? Anything?

[rugal0123]

Hi ofekp, i did try to mess with the port number from your source code. I just read that Asus router doesn't allow Upnp feature to the WAN, LAN devices only... I guess that's why no matter if i turn on or off Upnp feature there's no response at all...
Here's the discussion:
https://www.snbforums.com/threads/new-upnp-exploit-affecting-most-asus-routers-upnproxy-blackhat-proxies-via-nat-injections.46011/
Maybe i will swap out with another router with different brand tomorrow to see how's the library works ...

[ofekp]

That looks unfortunate...
But since you are contacting the router from the LAN I still don't think it should be a problem. The router should respond to M-SEARCH messages coming from the LAN, wifi or not...

I am willing to keep looking for the issue, but we will need to execute upnpc commands from within the LAN.

Do you have any other device in your network that is using UPnP and puts rules dynamically in your router?

[rugal0123]

yes, you are right. I just tried with a Netgear router and still no different... Currently, i don't have any device that using UPnP in my network.

[ofekp]

The plot thickens!
Please describe your network to me. Nice and easy, I want to know how many routers you have specifically. How many switches, how many devices. Anything you can think of please.

I found a workaround for upnpc command in windows environment:

• go to http://miniupnp.free.fr/files/
• look for "latest files"
• in the table, find the one file that is called "upnpc-exe" (upnpc-exe-win32-20150918)
• download the folder and extract it somewhere (no installation needed)
• open win terminal (command prompt)
• enter the following command (change the folder path as needed):
  C:\Users\user-name>C:\Users\user-name\Downloads\upnpc-exe-win32-20150918\upnpc-shared.exe -l

Then please copy paste the output here.

I still need to know the layout of your network too.

[rugal0123]

I use Comcast cable internet and have only 1 modem and 1 router. Currently running with 1 raspberry Pi that i can connect remotely with tightvnc service, 2 smart plug devices that i can control from anywhere, 1 PS4 and several mobile devices (iphones) and Smart TVs... very standard i think.

[ofekp]

OK, sounds standard, please try the upnpc for windows and and let's see what we get.

[rugal0123]

Hi ofekp, I just ran the tool you suggested and here's the output...

[ofekp]

Made a PR for you, if you feel adventurous (I have yet to test on my device) please give it a shot:
#53

Updated the PR. Please make sure to use commit with hash 94dc6cb, thanks.

[rugal0123]

I've compiled and use the new version #94dc6cb you provided but same error...

[ofekp]

Darn, I made a mistake that caused it to behave the same... new hash is b375e92

But please note that the output indicates that you're using the library from Arduino Library Manager and not from the branch I created.

You'll need to go to the folder where the package is and override the code there in both .h file and .cpp file.
Folder should look something like this:
D:\arduino-1.8.9\libraries\TinyUPnP\src
There you'll find the files that you need to change.

[rugal0123]

I've deleted the old library from Arduino Library Manager and use the branch that you created. The output displayed different but still no luck...

[ofekp]

Still timeout?
Can you please attach a text file of the output in following posts?

Seems like your Netgear router is also prone to UPnP issues:
https://community.netgear.com/t5/General-WiFi-Routers-Non/UPnP-still-broken-on-R6400/m-p/1639401

The claim that:
"UPnP is fixed by R6400 firmware version 1.0.1.46."

Can you please check what firmware you're running there?

[rugal0123]

Here's the output file
output.txt
The version i got is newer 1.0.4.84.

[ofekp]

so no "timeout" message after the M-SEARCH is sent? Did you wait long enough?

[rugal0123]

Yes, i didn't capture all of them in the text file. My bad. This is the full log.
output.txt

[ofekp]

Added another device type, please give the PR another try.

[rugal0123]

Just did. Same result. Here's the log.
output2.txt

[ofekp]

Can you type in the path to the xml file in your browser that was printed by the upnpc command that you attached in this comment and attach the result here please?

[rugal0123]

Hi ofekp, this is what i got after entered the info you asked
logfile.txt

[ofekp]

Now this please:
http://192.168.1.1:5000/Public_UPNP_WANIPConn.xml

And this:
http://192.168.1.1:5000/Public_UPNP_WANPPPConn.xml

And this:
http://192.168.1.1:5000/Public_UPNP_Layer3F.xml

😄

[rugal0123]

Here you go !
logfile_WAN.txt
logfile_WANPP.txt
UPNP_layer3F.txt

[ofekp]

Can you give it a try with master branch but with the two changes in the ino file?
Until I'll find new things to check, which is getting pretty hard BTW 😄

Just to be clear, talking about these two additions in the ino file:

1. #include "esp_wifi.h" // needed to call esp_wifi_set_ps`
2. esp_wifi_set_ps(WIFI_PS_NONE)

[rugal0123]

Just tried with master branch, specifically with two additions in the ino file as you suggested. Same result. Don't give up !

[ofekp]

Is this a dual band router? If so, can you try to disable 5Ghz band and try again?
If that won't work (which I guess it won't) please try this new branch.

[rugal0123]

Yes, both of them are dual band routers. I tried with new branch but no luck ...

[ofekp]

Please try the same PR again, this should do the trick this time.

I extended the ino example for ESP32 too, so feel free to use that too.

[rugal0123]

Hi ofekp, i tried the new PR and look like it worked with some conditions after a few hit and miss. Would you take a look at the error_log. It caused the esp32 processor reset a few times with error "Guru Meditation Error: Core 1 panic'ed (LoadProhibited). Exception was unhandled." It worked when i use Listen_port 17996 but no luck with port 80.
log_db37355_new_ino.txt

[ofekp]

Did you mean that no issue and no exception occurred when using port 17996, and that flakiness was only when using port 80, meaning port 80 sometimes was OK? Can you elaborate on this if I got it wrong?

Can you check this for me:

1. Keep using #define LISTEN_PORT 80
2. In the ino file, in the line WebServer server(LISTEN_PORT); change the port to hard coded 17996

This will tell us if the exception is from TinyUPnP or from ESP32 WebServer package.

The decoded stack trace made no sense to me:

Decoding stack results
0x4014da42: tcp_receive at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/lwip/lwip/src/core/tcp_in.c line 1206
0x400db0b1: TinyUPnP::getIGDEventURLs(_gatewayInfo*) at D:\arduino-1.8.9\libraries\TinyUPnP\src\TinyUPnP.cpp line 768
0x400db149: TinyUPnP::getIGDEventURLs(_gatewayInfo*) at D:\arduino-1.8.9\libraries\TinyUPnP\src\TinyUPnP.cpp line 783
0x400d8316: TinyUPnP::upnpRuleToString(_upnpRule*) at D:\arduino-1.8.9\libraries\TinyUPnP\src\TinyUPnP.cpp line 1045
0x400da2d6: TinyUPnP::printAllPortMappings() at D:\arduino-1.8.9\libraries\TinyUPnP\src\TinyUPnP.cpp line 985
0x400dbf7b: HTTPClient::beginInternal(String, char const*) at Arduino15\packages\esp32\hardware\esp32\1.0.4\libraries\HTTPClient\src\HTTPClient.cpp line 271
0x40088b7d: spi_flash_protected_read_mmu_entry at /home/runner/work/esp32-arduino-lib-builder/esp32-arduino-lib-builder/esp-idf/components/spi_flash/flash_mmap.c line 303

[rugal0123]

No, the flakiness happened on both port 17996 and 80 but port 17996 is able to mapped after multiple attempts, port 80 always failed. In both case, i noticed there are errors "Guru Meditation Error: Core 1 panic'ed (LoadProhibited).
I just checked as you suggested:
"Keep using #define LISTEN_PORT 80
In the ino file, in the line WebServer server(LISTEN_PORT); change the port to hard coded 17996"
i got the same error, here's the log file.
log_port80.txt

[ofekp]

Can you try the branch again, I added a few logs that may help.
Can you also attach here the available space on the ESP32 after flushing the the code (printed by the Arduino IDE after the code is uploaded).
Thanks.

[rugal0123]

Here's the available space:

Sketch uses 779254 bytes (37%) of program storage space. Maximum is 2097152 bytes.
Global variables use 52480 bytes (16%) of dynamic memory, leaving 275200 bytes for local variables. Maximum is 327680 bytes.
Here's the log:
log_48aa06d.txt

[ofekp]

Hey, sorry for the delay, this is quite hard to figure out, since it originates from the WiFiClient package which is not mine.

Please make sure:

1. All boards in "Boards Manager" are updated
2. All libraries in "Library Manager" are updated
3. No pins are connected to the ESP32

Now, please try the last branch (issue51_3) again.

[rugal0123]

Hi ofekp, no problem. I just tried the last branch and made sure everything as you suggested but still no luck. Here's the log file.
log_af9fa.txt

[ofekp]

Can you do a Postman call please?

Method:

POST

URL:

192.168.29.1:57515/ctl/IPConn

Headers:

Connection:close
Content-Type:text/xml; charset="utf-8"
Host:192.168.29.1:57515
SOAPAction:"urn:schemas-upnp-org:service:WANIPConnection:1#GetSpecificPortMappingEntry"

Body:

<?xml version="1.0"?>
<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/" s:encodingStyle="http://schemas.xmlsoap.org/soap/encoding/">
<s:Body>
<u:GetSpecificPortMappingEntry xmlns:u="urn:schemas-upnp-org:service:WANIPConnection:1">
<NewRemoteHost></NewRemoteHost>
<NewExternalPort>80</NewExternalPort>
<NewProtocol>TCP</NewProtocol>
</u:GetSpecificPortMappingEntry>
</s:Body>
</s:Envelope>

[rugal0123]

This is what i got. Let me know if i missed anything with Postman setup:

[ofekp]

Good job there :) this is perfect.

1. Are you using a good power supply? What kind please? Any way to replace it with something else just for the try?
2. This issue was not present in ESP8266, right?
3. Are you trying both port 80 and some other port?
4. How many times does the process succeeds without rebooting? Can you run this for a while and give a rough probability for success?
5. Please try the branch again, few modifications, unlikely to solve this if you ask me but let's try.

[ofekp]

Kind reminder, I hope all is well.

[rugal0123]

Hi ofekp, Sorry for the late response. I hope you are doing well also. I just tried the new branch build but still experienced the crash issue when loading router information into esp32. As you can see from the log:
log_f267f593.txt
1)The power supply is good. I've never had any problem running with it.
2)I never tried this with esp8266. As far as we tried last time, the library worked with esp8266 so we focused on fixing with esp32.
3) It worked with other ports such as 17997, 17996 but only after several attempt and rebooting. Port 80 never worked.
4) 0 success on port 80. Running for a while wouldn't increase the probability of success either.

[rugal0123]

Update for the esp8266:
_ There's no crash or reboot when running it. I'm able to map other ports such as 17997, 17996, 17995 except port 80. Not sure why but when i tried with port 80 it's registered as 17997...

log_f267f593_esp8266.txt

[ofekp]

Sorry for my delay.

Not sure why but when i tried with port 80 it's registered as 17997

I think it is just a rule that remained on the router, they stay there for a while before being cleaned up, do you think this is what happened?

I am unable to reconstruct this issue, even with port 80 and running for few hours. It seems to be an internal issue related to the ESP-32.

Can you try to run the code on the second core?

Refer to t this: https://techtutorialsx.com/2017/05/09/esp32-running-code-on-a-specific-core/
Relevant code (note that taskCore should be 1 as this is the second core):

static int taskCore = 1;
 
void coreTask( void * pvParameters ){
 
    String taskMessage = "Task running on core ";
    taskMessage = taskMessage + xPortGetCoreID();
 
    while(true){
        Serial.println(taskMessage);
        delay(1000);
    }
}
 
void setup() {
  Serial.begin(112500);
  delay(1000);
 
  Serial.print("Starting to create task on core ");
  Serial.println(taskCore);
 
  xTaskCreatePinnedToCore(
                    coreTask,   /* Function to implement the task */
                    "coreTask", /* Name of the task */
                    10000,      /* Stack size in words */
                    NULL,       /* Task input parameter */
                    0,          /* Priority of the task */
                    NULL,       /* Task handle. */
                    taskCore);  /* Core where the task should run */
 
  Serial.println("Task created...");
}
 
void loop() {
  Serial.println("Starting main loop...");
  while (true){}
}

[ofekp]

@rugal0123 Kind reminder :)

[rugal0123]

Hi ofekp, sorry it's been a while ! I tried your latest version and it worked after a few times of hit and miss. The only issue i saw now is it crashed sometimes when reading the xml file and the esp32 had to reboot. Have you experienced this issue before ? Thanks !

[rugal0123]

Here's the exception decoder:

[ofekp]

Please refrain from posting comments containing the references to other issues on threads that are not related to your specific problem as it may confuse other readers, I am referring to the comment I deleted in issue 70. No worries, just try to avoid that please.

What commit were you actually using?
So sometimes it works and sometimes not, right?
What is the percentage of success after a significant number of iterations?

It is falling on this line
So, this does not seem to be related to TinyUPnP as far as I can tell now.

Let's start from:

1. Run on the second core, as I said in this comment
2. Update all your libraries (Library Manager in Arduino IDE)
3. Update all your boards (Board Manager in Arduino IDE)
4. Try again

I am also noticing that you are trying a different port every time, is that correct? Maybe the issue is occurring only for specific set of ports?

[rugal0123]

1. What commit were you actually using? I'm running TinyUPnP v3.1.4 downloaded from Arduino Library Manager first then tried with branch issue 70, Arduino IDE v1.8.13.
2. So sometimes it works and sometimes not, right? It works eventually after few times of crash and reboot, which is weird.
3. What is the percentage of success after a significant number of iterations? After 4 or 5 times of reboot at least
4. Yes it failed every time on this line :

"String line = _wifiClient.readStringUntil('\r');"

when running verifyPortMapping.
5) I tried to run on different core already and it failed exact on same line as mentioned above.
6) Maybe the issue is occurring only for specific set of ports? No different ports work fine, after few times of crash and reboot as mentioned above...
7) I also removed all the server code and run the upnp alone... Same result, working after few times of crash and reboot.

[rugal0123]

So i read one of your closed issues "ESP32 support - crashs #38" and tried comment out UPNP_DEBUG. The result getting better.

Now my esp32 only crashed once before succeed mapping when addPortMappingEntry, exact on same line as well:
"String line = _wifiClient.readStringUntil('\r');"

We're starting to see interesting pattern here.

[ofekp]

Oh, I am so glad to read that :)
That's great!

I might consider turning it off by default due to this.

Thank you very much for letting me know.

[rugal0123]

Yup. Now if you have any suggestion so that we can reduce the # of reset/crash to 0 it would be even better !

[ofekp]

I will try to think of something, if you happen to be able to run on a different ESP32 or even ESP8266 please do so and let me know if this was reconstructed.
let me know if you'll find out more information.