Suggestion for Lower Memory Usage

[Pecacheu]

This isn't so much an issue but I have a question. This library really seems to be a massive ROM/RAM hog and I think part of the reason is all those non-PROGMEM global const vars, why use them instead of macros? Such things as the long lists of const uint8_t kDefaultBlahBlah = x found in all the header files.

RAM is a precious resource on ESP CPUs, and while this lib is well within the limits (so this is hardly of immidiate concern), I think it could be improved if the global vars were kept to a minimum, using compiler macro #defines or PROGMEM where possible.

[NiKiZe]

You might want to look into all the work that already has gone into efforts relating to the size.

Please check what is actually included in your build, maybe you should poke your compiler to do the right thing.

[Pecacheu]

Are you saying that those consts don't get compiled into the global namespace to take up space even if they aren't used? Because on my ESP, with default Arduino settings, GDB seems to disagree.

Besides, even for the ones that are used, they could be taking up zero global space if they were a compiler constant instead, it'd only hog memory while calling functions that say, have default parameters set, which would be initialized to that value at runtime (assuming no further optimization occurs), then disposed afterwards.

I mean, I appreciate it if work has already gone in to optimizing this library, and it works well, but I'm just making a suggestion. I haven't observed every inch of the code, so I don't know if there's another reason it has to be done this way. If that's the case then fine, but generally speaking, this is well known as a bad practice in C, especially on embedded systems.

There's a reason that most libraries don't use globals in this way, including the ESP's own SDK libs which use defines and enums. C compilers just can't be trusted to optimize globals well, partly due to the way linking works.

I mean for my main project, I have 6 or 7 other libraries, including OTA code and an HTTP page in PROGMEM for establishing WiFi connection, yet my code was only taking up 28% space and 20% RAM (for globals). This library (IR send + receive) literally doubled that all the way to 40% and 42%! (Granted that includes some extra Serial printing and reading code to go along with, but that shouldn't fill too much space, as I already had a fair bit of Serial code, all strings in PROGMEM of course)

[NiKiZe]

Some documentation
https://github.com/crankyoldgit/IRremoteESP8266#upgrading-from-pre-v25

[crankyoldgit]

This isn't so much an issue but I have a question. This library really seems to be a massive ROM/RAM hog and I think part of the reason is all those non-PROGMEM global const vars, why use them instead of macros? Such things as the long lists of const uint8_t kDefaultBlahBlah = x found in all the header files.

Depending how you use the library, yes it is a huge RAM/ROM hog. There is a shirt-tonne of code in it. As you indicate in a latter comment, you're using the decoder (IRrecv class) which is the most expensive, ram, flash PROGMEM, & CPU-wise.

To reduce the memory/flash/cpu footprint, take a look at these options/comments:

IRremoteESP8266/src/IRremoteESP8266.h

Lines 62 to 87 in 9573cb2

	// Do we enable all the protocols by default (true), or disable them (false)?
	// This allows users of the library to disable or enable all protocols at
	// compile-time with `-D_IR_ENABLE_DEFAULT_=true` or
	// `-D_IR_ENABLE_DEFAULT_=false` compiler flags respectively.
	// Everything is included by default.
	// e.g. If you only want to enable use of he NEC protocol to save program space,
	// you would use something like:
	// `-D_IR_ENABLE_DEFAULT_=false -DDECODE_NEC=true -DSEND_NEC=true`
	//
	// or alter your 'platform.ini' file accordingly:
	// ```
	// build_flags = -D_IR_ENABLE_DEFAULT_=false
	// -DDECODE_NEC=true
	// -DSEND_NEC=true
	// ```
	// If you want to enable support for every protocol *except* _decoding_ the
	// Kelvinator protocol, you would use:
	// `-DDECODE_KELVINATOR=false`
	#ifndef _IR_ENABLE_DEFAULT_
	#define _IR_ENABLE_DEFAULT_ true // Unless set externally, the default is on.
	#endif // _IR_ENABLE_DEFAULT_
	// Supported IR protocols
	// Each protocol you include costs memory and, during decode, costs time
	// Disable (set to false) all the protocols you do not need/want!
	// The Air Conditioner protocols are the most expensive memory-wise.

Using the IRac class is the most expensive of all, as it pretty much includes/uses all the text symbols/objects & code.
Next is using an A/C protocol via it's specific class object. e.g. IRKelvinatorAC, especially if you use A/C class's .toString() method

If you use the IRrecv::decode() method without making any changes to what protocols are supported, then you're including a huge amount of code. i.e. EVERY protocol decoder. Also study that method/class, as there are memory tuning options in there too.

For sending, it's similar. IRac is by far the most expensive, then the individual A/C class, then the lower level IRsend::send() method, which includes ALL of the supported protocols code, then last but not least, the low-level IRsend::sendBlah() which is the least expensive flash/memory wise.

Are you saying that those consts don't get compiled into the global namespace to take up space even if they aren't used? Because on my ESP, with default Arduino settings, GDB seems to disagree.

Now, to your "const vs DEFINE" point. The compiler should be treating these as literals and putting them in the symbol table. They should (in theory) not be taking up stack/variable space. They are(should be) behaving the same as a #DEFINE except they are typed, thus safer. If you're using it with a debugger (e.g. gdb), you're likely including the symbol table with the resultant debug binary, thus including a heap of bloat with your binary. You can't compare a debug binary's memory using with a stripped binary's size/memory usage etc. Also note, namespace is not the same as the global variable space. You may be conflating two different things.

If they were true global variables, yes, I'd agree with you 100%, but const(ants) are not variables.
If g++/the ESP is storing them in memory, or in the global (in-ram) variable memory, rather than flash space, then yes. There is probably an argument for using PROGMEM on all the constants. If you can prove/show me with hard data that is the case for a stripped binary then I'll gladly do that to save space. The intention is for these literals to not take up variable space & to increase code readability.
Note: If a constant/literal isn't used it shouldn't be included in the final binary. i.e. It should be excluded during the object code linking phase of building the executable at the latest, if not prior to that.

[crankyoldgit]

I mean, I appreciate it if work has already gone in to optimizing this library, and it works well, but I'm just making a suggestion. I haven't observed every inch of the code, so I don't know if there's another reason it has to be done this way. If that's the case then fine, but generally speaking, this is well known as a bad practice in C, especially on embedded systems.

FYI, I came across this today on the Arduino Reference site:
Constants defined with the const keyword obey the rules of variable scoping that govern other variables. This, and the pitfalls of using #define, makes the const keyword a superior method for defining constants and is preferred over using #define.
&
In general, the const keyword is preferred for defining constants and should be used instead of #define.

[crankyoldgit]

FWIW, I just did some tests with const variables with variable attribute of PROGMEM over a large portion of the library.
e.g.

const uint64_t kEcoclimDefaultState PROGMEM = 0x11063000FFFF02;

or

const static uint64_t kEcoclimDefaultState PROGMEM = 0x11063000FFFF02;

or

const PROGMEM uint64_t kEcoclimDefaultState = 0x11063000FFFF02;

There was zero change in the size of the RAM use.
i.e.

IRMQTTServer compiled for the D1_mini:
Without:

RAM:   [=====     ]  52.9% (used 43372 bytes from 81920 bytes)
Flash: [=====     ]  52.5% (used 548792 bytes from 1044464 bytes)

With:

RAM:   [=====     ]  52.9% (used 43372 bytes from 81920 bytes)
Flash: [=====     ]  52.5% (used 548792 bytes from 1044464 bytes)

IRrecvDumpV2 compiled for en-AU:
Without:

RAM:   [====      ]  36.9% (used 30216 bytes from 81920 bytes)
Flash: [===       ]  34.9% (used 364940 bytes from 1044464 bytes)

With:

RAM:   [====      ]  36.9% (used 30216 bytes from 81920 bytes)
Flash: [===       ]  34.9% (used 364940 bytes from 1044464 bytes)

So to me, the data/evidence says either the compiler is doing the correct thing already (as expected), or PROGMEM makes no difference to "global" constants. 🤷 I'm out of ideas.

[Pecacheu]

@crankyoldgit That's weird, considering that GDB found that those const variables inhabited the RAM address space and were all taking up room there. But keep in mind, the compiler's RAM usage numbers are just an estimate, they're different than the true numbers that you get at runtime.

It could be that PROGMEM actually does improve the situation at runtime (#define did for me when I did a CTRL+F replace). I would assume that a PROGMEM const would be just as good as define with none of the pitfalls.

Though frankly there's a lot of messy code in this library and bugs I've had with fragmentation after running for a long time, not to mention the codes in the AC class for the LG AC not being quite right for some of the functions, despite being for the exact same remote model I have, so I had to fix them. Oh yeah and a fahrenheit mode was missing too and it was a pain to implement because the library wasn't really set up with the best practices of code modularity. And that's depsite that craptastic Google code style guide (yeah read though that thing and it was an absolute disaster... All I'll say is, no wonder Stadia was a mess on launch lol)

The other thing is you can't dynamically load in/out different classes of device/AC, or even use multiple at once on the same IR instance, which seems like a fairly big oversight and was something I need to do. At this rate I'm considering just stealing the decoders I need (with credit of course) and putting them in a more basic IR library.

[crankyoldgit]

@crankyoldgit That's weird, considering that GDB found that those const variables inhabited the RAM address space and were all taking up room there. But keep in mind, the compiler's RAM usage numbers are just an estimate, they're different than the true numbers that you get at runtime.

Like I said earlier, are you including the symbol table (i.e. a non-stripped binary) when you're running GDB? That might be your issue.
I spent a fare few hours yesterday going over ways to reduce memory usage, espectially SRAM etc on the arduino platform/framework. We do most/all of what they suggest. A lot of the guides seem to indicate the compiler's RAM estimates are accurate, given that is typically the block/section of the binary that is loaded directly into memory for "global" / SRAM memory at the start of execution. Thus, it makes sense that value is accurate.

It could be that PROGMEM actually does improve the situation at runtime (#define did for me when I did a CTRL+F replace). I would assume that a PROGMEM const would be just as good as define with none of the pitfalls.

It could be the PROGMEM variable attribute just doesn't work on Integers etc. Just on strings/arrays. i.e. It's not forcing those values to be stored in flash.
The best way to see what is going on is to disassemble the final(stripped) binary and see how it is referencing the values.

Though frankly there's a lot of messy code in this library and bugs I've had with fragmentation after running for a long time, not to mention the codes in the AC class for the LG AC not being quite right for some of the functions, despite being for the exact same remote model I have, so I had to fix them.

Well, this is open source, feel free to contribute back the fixes. I look forward to your PRs.

I am curious about the fragmentation issues you are referring to. There is very little dynamic memory usage in the library. The only fragmentation issues we've ever had reported are with the toString() functions. Which I thought I had dealt with mostly.
If you can isolate or even point to the area where it is happening, I'll try to address it.

Oh yeah and a fahrenheit mode was missing too and it was a pain to implement because the library wasn't really set up with the best practices of code modularity.

There are a number of protocols that have support Fahrenheit, you can see from those how I tend to handle Fahrenheit. If no one provides raw data for Fahrenheit it is kind of hard to reverse engineer it. Turns out people using these devices tend to come from places that don't use Fahrenheit. I wonder why that is. ;-)

And that's depsite that craptastic Google code style guide (yeah read though that thing and it was an absolute disaster... All I'll say is, no wonder Stadia was a mess on launch lol)

Well, it is one of the better public style guides out there. None are perfect. Better to have a style guide than no style guide IMHO. Perhaps you've better. I'm not sure you can blame Stadia on the style guide. ;-)

The other thing is you can't dynamically load in/out different classes of device/AC, or even use multiple at once on the same IR instance, which seems like a fairly big oversight and was something I need to do.

We could add the ability to dynamically enable/disable protocols, but that turns out to be next to useless. As the code has to be stored in the flash, and flash space when all the "modules" are compiled is the main issue people have. i.e. limited flash space.
If you have a way of "loading in/out" code without storing it in flash, I'm all ears. I've been unable to come up with one that is acceptable. Loading in remote interpreted code across a network isn't an acceptable solution IMHO. ;-)

I'm not sure what you're doing, but you can have multiple classes/protocols/etc on the same "IR instance". The library doesn't stop you in any way with respect to supporting/controlling multiple devices from a single ESP chip or via a single IR led.
i.e. I control at least three devices using different protocols from the same "instance".

At this rate I'm considering just stealing the decoders I need (with credit of course) and putting them in a more basic IR library.

Feel free. It's open source, after all. Please respect the LGPL license. There is plenty of scope to create a simpler/smaller send-only library. I wish you luck.

[Pecacheu]

Like I said earlier, are you including the symbol table (i.e. a non-stripped binary) when you're running GDB? That might be your issue.
I spent a fare few hours yesterday going over ways to reduce memory usage, espectially SRAM etc on the arduino platform/framework. We do most/all of what they suggest. A lot of the guides seem to indicate the compiler's RAM estimates are accurate, given that is typically the block/section of the binary that is loaded directly into memory for "global" / SRAM memory at the start of execution. Thus, it makes sense that value is accurate.

I guess it's possible that the act of including GDB affects the program. Though what I meant by the RAM estimate not being accurate is, I've noticed that in many programs the readout of the RAM usage at runtime, even right at the start of the program, can be much higher than the estimate. I don't think the compiler can really predict various dynamic allocations that go on during initializations of classes and libraries, etc.

Well, it is one of the better public style guides out there. None are perfect. Better to have a style guide than no style guide IMHO. Perhaps you've better. I'm not sure you can blame Stadia on the style guide. ;-)

Heh, I suppose that's true. But hey I just like making fun of Statia lol

I am curious about the fragmentation issues you are referring to. There is very little dynamic memory usage in the library. The only fragmentation issues we've ever had reported are with the toString() functions.

That's what I think is happening cause I'm using some of the debug functions of the library related to the IRRecv, and I've noticed that if I go crazy pressing buttons on my remote for 20 minutes straight I can start getting issues much faster. So my theory is that this miiiight be more the fault of Arduino's String class than your library's fault. I noticed that the library does try to do some pre-allocation of the strings to prevent fragmentation. I'm sure that helps but it seems like that... isn't perfect.

I'm not sure what you're doing, but you can have multiple classes/protocols/etc on the same "IR instance". The library doesn't stop you in any way with respect to supporting/controlling multiple devices from a single ESP chip or via a single IR led.

It does though, the IRSend instance in each protocol class is a private variable you can't access. And maybe I'm doing something wrong but having more than one IRSend for the same IR LED seems to conflict with each other. Whereas when I modify the library to make the IRSend instance public, and use the same one for multiple protocols, it does work.

If you have a way of "loading in/out" code without storing it in flash, I'm all ears. I've been unable to come up with one that is acceptable. Loading in remote interpreted code across a network isn't an acceptable solution IMHO. ;-)

That's true, like literally loading a binary of each module over the network would be all kinds of insane, but I am trying to think of a way to do something like that... Like being able to dynamically add support for a new protocol without a full OTA update would be nice, through some sort of like... JSON file type thing. But then the parsing for that would take up space and all but idk... Maybe a silly/dumb feature lol

[crankyoldgit]

I guess it's possible that the act of including GDB affects the program. Though what I meant by the RAM estimate not being accurate is, I've noticed that in many programs the readout of the RAM usage at runtime, even right at the start of the program, can be much higher than the estimate. I don't think the compiler can really predict various dynamic allocations that go on during initializations of classes and libraries, etc.

The "RAM Estimate" is the floor value. It's all the globals and statics in the code. The RAM usage during runtime will be higher and of course can't be predicted accurately via static analysis. Dynamic memory, stack usage etc is not included in that estimate. Globals etc, can be accurately predicted, well, the minimum at least.

I am curious about the fragmentation issues you are referring to. There is very little dynamic memory usage in the library. The only fragmentation issues we've ever had reported are with the toString() functions.

That's what I think is happening cause I'm using some of the debug functions of the library related to the IRRecv, and I've noticed that if I go crazy pressing buttons on my remote for 20 minutes straight I can start getting issues much faster. So my theory is that this miiiight be more the fault of Arduino's String class than your library's fault. I noticed that the library does try to do some pre-allocation of the strings to prevent fragmentation. I'm sure that helps but it seems like that... isn't perfect.

I've no idea what you're doing with the code so I can't comment, and if you've enabled DEBUG well, all bets are off. It's not exactly designed to run optimally with that enabled. ;-)

I've just added commit 14e0e42 that does some more string reservation etc to try to reduce fragmentation if you are using some of the routines used by the examples etc. But, as I've got no idea what you're doing, that's a long shot.
It could be the Arduino String library, it could be the Serial library, it could be your code, it could be the libraries. Without concrete examples it's hard to tell. e.g. If you are constantly sending a stream of IR to IRrecvDumpV2, of course it can't keep up as it's limited to the serial baud rate. You can easily saturate that.

It does though, the IRSend instance in each protocol class is a private variable you can't access. And maybe I'm doing something wrong but having more than one IRSend for the same IR LED seems to conflict with each other. Whereas when I modify the library to make the IRSend instance public, and use the same one for multiple protocols, it does work.

If you can provide a simplified case where this happens, please log it as an issue. Off the top of my head, I can't see why that wouldn't work as expected. (Unless you were trying to send two messages simultaneously, producing garbage)
After all, all it is doing is bit-banging the GPIO. Nothing fancy at all.

[crankyoldgit]

That's true, like literally loading a binary of each module over the network would be all kinds of insane, but I am trying to think of a way to do something like that... Like being able to dynamically add support for a new protocol without a full OTA update would be nice, through some sort of like... JSON file type thing. But then the parsing for that would take up space and all but idk... Maybe a silly/dumb feature lol

I can see a "Parse a spec and implement it" method for the simple protocols. i.e. send X many bits in a certain way. But for the complexities of the A/C protocols, I can't see it being implemented without a Turing complete language of sorts. Bags not implementing that on an ESP chip. ;-)

[crankyoldgit]

FWIW, I've been looking into the entire mess that is F(), PSTR(), __FlashStringHelper, PROGMEM, etal.

It's complicated, but it looks like an entire re-write of how we do strings & constants would be required to store things in flash, rather than SRAM.

I'm not even sure how to even tackle it all. I've been reading lots of docs on how to do it, and it's full of dark-twisty-passages-all-alike.

If anyone has any suggestions, or wants to take this on, I'm more than happy to accept help.

[crankyoldgit]

More feedback/info:

I converted the consts to literal #defines .. and the SRAM usage was exactly the same. I did it for almost every aircon protocol.

So I think that eliminates the "them being globals is why it's taking up SRAM" theory.

e.g.

// Constants
-const uint64_t kAirwellKnownGoodState = 0x140500002;  // Mode Fan, Speed 1, 25C
+#define kAirwellKnownGoodState (uint64_t)0x140500002
 // Temperature
-const uint8_t kAirwellMinTemp = 16;  // Celsius
-const uint8_t kAirwellMaxTemp = 30;  // Celsius
+#define kAirwellMinTemp (uint8_t)16
+#define kAirwellMaxTemp (uint8_t)30
 // Fan
-const uint8_t kAirwellFanLow = 0;     // 0b00
-const uint8_t kAirwellFanMedium = 1;  // 0b01
-const uint8_t kAirwellFanHigh = 2;    // 0b10
-const uint8_t kAirwellFanAuto = 3;    // 0b11
+#define kAirwellFanLow (uint8_t)0
+#define kAirwellFanMedium (uint8_t)1
+#define kAirwellFanHigh (uint8_t)2
+#define kAirwellFanAuto (uint8_t)3

[Pecacheu]

I can see a "Parse a spec and implement it" method for the simple protocols. i.e. send X many bits in a certain way. But for the complexities of the A/C protocols, I can't see it being implemented without a Turing complete language of sorts. Bags not implementing that on an ESP chip. ;-)

Lol yeah, that's what I was worried about, pretty much would have to 're-invent the wheel' so to speak to implement that for the complex protocols.

More feedback/info:

I converted the consts to literal #defines .. and the SRAM usage was exactly the same. I did it for almost every aircon protocol.

Hmm, are you sure the SRAM usage was the same at runtime, or only at compile time? Cause the phenomenon I was noticing was happening at runtime. But also, in hindsight, the usage of the consts wasn't as big as I thought it was as I realized that some of my code was also taking up more space than in should in SRAM too.

Also, was using your latest version of the library recently and I didn't notice any string RAM leakage problems over time anymore afaict.

FWIW, I've been looking into the entire mess that is F(), PSTR(), __FlashStringHelper, PROGMEM, etal.
It's complicated, but it looks like an entire re-write of how we do strings & constants would be required to store things in flash, rather than SRAM.

Well I can show you what I learned from using them in my ESP programs, I don't know everything about them but here's how I've been using them:

PSTR() is a macro to put a string in PROGMEM in-line. Access time to PROGMEM strings seems to be just slightly higher since they have to get moved to RAM to actually use them, but they don't take up any RAM space except while they're in use.

Some functions, like printf, have a special version you must call to use PROGMEM. Not using this function will give you no error at compile time, but at runtime it'll cause a crash.
Serial.printf_P(PSTR("Hello %s\n"), name);

That's why they created FlashStringHelper. That way, you can be sure that a string IS or IS NOT stored in Flash and call the right version of a function. That's also where F() comes in. It's basically a combo-macro that stores a string in Flash using PSTR(), then pulls it out from the Flash as a FlashStringHelper.

Using it directly: Serial.println(F("Hello World"));
Storing in a global: FlashString str = F("Hello World");
(I used typedef const __FlashStringHelper FlashString; elsewhere to make things more readable for my sake)

Also, if you need to pull a FlashString into RAM to manipulate it directly (instead of just printing with print or printf), use the strcpy_P function. That was one part that took me a while to figure out, until I realized that all these 'P' functions exist for stuff like this.

[crankyoldgit]

Yeah, I got most of that, but for it work in normal C++ (unit tests) requires a lot more effort on top of that too. Plus it needs to be a all or nothing approach.

But hey, at least it's just the strings part, the numerical constants/literals I can say with some confidence now are not affected. Eg. See my previous update

[Pecacheu]

Yeah, I got most of that, but for it work in normal C++ (unit tests) requires a lot more effort on top of that too. Plus it needs to be a all or nothing approach.

Oh yeah lol, I wouldn't have thought of that.

In other news hopefully I can continue to test this library at all cause I think my not-even-that-old air con is angry at me for experimenting with it, and doesn't want to work with any remote controls anymore some days XD

That and I need better IR LEDs cause the ones I have are super-weak. Might try those Adafruit "high power" ones, but damn are they expensive (as far as LEDs go).

[crankyoldgit]

I got super lucky I guess when I bought about ~30 x 940nm IR LEDs (~30deg clear 5mm ones) of AliExpress or EBay for a few bucks. Real no name ones. I overdrive them with 5V via an 2N3904 TO-92 NPN General Purpose Transistor (which limits it to 200mA. They've worked fine for years. You can get away with the voltage due to it effectively getting a PWM signal. So, very short out-of-spec spikes and then it spends 99.9999% of the time off. I get about 5-10 meters out of it depending on the amount a direct sunlight in the room and ambient temp. It gets pretty bright and hot down here too. ;-)

[crankyoldgit]

FYI, the some of the changes mentioned above have been included in the just released v2.7.20 of the library.

[mcspr]

FWIW, I've been looking into the entire mess that is F(), PSTR(), __FlashStringHelper, PROGMEM, etal.

It's complicated, but it looks like an entire re-write of how we do strings & constants would be required to store things in flash, rather than SRAM.

I'm not even sure how to even tackle it all. I've been reading lots of docs on how to do it, and it's full of dark-twisty-passages-all-alike.

If anyone has any suggestions, or wants to take this on, I'm more than happy to accept help.

re. #1614, and something only related to strings and PROGMEM
(edit: typos, formatting)

The basics were already described above, but I would also like to point out the actual implementation for esp8266:
https://github.com/earlephilhower/newlib-xtensa/blob/xtensa-4_0_0-lock-arduino/newlib/libc/sys/xtensa/sys/pgmspace.h
https://github.com/esp8266/Arduino/blob/64e87f1149a0badf9438b29306d924db75eebf82/cores/esp8266/WString.h#L38-L40
https://github.com/esp8266/Arduino/blob/master/tools/sdk/ld/eagle.app.v6.common.ld.h

The gist of it is a statically allocated value, where we explicitly set the 'section' of the .elf (in our case, starting with '.irom0.text...' or '.irom0.pstr...' for char arrays) that maps to the flash and does not get loaded into RAM automatically, like the usual '.rodata' & '.data' (consts, static consts, global vars). And, some things already are stored in '.irom...' section automatically via the .ld script.

Reading from flash requires 4byte-aligned addresses and specific to strings this means we need to use the str_...P suffix'ed functions, which in turn will use pgm... funcs that deal with the pointer accordingly. e.g. reading 1byte of data means we read the whole neighbouring chunk as 'u32' starting from the nearest address to the left modulo 4 == 0 and then extract the resulting 'char' from the read 'u32'. (and also see the variants for u16 and u32)

As also mentioned above, we have:

• PROGMEM attribute macro for global variables or local ones marked with 'static'.
• For strings, there are PSTR("...") or F("...") when inside of a function block (also notice #define PSTR(...) PSTRN(..., PSTR_ALIGN), enabled by default, which places strings on 4byte-aligned addresses and which also results in some padding))
• const __FlashStringHelper* is just an 'alias' for const char* that points to the 1st char of static const char __pstr__[] PROGMEM = "..." , so it can be safely reinterpret_cast'ed back into a basic char pointer and is just a hint for the type-system to load it using ..._P funcs
• Just like string literals, flash strings are de-duped, so there should be no worry with unique or almost-unique (with shared suffix) strings bloating the ROM when they are not gathered in a single place and spread out across different files.

One could also avoid using the ..._P in some cases:

static const char __pstr__[] __attribute__((aligned(4))) PROGMEM = "blahblahblah";
Serial.printf("pstr stored on flash: %p\n", &__pstr__[0]);
Serial.printf("actual length: %lu\n", sizeof(__pstr__) - 1); // since C strings always have '\0' at the end, even when empty 

And also note of objdump -t -z -C on individual .o, .elf of the library .a, that would describe the available symbols, which may help in understanding the things that get loaded into RAM and the ones that are inlined or discarded

---

PS. Unlike esp8266, esp32 implementation does not require any special handling of the values to either place them into flash or any special way to access them and things like PROGMEM and PSTR are just no-ops for backwards compatibility:
https://github.com/espressif/arduino-esp32/blob/371f382db7dd36c470bb2669b222adf0a497600d/cores/esp32/pgmspace.h

[crankyoldgit]

Yeah, it's complicated. I wish it wasn't. I'm really open to a framework to use that works on all the platforms we need (ESP8266, ESP32, & X86/Arm etc)

Basically my requirements are:

1. Works with "standard" interoperable String manipulation for both Arduino's framework, and for normal C++ (e.g. Unit tests/CI etc.) without too much bother.
2. Allows us to do simple string replacement for Internationalisation via a compile-time argument flip.
3. Allows the code to still be readable. i.e. Not a million "per architecture" #ifdefs etc.

If someone knows of a "Gold Standard" way of doing that, or tool, or library, or has pointers to "simple to understand" example code that covers the above bases, I'm all ears.

I've gotten quite frustrated at the number of attempts to get it to play right by using flash on the ESP. So a system that "just works" is highly prized. ;-)

[mcspr]

If someone knows of a "Gold Standard" way of doing that, or tool, or library, or has pointers to "simple to understand" example code that covers the above bases, I'm all ears.

I've gotten quite frustrated at the number of attempts to get it to play right by using flash on the ESP. So a system that "just works" is highly prized. ;-)

I also hope I am not missing the actual point of the discussion by going into specifics that may be already apparent :) And I am not sure I'd find any 'standard' way to handle these things (or more or less a complete guide, besides referring to style-guide for c++ software that deals with consts, and then wrap certain things to place them into PROGMEM sections when possible). At least right now, I could only resort to examples I used in my own projects that dealt with flash strings and flash-mapped static data with the esp8266.

But, let me prepare those first. The gist is almost the same though - String typeToString(enum); with inline out = F(D_SOME_THING); strings and the same in reverse by using strncasecmp_P on the individual PSTR(D_SOME_STRING) pointers (since both point to the same thing); or, const char* typeToString(enum); if allocation should not be involved. libc modifications also allow %s / %.*s formatters to seamlessly work flash strings. Another thing is that String always handles const char* through the str..._P funcs anyway, so there is no specific need for const __FlashStringHelper* if it is always converted to String at some point (and to help to avoid injecting it into the API at all).

Another nice option is to have custom std::string_view-like class, that works as a proxy for the const char buf[] PROGMEM and avoids a lot of boilerplate that is neighboring the C strings. Plus, it is possible to use everywhere, unlike the F(...) and PSTR(...) bound to function blocks.

---

I still strongly suggest to check out the binutils toolset with 'objdump', 'nm' and 'gdb' to see the actual effects of various attributes / source code changes / compilation flags.

re. #1493 (comment)
With a small modification to the IR_Ecoclim.h that sets __attribute__((noinline)) on the stateReset() method:

$ gdb .pio/build/test/firmware.elf

(gdb) disassemble 'IREcoclimAc::stateReset()'
Dump of assembler code for function IREcoclimAc::stateReset():
   0x40235cc4 <+0>:     l32r    a4, 0x40235cbc
   0x40235cc7 <+3>:     l32r    a5, 0x40235cc0
   0x40235cca <+6>:     s32i.n  a4, a2, 16
   0x40235ccc <+8>:     s32i.n  a5, a2, 20
   0x40235cce <+10>:    ret.n
End of assembler dump.
(gdb) x 0x40235cbc
0x40235cbc:     0x00ffff02
(gdb) x 0x40235cc0
0x40235cc0:     0x00110630

There was no change in the PIO log, since the value was already in the flash even without the PROGMEM section attribute specifier.

Post-build info uses size, however it only searches for section info (both -A and -B flags) and does not know about the specifics of the ld script used to build the .elf (and that there are more things than just .text).

[crankyoldgit]

Okay, help me out here. I built IRMQTTServer using:

IRremoteESP8266/examples/IRMQTTServer$ pio run --jobs 2 --project-dir . --environment d1_mini

And here is a sample of output from: nm .pio/build/d1_mini/firmware.elf

...
402792ac T kCoolStr
40279218 T kDaysStr
4027921c T kDayStr
402792b8 T kDownStr
402792a4 T kDryStr
...

And:

$ objdump -h .pio/build/d1_mini/firmware.elf | head -17

.pio/build/d1_mini/firmware.elf:     file format elf32-little

Sections:
Idx Name          Size      VMA       LMA       File off  Algn
  0 .data         000005ec  3ffe8000  3ffe8000  00001000  2**4
                  CONTENTS, ALLOC, LOAD, DATA
  1 .noinit       00000004  3ffe85ec  3ffe85ec  000015ec  2**2
                  CONTENTS, ALLOC, LOAD, DATA
  2 .text         00000158  40100000  40100000  00005000  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  3 .irom0.text   0007d3b8  40201010  40201010  0000d010  2**4
                  CONTENTS, ALLOC, LOAD, CODE
  4 .text1        00006edc  40100158  40100158  00005158  2**2
                  CONTENTS, ALLOC, LOAD, READONLY, CODE
  5 .rodata       00002c2c  3ffe85f0  3ffe85f0  000015f0  2**4
                  CONTENTS, ALLOC, LOAD, DATA

If I am reading it correctly (and I'm new to this so don't count on it ;-), that puts a large chunk of the constant strings in .irom0.text which has a range of 0x40201010 to 0x4027E3C8. e.g. kCoolStr is at 0x402792ac
The VM and LM seem to indicate it should got to those addresses.

If I'm reading this right in: #1614 (comment)

Will print 0x3ffxxxxx instead of something in the 0x402xxxxx range of the teststring[] var, where the flash strings are usually stored with the esp8266

i.e. It looks like they are being stored in the flash.

What am I missing? To me it looks like the strings are being stored in Flash. That doesn't mean something else isn't loading them into RAM etc.

Or am I reading the info wrong?

[mcspr]

From the compiler POV, there are two objects

IRremoteESP8266/src/IRtext.cpp

Line 108 in 10706fc

const PROGMEM char* kCoolStr = D_STR_COOL; ///< "Cool"

With a small example

const char* const hello_world PROGMEM = "aaaaaaaaaaaaaaaa";

The first object contains the pointer to the string data, which is stored into flash. And the other one is with the actual data, where the string is placed into shared strings pool which is them mapped to ram

$ xtensa-lx106-elf-gdb.exe .\.pio\build\esp8266-1m-git-base\firmware.elf --batch --ex 'p /a hello_world' --ex 'p /a &hello_world'
$1 = 0x3ffe8e85
$2 = 0x4027a9d0 <_ZL11hello_world>

[crankyoldgit]

Ah, so the pointer to the array is being stored in Flash, not the array itself.

[crankyoldgit]

With a small example

const char* const hello_world PROGMEM = "aaaaaaaaaaaaaaaa";

The first object contains the pointer to the string data, which is stored into flash. And the other one is with the actual data, where the string is placed into shared strings pool which is them mapped to ram

So, using your example, how do I force "aaaaaaaaaaaaaaaa" into Flash?
Especially if it is outside a function.
e.g. PSTR() etc bitterly complains when it isn't.

[crankyoldgit]

e.g. Modifying IRtext.cpp to:

const char* kCoolStr PROGMEM = D_STR_COOL;  ///< "Cool"

Still seems to put the data in mem, yet the pointer still in flash.
i.e.

Reading symbols from .pio/build/d1_mini/firmware.elf...done.
(gdb) p /a kCoolStr
$1 = 0x3ffea884
(gdb) p /a &kCoolStr
$2 = 0x40279174 <kCoolStr>
(gdb) 

[mcspr]

#1614 (comment)? it could either be:

• const char kCoolStr[] PROGMEM = D_STR_COOL; in the .cpp plus replacing extern const char kCoolStr[]; in the header (since the header does not need to know the section). And the .cpp assigning the var needs to see the 'extern', so that means the 'IRtext.cpp' needs a #include "IRtext.h"
• from the comment, PSTR-like macro returning a static {ptr, length} struct. something like const MyPgmString kCoolStr = MY_PSTR("Cool"); and then extern const MyPgmString kCoolStr; for the header. minimal MyPgmString implementation can have c_str() and length() methods. but, I might need to work on that to be more accessible and useful

[crankyoldgit]

Hmmm. Sigh. Perhaps the best/better approach is it give in; wrap all references to kBlahStr's via a function.
e.g. (pseudo code)

enum class strings_enum_t {
    kUNUSED  = 0,
    ...
    kCool,
    ...,
};

String enumToString(strings_enum_t str_num) {
  static const char kStringMegaIndex[] const = {
   F(D_STR_UNUSED),
   ...
   F(D_STR_COOL),
   ..};
  return String(kStringMegaIndex[str_num]);
}

to replace all kCoolStr etc instances. e.g.

result += enumToString(strings_enum_t::kCool);

[crankyoldgit]

• And the .cpp assigning the var needs to see the 'extern', so that means the 'IRtext.cpp' needs a #include "IRtext.h"

Isn't that double defining it? or do you mean "ir_Protocols.cpp" instead of "IRtest.cpp" above?
I haven't tested it, but I would have thought if/when IRtext.h gets included in IRtext.cpp you'd effectively get:

extern const char kCoolStr[];
const char kCoolStr[] PROGMEM = D_STR_COOL;

i.e. You're defining it twice. (I vaguely remember having this issue when I was trying to implement it at the time)

[mcspr]

• And the .cpp assigning the var needs to see the 'extern', so that means the 'IRtext.cpp' needs a #include "IRtext.h"

Isn't that double defining it? or do you mean "ir_Protocols.cpp" instead of "IRtest.cpp" above? I haven't tested it, but I would have thought if/when IRtext.h gets included in IRtext.cpp you'd effectively get:

extern const char kCoolStr[];
const char kCoolStr[] PROGMEM = D_STR_COOL;

i.e. You're defining it twice. (I vaguely remember having this issue when I was trying to implement it at the time)

Only tried it in the other issue and a host-built example, and it seemed to work ok. 'extern' works as a declaration, and linking will fail with 'undefined reference' otherwise, but at least one file must assign something to it. There's also 'multiple definition of `...'' message when more than one file assigns something to the array.

result += enumToString(strings_enum_t::kCool);

Might be an overkill? global consts approach seems to be fine as-is (barring the fact that it is in the global namespace, but idk if these are supposed to be seen by the user and are not exclusively internal values?)

[crankyoldgit]

i.e. You're defining it twice. (I vaguely remember having this issue when I was trying to implement it at the time)

Only tried it in the other issue and a host-built example, and it seemed to work ok. 'extern' works as a declaration, and linking will fail with 'undefined reference' otherwise, but at least one file must assign something to it. There's also 'multiple definition of `...'' message when more than one file assigns something to the array.

Just had the chance to test it. You're right. I was wrong. I'm going to blame a long day and being too tired. ;-)

extern const char kCoolStr[];
const char kCoolStr[] PROGMEM = D_STR_COOL;

FYI, added #include "IRtext.h" to the top of IRtext.cpp, which effectively does the above. Still doesn't place it in flash. :-(

I'll try your macro before I give up and try to force it via containing it in a function.

[crankyoldgit]

Zomg! Changing it from char * to char array[] fixes it and puts it in flash! I want to hit something. :-/

[crankyoldgit]

@Pecacheu @mcspr, Can you please test out the branch https://github.com/crankyoldgit/IRremoteESP8266/tree/strings_in_flash (PR #1623)
Fingers crossed, it should give you what you want (or at least some of it) which is a reduction in global mem.

Let me know how it goes, and if you can confirm it's now storing them in Flash for you, and if everything still works etc.

[crankyoldgit]

FYI, the changes mentioned above have now been included in the new v2.8.0 release of the library.