Support for large databases on 32 bit embedded systems #10486
Comments
3 tasks
|
This issue is superseded by #12362. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
This issue has been automatically closed because it has not had recent activity. Please reopen if this issue is still important to you. Thank you for your contributions. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi, for this proposal, I am also offering notes on a solution I am currently working on, but had to temporarily shelve to work on other things. I am hoping this thread can serve as a discussion of my solution-in-progress and also possible alternatives.
This is my first time working with Go and influxdb's source. I'm very open to suggestions or recommendations if something is not done idiomatically.
Proposal: Enable database on 32-bit embedded architectures to have maximum aggregate TSM file size greater than e.g. 3.4 GB.
Current behavior: Currently, influx memory maps all TSM files into its userland address space. On 64 bit architectures, this not a problem because the userland address space is huge. On 32 bit architectures, it can be between 2 and 3.4 GB, depending on the system. When the TSM files approach this limit, Influx fails with "Cannot allocate memory" errors.
Desired behavior: TSM limit of at least 100 GB.
Use case: Data logging locally on a piece of industrial equipment, not the cloud. See multiple times this has come up previously in similar applications: #10160, #6171
Question: Would the following solution be amenable to the influx team for inclusion into the main line? If not, what alternative solution would be? My organization would prefer to not maintain a separate fork of influx.
My in-progress solution is here: https://github.com/fluffynukeit/influxdb/tree/dual_mmap.
The implemented changes are described below. Most changes are associated with the mmapAccessor in reader.go. If there is a better way to do this, I'm open to working on it. This is what I was able to figure out from the code.
This solution did a good job of powering through the address space limit. Care needs to be taken to set the shard duration low enough that compaction jobs don't steal a too large chunk of the memory space, and that individual TSM files are well below the 3.4 GB limit. See the monitoring output below, specifically the 6.3 GB aggregate size and the sub-GB VM peak usage. To generate the database, I used influx-stress with a modification to use randomly changing data instead of incrementing data that would otherwise be compacted to almost nothing via RLE.
While writing data to the database was great, I had serious problem with querying the data. I encountered frequent, unpredictable SIGSEGV when running queries with the influx client. I believe this had something to do with cursors entering the same reader.go functions and some problem happening with the access counters that allowed the free() function to release an mmap when another cursor still needed to use it. I'm not super competent with the synchronization methods in Go, so I could have messed this part up.
The text was updated successfully, but these errors were encountered: