BUG REPORT - Too many open files, bearclaw fails silently

[ekacoei]

What happened?
I have more than 600 blog posts in the input folder. For each file bearclaw opens a Markdown input file and an HTML output file. After opening 1024 files the operating system (Debian) default limit is reached, no more file will be opened. Bearclaw failes silently by not creating HTML output files.

What should have happened?

1. os.Open() returns an error. Any error should panic the programm.
2. closing markdownInputFile and closing HTMLoutputFile cannot be deferred, otherwise the operating-system-open-file-limit will be reached

More information:
Improved main.go attached. This is how I found the bug

			// create the html file
			htmlFile, err := os.Create(outFolder + "/" + file.Name() + ".html")
      if err != nil {
        fmt.Println("Error creating file", outFolder + "/" + file.Name() + ".html" )
        panic(err)
/*
$ ./bearclaw
Error creating file ../html/2023-01-30_post.mdwn.html
panic: open ../html/2023-01-30_post.mdwn.html: too many open files
$ ls -l ../markdownposts/ |wc -l
567
$ ls /proc/$(pidof bearclaw)/fd |wc -l
1024
$ ulimit -n
1024
*/

The time the Markdown input needs to be closed

			// read the md
			reader := bufio.NewReader(markdownFile)
			markdown, _ := io.ReadAll(reader)
			// don't forget to close it when done
			markdownFile.Close()

The time the HTML file needs to be closed

			fmt.Fprintln(htmlFile, htmlAfterPlugins)
			htmlFile.Close() //you cant defer closing the file, because that postpones it after the end of markdownToHTML

Minor improvement (open htmlTemplateHeader only once per run instead of once per input file) included

// markdownToHTML converts markdown documents to HTML
func markdownToHTML(inFolder, outFolder, templateFolder string) {
	files, _ := os.ReadDir(inFolder)

	// read in our templates, but not in a loop on every file, just once for all markdown files to be converted of them
	header, _ := os.ReadFile(templateFolder + "/header.html")
	footer, _ := os.ReadFile(templateFolder + "/footer.html")

[donuts-are-good]

This is a good catch, thanks! It makes sense to add better concurrency management, as well as managing the defer as you mentioned. Thank you for putting this issue together. Since it already runs so fast, it'd probably make sense to define a threshold, say 100 posts, and do a worker queue to chew through them in chunks to prevent passing the os limit for open files.

[donuts-are-good]

Thanks to @mpldr for this patch!

[ekacoei]

I am very sorry to annoy you, but rss.go with its function func CreateXMLRSSFeed(inFolder, outFolder string) suffers from the same issue -- too many open files at once.

[mpldr]

I'll take a look