Skip to content

Commit

Permalink
Merge branch 'master' into master-with-d
Browse files Browse the repository at this point in the history
  • Loading branch information
@xtne6f
xtne6f committed May 1, 2024
2 parents b6d5f11 + bbc1ec3 commit 4850e9c
Show file tree
Hide file tree
Showing 2 changed files with 128 additions and 36 deletions.
55 changes: 54 additions & 1 deletion Readme.txt
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ tsmemseg - In-memory transport stream segmenter mainly for HLS/LL-HLS

Usage:

tsmemseg [-4][-i inittime][-t time][-p ptime][-a acc_timeout][-c cmd][-r readrate][-f fill_readrate][-s seg_num][-m max_kbytes][-d flags] seg_name
tsmemseg [-4][-i inittime][-t time][-p ptime][-a acc_timeout][-c cmd][-r readrate][-f fill_readrate][-s seg_num][-m max_kbytes][-g dir][-d flags] seg_name

-4
Convert to fragmented MP4.
Expand Down Expand Up @@ -34,6 +34,10 @@ tsmemseg [-4][-i inittime][-t time][-p ptime][-a acc_timeout][-c cmd][-r readrat
-m max_kbytes (kbytes), 32<=range<=32768, default=4096
Maximum size of each segment. If segment length exceeds this limit, the segment is forcibly cut whether on a key packet or not.

-g dir, default=""
Specify the directory for creating FIFOs. If not specified, created in "/tmp" with 0600 permission.
This option is ignored on Windows.

-d flags, range=0 or 1 or 3, default=0
Convert ARIB caption/superimpose streams to an ID3 timed-metadata stream that https://github.com/monyone/aribb24.js can recognize.
This feature is enabled when "flags" is 1 or 3.
Expand Down Expand Up @@ -77,8 +81,34 @@ Information about MP4 fragments (16 bytes units) are placed in the extra readabl
The 0-3rd byte of the units stores the duration of fragment in milliseconds.
Besides the fragment information, if there is space in the extra readable area, it is MP4 header box (ftyp/moov).

For Unix FIFO only, there is a 64-bytes field preceding the information units to store the seg_name.
The field can be used as a signature to prevent multiple processes from reading the FIFO in a race condition.

All other unused bytes are initialized with 0.

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
+----------------------+-----------------------------+----------------------------------+-----------------+
|seg_name field (Unix FIFO only) :
+----------------------+-----------------------------+----------------------------------+-----------------+
: :
+----------------------+-----------------------------+----------------------------------+-----------------+
: :
+----------------------+-----------------------------+----------------------------------+-----------------+
: |
+----------------------+-----------------------------+----------------------------------+-----------------+
|seg_num 0 0 0|UNIX_time_updated |no_longer_updated incomplete MP4 0|extra_area_length|
+----------------------+-----------------------------+----------------------------------+-----------------+
|seg_index 0 frag_num 0|sequential_number unavailable|seg_duration_msec |sum_of_durations |
+----------------------+-----------------------------+----------------------------------+-----------------+
...
+----------------------+-----------------------------+----------------------------------+-----------------+
|frag_duration_msec |0 0 0 0 |0 0 0 0|0 0 0 0 |
+----------------------+-----------------------------+----------------------------------+-----------------+
...
+----------------------+-----------------------------+----------------------------------+-----------------+
|ftyp/moov :
...

Specification of "segment pipe":

"segment pipe" contains MPEG-TS packets or MP4 moof boxes.
Expand All @@ -89,8 +119,31 @@ The sequence of 4-6th bytes (immediately after TS-NULL header) stores the sequen
12th stores whether this segment is MPEG-TS (0) or MP4 (1).
32-35th, and subsequent 4 bytes units (until its value is 0) store the size of all fragments contained in the stream.

For Unix FIFO only, there is a 188-bytes field preceding the information packet to store the seg_name.

All other unused bytes are initialized with 0.

0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5
(The following 188 bytes are Unix FIFO only)
+-------------------+-----------------------------+------------------------+-----------+
|0x47 0x1f 0xff 0x10|seg_name field :
...(188 bytes)
+-------------------+-----------------------------+------------------------+-----------+
|0x47 0x1f 0xff 0x10|sequential_number unavailable|number_of_units_or_bytes|MP4 0 0 0 |
+-------------------+-----------------------------+------------------------+-----------+
|0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |
+-------------------+-----------------------------+------------------------+-----------+
|0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |
+-------------------+-----------------------------+------------------------+-----------+
|0 0 0 0 |0 0 0 0 |0 0 0 0 |0 0 0 0 |
+-------------------+-----------------------------+------------------------+-----------+
|frag_size_0 |frag_size_1 |frag_size_2 |frag_size_3|
+-------------------+-----------------------------+------------------------+-----------+
...(188 bytes)
+-------------------+-----------------------------+------------------------+-----------+
|MPEG-TS/MP4 stream :
...

Notes:

This tool currently only supports Windows and Linux.
Expand Down
109 changes: 74 additions & 35 deletions tsmemseg.cpp
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,7 @@
#else
#include <errno.h>
#include <signal.h>
#include <string.h>
#include <sys/select.h>
#include <sys/stat.h>
#include <sys/types.h>
Expand Down Expand Up @@ -96,7 +97,7 @@ struct SEGMENT_PIPE_CONTEXT

struct SEGMENT_CONTEXT
{
char path[128];
char path[256];
SEGMENT_PIPE_CONTEXT pipes[2];
std::vector<uint8_t> buf;
std::vector<uint8_t> backBuf;
Expand Down Expand Up @@ -358,49 +359,68 @@ void WriteUint32(uint8_t *buf, uint32_t n)
buf[3] = static_cast<uint8_t>(n >> 24);
}

void AssignSegmentList(std::vector<uint8_t> &buf, const std::vector<SEGMENT_CONTEXT> &segments, size_t segIndex,
void AssignSegmentList(std::vector<uint8_t> &buf, const char *signature, const std::vector<SEGMENT_CONTEXT> &segments, size_t segIndex,
bool endList, bool incomplete, bool isMp4, const std::vector<uint8_t> &mp4Header)
{
buf.assign(segments.size() * 16, 0);
WriteUint32(&buf[0], static_cast<uint32_t>(segments.size() - 1));
WriteUint32(&buf[4], GetCurrentUnixTime());
buf[8] = endList;
buf[9] = incomplete;
buf[10] = isMp4;
buf.assign(segments.size() * 16 + (signature ? 64 : 0), 0);
size_t ofs = 0;
if (signature) {
for (size_t i = 0; i < 64 && signature[i]; ++i) {
buf[i] = signature[i];
}
ofs = 64;
}
WriteUint32(&buf[ofs], static_cast<uint32_t>(segments.size() - 1));
WriteUint32(&buf[ofs + 4], GetCurrentUnixTime());
buf[ofs + 8] = endList;
buf[ofs + 9] = incomplete;
buf[ofs + 10] = isMp4;
for (size_t i = segIndex, j = 1; j < segments.size(); ++j) {
WriteUint32(&buf[j * 16], static_cast<uint32_t>(i));
WriteUint32(&buf[j * 16 + 2], static_cast<uint32_t>(segments[i].fragDurationsMsec.size()));
WriteUint32(&buf[j * 16 + 4], segments[i].segCount);
WriteUint32(&buf[j * 16 + 8], segments[i].segDurationMsec);
WriteUint32(&buf[j * 16 + 12], static_cast<uint32_t>(segments[i].segTimeMsec / 10));
WriteUint32(&buf[ofs + j * 16], static_cast<uint32_t>(i));
WriteUint32(&buf[ofs + j * 16 + 2], static_cast<uint32_t>(segments[i].fragDurationsMsec.size()));
WriteUint32(&buf[ofs + j * 16 + 4], segments[i].segCount);
WriteUint32(&buf[ofs + j * 16 + 8], segments[i].segDurationMsec);
WriteUint32(&buf[ofs + j * 16 + 12], static_cast<uint32_t>(segments[i].segTimeMsec / 10));
for (size_t k = 0; k < segments[i].fragDurationsMsec.size(); ++k) {
buf.insert(buf.end(), 16, 0);
WriteUint32(&buf[buf.size() - 16], segments[i].fragDurationsMsec[k]);
}
i = i % (segments.size() - 1) + 1;
}
buf.insert(buf.end(), mp4Header.begin(), mp4Header.end());
WriteUint32(&buf[12], static_cast<uint32_t>(buf.size() - segments.size() * 16));
WriteUint32(&buf[ofs + 12], static_cast<uint32_t>(buf.size() - segments.size() * 16 - ofs));
}

void WriteSegmentHeader(std::vector<uint8_t> &buf, uint32_t segCount, bool isMp4, const std::vector<size_t> &fragSizes)
void WriteSegmentHeader(std::vector<uint8_t> &buf, const char *signature, uint32_t segCount, bool isMp4, const std::vector<size_t> &fragSizes)
{
size_t ofs = 0;
if (signature) {
// NULL TS header
buf[0] = 0x47;
buf[1] = 0x1f;
buf[2] = 0xff;
buf[3] = 0x10;
for (size_t i = 0; i < 184 && signature[i]; ++i) {
buf[i + 4] = signature[i];
}
ofs = 188;
}
// NULL TS header
buf[0] = 0x47;
buf[1] = 0x1f;
buf[2] = 0xff;
buf[3] = 0x10;
WriteUint32(&buf[4], segCount);
WriteUint32(&buf[8], static_cast<uint32_t>((buf.size() - 188) / (isMp4 ? 1 : 188)));
buf[12] = isMp4;
buf[ofs] = 0x47;
buf[ofs + 1] = 0x1f;
buf[ofs + 2] = 0xff;
buf[ofs + 3] = 0x10;
WriteUint32(&buf[ofs + 4], segCount);
WriteUint32(&buf[ofs + 8], static_cast<uint32_t>((buf.size() - 188 - ofs) / (isMp4 ? 1 : 188)));
buf[ofs + 12] = isMp4;
if (isMp4) {
size_t remainSize = buf.size() - 188;
size_t remainSize = buf.size() - 188 - ofs;
size_t i = 0;
for (; i + 1 < std::min(fragSizes.size(), MP4_FRAG_MAX_NUM) && remainSize >= fragSizes[i]; ++i) {
WriteUint32(&buf[i * 4 + 32], static_cast<uint32_t>(fragSizes[i]));
WriteUint32(&buf[ofs + i * 4 + 32], static_cast<uint32_t>(fragSizes[i]));
remainSize -= fragSizes[i];
}
WriteUint32(&buf[i * 4 + 32], static_cast<uint32_t>(remainSize));
WriteUint32(&buf[ofs + i * 4 + 32], static_cast<uint32_t>(remainSize));
}
}

Expand Down Expand Up @@ -661,6 +681,9 @@ int main(int argc, char **argv)
int nextReadRatePerMille = 0;
size_t segNum = 8;
size_t segMaxBytes = 4096 * 1024;
#ifndef _WIN32
const char *fifoDir = "";
#endif
const char *destName = "";
CID3Converter id3conv;
CMp4Fragmenter mp4frag;
Expand All @@ -671,7 +694,7 @@ int main(int argc, char **argv)
c = argv[i][1];
}
if (c == 'h') {
fprintf(stderr, "Usage: tsmemseg [-4][-i inittime][-t time][-p ptime][-a acc_timeout][-c cmd][-r readrate][-f fill_readrate][-s seg_num][-m max_kbytes][-d flags] seg_name\n");
fprintf(stderr, "Usage: tsmemseg [-4][-i inittime][-t time][-p ptime][-a acc_timeout][-c cmd][-r readrate][-f fill_readrate][-s seg_num][-m max_kbytes][-g dir][-d flags] seg_name\n");
return 2;
}
bool invalid = false;
Expand Down Expand Up @@ -714,6 +737,12 @@ int main(int argc, char **argv)
segMaxBytes = static_cast<size_t>(strtol(argv[++i], nullptr, 10) * 1024);
invalid = segMaxBytes < 32 * 1024 || 32 * 1024 * 1024 < segMaxBytes;
}
else if (c == 'g') {
++i;
#ifndef _WIN32
fifoDir = argv[i];
#endif
}
else if (c == 'd') {
id3conv.SetOption(static_cast<int>(strtol(argv[++i], nullptr, 10)));
}
Expand Down Expand Up @@ -814,6 +843,9 @@ int main(int argc, char **argv)
#ifdef _WIN32
// Used for asynchronous writing of segments.
std::vector<std::unique_ptr<CManualResetEvent>> events;
const char *signature = nullptr;
#else
const char *signature = destName;
#endif

while (segments.size() < 1 + segNum) {
Expand All @@ -836,15 +868,22 @@ int main(int argc, char **argv)
break;
}
#else
sprintf(seg.path, "/tmp/tsmemseg_%s%02d.fifo", destName, static_cast<int>(segments.size()));
if (mkfifo(seg.path, S_IRWXU) != 0) {
size_t dirLen = strlen(fifoDir);
if ((dirLen ? dirLen + (fifoDir[dirLen - 1] != '/' ? 1 : 0) : 5) + strlen(destName) + 16 >= sizeof(seg.path)) {
// path too long
break;
}
sprintf(seg.path, "%s%stsmemseg_%s%02d.fifo", dirLen ? fifoDir : "/tmp/",
dirLen && fifoDir[dirLen - 1] != '/' ? "/" : "",
destName, static_cast<int>(segments.size()));
if (mkfifo(seg.path, S_IRUSR + S_IWUSR + (dirLen ? S_IRGRP + S_IWGRP + S_IROTH + S_IWOTH : 0)) != 0) {
break;
}
#endif
seg.segCount = SEGMENT_COUNT_EMPTY;
if (!segments.empty()) {
seg.buf.assign(188, 0);
WriteSegmentHeader(seg.buf, seg.segCount, isMp4, mp4frag.GetFragmentSizes());
seg.buf.assign(signature ? 376 : 188, 0);
WriteSegmentHeader(seg.buf, signature, seg.segCount, isMp4, mp4frag.GetFragmentSizes());
}
segments.push_back(std::move(seg));
}
Expand All @@ -853,7 +892,7 @@ int main(int argc, char **argv)
fprintf(stderr, "Error: pipe/fifo creation failed.\n");
return 1;
}
AssignSegmentList(segments.front().buf, segments, 1, false, false, isMp4, mp4frag.GetHeader());
AssignSegmentList(segments.front().buf, signature, segments, 1, false, false, isMp4, mp4frag.GetHeader());

#ifndef _WIN32
struct sigaction sigact = {};
Expand Down Expand Up @@ -956,7 +995,7 @@ int main(int argc, char **argv)
}

std::vector<uint8_t> &segBuf = SelectWritableSegmentBuffer(seg);
segBuf.assign(188, 0);
segBuf.assign(signature ? 376 : 188, 0);

if (isMp4) {
seg.fragDurationsMsec = mp4frag.GetFragmentDurationsMsec();
Expand All @@ -981,12 +1020,12 @@ int main(int argc, char **argv)
segBuf.insert(segBuf.end(), packets.begin(), packets.end());
}

WriteSegmentHeader(segBuf, seg.segCount, isMp4, mp4frag.GetFragmentSizes());
WriteSegmentHeader(segBuf, signature, seg.segCount, isMp4, mp4frag.GetFragmentSizes());
if (!segIncomplete) {
mp4frag.ClearFragments();
}
std::vector<uint8_t> &segfrBuf = SelectWritableSegmentBuffer(segments.front());
AssignSegmentList(segfrBuf, segments, segIndex, false, segIncomplete, isMp4, mp4frag.GetHeader());
AssignSegmentList(segfrBuf, signature, segments, segIndex, false, segIncomplete, isMp4, mp4frag.GetHeader());
return false;
});

Expand All @@ -995,7 +1034,7 @@ int main(int argc, char **argv)

// End list
std::vector<uint8_t> &segfrBuf = SelectWritableSegmentBuffer(segments.front());
AssignSegmentList(segfrBuf, segments, segIndex, true, false, isMp4, mp4frag.GetHeader());
AssignSegmentList(segfrBuf, signature, segments, segIndex, true, false, isMp4, mp4frag.GetHeader());
}

if (syncError) {
Expand Down

0 comments on commit 4850e9c

Please sign in to comment.