Help with Push Publishing + stream map recreates new entries

[mvximvs]

Hi everybody, as the title says, I need help concerning a strange behavior of the push publishing feature.
We have this feature running in our Oven Media, with a stream map that contains the definitions of the push.
We push streams from FFmpeg to the Oven Media instance, but if we stop FFmpeg, and then we start again, the POST/ pushes api shows us a new entry for the same existent stream with different sequence numbers each time, different ids... Sometimes the two streams are in a pushing state, sometimes one is pushing, and the other one is ready.

So for our workflow, the main problem is, that in case of an unstable network or if we lost the stream for a couple of seconds, and then it starts streaming again the push publishing feature doesn't use the same existent push that has been registered previously, so it recreates a new entry each time.

Can you please tell us if this is a normal behavior or if this could be fixed on next releases?
Thanks a lot,

[bchah]

This could be a problem with StreamMap, because I don't see the ability there to set a static ID for each push. When you say the entries come back with different IDs, are you aware of how these IDs are being generated?

In my scenario we control push publishing entirely through API and it works as intended. In case it inspires or helps, here are the steps we take:

• In our app, a user says they would like to push to a destination
• In our database, we devise an ID and store the parameters of the push
• When the user clicks "start push" we make a POST to api/.../startPush
• OME automatically pauses and resumes the push when the stream starts and stops
• We store the status of the push (essentially, whether the user has clicked "stop push" or not) and when OME restarts we call the startPush endpoint for any pushes which had not previously been stopped, so they resume smoothly
• When the user does finally click "stop push", we send a POST to api/.../stopPush and the process begins again

This all works smoothly and as expected, so again I wonder if the issue has to do with StreamMap and whether it is possible to set a static Push ID to avoid duplication of pushes.

[bchah]

OK - it sounds like the ability to specify a static ID in the PushMap (which OME would check before creating another push definition) would solve this.

In the meantime as a workaround, you could use AdmissionWebhooks to check/create the pushes to your other OME instances.

[mvximvs]

That's entirely new to me, thanks for pointing it out, I've read the page on the airensoft site, but I'm not sure to understand how I can replace StreamMap with AdmissionWebhooks in order to achieve something similar. It will be possible to give me an example of this?
Thanks a lot!

[bchah]

The gist of it is... AdmissionWebhooks fires POST requests to your specified "control server" (or URL within your app) whenever a stream comes in or plays out. Receiving these requests, you can parse out things like the protocol, whether it is incoming or outgoing, what the stream name is etc... and apply your own logic to decide whether to allow or deny the request.

At this same time you can trigger other things like your push action. Here's my entire source code for my "control server". Some stuff like the DB calls and certain functions are specific to my app but this should be a good enough example to get your solution going:

<?php

require_once "db.php";
session_abort();
deny_api_access();
function respond($allowed, $reason = "allowed", $new_url = null)
{
    $response = ['allowed' => $allowed, 'reason' => $reason];
    if (isset($new_url)) {
    $response['new_url'] = $new_url;
    }
    header('Content-Type: application/json');
    echo json_encode($response);
    die();
}

function trim_signature($key_to_cut, $cut_point = "?p=")
{   
    if (strpos($key_to_cut, $cut_point) !== false) {
    $signaturePos = strrpos($key_to_cut, $cut_point);
    $cut_key = substr($key_to_cut, 0, $signaturePos);
    return $cut_key;
    }
}

function check_protocol_enabled($protocol)
{   
    $protocol_setting = "";

    $protocol_setting = match ($protocol) {
      "SRT" => "srt_input_enabled",
      "SRT_Playback" => "srt_playback_enabled",
      "RTMP" => "rtmp_input_enabled",
      "RTMPS" => "rtmps_input_enabled",
      "WHIP" => "whip_input_enabled",
      "Voice Chat" => "voice_chat_enabled",
      "Control Room" => "control_room",
    };
    
    if (get_setting($protocol_setting) != 1) {
    $allowed = false;
    $message = "$protocol is not permitted. Denying request.";
    logMe($message);
    respond($allowed, $message);
        return false;
    } else {
        return true;
    };
}

if ($_SERVER["REQUEST_METHOD"] == "POST" && $_SERVER["REMOTE_ADDR"] == "127.0.0.1") {

    $jsonData = file_get_contents('php://input');

    // Decode JSON data to PHP array
    $data = json_decode($jsonData, true);

    if ($data !== null && isset($data['request'])) {

        $protocol = $data['request']['protocol'];
        $direction = $data['request']['direction'];
        $status = $data['request']['status'];
        $url = $data['request']['url'];
        $address = $data['client']['address'];
        $new_url = null;

        $lastSlashPos = strrpos($url, '/');
        $stream_key = substr($url, $lastSlashPos + 1);

        if ($direction == "outgoing") {
            
            $allowed = false;
            if ($protocol == "SRT") {
                logMe("SRT playback requested by $address");
                check_protocol_enabled("SRT_Playback");
                $allowed = true;
                logMe("Allowing SRT playback to $address");
                respond($allowed);
            } else if ($protocol == "HLS") {
                // TODO: Improve check_protocol_enabled to handle multiple settings checks
                if (get_setting("hls_playback_option") != 1 && get_setting("hls_links_disabled") != 0) {
                    $allowed = false;
                    logMe($reason);
                    respond($allowed, $reason);
                } else {
                    $allowed = true;
                    logMe("Starting HLS playback to $address");
                    respond($allowed);
                }
            }

        } else if ($direction == "incoming") {

            logMe("Incoming $protocol stream from $address");
            logMe("Request URL: $url");

            if ($protocol == "WebRTC") {

                if (preg_match('/^http.*/', $url)) {
                    $protocol_name = "WHIP";
                    check_protocol_enabled($protocol_name);
                    $stream_key = trim_signature($stream_key);
                } else if (preg_match('/^ws.*/', $url) && strpos($url, "_TALK") == false) {
                    $protocol_name = "Control Room";
                    check_protocol_enabled($protocol_name);
                    $stream_key = trim_signature($stream_key);
                } else if (strpos($stream_key, '_TALK')) {
                    $protocol_name = "Voice Chat";
                    check_protocol_enabled($protocol_name);
                    $stream_key = trim_signature($stream_key, "_TALK");
                }

                logMe("Checking $protocol_name (WebRTC) stream key: $stream_key");

            } else if (($protocol == "RTMP" || $protocol == "SRT")) {
                $stream_key = trim_signature($stream_key);
                logMe("Checking $protocol stream key: $stream_key");
            } else {
                $allowed = false;
                logMe("Bad request from $address: $url");
                $reason = "Bad request!";
                respond($allowed, $reason);
            }

            if (db()->query("SELECT id FROM channels WHERE stream_key = ?", $stream_key)->numRows() > 0) {
                logMe("Stream key $stream_key exists, incoming stream allowed");
                
                // Reset max viewer count
                // Workaround for OME including recording/push stats in the stream stats (0.17.2 and up)
                db()->query("UPDATE channels SET viewer_limit = 0 WHERE stream_key = ?", $stream_key);
                
                $allowed = true;
                respond($allowed);
            } else {
                $allowed = false;
                logMe("Stream key $stream_key does not exist!");
                $reason = "Stream key $stream_key does not exist!";
                if ($stream_key == "test" && strpos($url, "rtmp:") !== false) {
                    // Allow test stream
                    $hostname = shell_exec("hostname -I");
                    $gateway = shell_exec("ip route | grep default | awk '{print $3}'");
                    if ($address == $hostname || $address == $gateway || $address == "172.17.0.1" || $address == "127.0.0.1" || $address == "192.168.65.1") {
                        logMe("However, this appears to be a bonafide test stream so we'll permit it.");
                        $reason = "We like tests.";
                        $allowed = true;
                    } else {
                        logMe("$address does not match $hostname or any other known-local IP.");
                    }
                }
                respond($allowed, $reason);
            }


        } else if ($status == "closing") {
            // All disconnections are allowed
            $allowed = true;
            respond($allowed);
        }

    }

}

[mvximvs]

Wow thanks a lot, I sincerely appreciate your help man. I'm going to discuss this topic and your example with my team.
In the meantime, should we open an issue around this topic?

[bchah]

Up to you whether you want to open an issue (please include debug logs, copies of your XML config files etc. if you do) or approach this as a feature request.

I wonder if, since you mentioned this happens when the stream only drops for a couple of seconds, the stream has not fully "timed out" on the server side when the new connection comes in. If you wait for 10, 20, 30 seconds between restarts does it behave better? If so I wonder if the <BlockDuplicateStreamName> option can help here: https://airensoft.gitbook.io/ovenmediaengine/live-source/rtmp

I also wonder if you were to adopt AdmissionWebhooks in conjunction with your existing StreamMap config, and forcibly DELETE any existing streams / STOP any existing pushes with the same stream key as the incoming stream... that might change the behaviour of the automated push too.

Good luck and keep us posted 👍

[Keukhan]

@mvximvs

In version 0.17.3, there's an issue where pushing through StreamMap causes session duplication or prevents deletion. This has been patched in the Master branch but hasn't been released yet. It should work properly once the next version is released.

b2783ad

Thanks :)

[mvximvs]

Good news! Thanks for the information, do you have any news concerning the release date of the new version?

[bchah]

There's the answer we wanted!!
@Keukhan do you think we could gain the ability to set static IDs (or IDs based on Macros) in the StreamMap XML?

[Keukhan]

@mvximvs
There are still many features being tested, so I’m not sure when the release will be. For now, please try the master version, and if you find any problems, please let me know.

@bchah
Using a static id isn't possible because it can cause duplicate id. but, we can use macros to create a predictable id.
Thanks for the great idea. I'll add it to the plan.