<div dir="ltr"><div dir="ltr"><div>We've continued analysing the issue with the growing number of mixer ports and here's what we've learned.</div><div><br></div><div>1.
We've been having multiple incoming SIP connections from unknown
(probably, attacker) SIP servers as our SylkServer instance was not
protected by a firewall. These connections were initiated and then
failed after a second or so.</div><div><br></div><div>2. We tried using
the 'trusted_peers' setting in the SylkServer config.ini to only allow
connections from our own servers. According to our logs, there is still a
new audio stream created for each incoming SIP connection from the
addresses, not listed in 'trusted_peers'. These streams don't seem to be
destroyed properly since, for each such stream, 2 ports are allocated
in the audio mixer and they're never removed.</div><div><br></div><div>So
enabling the 'trusted_peers' option in our case only lead to a steadily
growing number of used audio ports, which, as we understand, would
eventually lead to the PJ_ETOOMANY exception from our previous email.</div><div><br></div><div>We ended up setting firewall rules on the machine so that unwanted SIP connections never reach SylkServer.</div><div><br></div><div>3.
We also discovered that whenever an incoming SIP connection is
canceled, it creates 2 extra ports in the audio mixer as well that stay
until the SylkServer restart.</div><div><br></div><div>I'm attaching a
patch that seems to have addressed (2) and (3) for us. We're not sure
though that this would cover all the cases when audio streams are not
terminated properly. Let me know if it looks correct to you.</div><div><br></div><div>Also,
we have no new info on the python logging exception from the previous
email, so there might still be another independent problem leading to
similar consequences.</div><div><br></div><div>Please let me know what you think.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Jan 30, 2019 at 4:18 PM Penelope Vennisse <<a href="mailto:studio.pqv9@gmail.com">studio.pqv9@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>We're using SylkServer as an audio conference server for webRTC and
SIP clients. Only the Conference and the WebRTC Gateway applications
are enabled in the config. Our SylkServer version is 5.0.0, checked out
from <a href="https://github.com/AGProjects/sylkserver/releases/tag/release-5.0.0" target="_blank">this GitHub tag</a>.</div><div><br></div><div>Once
in several days, we observe the following problem: the CPU usage by the
SylkServer process starts to grow and all the webRTC client connection
attempts fail. After some time (usually a few hours), the following
exception appears in the logs:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">ERROR
[sylk.core] Exception occurred while setting up stream from SDP:
Could not add audio object to audio mixer: Too many objects of the
specified type (PJ_ETOOMANY)<br>ERROR [sylk.core] Traceback (most recent call last):<br>ERROR [sylk.core] File "/srv/sylkserver/sylk/session.py", line 394, in init_incoming<br>ERROR [sylk.core] stream = stream_type.new_from_sdp(self, remote_sdp, index)<br>ERROR [sylk.core] File "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/__init__.py", line 425, in new_from_sdp<br>ERROR [sylk.core] stream = cls()<br>ERROR [sylk.core] File "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/audio.py", line 24, in __init__<br>ERROR [sylk.core] self.bridge = AudioBridge(self.mixer)<br>ERROR [sylk.core] File "/usr/lib/python2.7/dist-packages/sipsimple/audio.py", line 129, in __init__<br>ERROR [sylk.core] self.multiplexer.start()<br>ERROR [sylk.core] File "sipsimple/core/_core.sound.pxi", line 1310, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38212)<br>ERROR [sylk.core] File "sipsimple/core/_core.sound.pxi", line 1307, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38143)<br>ERROR [sylk.core] File "sipsimple/core/_core.sound.pxi", line 471, in sipsimple.core._core.AudioMixer._add_port (sipsimple/core/_core.c:24844)</blockquote><div><br></div><div>We've
added some logs to the SylkServer and the Python SipSimple library to
clear things up a bit. What we discovered is that during the period when
the connection attempts fail, the number of used audio mixer ports (we
were logging the conf->port_cnt counter in pjsip/pjmedia/src/pjmedia/conference.c)
grows steadily and reaches the limit (9999) in the end. Right before
this period starts there are these two lines in the logs:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Traceback (most recent call last):<br>File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit</blockquote><div><br></div><div>This
looks like a cropped exception message, but there is nothing more. It
might be that some log message is ill-formatted but we're not sure where
to look for it.</div><div><br></div><div>Please let us know if this is a known issue or if you have any advice on how we could track it down.</div><div><br></div><div>Thanks.</div></div>
</blockquote></div></div>