[SIP Beyond VoIP] Issue causes Sylkserver to refuse new connections

[Penelope Vennisse]

We've continued analysing the issue with the growing number of mixer ports
and here's what we've learned.

1. We've been having multiple incoming SIP connections from unknown
(probably, attacker) SIP servers as our SylkServer instance was not
protected by a firewall. These connections were initiated and then failed
after a second or so.

2. We tried using the 'trusted_peers' setting in the SylkServer config.ini
to only allow connections from our own servers. According to our logs,
there is still a new audio stream created for each incoming SIP connection
from the addresses, not listed in 'trusted_peers'. These streams don't seem
to be destroyed properly since, for each such stream, 2 ports are allocated
in the audio mixer and they're never removed.

So enabling the 'trusted_peers' option in our case only lead to a steadily
growing number of used audio ports, which, as we understand, would
eventually lead to the PJ_ETOOMANY exception from our previous email.

We ended up setting firewall rules on the machine so that unwanted SIP
connections never reach SylkServer.

3. We also discovered that whenever an incoming SIP connection is canceled,
it creates 2 extra ports in the audio mixer as well that stay until the
SylkServer restart.

I'm attaching a patch that seems to have addressed (2) and (3) for us.
We're not sure though that this would cover all the cases when audio
streams are not terminated properly. Let me know if it looks correct to you.

Also, we have no new info on the python logging exception from the previous
email, so there might still be another independent problem leading to
similar consequences.

Please let me know what you think.


On Wed, Jan 30, 2019 at 4:18 PM Penelope Vennisse <studio.pqv9 at gmail.com>
wrote:

> We're using SylkServer as an audio conference server for webRTC and SIP
> clients. Only the Conference and the WebRTC Gateway applications are
> enabled in the config. Our SylkServer version is 5.0.0, checked out from this
> GitHub tag
> <https://github.com/AGProjects/sylkserver/releases/tag/release-5.0.0>.
>
> Once in several days, we observe the following problem: the CPU usage by
> the SylkServer process starts to grow and all the webRTC client connection
> attempts fail. After some time (usually a few hours), the following
> exception appears in the logs:
>
> ERROR    [sylk.core]     Exception occurred while setting up stream from
>> SDP: Could not add audio object to audio mixer: Too many objects of the
>> specified type (PJ_ETOOMANY)
>> ERROR    [sylk.core]     Traceback (most recent call last):
>> ERROR    [sylk.core]       File "/srv/sylkserver/sylk/session.py", line
>> 394, in init_incoming
>> ERROR    [sylk.core]         stream = stream_type.new_from_sdp(self,
>> remote_sdp, index)
>> ERROR    [sylk.core]       File
>> "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/__init__.py", line
>> 425, in new_from_sdp
>> ERROR    [sylk.core]         stream = cls()
>> ERROR    [sylk.core]       File
>> "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/audio.py", line 24,
>> in __init__
>> ERROR    [sylk.core]         self.bridge = AudioBridge(self.mixer)
>> ERROR    [sylk.core]       File
>> "/usr/lib/python2.7/dist-packages/sipsimple/audio.py", line 129, in __init__
>> ERROR    [sylk.core]         self.multiplexer.start()
>> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
>> 1310, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38212)
>> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
>> 1307, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38143)
>> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
>> 471, in sipsimple.core._core.AudioMixer._add_port
>> (sipsimple/core/_core.c:24844)
>
>
> We've added some logs to the SylkServer and the Python SipSimple library
> to clear things up a bit. What we discovered is that during the period when
> the connection attempts fail, the number of used audio mixer ports (we were
> logging the conf->port_cnt counter in
> pjsip/pjmedia/src/pjmedia/conference.c) grows steadily and reaches the
> limit (9999) in the end. Right before this period starts there are these
> two lines in the logs:
>
> Traceback (most recent call last):
>> File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
>
>
> This looks like a cropped exception message, but there is nothing more. It
> might be that some log message is ill-formatted but we're not sure where to
> look for it.
>
> Please let us know if this is a known issue or if you have any advice on
> how we could track it down.
>
> Thanks.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ag-projects.com/pipermail/sipbeyondvoip/attachments/20190213/f52fca7f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: session.patch
Type: application/octet-stream
Size: 2609 bytes
Desc: not available
URL: <http://lists.ag-projects.com/pipermail/sipbeyondvoip/attachments/20190213/f52fca7f/attachment.obj>