[SIP Beyond VoIP] Issue causes Sylkserver to refuse new connections

[Dan Pascu]

Thanks for the patch. I'll check it and integrate it.

On 13 Feb 2019, at 16:55, Penelope Vennisse wrote:

> We've continued analysing the issue with the growing number of mixer ports and here's what we've learned.
> 
> 1. We've been having multiple incoming SIP connections from unknown (probably, attacker) SIP servers as our SylkServer instance was not protected by a firewall. These connections were initiated and then failed after a second or so.
> 
> 2. We tried using the 'trusted_peers' setting in the SylkServer config.ini to only allow connections from our own servers. According to our logs, there is still a new audio stream created for each incoming SIP connection from the  addresses, not listed in 'trusted_peers'. These streams don't seem to be destroyed properly since, for each such stream, 2 ports are allocated in the audio mixer and they're never removed.
> 
> So enabling the 'trusted_peers' option in our case only lead to a steadily growing number of used audio ports, which, as we understand, would eventually lead to the PJ_ETOOMANY exception from our previous email.
> 
> We ended up setting firewall rules on the machine so that unwanted SIP connections never reach SylkServer.
> 
> 3. We also discovered that whenever an incoming SIP connection is canceled, it creates 2 extra ports in the audio mixer as well that stay until the SylkServer restart.
> 
> I'm attaching a patch that seems to have addressed (2) and (3) for us. We're not sure though that this would cover all the cases when audio streams are not terminated properly. Let me know if it looks correct to you.
> 
> Also, we have no new info on the python logging exception from the previous email, so there might still be another independent problem leading to similar consequences.
> 
> Please let me know what you think.
> 
> 
> On Wed, Jan 30, 2019 at 4:18 PM Penelope Vennisse <studio.pqv9 at gmail.com> wrote:
> We're using SylkServer as an audio conference server for webRTC and SIP clients. Only the Conference and the WebRTC Gateway applications are enabled in the config. Our SylkServer version is 5.0.0, checked out from this GitHub tag.
> 
> Once in several days, we observe the following problem: the CPU usage by the SylkServer process starts to grow and all the webRTC client connection attempts fail. After some time (usually a few hours), the following exception appears in the logs:
> 
> ERROR    [sylk.core]     Exception occurred while setting up stream from SDP: Could not add audio object to audio mixer: Too many objects of the specified type (PJ_ETOOMANY)
> ERROR    [sylk.core]     Traceback (most recent call last):
> ERROR    [sylk.core]       File "/srv/sylkserver/sylk/session.py", line 394, in init_incoming
> ERROR    [sylk.core]         stream = stream_type.new_from_sdp(self, remote_sdp, index)
> ERROR    [sylk.core]       File "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/__init__.py", line 425, in new_from_sdp
> ERROR    [sylk.core]         stream = cls()
> ERROR    [sylk.core]       File "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/audio.py", line 24, in __init__
> ERROR    [sylk.core]         self.bridge = AudioBridge(self.mixer)
> ERROR    [sylk.core]       File "/usr/lib/python2.7/dist-packages/sipsimple/audio.py", line 129, in __init__
> ERROR    [sylk.core]         self.multiplexer.start()
> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line 1310, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38212)
> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line 1307, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38143)
> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line 471, in sipsimple.core._core.AudioMixer._add_port (sipsimple/core/_core.c:24844)
> 
> We've added some logs to the SylkServer and the Python SipSimple library to clear things up a bit. What we discovered is that during the period when the connection attempts fail, the number of used audio mixer ports (we were logging the conf->port_cnt counter in pjsip/pjmedia/src/pjmedia/conference.c) grows steadily and reaches the limit (9999) in the end. Right before this period starts there are these two lines in the logs:
> 
> Traceback (most recent call last):
> File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
> 
> This looks like a cropped exception message, but there is nothing more. It might be that some log message is ill-formatted but we're not sure where to look for it.
> 
> Please let us know if this is a known issue or if you have any advice on how we could track it down.
> 
> Thanks.
> <session.patch>_______________________________________________
> SIPBeyondVoIP mailing list
> SIPBeyondVoIP at lists.ag-projects.com
> http://lists.ag-projects.com/mailman/listinfo/sipbeyondvoip


--
Dan