[SIP Beyond VoIP] Issue causes Sylkserver to refuse new connections

Penelope Vennisse studio.pqv9 at gmail.com
Thu Apr 11 16:09:10 CEST 2019


Thanks for implementing the fix! We've updated our Python SIPSimple version
to v3.4.1 and the SylkServer to v5.2.0. Looks like the problem with the
growing number of ports is now gone - we haven't seen the PJ_ETOOMANY
exception for more than a month now.

However, recently we saw the logging error that we mentioned in the
previous emails. Looks like this might be an independent problem. Our logs
show the following two lines:

Traceback (most recent call last):
> File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit


The traceback ends there. After this happens, SylkServer stops responding
to session create requests until we restart it.

I understand the error is not very helpful and this might even be an issue
with the Python logging library or something related, but I thought it
makes sense to ask you anyway in case you might have some advice.

Our operating system info is as follows:

Debian GNU/Linux 9.6 (stretch)
Kernel: Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.130-2 (2018-10-27)

The Python version is 2.7.13.

Please let me know if you have any clue as to what might be the problem.

Thanks for your time!

On Tue, Feb 19, 2019 at 7:52 PM Dan Pascu <dan at ag-projects.com> wrote:

> It turns out that the fix was completely different that what was proposed
> in the patch and it had to be applied in python-sipsimple. The whole issue
> originated with a memory leak in sipsimple that happened with streams that
> were created but ended up not being used (like a canceled or refused call).
>
> To fix your issue you'll need to get and build the latest version of
> python-sipsimple (either from darcs or github). While you're at it, you can
> also grab the latest version of sylkserver from the same sources, as it
> contains a few additional fixes, including a smaller memory leak that
> affects canceling mid-session proposals (re-invites).
>
> Let me know if it fixes your problems. If nothing else shows up, I'll make
> new releases for both.
>
> On 14 Feb 2019, at 14:56, Dan Pascu wrote:
>
> > Thanks for the patch. I'll check it and integrate it.
> >
> > On 13 Feb 2019, at 16:55, Penelope Vennisse wrote:
> >
> >> We've continued analysing the issue with the growing number of mixer
> ports and here's what we've learned.
> >>
> >> 1. We've been having multiple incoming SIP connections from unknown
> (probably, attacker) SIP servers as our SylkServer instance was not
> protected by a firewall. These connections were initiated and then failed
> after a second or so.
> >>
> >> 2. We tried using the 'trusted_peers' setting in the SylkServer
> config.ini to only allow connections from our own servers. According to our
> logs, there is still a new audio stream created for each incoming SIP
> connection from the  addresses, not listed in 'trusted_peers'. These
> streams don't seem to be destroyed properly since, for each such stream, 2
> ports are allocated in the audio mixer and they're never removed.
> >>
> >> So enabling the 'trusted_peers' option in our case only lead to a
> steadily growing number of used audio ports, which, as we understand, would
> eventually lead to the PJ_ETOOMANY exception from our previous email.
> >>
> >> We ended up setting firewall rules on the machine so that unwanted SIP
> connections never reach SylkServer.
> >>
> >> 3. We also discovered that whenever an incoming SIP connection is
> canceled, it creates 2 extra ports in the audio mixer as well that stay
> until the SylkServer restart.
> >>
> >> I'm attaching a patch that seems to have addressed (2) and (3) for us.
> We're not sure though that this would cover all the cases when audio
> streams are not terminated properly. Let me know if it looks correct to you.
> >>
> >> Also, we have no new info on the python logging exception from the
> previous email, so there might still be another independent problem leading
> to similar consequences.
> >>
> >> Please let me know what you think.
> >>
> >>
> >> On Wed, Jan 30, 2019 at 4:18 PM Penelope Vennisse <
> studio.pqv9 at gmail.com> wrote:
> >> We're using SylkServer as an audio conference server for webRTC and SIP
> clients. Only the Conference and the WebRTC Gateway applications are
> enabled in the config. Our SylkServer version is 5.0.0, checked out from
> this GitHub tag.
> >>
> >> Once in several days, we observe the following problem: the CPU usage
> by the SylkServer process starts to grow and all the webRTC client
> connection attempts fail. After some time (usually a few hours), the
> following exception appears in the logs:
> >>
> >> ERROR    [sylk.core]     Exception occurred while setting up stream
> from SDP: Could not add audio object to audio mixer: Too many objects of
> the specified type (PJ_ETOOMANY)
> >> ERROR    [sylk.core]     Traceback (most recent call last):
> >> ERROR    [sylk.core]       File "/srv/sylkserver/sylk/session.py", line
> 394, in init_incoming
> >> ERROR    [sylk.core]         stream = stream_type.new_from_sdp(self,
> remote_sdp, index)
> >> ERROR    [sylk.core]       File
> "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/__init__.py", line
> 425, in new_from_sdp
> >> ERROR    [sylk.core]         stream = cls()
> >> ERROR    [sylk.core]       File
> "/usr/lib/python2.7/dist-packages/sipsimple/streams/rtp/audio.py", line 24,
> in __init__
> >> ERROR    [sylk.core]         self.bridge = AudioBridge(self.mixer)
> >> ERROR    [sylk.core]       File
> "/usr/lib/python2.7/dist-packages/sipsimple/audio.py", line 129, in __init__
> >> ERROR    [sylk.core]         self.multiplexer.start()
> >> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
> 1310, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38212)
> >> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
> 1307, in sipsimple.core._core.MixerPort.start (sipsimple/core/_core.c:38143)
> >> ERROR    [sylk.core]       File "sipsimple/core/_core.sound.pxi", line
> 471, in sipsimple.core._core.AudioMixer._add_port
> (sipsimple/core/_core.c:24844)
> >>
> >> We've added some logs to the SylkServer and the Python SipSimple
> library to clear things up a bit. What we discovered is that during the
> period when the connection attempts fail, the number of used audio mixer
> ports (we were logging the conf->port_cnt counter in
> pjsip/pjmedia/src/pjmedia/conference.c) grows steadily and reaches the
> limit (9999) in the end. Right before this period starts there are these
> two lines in the logs:
> >>
> >> Traceback (most recent call last):
> >> File "/usr/lib/python2.7/logging/__init__.py", line 861, in emit
> >>
> >> This looks like a cropped exception message, but there is nothing more.
> It might be that some log message is ill-formatted but we're not sure where
> to look for it.
> >>
> >> Please let us know if this is a known issue or if you have any advice
> on how we could track it down.
> >>
> >> Thanks.
> >> <session.patch>_______________________________________________
> >> SIPBeyondVoIP mailing list
> >> SIPBeyondVoIP at lists.ag-projects.com
> >> http://lists.ag-projects.com/mailman/listinfo/sipbeyondvoip
> >
> >
> > --
> > Dan
> >
> >
> >
> >
> > _______________________________________________
> > SIPBeyondVoIP mailing list
> > SIPBeyondVoIP at lists.ag-projects.com
> > http://lists.ag-projects.com/mailman/listinfo/sipbeyondvoip
> >
>
>
> --
> Dan
>
>
>
>
> _______________________________________________
> SIPBeyondVoIP mailing list
> SIPBeyondVoIP at lists.ag-projects.com
> http://lists.ag-projects.com/mailman/listinfo/sipbeyondvoip
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.ag-projects.com/pipermail/sipbeyondvoip/attachments/20190411/b443ac9e/attachment.html>


More information about the SIPBeyondVoIP mailing list