Kamailio 4.4.4 crash in tcp_read_headers()

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Kamailio 4.4.4 crash in tcp_read_headers()

Armen Babikyan
Hello,

Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:


Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf.  That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced.  In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.

Other info: This is Kamailio 4.4.4 on x86_64.  I've not had success trying to reproduce this yet.  Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.

Thoughts are welcome.  Thanks!

Armen


_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 4.4.4 crash in tcp_read_headers()

Daniel-Constantin Mierla-6

Hello,

I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there?

Can you also give the locals?

frame 0

info locals

Cheers,
Daniel


On 05.06.17 05:19, Armen Babikyan wrote:
Hello,

Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:


Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf.  That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced.  In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.

Other info: This is Kamailio 4.4.4 on x86_64.  I've not had success trying to reproduce this yet.  Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.

Thoughts are welcome.  Thanks!

Armen



_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 4.4.4 crash in tcp_read_headers()

Armen Babikyan
Hi Daniel,

The server is running other protocols as well, yes, but those requests are handled on other ports (e.g. WSS on 443/tcp, TLS on 5061/tcp).

Regarding the locals, I have updated the pastebin.

Many thanks!

Armen


On Mon, Jun 5, 2017 at 1:23 AM, Daniel-Constantin Mierla <[hidden email]> wrote:

Hello,

I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there?

Can you also give the locals?

frame 0

info locals

Cheers,
Daniel


On 05.06.17 05:19, Armen Babikyan wrote:
Hello,

Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:


Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf.  That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced.  In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.

Other info: This is Kamailio 4.4.4 on x86_64.  I've not had success trying to reproduce this yet.  Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.

Thoughts are welcome.  Thanks!

Armen



_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users

-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users



_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Reply | Threaded
Open this post in threaded view
|

Re: Kamailio 4.4.4 crash in tcp_read_headers()

Daniel-Constantin Mierla-6

Hello,

I analyzed the code and couldn't find a reason for that pointer to be out of range. It could be some memory corruption, independent of the software, given that's one bit shifted one position in 'parsed', all other bits are the same as for buf/start/pos, which should be the good value.

But first to dig into it a bit more ...

  - from frame 0, let's see if there is something in the read buffer, get:

p r->buf[0]
p r->buf[1]
p r->buf[2]
p r->buf[3]

  - from frame 3, get:

info locals
p *h
p *fm

Cheers,
Daniel

On 05.06.17 16:58, Armen Babikyan wrote:
Hi Daniel,

The server is running other protocols as well, yes, but those requests are handled on other ports (e.g. WSS on 443/tcp, TLS on 5061/tcp).

Regarding the locals, I have updated the pastebin.

Many thanks!

Armen


On Mon, Jun 5, 2017 at 1:23 AM, Daniel-Constantin Mierla <[hidden email]> wrote:

Hello,

I see port is 5060, is it a possibility that you have multiplexing of websocket or other protocol (http, msrp) there?

Can you also give the locals?

frame 0

info locals

Cheers,
Daniel


On 05.06.17 05:19, Armen Babikyan wrote:
Hello,

Over the past few months, I've seen a smattering of kamailio crashes on various systems with identical backtraces: SIGSEGV in tcp_read_headers(), at tcp_read.c line 628. Example here:


Note that in frame 0, print *c shows that req->parsed is pointing to an address exactly 8GB lower than req->buf.  That req->parsed is pointing to an invalid memory location, crashing kamailio when the location is dereferenced.  In other coredumps, I see that req->parsed is pointing to an address exactly 4GB lower than req->buf.

Other info: This is Kamailio 4.4.4 on x86_64.  I've not had success trying to reproduce this yet.  Also worth noting that the crashes seem to be consistently associated with processing traffic from a UA connected over SIP/TCP; I've seen no other transport associated with this crash.

Thoughts are welcome.  Thanks!

Armen



_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com
_______________________________________________ Kamailio (SER) - Users Mailing List [hidden email] https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- 
Daniel-Constantin Mierla
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Kamailio Advanced Training - www.asipto.com
Kamailio World Conference - www.kamailioworld.com

_______________________________________________
Kamailio (SER) - Users Mailing List
[hidden email]
https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users