### Description
We send HEP3 data to the UDP listener in kamailio. In this HEP3 data, there are SIP requests and replies. We want to route these SIP messages through `request_route` and `onreply_route`. For this, we use the SipCapture module with the following settings:
``` modparam("sipcapture", "db_url", DBURL) modparam("sipcapture", "capture_on", 1) modparam("sipcapture", "hep_capture_on", 1) modparam("sipcapture", "insert_retries", 0) modparam("sipcapture", "insert_retry_timeout", 0)
request_route { xlog("L_INFO", "Got new request ($ci) $rm: $si => $Ri\n"); }
onreply_route { xlog("L_INFO", "Got new reply ($ci) $rm: $si => $Ri - Reply Code $rs\n"); } ```
In the past (version kamailio 5.1.x - 5.4.x), this worked as expected. From the logs:
``` 2(8) INFO: <script>: Got new request (7xkVDrAWegvcZl-ACaUb1HWHwiCYhAJR) INVITE: 217.116.121.230 => 172.20.21.4 3(9) INFO: <script>: Got new reply (7xkVDrAWegvcZl-ACaUb1HWHwiCYhAJR) INVITE: 172.20.21.4 => 217.116.121.230 - Reply Code 407 ```
Starting with kamailio 5.5.x, some pseudo variables are broken. From the logs:
``` 1(7) INFO: <script>: Got new request (7xkVDrAWegvcZl-ACaUb1HWHwiCYhAJR) INVITE: 217.116.121.230 => 217.116.121 2(8) INFO: <script>: Got new reply (7xkVDrAWegvcZl-ACaUb1HWHwiCYhAJR) INVITE: 172.20.21.4 => 172.20.21.4 3(9) INFO: <script>: ... ```
As you can see, the `$Ri` in the request is wrong. It looks like the first three octets of the `$si` and the fourth octet is missing entirely. And for the reply:
- The `$Ri` is the `$si`. - The line is missing the static text `Reply Code`, the actual reply code. - The next log is on the same line (the new line is gone). (In this case, we simply sent a third HEP3 message.)
To my untrained eye this looks like possible memory corruption.
### Troubleshooting
#### Reproduction
1. Set up a kamailio 5.8.1 using the attached [kamailio.cfg](https://github.com/kamailio/kamailio/files/15161868/kamailio.cfg.zip) `/usr/sbin/kamailio -E -DD -dd -f "/etc/kamailio/kamailio.cfg"` 2. Send the HEP3 data using [heplify](https://github.com/sipcapture/heplify) `./heplify -t pcap -rf "sipcapture-bug.pcap" -hs localhost:9060 -e -eof-exit`
#### Debugging Data
#### Log Messages
see above
#### SIP Traffic
see attached [sipcapture-bug.pcap](https://github.com/kamailio/kamailio/files/15161698/sipcapture-bug.zip)
### Possible Solutions
Use kamailio 5.4.x or earlier versions.
### Additional Information
When setting `nonsip_hook` = `1`, then the `$hep(...)` pseudo variables look fine. Unfortunately, we don't know how process the SIP messages after that point.
``` modparam("sipcapture", "nonsip_hook", 1)
...
event_route[sipcapture:request] { xlog("L_INFO", "version: $hep(version)\n"); xlog("L_INFO", "src_ipv4: $hep(0x003)\n"); xlog("L_INFO", "dst_ipv4: $hep(0x004)\n"); xlog("L_INFO", "src_proto: $hep(0x007)\n"); xlog("L_INFO", "dst_proto: $hep(0x008)\n"); xlog("L_INFO", "proto: $hep(0x00b)\n"); }
```
* **Kamailio Version** - output of `kamailio -v`
``` version: kamailio 5.8.1 (x86_64/linux) flags: USE_TCP, USE_TLS, USE_SCTP, TLS_HOOKS, USE_RAW_SOCKS, DISABLE_NAGLE, USE_MCAST, DNS_IP_HACK, SHM_MMAP, PKG_MALLOC, MEM_JOIN_FREE, Q_MALLOC, F_MALLOC, TLSF_MALLOC, DBG_SR_MEMORY, USE_FUTEX, FAST_LOCK-ADAPTIVE_WAIT, USE_DNS_CACHE, USE_DNS_FAILOVER, USE_NAPTR, USE_DST_BLOCKLIST, HAVE_RESOLV_RES, TLS_PTHREAD_MUTEX_SHARED ADAPTIVE_WAIT_LOOPS 1024, MAX_RECV_BUFFER_SIZE 262144, MAX_SEND_BUFFER_SIZE 262144, MAX_URI_SIZE 1024, BUF_SIZE 65535, DEFAULT PKG_SIZE 8MB poll method support: poll, epoll_lt, epoll_et, sigio_rt, select. id: compiled with gcc 12.2.0 ```
Also tested with same error: 5.5.x, 5.6.x, 5.7.x
* **Operating System**:
``` # lsb_release -a No LSB modules are available. Distributor ID: Debian Description: Debian GNU/Linux 12 (bookworm) Release: 12 Codename: bookworm ```
another way to do it (just as a workaround)
`` modparam("sipdump", "event_callback", "sipdump_event")
//you can check everything here - method, si, RI and send to a destination. event_route[sipdump_event] { sip_trace("sip:10.1.1.2:5085", "$ci-abc"); }
``
btw, same you can do for sipcapture:request - execute manual sip_trace and after send return 0 or exit;
@adubovikov: as I see it right now, by replicating at SIP level (e.g., via sip dump) to another system, the original source ip/destination IP are lost in the receiving system. They can eventually be added as custom headers.
@adubovikov: as I see it right now, by replicating at SIP level (e.g., via sip dump) to another system, the original source ip/destination IP are lost in the receiving system. They can eventually be added as custom headers.
or we should set the hep variables. Anyway @sipgate-uhlig is talking about the corrupted $Ri and $si and unfortunately this is not related to the siptrace module. Do you have any idea ?
Thank you for your input so far. All the alternatives seem to work-around the issue but not fulfill our needs.
We have debugged a little using `gdb`. We noticed that the value of `msg->rcv->bind_address->address_str` is correct before calling `do_action`. Afterwards, it is wrong.
Before: ``` s = "172.20.21.3" len = 11 ```
After: ``` s = "217.116.120.247" len = 11 (sic!) ```
We took a closer look (step by step) and the problem seems to be using `ip_addr2a`. It returns a pointer to a local static `char[]`. This is assigned to the original struct of `address_str` inside of `parsing_hepv3_message`.
Later, when `ip_addr2a` is called again for xlog, then the same address is used again and hence the value of `address_str.s` is overwriten.
A quick look at the git history for the functions did not show any relevant changes to my untrained eyes.
As the jump from 5.1/5.4 (debian jessie) to 5.5/5.8 (debian bookworm) is kind of huge, maybe this is related to changes in the compiler?
Thank you for your input so far. All the alternatives seem to work-around the issue but not fulfill our needs.
We have debugged a little using `gdb`. We noticed that the value of `msg->rcv->bind_address->address_str` is correct before calling `do_action`. Afterwards, it is wrong.
Before:
s = "172.20.21.3" len = 11After:
s = "217.116.120.247" len = 11 (sic!)We took a closer look (step by step) and the problem seems to be using `ip_addr2a`. It returns a pointer to a local static `char[]`. This is assigned to the original struct of `address_str` inside of `parsing_hepv3_message`.
Later, when `ip_addr2a` is called again for xlog, then the same address is used again and hence the value of `address_str.s` is overwriten.
A quick look at the git history for the functions did not show any relevant changes to my untrained eyes.
As the jump from 5.1/5.4 (debian jessie) to 5.5/5.8 (debian bookworm) is kind of huge, maybe this is related to changes in the compiler?
ok, it can be related to the struct pack, need to check it the union works correct. What is the compiler ?
https://github.com/kamailio/kamailio/blob/master/src/core/ip_addr.h#L74-L80
For kamailio 5.8 we have used ``` # gcc --version gcc (Debian 12.2.0-14) 12.2.0 ```
I am not a C developer so I have to guess here.
For me it looks like the `static` address of `buff` is returned by `ip_addr2a`.
```c char *ip_addr2a(struct ip_addr *ip) { static char buff[IP_ADDR_MAX_STR_SIZE]; // ... return buff; } ```
-- https://github.com/kamailio/kamailio/blob/1535031a6c992c23270050793b23d290a6...
If that's the case, I think every user of that function should copy the value from that address instead of remembering the address. Otherwise it is always the same address for every call of `ip_addr2a` and subsequent calls overwrite the value of the previous call.
Unfortunately, the sipcapture module uses the returned value directly: ```c si->address_str.s = ip_addr2a(&si->address); ``` -- https://github.com/kamailio/kamailio/blob/1535031a6c992c23270050793b23d290a6...
I tested with the following code and I think it confirms my theory because `foo` changes to "bar".
```c #include <stdio.h> #include <string.h>
char *a(char* in);
int main() { char *foo = a("foo"); printf("foo %08x -> %08x = %s\n", &foo, foo, foo);
char *bar = a("bar"); printf("foo %08x -> %08x = %s\n", &foo, foo, foo); printf("bar %08x -> %08x = %s\n", &bar, bar, bar); }
char *a(char* in) { static char out[10]; strcpy(out, in); return out; } ```
Output: ```plain foo 0c3399d8 -> b37ba018 = foo foo 0c3399d8 -> b37ba018 = bar bar 0c3399e0 -> b37ba018 = bar ```
However, I can't explain why this should be any different in previous kamailio versions.
Can you try with master branch or the patch from the commit references above?
I've worked with @sipgate-uhlig on this topic. He's out of office this week so I took a look at it together with another colleague. We can confirm that the latest master branch does no longer suffer from this issue . Thank you @miconda for resolving this so quickly!
Can we assume that it will be part of Kamailio 5.8.2 and if so, is there an ETA?
Thanks for testing and reporting back. Probably it's time do plan the 5.8.2, max 1-2 weeks.
Closing this one.
Closed #3835 as completed.
-
Alexandr Dubovikov -
Daniel-Constantin Mierla
-
Lennart Rosam
-
Sven Uhlig