Damm that was a rabbit hole...

So the key pointers were found thanks to reading two very helpful links [1] [2].

The TL;DR is that I use setcap to add capabilities to Kamailio to allow to listen on ports <1024 without root. Once you add capabilities, any LD_* env var gets stripped out and is not accessible to the process for security reasons. The solution is to have the sslkeylogger.so lib in a system LD path with setuid bit added (chmod +s), and load it without any "/" in the name.

So basically doing this (pseudo commands):

mv keylogger.so /system/ld/path/keylogger.so
chmod u+s /system/ld/path/keylogger.so

And then have the /etc/default/kamailio.d/voipmonitor file as:

SSLKEYLOG_UDP='127.0.0.1:1234'
LD_PRELOAD="sslkeylog.so libssl.so.1.1"

Restart and boom, sslkeylogger is loaded:

root@csbc03:~# fgrep ssl /proc/2633948/maps
7f97ffb92000-7f97ffbaf000 r--p 00000000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f97ffbaf000-7f97ffbfd000 r-xp 0001d000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f97ffbfd000-7f97ffc17000 r--p 0006b000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f97ffc17000-7f97ffc18000 ---p 00085000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f97ffc18000-7f97ffc21000 r--p 00085000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f97ffc21000-7f97ffc25000 rw-p 0008e000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
7f9800173000-7f9800174000 r--p 00000000 08:06 262170                     /usr/lib/x86_64-linux-gnu/sslkeylog.so
7f9800174000-7f9800175000 r-xp 00001000 08:06 262170                     /usr/lib/x86_64-linux-gnu/sslkeylog.so
7f9800175000-7f9800176000 r--p 00002000 08:06 262170                     /usr/lib/x86_64-linux-gnu/sslkeylog.so
7f9800176000-7f9800177000 r--p 00002000 08:06 262170                     /usr/lib/x86_64-linux-gnu/sslkeylog.so
7f9800177000-7f9800178000 rw-p 00003000 08:06 262170                     /usr/lib/x86_64-linux-gnu/sslkeylog.so
root@csbc03:~#

I have one last question for you Calvin, 

Can you share the settings you have on your local and remote voipmonitor-sniffers to allow Kamailio to send keys to 127.0.0.1:1234, and then have the voipmonitor-client forward that to voipmonitor-server for processing?

These are mine:

-CLIENT- 

[general]
id_sensor = 23
query_cache = yes
server_destination = XXX
server_destination_port = XXX
server_password = XXX
packetbuffer_sender             = yes
packetbuffer_enable             = yes
packetbuffer_total_maxheap      = 512 #in MB
packetbuffer_compress           = yes
packetbuffer_file_totalmaxsize  = 2000 #MB. Default is disabled.
packetbuffer_file_path          = /var/spool/voipmonitor/packetbuffer
interface = eno1,lo
sipport = 5060
sipport = 5061
sipport = 5062

NOTES: 5060 is regular UDP, 5061 and 5062 are both TLS ports.

-SERVER- (only the ssl_* settings)

ssl = yes
ssl_ipport = A.B.C.D : 5061
ssl_ipport = A.B.C.E : 5061
ssl_ipport = A.B.C.F : 5061
ssl_ipport = A.B.C.G : 5061
ssl_ipport = A.B.C.D : 5062
ssl_ipport = A.B.C.E : 5062
ssl_ipport = A.B.C.F : 5062
ssl_ipport = A.B.C.G : 5062
ssl_store_sessions_expiration_hours = 48
ssl_sessionkey_udp = yes
ssl_sessionkey_udp_port = 1234
ssl_sessionkey_udp_ip = 192.168.1.0/24
ssl_sessionkey_udp_maxwait_ms = 10000
ssl_store_sessions = persistent
ssl_ignore_error_invalid_mac = yes

NOTES: All the A.B.C.X are Kamailio instances Public IPs.

I know I'm missing something to get the combo Kamailio->Local-Sniffer->Remote-Sniffer to work, any hints there?

Thanks, 
Joel.


[1] https://stackoverflow.com/questions/18058426/does-using-linux-capabilities-disable-ld-preload
[2] https://unix.stackexchange.com/questions/757484/ld-preload-does-not-work-and-ld-debug-shows-nothing
 

On Thu, Mar 7, 2024 at 4:33 PM Calvin E. <calvine@gmail.com> wrote:
Does your sslkeylog.so work on that same host with the openssl test? I
noticed you're using ansible, so I'm curious if you're compiling on
some other host that could have different versions of the openssl-dev
stuff. Other things could be file or path permissions, or maybe a
security tool blocking it (would auditd do that?).

At this point I'd reach out to their support.

On Tue, Mar 5, 2024 at 10:24 PM Joel Serrano <joel@textplus.com> wrote:
>
> Hi Calvin,
>
> Thanks for the tip on capturing on LO interface, I'm sure you just saved me some headaches ;)
>
> Interestingly when I check the environ I do see the env vars being set, but in the maps I don't see the keylogger:
>
> root@csbc03:~# cat /proc/2216899/environ
> LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binPIDFILE=/run/kamailio/kamailio.pidHOME=/run/kamailioLOGNAME=kamailioUSER=kamailioINVOCATION_ID=fb5d2818a5434319ab2381262737dcffJOURNAL_STREAM=8:1642042024RUNTIME_DIRECTORY=/run/kamailioCFGFILE=/etc/kamailio/csbc.cfgSHM_MEMORY=512PKG_MEMORY=32SSLKEYLOG_UDP=10.2.1.19:1234LD_PRELOAD=/opt/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so /usr/lib/x86_64-linux-gnu/libssl.so.1.1RUN_KAMAILIO=yesGROUP=kamailioDUMP_CORE=yes
> root@csbc03:~#
>
> root@csbc03:~# fgrep ssl /proc/2216899/maps
> 7f1ceef99000-7f1ceefb6000 r--p 00000000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> 7f1ceefb6000-7f1cef004000 r-xp 0001d000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> 7f1cef004000-7f1cef01e000 r--p 0006b000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> 7f1cef01e000-7f1cef01f000 ---p 00085000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> 7f1cef01f000-7f1cef028000 r--p 00085000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> 7f1cef028000-7f1cef02c000 rw-p 0008e000 08:06 266231                     /usr/lib/x86_64-linux-gnu/libssl.so.1.1
> root@csbc03:~#
>
> This is on a debian 10 box. I have another box for testing on debian12, I set the exact same config as you and I still don't see the keylogger being loaded:
>
> root@csbc01:~# lsb_release -a
> No LSB modules are available.
> Distributor ID: Debian
> Description: Debian GNU/Linux 12 (bookworm)
> Release: 12
> Codename: bookworm
> root@csbc01:~#
>
> root@csbc01:~# cat /etc/default/kamailio.d/voipmonitor
> # ANSIBLE_MANAGED_FILE - Do NOT edit this file as it is auto-generated by Ansible.
> SSLKEYLOG_UDP='127.0.0.1:1234'
> LD_PRELOAD="/usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so /usr/lib/x86_64-linux-gnu/libssl.so.3"
> root@csbc01:~#
>
> root@csbc01:~# file /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
> /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f1a884cad7648cc38a579b1d00a9ad523297b78c, with debug_info, not stripped
> root@csbc01:~#
>
> root@csbc01:~# file /usr/lib/x86_64-linux-gnu/libssl.so.3
> /usr/lib/x86_64-linux-gnu/libssl.so.3: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=dd6b0615fc5d03f9c698d6d0c9d2da1b1e8f2d24, stripped
> root@csbc01:~#
>
> root@csbc01:~# cat /proc/181454/environ
> LANG=en_US.UTF-8PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/binPIDFILE=/run/kamailio/kamailio.pidHOME=/run/kamailioLOGNAME=kamailioUSER=kamailioINVOCATION_ID=059a5e15f1bb4e2bae17c0b5ec9c731eJOURNAL_STREAM=8:2661302RUNTIME_DIRECTORY=/run/kamailioSYSTEMD_EXEC_PID=181394CFGFILE=/etc/kamailio/csbc.cfgSHM_MEMORY=512PKG_MEMORY=32RUN_KAMAILIO=yesGROUP=kamailioDUMP_CORE=yesSSLKEYLOG_UDP=127.0.0.1:1234LD_PRELOAD=/usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so /usr/lib/x86_64-linux-gnu/libssl.so.3
> root@csbc01:~#
>
> root@csbc01:~# fgrep ssl /proc/181454/maps
> 7f0c537b6000-7f0c537d5000 r--p 00000000 08:01 3674382                    /usr/lib/x86_64-linux-gnu/libssl.so.3
> 7f0c537d5000-7f0c53833000 r-xp 0001f000 08:01 3674382                    /usr/lib/x86_64-linux-gnu/libssl.so.3
> 7f0c53833000-7f0c53852000 r--p 0007d000 08:01 3674382                    /usr/lib/x86_64-linux-gnu/libssl.so.3
> 7f0c53852000-7f0c5385c000 r--p 0009c000 08:01 3674382                    /usr/lib/x86_64-linux-gnu/libssl.so.3
> 7f0c5385c000-7f0c53860000 rw-p 000a6000 08:01 3674382                    /usr/lib/x86_64-linux-gnu/libssl.so.3
> root@csbc01:~#
>
> Any other ideas of what I can be missing?
>
> On Tue, Mar 5, 2024 at 2:30 PM Calvin E. <calvine@gmail.com> wrote:
>>
>> Make sure you are preloading the correct OpenSSL library. On my Debian
>> 12 box it is libssl.so.3 not libssl.so.1.1. You can confirm which is
>> loaded by checking the "maps" of a running proc:
>>
>> $ sudo fgrep ssl /proc/2951676/maps
>> 7f26647a4000-7f26647c3000 r--p 00000000 08:01 131274
>>   /usr/lib/x86_64-linux-gnu/libssl.so.3
>> 7f26647c3000-7f2664821000 r-xp 0001f000 08:01 131274
>>   /usr/lib/x86_64-linux-gnu/libssl.so.3
>> 7f2664821000-7f2664840000 r--p 0007d000 08:01 131274
>>   /usr/lib/x86_64-linux-gnu/libssl.so.3
>> 7f2664840000-7f266484a000 r--p 0009c000 08:01 131274
>>   /usr/lib/x86_64-linux-gnu/libssl.so.3
>> 7f266484a000-7f266484e000 rw-p 000a6000 08:01 131274
>>   /usr/lib/x86_64-linux-gnu/libssl.so.3
>> 7f266484e000-7f266484f000 r--p 00000000 08:01 154916
>>   /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>> 7f266484f000-7f2664850000 r-xp 00001000 08:01 154916
>>   /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>> 7f2664850000-7f2664851000 r--p 00002000 08:01 154916
>>   /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>> 7f2664851000-7f2664852000 r--p 00002000 08:01 154916
>>   /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>> 7f2664852000-7f2664853000 rw-p 00003000 08:01 154916
>>   /usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>>
>> My systemd /lib/systemd/system/kamailio.service has a line
>> "EnvironmentFile=-/etc/default/kamailio.d/*" so I dropped a file
>> there:
>>
>> $ cat /etc/default/kamailio.d/voipmonitor
>> SSLKEYLOG_UDP='127.0.0.1:1234'
>> LD_PRELOAD="/usr/local/src/voipmonitor-git/tools/ssl_keylogger/sslkeylog.so
>> /usr/lib/x86_64-linux-gnu/libssl.so.3"
>>
>> In my environment we're using "packetbuffer_sender = yes" to copy all
>> packets to a central processor. I'm sending the keys to localhost so
>> they can get picked up by the sniffer instead of sending them
>> separately to the central processor. For this to work, the sniffer
>> also must capture the "lo" interface.