<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
Hello </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
I have updated Kamailio from 5.2.0 to 5.4.2 but the service is still crashing with sig 11 and sig 6. Please find the attached 'bt full' traces.</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<br>
</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
Sig 11:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<pre style="box-sizing:border-box;font-family:SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size:11.9px;margin-top:0px !important;margin-bottom:0px !important;padding:16px;overflow:auto;line-height:1.45;background-color:var(--color-bg-tertiary);border-radius:6px;color:rgb(36, 41, 46)"><code style="box-sizing:border-box;margin:0px;border-radius:6px;display:inline">Core was generated by `/usr/local/sbin/kamailio -f /usr/local/etc/kamailio_pcscf/kamailio.cfg -P /var/'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00000000006a16fc in timer_list_expire (t=1339518316, h=0x7facee484fd0, slow_l=0x7facee488208, slow_mark=7936) at core/timer.c:846
846 _timer_rm_list(tl); /* detach */
(gdb) bt
#0 0x00000000006a16fc in timer_list_expire (t=1339518316, h=0x7facee484fd0, slow_l=0x7facee488208, slow_mark=7936) at core/timer.c:846
#1 0x00000000006a1c19 in timer_handler () at core/timer.c:922
#2 0x00000000006a20ba in timer_main () at core/timer.c:961
#3 0x000000000042ac0a in main_loop () at main.c:1753
#4 0x000000000043400c in main (argc=13, argv=0x7ffc256fe358) at main.c:2856</code></pre>
Sig 6:</div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<pre style="box-sizing:border-box;font-family:SFMono-Regular, Consolas, "Liberation Mono", Menlo, monospace;font-size:11.9px;margin-top:0px !important;margin-bottom:0px !important;padding:16px;overflow:auto;line-height:1.45;background-color:var(--color-bg-tertiary);border-radius:6px;color:rgb(36, 41, 46)"><code style="box-sizing:border-box;margin:0px;border-radius:6px;display:inline">Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/local/sbin/kamailio -f /usr/local/etc/kamailio_pcscf/kamailio.cfg -P /var/'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007fad95f56c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
(gdb) bt
#0 0x00007fad95f56c37 in raise () from /lib/x86_64-linux-gnu/libc.so.6
#1 0x00007fad95f5a028 in abort () from /lib/x86_64-linux-gnu/libc.so.6
#2 0x000000000041e39d in sig_alarm_abort (signo=14) at main.c:687
#3 <signal handler called>
#4 0x00007fad960183c7 in syscall () from /lib/x86_64-linux-gnu/libc.so.6
#5 0x00007fad6f5c024b in futex_get (lock=0x7facee70baa0) at ../../core/mem/../futexlock.h:121
#6 0x00007fad6f5c2167 in get_peer_by_fqdn (fqdn=0x7faceeb377b0) at peermanager.c:240
#7 0x00007fad6f5e6c31 in get_first_connected_route (cdp_session=0x7faceeb37768, r=0x7facee70a220, app_id=16777236, vendor_id=10415) at routing.c:110
#8 0x00007fad6f5ea8b2 in get_routing_peer (cdp_session=0x7faceeb37768, m=0x7facef617fb0) at routing.c:272
#9 0x00007fad6f5ac72d in AAASendMessage (message=0x7facef617fb0, callback_f=0x0, callback_param=0x0) at diameter_comm.c:137
#10 0x00007fad6ef05dbb in rx_send_str (rx_session_id=0x7facf0849d50) at rx_str.c:157
#11 0x00007fad6eeb25c4 in callback_dialog (dlg=0x7facf0036d70, type=8192, params=0x7fad72b2a1c0 <params>) at ims_qos_mod.c:453
#12 0x00007fad728973af in run_dlg_callbacks (type=8192, dlg=0x7facf0036d70, req=0x0, rpl=0x0, dir=0, dlg_data=0x0) at dlg_cb.c:274
#13 0x00007fad728d5315 in destroy_dlg (dlg=0x7facf0036d70) at dlg_hash.c:219
#14 0x00007fad728d5b1b in destroy_dlg_table () at dlg_hash.c:291
#15 0x00007fad7290b71e in mod_destroy () at ims_dialog.c:627
#16 0x0000000000616ae1 in destroy_modules () at core/sr_module.c:746
#17 0x000000000041ce17 in cleanup (show_status=1) at main.c:563
#18 0x000000000041e6ef in shutdown_children (sig=15, show_status=1) at main.c:706
#19 0x0000000000421755 in handle_sigs () at main.c:806
#20 0x000000000042bc85 in main_loop () at main.c:1817
#21 0x000000000043400c in main (argc=13, argv=0x7ffc256fe358) at main.c:2856</code></pre>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<br>
</div>
PS: I am using ubuntu 14.04. </div>
<div style="font-family: Calibri, Helvetica, sans-serif; font-size: 12pt; color: rgb(38, 114, 236);">
<br>
</div>
<div>
<div id="Signature">
<div>
<div></div>
<p class="ecxMsoNormal">Regards</p>
<p class="ecxMsoNormal"><a name="_MailAutoSig" target="_blank" style="font-size:12pt"><b><span style="font-size:12pt; font-family:Verdana,sans-serif; font-variant:small-caps; color:navy">Hamid R. Hashmi</span></b></a></p>
</div>
</div>
</div>
<div id="appendonsend"></div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Henning Westerholt <hw@skalatan.de><br>
<b>Sent:</b> Saturday, September 26, 2020 12:04 PM<br>
<b>To:</b> Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org><br>
<b>Cc:</b> Hamid Hashmi <hamid2kviii@hotmail.com><br>
<b>Subject:</b> Re: kamailio-5.2.0 Crash - udp_send failed</font>
<div> </div>
</div>
<div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Hello,<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
you are using a old and non released dev version of kamailio. These are meant only for development purposes.<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Please update to a proper version (like the latest one based on 5.3 or 5.4 branch.<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0px; padding:0px; font-family:sans-serif; font-size:11pt; color:black; text-align:left">
Then try to reproduce the bug again and report on the list or by creating an issue.<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Cheers,<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Henning<br>
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
<span id="x_OutlookSignature">
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
<br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
-- <br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Henning Westerholt - <a href="https://skalatan.de/blog/">https://skalatan.de/blog/</a><br>
</div>
<div dir="auto" style="direction:ltr; margin:0; padding:0; font-family:sans-serif; font-size:11pt; color:black">
Kamailio services - <a href="https://skalatan.de/services">https://skalatan.de/services</a></div>
</span><br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> sr-users <sr-users-bounces@lists.kamailio.org> on behalf of Hamid Hashmi <hamid2kviii@hotmail.com><br>
<b>Sent:</b> Friday, September 25, 2020 9:28:07 PM<br>
<b>To:</b> sr-users@lists.kamailio.org <sr-users@lists.kamailio.org><br>
<b>Subject:</b> [SR-Users] kamailio-5.2.0 Crash - udp_send failed</font>
<div> </div>
</div>
<style type="text/css" style="display:none">
<!--
p
{margin-top:0;
margin-bottom:0}
-->
</style>
<div dir="ltr">
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
I am using "<span style="background-color:rgba(27,31,35,0.05); font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace; font-size:11.9px; color:rgb(36,41,46)">kamailio 5.2.0-dev6 (x86_64/linux) 48ed62-dirty</span>" what could be the reason of
the crash<br>
</div>
<div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
its usually crash with the following logs</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<pre style="box-sizing:border-box; font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace; font-size:11.9px; margin-top:0px; margin-bottom:16px; padding:16px; overflow:auto; line-height:1.45; background-color:rgb(246,248,250); border-radius:6px; color:rgb(36,41,46)"><code style="box-sizing:border-box; margin:0px; border-radius:6px; display:inline">Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36eee8c560,418,0,10.100.227.202:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: ERROR: tm [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20771]: WARNING: tm [t_reply.c:1581]: t_retransmit_reply(): send pr buffer failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20756]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8eb948,358,0,10.100.177.91:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20756]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20756]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20744]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8b4a80,358,0,10.100.72.230:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20744]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20744]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20783]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8ac740,416,0,10.100.72.230:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20783]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20783]: ERROR: tm [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20752]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8b5098,356,0,10.100.56.38:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20805]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8e8f18,416,0,10.100.177.91:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20805]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20805]: ERROR: tm [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20752]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20752]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20777]: ERROR: <script>: Re-Transmit: REGISTER (sip:410060269353579@ims.mnc006.mcc410.3gppnetwork.org (10.100.236.57:5060) to sip:410060269353579@ims.mnc006.mcc410.3gppnetwork.org, 1SDq5cuY3iDYFKLATTp)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20761]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8e8a08,354,0,10.100.2.76:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20761]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20761]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20767]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8ad100,360,0,10.100.144.137:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20767]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20767]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:41 pcscf [Proxy-CSCF][20777]: ERROR: <core> [core/udp_server.c:596]: udp_send(): sendto(sock,0x7f36ef8eb860,360,0,10.100.140.226:5060,16): Invalid argument(22)
Sep 25 16:58:41 pcscf [Proxy-CSCF][20777]: CRITICAL: <core> [core/udp_server.c:601]: udp_send(): invalid sendtoparameters#012one possible reason is the server is bound to localhost and#012attempts to send to the net
Sep 25 16:58:41 pcscf [Proxy-CSCF][20777]: ERROR: sl [../../core/forward.h:219]: msg_send_buffer(): udp_send failed
Sep 25 16:58:43 pcscf [Proxy-CSCF][20861]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 77
Sep 25 16:58:44 pcscf [Proxy-CSCF][20853]: ERROR: <core> [core/tcp_read.c:301]: tcp_read_data(): error reading: Connection timed out (110) ([10.100.64.49]:32888 ->
Sep 25 16:58:44 pcscf [Proxy-CSCF][20853]: ERROR: <core> [core/tcp_read.c:302]: tcp_read_data(): -> [10.161.224.228]:5060)
Sep 25 16:58:44 pcscf [Proxy-CSCF][20853]: ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f36a739fe78 r: 0x7f36a739fef8
Sep 25 16:58:44 pcscf [Proxy-CSCF][20742]: ALERT: <core> [main.c:748]: handle_sigs(): child process 20815 exited by a signal 11
Sep 25 16:58:44 pcscf [Proxy-CSCF][20742]: ALERT: <core> [main.c:751]: handle_sigs(): core was generated
Sep 25 16:58:44 pcscf kernel: [52607827.581845] net_ratelimit: 16 callbacks suppressed
Sep 25 16:58:44 pcscf [Proxy-CSCF][20742]: CRITICAL: cdp [diameter_peer.c:423]: diameter_peer_destroy(): destroy_diameter_peer(): Bye Bye from C Diameter Peer test
Sep 25 16:59:01 pcscf CRON[24805]: (root) CMD (/bin/bash /root/volumes/monitoring/src/services_status.sh -p http://10.161.224.25:9091 -s kamailio_pcscf,sems -n pcscf)
Sep 25 16:59:01 pcscf CRON[24804]: (CRON) info (No MTA installed, discarding output)
Sep 25 16:59:44 pcscf [Proxy-CSCF][20742]: CRITICAL: <core> [main.c:654]: sig_alarm_abort(): shutdown timeout triggered, dying...</code></pre>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
but last time its also crashed with the following logs as well</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<pre style="box-sizing:border-box; font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,monospace; font-size:11.9px; margin-top:0px; margin-bottom:0px!important; padding:16px; overflow:auto; line-height:1.45; background-color:rgb(246,248,250); border-radius:6px; color:rgb(36,41,46)"><code style="box-sizing:border-box; margin:0px; border-radius:6px; display:inline">Sep 25 18:39:59 pcscf [Proxy-CSCF][28015]: ERROR: <core> [core/tcp_read.c:301]: tcp_read_data(): error reading: Connection timed out (110) ([10.100.183.34]:39022 ->
Sep 25 18:39:59 pcscf [Proxy-CSCF][28015]: ERROR: <core> [core/tcp_read.c:302]: tcp_read_data(): -> [10.161.224.228]:5060)
Sep 25 18:39:59 pcscf [Proxy-CSCF][28015]: ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f92ca8d0150 r: 0x7f92ca8d01d0
Sep 25 18:39:59 pcscf [Proxy-CSCF][28010]: ERROR: cdp [authstatemachine.c:404]: auth_client_statefull_sm_process(): auth_client_statefull_sm_process(): Received invalid event 6 while in state Open!
Sep 25 18:39:59 pcscf [Proxy-CSCF][28010]: ERROR: cdp [authstatemachine.c:404]: auth_client_statefull_sm_process(): auth_client_statefull_sm_process(): Received invalid event 6 while in state Open!
Sep 25 18:40:00 pcscf [Proxy-CSCF][28034]: CRITICAL: <core> [core/pass_fd.c:277]: receive_fd(): EOF on 77
Sep 25 18:40:01 pcscf [Proxy-CSCF][28026]: ERROR: <core> [core/tcp_read.c:301]: tcp_read_data(): error reading: Connection timed out (110) ([10.100.100.218]:1337 ->
Sep 25 18:40:01 pcscf [Proxy-CSCF][28026]: ERROR: <core> [core/tcp_read.c:302]: tcp_read_data(): -> [10.161.224.228]:5060)
Sep 25 18:40:01 pcscf [Proxy-CSCF][28026]: ERROR: <core> [core/tcp_read.c:1485]: tcp_read_req(): ERROR: tcp_read_req: error reading - c: 0x7f92ca87f070 r: 0x7f92ca87f0f0
Sep 25 18:40:01 pcscf [Proxy-CSCF][27919]: ALERT: <core> [main.c:748]: handle_sigs(): child process 27989 exited by a signal 11
Sep 25 18:40:01 pcscf [Proxy-CSCF][27919]: ALERT: <core> [main.c:751]: handle_sigs(): core was generated
Sep 25 18:40:01 pcscf kernel: [52613904.372973] net_ratelimit: 29 callbacks suppressed
Sep 25 18:40:01 pcscf [Proxy-CSCF][27919]: CRITICAL: cdp [diameter_peer.c:423]: diameter_peer_destroy(): destroy_diameter_peer(): Bye Bye from C Diameter Peer test
Sep 25 18:40:01 pcscf CRON[30581]: (root) CMD (/bin/bash /root/volumes/monitoring/src/services_status.sh -p http://10.161.224.25:9091 -s kamailio_pcscf,sems -n pcscf)
Sep 25 18:40:01 pcscf CRON[30580]: (CRON) info (No MTA installed, discarding output)
Sep 25 18:40:02 pcscf sems[2858]: [#7f2080044700] [update_uac_request, trans_layer.cpp:2080] WARNING: While sending 200 ACK: no matching transaction
Sep 25 18:41:01 pcscf [Proxy-CSCF][27919]: CRITICAL: <core> [main.c:654]: sig_alarm_abort(): shutdown timeout triggered, dying...</code></pre>
<br>
</div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
PS: Kamailio is not listening on localhost. </div>
<div style="font-family:Calibri,Helvetica,sans-serif; font-size:12pt; color:rgb(38,114,236)">
<br>
</div>
<div id="x_x_Signature">
<div>
<div></div>
<p class="x_x_ecxMsoNormal">Regards</p>
<p class="x_x_ecxMsoNormal"><a name="x_x__MailAutoSig" target="_blank" style="font-size:12pt"><b><span style="font-size:12pt; font-family:Verdana,sans-serif; font-variant:small-caps; color:navy">Hamid R. Hashmi</span></b></a></p>
</div>
</div>
</div>
</div>
</div>
</body>
</html>