[kamailio/kamailio] KEMI: Add htable count/match API equivalent to $shtcn() (to avoid pv_cache growth with dynamic PV strings) (Issue #4426)

8 Oct 2025

      wedhornsipgate created an issue (kamailio/kamailio#4426)

**Description:**

We're missing a KEMI function in the htable module that provides the same functionality as the PV function `$shtcn(table=>pattern)` to count the number of keys in an htable that match a given pattern (prefix/suffix/regex). The goal is to let KEMI scripts perform pattern counts without constructing dynamic PV expressions, which currently causes unbounded pv_cache growth in PKG memory.

**Use case / motivation:**

We implemented rolling-window rate limiting by writing short-lived, per-call entries into an htable and then counting keys that match a number-derived pattern. Using $shtcn() works well for the logic, but in KEMI (JS/TypeScript) we must call it via pv.gete("$shtcn(...)"). Because the PV string varies per INVITE (different pattern/key each time), these dynamic PV expressions accumulate in the PV cache, growing PKG/private memory over time.

We’d like a native KEMI function to perform the same count/match so we can avoid the PV layer entirely.

**Environment**

Kamailio: 5.8.6 on Debian 12

Scripting: KEMI (JavaScript/TypeScript)

Symptom: steady PKG/private memory increase over ~24h under load when using pv.gete("$shtcn(...)"). Disabling the $shtcn() call stops the growth.

Related log line: the one discussed in [#3440](https://github.com/kamailio/kamailio/issues/3440) (“pv cache … pkg memory may get filled with pv declarations”).

Current line in our script:

```if (ksr.pv.gete(`$shtcn(${htable_count}=>%~${number})`) > threshold) { ... }```

**Current workaround**

Restarting the Kamailio every day or disabling rate limit on instances with KEMI.

**Idea for API**

Minimal, PV-parity:

```
// KEMI export
int shtcn_match(str table, str pattern, int mode /* 0=exact, 1=prefix, 2=suffix, 3=regex */);
```

JS example:

`const n = KSR.htable.shtcn_match("from_number_count", number, 3); // regex`

More explicit helpers (avoids integer modes):

```
KSR.htable.shtcn_exact(table, key)      -> int
KSR.htable.shtcn_prefix(table, prefix)  -> int
KSR.htable.shtcn_suffix(table, suffix)  -> int
KSR.htable.shtcn_regex(table, pattern)  -> int
```

**Behavioral notes / expectations**

Should mirror $shtcn() semantics where reasonable (esp. regex behavior), but returning an integer directly to KEMI.

Must not allocate PV expressions or interact with pv_cache.

**Why this belongs in htable KEMI**

$shtcn() is already offered at the PV level; parity in KEMI keeps feature consistency across scripting engines.

It enables memory-safe implementations of rolling-window rate limits and similar patterns (without touching PV).

**Alternatives considered**

Replace pattern counting with per-bucket counters (sht_inc), which leads to other problems like increment/decrement handling and locking. 

Use ratelimit/pike modules—useful for IP/method limits, but our use case is PAI/From/To based and currently hinges on pattern counts.

**Acceptance criteria**

A KEMI function exists to count matching keys in an htable without PV parsing.

Verified that repeated calls with dynamic patterns do not grow PV cache / PKG memory.

Basic tests for exact/prefix/suffix/regex modes (and skip_expired if implemented).

Thank you!

This would let KEMI users keep $shtcn()-style logic with the same performance characteristics, but without the PV-cache side effects.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/kamailio/kamailio/issues/4426
You are receiving this because you are subscribed to this thread.

Message ID: <kamailio/kamailio/issues/4426@github.com>

Hendrik Wedhorn

Xenofon Karamanos

Jonas

Daniel-Constantin Mierla

Daniel-Constantin Mierla

Sebastian Damm

tags

participants (5)