Hello,
kamailio tools (kamctl and kamdbctl) have not been imported yet. The usual place for them was 'scripts' folder but now there are overlapping names, related to the folders holding sql scripts.
Shall we create a subfolder in scripts directory, e.g., scripts/kamctl?
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent of DB api but of what DB structure to be used (some modules using db were moved in common folder already)?
Cheers, Daniel
On 27-05 10:27, Daniel-Constantin Mierla wrote:
Hello,
kamailio tools (kamctl and kamdbctl) have not been imported yet. The usual place for them was 'scripts' folder but now there are overlapping names, related to the folders holding sql scripts.
Shall we create a subfolder in scripts directory, e.g., scripts/kamctl?
What about importing them into the tools directly?
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent of DB api but of what DB structure to be used (some modules using db were moved in common folder already)?
I would suggest that we keep them there for now. The scripts that are used to generate db schema need, in my opinion, an overhaul, so we might just do it in one go.
By the way, what else is missing in the git repository and should be imported from the SVN repository? Could you (or somebody who is familiar with kamailio svn repository) make a list of stuff that should be imported into git, along with the destination directory where you want to have them?
I could then import them with full history all at once.
Jan.
On Mittwoch, 27. Mai 2009, Jan Janak wrote:
kamailio tools (kamctl and kamdbctl) have not been imported yet. The usual place for them was 'scripts' folder but now there are overlapping names, related to the folders holding sql scripts.
Shall we create a subfolder in scripts directory, e.g., scripts/kamctl?
Hi Jan,
What about importing them into the tools directly?
do you mean the tools directory? Hm, not sure what then the purpose of the scripts directory would be, perhaps this could be then removed?
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent of DB api but of what DB structure to be used (some modules using db were moved in common folder already)?
I would suggest that we keep them there for now. The scripts that are used to generate db schema need, in my opinion, an overhaul, so we might just do it in one go.
I refactored the k XSL scripts (which were imported from SER) in the last year somewhat and also improved/ added support for DBs!=mysql. They do IMHO a good job now to generate the needed database scripts from XML sources, so i don't see a big need to rework them. What i removed was the possibility to also create insert statements, i think, not sure if this is needed from SER.
By the way, what else is missing in the git repository and should be imported from the SVN repository? Could you (or somebody who is familiar with kamailio svn repository) make a list of stuff that should be imported into git, along with the destination directory where you want to have them?
I could then import them with full history all at once.
The utils directory content is also missing.
Henning
Hello,
On 05/27/2009 04:04 PM, Henning Westerholt wrote:
On Mittwoch, 27. Mai 2009, Jan Janak wrote:
kamailio tools (kamctl and kamdbctl) have not been imported yet. The usual place for them was 'scripts' folder but now there are
overlapping
names, related to the folders holding sql scripts.
Shall we create a subfolder in scripts directory, e.g.,
scripts/kamctl?
Hi Jan,
What about importing them into the tools directly?
do you mean the tools directory? Hm, not sure what then the purpose of the scripts directory would be, perhaps this could be then removed?
tools looks like a good place, imo -- scripts can be removed completely or kept for small scripts used in migration/mass update.
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent
of DB
api but of what DB structure to be used (some modules using db were moved in common folder already)?
I would suggest that we keep them there for now. The scripts that
are used
to generate db schema need, in my opinion, an overhaul, so we might
just do
it in one go.
I refactored the k XSL scripts (which were imported from SER) in the last year somewhat and also improved/ added support for DBs!=mysql. They do IMHO a good job now to generate the needed database scripts from XML sources, so i don't see a big need to rework them. What i removed was the possibility to also create insert statements, i think, not sure if this is needed from SER.
By the way, what else is missing in the git repository and should be imported from the SVN repository? Could you (or somebody who is familiar with kamailio svn repository) make a list of stuff that should be
imported
into git, along with the destination directory where you want to
have them?
I could then import them with full history all at once.
The utils directory content is also missing.
I think this one needs a bit of cleanup and moved under tools folder as well. Perhaps kamunix can be moved within tools/kamctl, being used by that one.
Henning, any idea what db_berkeley and db_oracle in utils/ offer? I never used them... maybe we can give better names, to reflect what they do, instead of db module name.
fifo_relay is in ser as well, not sure if they suffered changes meanwhile. Anyhow, imo, tools, utils, scripts can be merged in one, say tools.
Default kamailio config should be imported, has good example of using presence, nat, auth with K modules. The examples directory has to be checked, I guess the files there are more or less what is now in sr, but with kamailio syntax.
doc/dbschema and doc/database - Henning, here is your call.
So, what I can summarize as being with higher priority: - scripts/ imported in tools/kamctl - etc/kamailio.cfg imported in etc/ - utils/kamunix in tools/kamctl/kamunix
Cheers, Daniel
On Mittwoch, 27. Mai 2009, Daniel-Constantin Mierla wrote:
[..]
Hi Daniel,
I could then import them with full history all at once.
The utils directory content is also missing.
I think this one needs a bit of cleanup and moved under tools folder as well. Perhaps kamunix can be moved within tools/kamctl, being used by that one.
Henning, any idea what db_berkeley and db_oracle in utils/ offer? I never used them... maybe we can give better names, to reflect what they do, instead of db module name.
In db_berkeley is a recovery tool for crash recovery or something like this, should be used from the admin. The db_oracle contains a basic command line client as replacement for SQLplus, i think, this is used from the oracle (db)ctl scripts.
doc/dbschema and doc/database - Henning, here is your call.
doc/database is an empty directory that is only used for the generated database documentation, thus there is nothing to import. The doc/dbschema contains the XSL schemes that are used to generate the SQL/ database sources, the table docs and this experimental code generation stuff that is used from a few modules.
Cheers, Henning
Henning,
On 27-05 15:04, Henning Westerholt wrote:
On Mittwoch, 27. Mai 2009, Jan Janak wrote:
kamailio tools (kamctl and kamdbctl) have not been imported yet. The usual place for them was 'scripts' folder but now there are
overlapping
names, related to the folders holding sql scripts.
Shall we create a subfolder in scripts directory, e.g.,
scripts/kamctl?
Hi Jan,
What about importing them into the tools directly?
do you mean the tools directory? Hm, not sure what then the purpose of the scripts directory would be, perhaps this could be then removed?
Currently the script directory contains all the database related script, and some stuff that is not being used much (or at all), i.e. filter_log.sh, serstats, kam_to_sr.sh.
I think we can use the scripts directory to store supporting scripts that people typically run once or occasionally.
The tools directory could than be used for all other admin related tools, such as sercmd, gen_ha1, kamdbctl.
We can also get rid of the utils directory.
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent of
DB
api but of what DB structure to be used (some modules using db were moved in common folder already)?
I would suggest that we keep them there for now. The scripts that are
used
to generate db schema need, in my opinion, an overhaul, so we might just
do
it in one go.
I refactored the k XSL scripts (which were imported from SER) in the last year somewhat and also improved/ added support for DBs!=mysql. They do IMHO a good job now to generate the needed database scripts from XML sources, so i don't see a big need to rework them. What i removed was the possibility to also create insert statements, i think, not sure if this is needed from SER.
I think that xsltproc (or XSLT) is the wrong tool for the job. If you take a look at the conversion scripts, they use lots of variables, conditional statements, recursions, and functions specified in the XPath specification. In other words, we use xlst as a real programming language, but unlike many other languages available today, it has horrible syntax. Heck, even Perl scripts are more readable than XSLT.
XSLT is probably OK if you want to generate something that looks like XML out of a XML document, but if you try to build an output with a less regular syntax, like the SQL scripts or dbtext scripts we generate, you are stuck with all those extension functions from XPath, here are prime examples of its cumbersome syntax:
<xsl:value-of select="normalize-space($select/name[@db=$db])"/> <xsl:value-of select="translate(normalize-space($select/type[@db=$db]), 'ABCDEFGHIJKLMNOPQRSTUVWXYZ', 'abcdefghijklmnopqrstuvwxyz')"/>
The first statement is an equivalent of the function trim which removes leading and trailing whitespace and the second statement converts a string to lower case.
Another major problem is that the scripts are not portable. They work with xsltproc but would not work with other xsl processors due to subtle differences in implementation. I couldn't even generate a consistent output across different versions of xsltproc.
If I had to write such scripts again, I would have chosen another language, probably Python or Perl. They are both wide-spread, they have decent XML parsers and less terryfing syntax (well, in case of perl this is questionable :-).
With python or perl we could do much more than just generate plain-text output, we could create the tables directly in the database, setup usernames and passwords, handle backups and upgrades conveniently in one tool, and so on.
While writing those scripts in XSLT was a good experience, I think we are using the language outside its domain. But at least we learned it, which is good if one needs to modify the docbook xslt scripts, for example.
Jan.
On Freitag, 29. Mai 2009, Jan Janak wrote:
[..]
Hi Jan,
sorry for the late reply, i was on vacation.
What about importing them into the tools directly?
do you mean the tools directory? Hm, not sure what then the purpose of the scripts directory would be, perhaps this could be then removed?
Currently the script directory contains all the database related script, and some stuff that is not being used much (or at all), i.e. filter_log.sh, serstats, kam_to_sr.sh.
I think we can use the scripts directory to store supporting scripts that people typically run once or occasionally.
The tools directory could than be used for all other admin related tools, such as sercmd, gen_ha1, kamdbctl.
We can also get rid of the utils directory.
Ok, sounds good.
The xml definition of db tables were in db directory, which now is library srdb1(same for ser, are now in srdb2). Do we let them there or is another more suitable place for them as they are not dependent of
DB
api but of what DB structure to be used (some modules using db were moved in common folder already)?
I would suggest that we keep them there for now. The scripts that are
used
to generate db schema need, in my opinion, an overhaul, so we might just
do
it in one go.
I refactored the k XSL scripts (which were imported from SER) in the last year somewhat and also improved/ added support for DBs!=mysql. They do IMHO a good job now to generate the needed database scripts from XML sources, so i don't see a big need to rework them. What i removed was the possibility to also create insert statements, i think, not sure if this is needed from SER.
I think that xsltproc (or XSLT) is the wrong tool for the job. If you take a look at the conversion scripts, they use lots of variables, conditional statements, recursions, and functions specified in the XPath specification. In other words, we use xlst as a real programming language, but unlike many other languages available today, it has horrible syntax. Heck, even Perl scripts are more readable than XSLT.
Hm, i agree that the xpath stuff is really a bit ugly. :-)
[..] Another major problem is that the scripts are not portable. They work with xsltproc but would not work with other xsl processors due to subtle differences in implementation. I couldn't even generate a consistent output across different versions of xsltproc.
Only used xsltproc so far, but i also got the impression that portability accross different parsers is sometimes a problem.
If I had to write such scripts again, I would have chosen another language, probably Python or Perl. They are both wide-spread, they have decent XML parsers and less terryfing syntax (well, in case of perl this is questionable
:-).
:-)
With python or perl we could do much more than just generate plain-text output, we could create the tables directly in the database, setup usernames and passwords, handle backups and upgrades conveniently in one tool, and so on.
While writing those scripts in XSLT was a good experience, I think we are using the language outside its domain. But at least we learned it, which is good if one needs to modify the docbook xslt scripts, for example.
I don't would write this stuff against in XSTL either, learning all this was a interesting experience.. ;-) I only wanted to say that i don't favor a complete rewrite of this stuff _with the same functionality_ just because it turned out to be not the perfect choice for the job. The existing implementation works so far ok, and IMHO there are other, more important problems right now. If we get more functionality the way i have nothing against it at all.
Henning
-
Daniel-Constantin Mierla -
Henning Westerholt -
Jan Janak -
jh＠tutpro.com