Eventually I get into producing something I think is useful for others. You'll find random technical knowledge in this Blog so please be patient and use the search function. Some of these software are rather lame and old, the list is ordered by the approximate age (newest first). Unfortunately my older stuff are lost or scattered around my old CDs, not readable anymore 8(.
As you may already know, logrotate
is an excellent utility to keep the size of your log directory under certain control. However in some cases it may be feasible to „undo” the rotated logs into a single monolithic file. Such is a case when you want to migrate your web server log analization from webalizer
to awstats
. Fortunately there is an utility called mergelog
that can this for you for a specific set of files. All I had to do was this small shell wrapper around mergelog
to have a complete log directory (containing many virtual host logfiles from apache) „anti-logrotated” into the current working directory.
#!/bin/bash if [ ! -d "$1" ]; then echo "usage: $0 [directory]" exit -1 fi ls -1 "$1" | sed 's/[.]gz$//g' | sed 's/[.][0-9]*$//g' | sort | uniq | while read logname; do echo "merging files of ${logname}..." zmergelog "$1/${logname}"* > "${logname}" done
This small script realizes the routine task of finding and archiving files older than a given date from a directory structure.
#!/bin/bash if [ ! -d "$2" ]; then echo "usage: $0 <timestamp> <directory>" exit 0 fi ts="$1" dir="$2" tmpdir=`mktemp -d` trap "rm -rf '$tmpdir'" exit touch -d "$ts" "$tmpdir/tsfile" || exit echo "+ finding files older than $ts in $dir" find "$dir" -type f -not -cnewer "$tmpdir/tsfile" > "$tmpdir/filelist" archive="archive-$ts-"`date +%s`".tgz" echo "+ packing contents to $archive" tar -czf "$archive" --files-from "$tmpdir/filelist" && { echo "+ removing files" ; cat "$tmpdir/filelist" | xargs -n1 rm -f ;} echo "+ done"
The following script solves the task of replicating a remote, ldap based, user and group database to a local ldap server. This approach is better than on-line ldap replication in scenarios where the local ldap server is behind a slow and unstable internet connection. The script utilizes extensive error reporting and emergency backups to keep everything in order under unexpected circumstances.
#!/bin/bash # GPLv3 by erno@rigo.info # log file placement LOG=/tmp/ldapmigrate.log # ldap database dir to backup and restore on emergency LDAPDIR=/var/lib/ldap # source ldap server MASTERHOST=ldap.example.com MASTERDN=dc=example,dc=com MASTERBIND=cn=admin,$MASTERDN MASTERPASS=********* # destination ldap server SLAVEHOST=localhost SLAVEDN=$MASTERDN SLAVEBIND=$MASTERBIND SLAVEPASS=$MASTERPASS # organizational units USERSOU=users GROUPSOU=groups ################ # do not touch bellow this line echo "+ $0 "`date "+%Y-%m-%d %H:%M:%S"`" starting..." | tee $LOG echo "+ logfile is at $LOG" | tee -a $LOG MASTERCONN="-h $MASTERHOST -D $MASTERBIND -x -w $MASTERPASS" SLAVECONN="-h $SLAVEHOST -D $SLAVEBIND -x -w $SLAVEPASS" do_delete() { echo "+ removing $1 (class $2) at $SLAVEHOST" | tee -a $LOG tmpfile=`mktemp` ldapsearch $SLAVECONN -LLL -b ou=$1,$MASTERDN "(objectClass=$2)" 'dn' 2>>$LOG | egrep "^dn" | cut -f2 -d':' > $tmpfile || return $? if [ -s $tmpfile ]; then ldapdelete -v $SLAVECONN -f $tmpfile >> $LOG 2>&1 || return $? count=`cat $tmpfile | wc -l` echo "+ successfully removed $count entries" | tee -a $LOG else echo "+ (no entries)" | tee -a $LOG fi rm -f $tmpfile } do_copy() { echo "+ copying $1 (class $2) from $MASTERHOST to $SLAVEHOST" | tee -a $LOG tmpfile=`mktemp` ldapsearch $MASTERCONN -b ou=$1,$MASTERDN "(objectClass=$2)" > $tmpfile 2>>$LOG || return $? if [ -s $tmpfile ]; then ldapmodify $SLAVECONN -a -f $tmpfile >> $LOG 2>&1 || return $? count=`cat $tmpfile | egrep "^dn:" | wc -l` echo "+ successfully copied $count entries" | tee -a $LOG else echo "+ (no entries)" | tee -a $LOG fi rm -f $tmpfile } do_migrate() { do_delete $1 $2 || return $? do_copy $1 $2 || return $? } try_migrate() { ( do_migrate $USERSOU posixAccount && do_migrate $GROUPSOU posixGroup ) || return $? } BACKUPDIR=`mktemp -d` do_backup() { echo "+ stoping slapd" | tee -a $LOG /etc/init.d/slapd stop >> $LOG 2>&1 || return $? echo "+ creating backup in $BACKUPDIR from $LDAPDIR" | tee -a $LOG cp -r $LDAPDIR/* $BACKUPDIR/ >> $LOG 2>&1 || return $? echo "+ starting slapd" | tee -a $LOG /etc/init.d/slapd start >> $LOG 2>&1 || return $? } do_restore() { echo "+ stoping slapd" | tee -a $LOG /etc/init.d/slapd stop >> $LOG 2>&1 || return $? echo "+ restoring backup from $BACKUPDIR to $LDAPDIR" | tee -a $LOG rm -rfv $LDAPDIR/* >> $LOG 2>&1 cp -r $BACKUPDIR/* $LDAPDIR/ >> $LOG 2>&1 || return $? chown -R openldap.openldap $LDAPDIR >> $LOG 2>&1 || return $? echo "+ starting slapd" | tee -a $LOG /etc/init.d/slapd start >> $LOG 2>&1 || return $? } do_cleanup() { echo "+ cleaning up backup directory $BACKUPDIR" | tee -a $LOG rm -rf $BACKUPDIR >> $LOG 2>&1 || return $? } ( do_backup && try_migrate && do_cleanup && echo "+ done and finishing successfully at "`date "+%Y-%m-%d %H:%M:%S"` ) || ( do_restore ; echo "+ FAILED! see: $LOG" ; exit -1 )
The following small script dumps all usable information from a trac project directory, then creates a .zip archive in the actual working directory. The zip will contain all lists from trac, a dump of the wiki structure, and a hot copy of the project dir itself.
#!/bin/bash #env tmpdir=`mktemp -d` tracdir=/var/lib/trac.project.directory/ #dumps trac-admin "$tracdir" hotcopy "${tmpdir}/full_backup" trac-admin "$tracdir" wiki dump "${tmpdir}/wiki_dump" #lists trac-admin "$tracdir" wiki list > $tmpdir/wiki_list.txt trac-admin "$tracdir" permission list > $tmpdir/permission_list.txt trac-admin "$tracdir" ticket_type list > $tmpdir/ticket_type_list.txt trac-admin "$tracdir" priority list > $tmpdir/priority_list.txt trac-admin "$tracdir" severity list > $tmpdir/severity_list.txt trac-admin "$tracdir" component list > $tmpdir/component_list.txt trac-admin "$tracdir" version list > $tmpdir/version_list.txt trac-admin "$tracdir" milestone list > $tmpdir/milestone_list.txt trac-admin "$tracdir" resolution list > $tmpdir/resolution_list.txt #zip'em currdir=`pwd` pushd "$tmpdir" zip -r "${currdir}/trac-"`date +%Y-%m-%d_%H%M`"-backup.zip" . popd rm -rf "$tmpdir"
I was requested to insert outline numbering into a Schematron rule file for debug purposes. Here is the resulting XSLT file:
<?xml version="1.0"?> <!-- Insert outline numbering into Schematron files - Ernő Rigó <erno@rigo.info> --> <xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:sch="http://www.ascc.net/xml/schematron" schemaVersion="1.0" > <xsl:template match="sch:pattern"> <xsl:variable name="counter"> <xsl:value-of select="count(preceding-sibling::sch:pattern) + 1"/> </xsl:variable> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:comment>PatternNUM:<xsl:number value="$counter" format="001" /></xsl:comment> <xsl:apply-templates> <xsl:with-param name="patternnum"> <xsl:number value="$counter" format="001" /> </xsl:with-param> </xsl:apply-templates> </xsl:copy> </xsl:template> <xsl:template match="sch:rule"> <xsl:param name="patternnum" /> <xsl:variable name="counter"> <xsl:value-of select="count(preceding-sibling::sch:rule) + 1"/> </xsl:variable> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:comment>RuleNUM:<xsl:value-of select="$patternnum" />.<xsl:number value="$counter" format="001" /></xsl:comment> <xsl:apply-templates> <xsl:with-param name="patternnum"> <xsl:number value="$patternnum" format="001" /> </xsl:with-param> <xsl:with-param name="rulenum"> <xsl:number value="$counter" format="001" /> </xsl:with-param> </xsl:apply-templates> </xsl:copy> </xsl:template> <xsl:template match="sch:assert"> <xsl:param name="patternnum" /> <xsl:param name="rulenum" /> <xsl:variable name="counter"> <xsl:value-of select="count(preceding-sibling::sch:assert) + 1"/> </xsl:variable> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:variable name="myid"> <xsl:value-of select="$patternnum" />.<xsl:value-of select="$rulenum" />.<xsl:number value="$counter" format="001" /> </xsl:variable> <xsl:comment>AssertNUM:<xsl:value-of select="$myid" /></xsl:comment> <xsl:apply-templates />(AssertNUM:<xsl:value-of select="$myid" />) </xsl:copy> </xsl:template> <xsl:template match="sch:report"> <xsl:param name="patternnum" /> <xsl:param name="rulenum" /> <xsl:variable name="counter"> <xsl:value-of select="count(preceding-sibling::sch:report) + 1"/> </xsl:variable> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:variable name="myid"> <xsl:value-of select="$patternnum" />.<xsl:value-of select="$rulenum" />.<xsl:number value="$counter" format="001" /> </xsl:variable> <xsl:comment>ReportNUM:<xsl:value-of select="$myid" /></xsl:comment> <xsl:apply-templates />(ReportNUM:<xsl:value-of select="$myid" />) </xsl:copy> </xsl:template> <xsl:template match="*"> <xsl:copy> <xsl:copy-of select="@*"/> <xsl:apply-templates/> </xsl:copy> </xsl:template> </xsl:stylesheet>