RIGO.INFO TechBlog

Eventually I get into producing something I think is useful for others. You'll find random technical knowledge in this Blog so please be patient and use the search function. Some of these software are rather lame and old, the list is ordered by the approximate age (newest first). Unfortunately my older stuff are lost or scattered around my old CDs, not readable anymore 8(.

Merging logrotate'd files back again

As you may already know, logrotate is an excellent utility to keep the size of your log directory under certain control. However in some cases it may be feasible to „undo” the rotated logs into a single monolithic file. Such is a case when you want to migrate your web server log analization from webalizer to awstats. Fortunately there is an utility called mergelog that can this for you for a specific set of files. All I had to do was this small shell wrapper around mergelog to have a complete log directory (containing many virtual host logfiles from apache) „anti-logrotated” into the current working directory.

#!/bin/bash
 
if [ ! -d "$1" ]; then
        echo "usage: $0 [directory]"
        exit -1
fi
 
ls -1 "$1" | sed 's/[.]gz$//g' | sed 's/[.][0-9]*$//g' | sort | uniq | while read logname; do
 
        echo "merging files of ${logname}..."
        zmergelog "$1/${logname}"* > "${logname}"
 
done
2009-09-05 19:54 · Erno Rigo · 0 Linkbacks

Archive and remove files older than a specified date

This small script realizes the routine task of finding and archiving files older than a given date from a directory structure.

#!/bin/bash
 
if [ ! -d "$2" ]; then
    echo "usage: $0 <timestamp> <directory>"
    exit 0
fi
 
ts="$1"
dir="$2"
 
tmpdir=`mktemp -d`
trap "rm -rf '$tmpdir'" exit
 
touch -d "$ts" "$tmpdir/tsfile" || exit
 
echo "+ finding files older than $ts in $dir"
find "$dir" -type f -not -cnewer "$tmpdir/tsfile" > "$tmpdir/filelist"
 
archive="archive-$ts-"`date +%s`".tgz"
 
echo "+ packing contents to $archive"
tar -czf "$archive" --files-from "$tmpdir/filelist" && { echo "+ removing files" ; cat "$tmpdir/filelist" | xargs -n1 rm -f ;}
 
echo "+ done"
2009-07-17 12:57 · Erno Rigo · 0 Linkbacks

Periodically mirror remote LDAP user and group database

The following script solves the task of replicating a remote, ldap based, user and group database to a local ldap server. This approach is better than on-line ldap replication in scenarios where the local ldap server is behind a slow and unstable internet connection. The script utilizes extensive error reporting and emergency backups to keep everything in order under unexpected circumstances.

#!/bin/bash
# GPLv3 by erno@rigo.info
 
# log file placement
LOG=/tmp/ldapmigrate.log
 
# ldap database dir to backup and restore on emergency
LDAPDIR=/var/lib/ldap
 
# source ldap server
MASTERHOST=ldap.example.com
MASTERDN=dc=example,dc=com
MASTERBIND=cn=admin,$MASTERDN
MASTERPASS=*********
 
# destination ldap server
SLAVEHOST=localhost
SLAVEDN=$MASTERDN
SLAVEBIND=$MASTERBIND
SLAVEPASS=$MASTERPASS
 
# organizational units
USERSOU=users
GROUPSOU=groups
 
################
# do not touch bellow this line
 
echo "+ $0 "`date "+%Y-%m-%d %H:%M:%S"`" starting..." | tee $LOG
 
echo "+ logfile is at $LOG" | tee -a $LOG
 
MASTERCONN="-h $MASTERHOST -D $MASTERBIND -x -w $MASTERPASS"
SLAVECONN="-h $SLAVEHOST -D $SLAVEBIND -x -w $SLAVEPASS"
 
do_delete() {
    echo "+ removing $1 (class $2) at $SLAVEHOST" | tee -a $LOG
    tmpfile=`mktemp`
    ldapsearch $SLAVECONN -LLL -b  ou=$1,$MASTERDN "(objectClass=$2)" 'dn' 2>>$LOG | egrep "^dn" | cut -f2 -d':' > $tmpfile || return $?
    if [ -s $tmpfile ]; then
        ldapdelete -v $SLAVECONN -f $tmpfile >> $LOG 2>&1 || return $?
        count=`cat $tmpfile | wc -l`
        echo "+ successfully removed $count entries" | tee -a $LOG
    else 
        echo "+ (no entries)" | tee -a $LOG
    fi
    rm -f $tmpfile
}
 
do_copy() {
    echo "+ copying $1 (class $2) from $MASTERHOST to $SLAVEHOST" | tee -a $LOG
    tmpfile=`mktemp`
    ldapsearch $MASTERCONN -b ou=$1,$MASTERDN "(objectClass=$2)" > $tmpfile 2>>$LOG || return $?
    if [ -s $tmpfile ]; then
        ldapmodify $SLAVECONN -a -f $tmpfile >> $LOG 2>&1 || return $?
        count=`cat $tmpfile | egrep "^dn:" | wc -l`
        echo "+ successfully copied $count entries" | tee -a $LOG
    else 
        echo "+ (no entries)" | tee -a $LOG
    fi
    rm -f $tmpfile
}
 
do_migrate() {
    do_delete $1 $2 || return $?
    do_copy $1 $2 || return $?
}
 
try_migrate() {
    ( do_migrate $USERSOU posixAccount && do_migrate $GROUPSOU posixGroup ) || return $?
}
 
BACKUPDIR=`mktemp -d`
do_backup() {
    echo "+ stoping slapd" | tee -a $LOG
    /etc/init.d/slapd stop >> $LOG 2>&1 || return $?
    echo "+ creating backup in $BACKUPDIR from $LDAPDIR" | tee -a $LOG
    cp -r $LDAPDIR/* $BACKUPDIR/ >> $LOG 2>&1 || return $?
    echo "+ starting slapd" | tee -a $LOG
    /etc/init.d/slapd start >> $LOG 2>&1 || return $?
}
 
do_restore() {
    echo "+ stoping slapd" | tee -a $LOG
    /etc/init.d/slapd stop >> $LOG 2>&1 || return $?
    echo "+ restoring backup from $BACKUPDIR to $LDAPDIR" | tee -a $LOG
    rm -rfv $LDAPDIR/* >> $LOG 2>&1
    cp -r $BACKUPDIR/* $LDAPDIR/ >> $LOG 2>&1 || return $?
    chown -R openldap.openldap $LDAPDIR >> $LOG 2>&1 || return $?
    echo "+ starting slapd" | tee -a $LOG
    /etc/init.d/slapd start >> $LOG 2>&1 || return $?
}
 
do_cleanup() {
    echo "+ cleaning up backup directory $BACKUPDIR" | tee -a $LOG
    rm -rf $BACKUPDIR >> $LOG 2>&1 || return $?
}
 
( do_backup && try_migrate && do_cleanup && echo "+ done and finishing successfully at "`date "+%Y-%m-%d %H:%M:%S"` ) || ( do_restore ; echo "+ FAILED! see: $LOG" ; exit -1 )
2009-07-13 10:53 · Erno Rigo · 0 Linkbacks

Archiving trac project environment into a zip file

The following small script dumps all usable information from a trac project directory, then creates a .zip archive in the actual working directory. The zip will contain all lists from trac, a dump of the wiki structure, and a hot copy of the project dir itself.

#!/bin/bash
#env
tmpdir=`mktemp -d`
tracdir=/var/lib/trac.project.directory/
#dumps
trac-admin "$tracdir" hotcopy "${tmpdir}/full_backup"
trac-admin "$tracdir" wiki dump "${tmpdir}/wiki_dump"
#lists
trac-admin "$tracdir" wiki list > $tmpdir/wiki_list.txt
trac-admin "$tracdir" permission list > $tmpdir/permission_list.txt
trac-admin "$tracdir" ticket_type list > $tmpdir/ticket_type_list.txt
trac-admin "$tracdir" priority list > $tmpdir/priority_list.txt
trac-admin "$tracdir" severity list > $tmpdir/severity_list.txt
trac-admin "$tracdir" component list > $tmpdir/component_list.txt
trac-admin "$tracdir" version list > $tmpdir/version_list.txt
trac-admin "$tracdir" milestone list > $tmpdir/milestone_list.txt
trac-admin "$tracdir" resolution list > $tmpdir/resolution_list.txt
#zip'em
currdir=`pwd`
pushd "$tmpdir"
zip -r "${currdir}/trac-"`date +%Y-%m-%d_%H%M`"-backup.zip" .
popd
rm -rf "$tmpdir"
2009-07-06 14:39 · Erno Rigo · 0 Linkbacks

Insert outline numbering into Schematron files

I was requested to insert outline numbering into a Schematron rule file for debug purposes. Here is the resulting XSLT file:

<?xml version="1.0"?>
<!--
  Insert outline numbering into Schematron files - Ernő Rigó <erno@rigo.info>
-->
<xsl:stylesheet version="1.0"
xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
xmlns:sch="http://www.ascc.net/xml/schematron" schemaVersion="1.0"
>
 
<xsl:template match="sch:pattern">
  <xsl:variable name="counter">
    <xsl:value-of select="count(preceding-sibling::sch:pattern) + 1"/>
  </xsl:variable>
  <xsl:copy>
    <xsl:copy-of select="@*"/>
    <xsl:comment>PatternNUM:<xsl:number value="$counter" format="001" /></xsl:comment>
    <xsl:apply-templates>
      <xsl:with-param name="patternnum">
        <xsl:number value="$counter" format="001" />
      </xsl:with-param>
    </xsl:apply-templates>
  </xsl:copy>
</xsl:template>
 
<xsl:template match="sch:rule">
  <xsl:param name="patternnum" />
  <xsl:variable name="counter">
    <xsl:value-of select="count(preceding-sibling::sch:rule) + 1"/>
  </xsl:variable>
  <xsl:copy>
    <xsl:copy-of select="@*"/>
    <xsl:comment>RuleNUM:<xsl:value-of select="$patternnum" />.<xsl:number value="$counter" format="001" /></xsl:comment>
    <xsl:apply-templates>
      <xsl:with-param name="patternnum">
        <xsl:number value="$patternnum" format="001" />
      </xsl:with-param>
      <xsl:with-param name="rulenum">
        <xsl:number value="$counter" format="001" />
      </xsl:with-param>
    </xsl:apply-templates>
  </xsl:copy>
</xsl:template>
 
<xsl:template match="sch:assert">
  <xsl:param name="patternnum" />
  <xsl:param name="rulenum" />
  <xsl:variable name="counter">
    <xsl:value-of select="count(preceding-sibling::sch:assert) + 1"/>
  </xsl:variable>
  <xsl:copy>
    <xsl:copy-of select="@*"/>
    <xsl:variable name="myid">
      <xsl:value-of select="$patternnum" />.<xsl:value-of select="$rulenum" />.<xsl:number value="$counter" format="001" />
    </xsl:variable>
    <xsl:comment>AssertNUM:<xsl:value-of select="$myid" /></xsl:comment>
    <xsl:apply-templates />(AssertNUM:<xsl:value-of select="$myid" />)
  </xsl:copy>
</xsl:template>
 
<xsl:template match="sch:report">
  <xsl:param name="patternnum" />
  <xsl:param name="rulenum" />
  <xsl:variable name="counter">
    <xsl:value-of select="count(preceding-sibling::sch:report) + 1"/>
  </xsl:variable>
  <xsl:copy>
    <xsl:copy-of select="@*"/>
    <xsl:variable name="myid">
      <xsl:value-of select="$patternnum" />.<xsl:value-of select="$rulenum" />.<xsl:number value="$counter" format="001" />
    </xsl:variable>
    <xsl:comment>ReportNUM:<xsl:value-of select="$myid" /></xsl:comment>
    <xsl:apply-templates />(ReportNUM:<xsl:value-of select="$myid" />)
  </xsl:copy>
</xsl:template>
 
<xsl:template match="*">
  <xsl:copy>
    <xsl:copy-of select="@*"/>
    <xsl:apply-templates/>
  </xsl:copy>
</xsl:template>
 
</xsl:stylesheet>
2009-05-18 14:32 · Erno Rigo · 0 Linkbacks

Linkbacks

Use the following URL for manually sending trackbacks: https://rigo.info/lib/plugins/linkback/exe/trackback.php/en:blog
en/blog.txt · Utolsó módosítás: 2009-05-15 00:00 (külső szerkesztés)
CC Attribution-Noncommercial-Share Alike 4.0 International
www.chimeric.de Valid CSS Driven by DokuWiki do yourself a favour and use a real browser - get firefox!! Recent changes RSS feed Valid XHTML 1.0