Bash scripting for RSync logging

The following bash script is used to email a snapshot of the rsync job status upon completion of the job:


#!/bin/sh

RSYNC=/usr/bin/rsync
SSH=/usr/bin/ssh
KEY=/home/rsync/cron/key
RUSER=rsync
RHOST=portal.rhost.com
RPATH=/media/DOC_CONTROL/
LPATH=/rhost/DOC_CONTROL
LOGFILE=/home/rsync/rsync.log

$RSYNC -azhh --stats --log-file=$LOGFILE -e "$SSH -i $KEY" $RUSER@$RHOST:$RPATH $LPATH

EMAIL=user@rhost.com
BODY=/home/rsync/email-body.tmp

if [ $? -eq 0 ]
then
STATUS=SUCCESS
else
STATUS=FAILURE
fi

echo -e "Backup Job Summary:\n" > $BODY
/usr/bin/tail -14 /home/rsync/rsync.log >> $BODY
echo -e "\nFile System Usage\n" >> $BODY
/bin/df -h | head -1 >> $BODY
/bin/df -h | grep startllc >> $BODY
echo -e "\n----END LOG----" >> $BODY

/bin/mail -s "$STATUS : Document Control Backup" "$EMAIL" < $BODY

and the email notification looks like:


From: xxxxx
Sent: xxxxx
To: xxxxx
Subject: SUCCESS : Document Control Backup

Job Summary:

2015/02/26 14:13:28 [7170] Number of files: 3282
2015/02/26 14:13:28 [7170] Number of files transferred: 0
2015/02/26 14:13:28 [7170] Total file size: 5.91G bytes
2015/02/26 14:13:28 [7170] Total transferred file size: 0 bytes
2015/02/26 14:13:28 [7170] Literal data: 0 bytes
2015/02/26 14:13:28 [7170] Matched data: 0 bytes
2015/02/26 14:13:28 [7170] File list size: 98.46K
2015/02/26 14:13:28 [7170] File list generation time: 0.002 seconds
2015/02/26 14:13:28 [7170] File list transfer time: 0.000 seconds
2015/02/26 14:13:28 [7170] Total bytes sent: 352
2015/02/26 14:13:28 [7170] Total bytes received: 99.74K
2015/02/26 14:13:28 [7170] sent 352 bytes received 99.74K bytes 28.60K
bytes/sec
2015/02/26 14:13:28 [7170] total size is 5.91G speedup is 61895.84
2015/02/26 14:13:28 [7168] sent 353 bytes received 102155 bytes total size
6343704911

File System Usage

Filesystem Size Used Avail Use% Mounted on
/dev/sdb1 2.7T 1.4T 1.2T 55% /rmount

----END LOG----

WLF Curve Fitting

A short script to fit a W-L-F model to experimental data. This is flaky, depending on the input guesses and convergence is poor. We simply stop the convergence at the point we have an acceptable fit. This is not robust!


options(scipen = 10)
aT <- structure(list(Temp.F = c(64.4, 86, 104, 122, 149, 176, 203,
230, 257, 284, 311), aT = c(16.7, 1.76, 1, 0.358, 0.00965, 0.000722,
0.000000308, 0.0000000733, 0.00000000357, 0.000000000202, 0.0000000000191
)), .Names = c("Temp.F", "aT"), class = "data.frame", row.names = c(NA,
-11L))
at.nls<-nls(log(aT) ~ ((-C1*(Temp.F-104))/(C2+(Temp.F-104))),
data = aT,
start = list(C1 = 100, C2 = 202),
trace = 1,
# weights = c(10,rep(1,length(aT)-1)),
nls.control(maxiter = 3,
tol = 1e-16,
minFactor = 1/1000000000024,
warnOnly = TRUE))
plot(log(aT$aT)~aT$Temp.F)
lines(predict(at.nls)~aT$Temp.F)
C1 <- summary(at.nls)$coefficients[[1]]
C2 <- summary(at.nls)$coefficients[[2]]
plot((aT$aT)~aT$Temp.F, log = "y", ylim = c(1e-20,100), xlim = c(0,500))
lines(exp(predict(at.nls))~aT$Temp.F)
curve(exp((-C1*(x-104))/(C2+(x-104))), add = TRUE)

Average of multiple curves

This R script is useful for creating average curves of experimental material property data. it simply breaks the multiple curve data into discrete chunks and averages the values within each chunk. The size of the window is customizable. Additionally, it also generates a lowess fit for the resultant dataset, applying additional smoothing if desired.


# R Script to compute an average curve from material test data
options(digits = 5)

# Read raw data (data should be in 2 colums) and sort by x-value.
# Data should be offset to the origin and all but a single 0,0 point should be included)
Data <- read.csv("Z:/Christopher/15/141531/Materials/Planar_23.csv", header=FALSE)
Data.sort <- Data[order(Data$V1),]

# Define the number of samples to generate
points <- 19
samples <- floor((length(Data.sort$V1)-1)/points)

# Initialize the matrix (we assume that the )
Result <- matrix(nrow = points, ncol = 2)
Result[1,] <- c(0,0)

# Loop over the data computing the average of groups of data points
# defined by the width of the sample window
i <- 1
x <- 1
for(x in 2:points){
V1 <- V2 <- 0.0
for(y in 1:samples){
V1 <- V1 + Data.sort[y+i,1]
V2 <- V2 + Data.sort[y+i,2]
}
i<-i+samples
Result[x,] <- c(V1/samples,V2/samples)
}

# Plot the raw data and the averaged curve in red
plot(Data.sort, pch = 1)
lines(Result, col = "red", lwd = 3)

# Create a lowess smoothed curve and plot in blue.
# This is helpful to remove noise in the average especially with a
# large number of sampled points
Smooth <- (lowess(Result, f = .3))
lines(Smooth, col = "blue", lwd = 3)

# Perform some data manipulation and export as tab delimited text files
Smooth<-(do.call(rbind, lapply(Smooth, unlist)))
Smooth[,1] <- c(0,0)
write(t(Result), file = "Planar_23_avg.txt", ncolumns = 2, sep = "\t")
write(Smooth, file = "Planar_23_smth.txt", ncolumns = 2, sep = "\t")

Rplot

A Simple R Program

Not only can R be used for statistical analysis, but I find it extremely useful for other tasks such as performing quick calculations, running simulations, and for general overall work that is suited to any scripting language. As an example of the simplicity of R, a small script is presented that calculates the value of PI by the Archimedian method of polygons. There are only 16 lines of code and 5 of those are used to print out the results to the console.


#------------------------------#
# Pi by the Archimedian Method #
#------------------------------#

# Calculate PI by approximating the circumfrence of a circle
# of unity radius with circumscribed and inscribed polygons.  
# This method provides upper and lower bounds of the value of PI

# Define the radius of the base circle
R <- 1

# Define an initial number of polygon sides
n <- 4

# Define a convergence tolerance
# The analysis will halt when either of the circumscribed or 
# inscribed polygon perimiters is within this tolerance value 
# of PI as determined by the machine.
conv <- 1e-5

# Initialize the residuals
R_i <- 2 * conv
R_c <- 2 * conv

# Begin loop
# The loop continues unitl one of the residuals is less than 
# the convergence tolerance.  Each iteration increases the number 
# of sides of the polygons by one.

while(R_i > conv || R_c > conv){

  # Perimiter of inscribed polygon
  L_i <- n*R*sin(pi/n)

  # Perimiter of circumscribed polygon
  L_c <- n*R*tan(pi/n)

  # Calculate the Residuals
  R_i <- pi - L_i
  R_c <- L_c - pi

  # Increment the number of polygon sides
  n <- n + 1
  }

# Print the results to the console
cat("The number of iterations was:  ", n - 4,"n")
cat("The number of polygon sides is:  ", n,"n")
cat("Upper Bound:  ", L_c,"n")
cat("Lower Bound:  ", L_i,"n")
cat("Maximum Residual:  ", max(R_i,R_c),"n")

# End Code

The number of iterations was:   1014 
The number of polygon sides is:   1018 
Upper Bound:   3.141603 
Lower Bound:   3.141588 
Maximum Residual:   9.992821e-06 

Xorg.conf for dual head fglrx

I’m running two monitors on an ATI HD5770 card and want a single spanned desktop. Unfortunately, the two monitors have different resolutions — the primary HP ZR24W runs at 1900 x 1200 while the secondary Dell 2208WFP runs at 1680 x 1050. This is enough to throw the ATI installer for a loop. Not to mention that the Catalyst Control Center crashes on apply on my Fedora 14 install.

The manual procedure for setting up the system is:

aticonfig --initial=dual-head --screen-layout=right --output=/etc/X11/xorg.conf

This gets me to a true dual-head configuration with two independent desktops. However, since we want a spanned desktop, we need to turn on Xinerama. This is achieved by add ing the line:

Option "Xinerama" "on"

to the Section “ServerLayout” area of the xorg.conf.

The full working xorg.conf is:


Section "ServerLayout"
        Identifier     "aticonfig Layout"
        Screen      0  "aticonfig-Screen[0]-0" 0 0
        Screen         "aticonfig-Screen[0]-1" RightOf "aticonfig-Screen[0]-0"
        Option  "Xinerama"      "on"
EndSection

Section "Module"
EndSection

Section "Monitor"
        Identifier   "aticonfig-Monitor[0]-0"
        Option      "VendorName" "ATI Proprietary Driver"
        Option      "ModelName" "Generic Autodetecting Monitor"
        Option      "DPMS" "true"
EndSection

Section "Monitor"
        Identifier   "aticonfig-Monitor[0]-1"
        Option      "VendorName" "ATI Proprietary Driver"
        Option      "ModelName" "Generic Autodetecting Monitor"
        Option      "DPMS" "true"
EndSection

Section "Device"
        Identifier  "aticonfig-Device[0]-0"
        Driver      "fglrx"
        BusID       "PCI:1:0:0"
EndSection

Section "Device"
        Identifier  "aticonfig-Device[0]-1"
        Driver      "fglrx"
        BusID       "PCI:1:0:0"
        Screen      1
EndSection

Section "Screen"
        Identifier "aticonfig-Screen[0]-0"
        Device     "aticonfig-Device[0]-0"
        Monitor    "aticonfig-Monitor[0]-0"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
        EndSubSection
EndSection

Section "Screen"
        Identifier "aticonfig-Screen[0]-1"
        Device     "aticonfig-Device[0]-1"
        Monitor    "aticonfig-Monitor[0]-1"
        DefaultDepth     24
        SubSection "Display"
                Viewport   0 0
                Depth     24
        EndSubSection
EndSection

Using ‘sed’ to repace text in a file

I have 100 .svg files in a directory that I wanted to crop by one pixel on each side. This was an automatically generated 1 px wide white border, so it was controlled by only a single line in the .svg file specifying a white box from 0,0 to 31,31. The hard way would be to use Inkscape to modify each file individually, but once again ‘sed’ can come to the rescue and modify all 100 files with one command.

The pertinent line that I needed to change was

<rect x=”0″ y=”0″ width=”31″ height=”31″ fill=”#FFFFFF” />
to
<rect x=”1″ y=”1″ width=”29″ height=”29″ fill=”#FFFFFF” />

so we invoke ‘sed’ with


# sed -i 's/x="0" y="0" width="31" height="31"/x="1" y="1" width="29" height="29"/1' *.svg

The ‘-i’ switch tells ‘sed’ to modify the files in place and the ‘s/regexp/replacement/’ defines the string to search for and replace. the ‘/1′ switch on the end tells ‘sed’ to operate only on the 1st instance of the matched expression.

Leave out the ‘-i’ switch and operate on a single file to preview the changes to stdout before committing.

And voila! 100 files fixed in no time!

Mount ISO disk image

I’m partial to FTP installs of all my linux OS’s. This is mainly due to buggy DVD writers causing me to generate a ton of silver coasters and to generally increase my stress levels. So I;ve been downloading ISO’s and mounting them under my FTP site to allow for network installations. This also seems to run quicker than from disk in some instances since my network throughput is greater than my DVD throughput on some of my laptops. This also helps for systems which do not have cd/DVD drives installed and installation is performed using a USB stick.

So without further ado… To mount an ISO run the following command as root:

mount -r -t iso9660 -o loop /source/disk.iso /target/directory/