How-to make cross-domain URL call work in javascript

I was quite busy in the past 2014. I only wrote one blog post in the whole year. Busy is a good excuse for lazy :). I hope I can post more in 2015 to help others and help me as well.

In my first post this year, I’ll share my experiences with cross domain URL call in JavaScript.

Case 1. Normal URL call with JQuery

url = "https://alexzeng.wordpress.com/api/report/";    
$.get(url, function(data) {
    //process data;
    process(data);
});

Case 2. Cross-domain URL call with JQuery
Just add a parameter “callback” with value “?” in your URL, JQuery will handle everything else.

url = "http://alezeng.blogspot.com/api/report/";
$.getJSON(url + "?callback=?", function (result) {
    // process data
    process(data);
}); 

Note: if your url already have some prarameters, use “&” to connect callback parameter like “&callback=?”

There’re some pre-requirements on the API at server side:
1. The API should return the data with “Content-Type:text/javascript”. Otherwise this error’ll show in javascript call:

Resource interpreted as Script but transferred with MIME type text/html

You can check the “Content-Type” in Response Headers by manually call the url in Chrome debug mode.

2. The API should handle the callback parameter, and wrapper the data in callback function. Otherwise you’ll get error because it cannot run as a javascript function
Many RESTful API using modern framework can support by default, for example Django, Spring MVC. But if you use an old framework or home-grown API, or even just a statistic text, it won’t work by default.
However, if you understand how it works, you can mimic what it does and make it work for any URL. Let’s practice it in Case 3.

Case 3.  Manually call Cross-domain URL
Scenario:
I have data at http://alezeng.blogspot.com/api/report.dat.
I want to use the data it in my another site https://alexzeng.wordpress.com/
Step 1. Wrapper data in a callback function
http://alezeng.blogspot.com/api/report.dat, content:

my_callback({
   'key1' : 'value1',
   'key2' : 'value2'
})

Note: you can enclose any value in my_callback as long as it’s a valid javascript parameter.

Step 2. Make your server response the URL as javascript.
In my case, apache is my web server. It’ll return the data as “Content-Type:text/html” by default.
Add below line in apache config file to let it response “.dat” as javascript:

AddType text/javascript .dat 

Note: if the URL is a RESTful API, wrote by programming language, like java or python, you can set the response content type in code directly.

Step 3. Write the callback function
In another site https://alexzeng.wordpress.com/, write javascript codes to handle the data:

function my_callback(data)
{
    //process data
    process(data);
}

function cross_domain_call() {
    url = 'http://alezeng.blogspot.com/api/report.dat'    
    var script = document.createElement('script');
    script.src = url +'?callback=my_callback'
    document.getElementsByTagName('head')[0].appendChild(script);
}

With this, we can successfully get the data from http://alezeng.blogspot.com in https://alexzeng.wordpress.com.
But it’s kind of troublesome if you have any URL that need to change at server side. The other way is, we can use our server to call cross-domain URL on behalf of client.

Case 4. Use server to call Cross-domain URL
Step 1. Write a generic URL call function in server side.
Let’s use Django server as example:
In views.py

from django.views.decorators.gzip import gzip_page

#gzip the response content, this's not a must
@gzip_page
def get_crossdomain_data(request):
    try:
        url = request.GET['url']
        url = urllib.unquote(url).decode('utf8')
        jsonData = urllib2.urlopen(url).readlines()
        return HttpResponse(jsonData)
    except Exception,e:
        result={};
        result['success']= 'false'
        result['message']= str(e)
        return HttpResponse(result)

In urls.py, map the URL

(r'^get_crossdomain_data',views.get_crossdomain_data),

Step 2. Use the server URL to call Cross-domain URL
We need to encode the original URL and transfer it as an parameter to server, and then parse the response data

url = "http://alezeng.blogspot.com/api/report.dat";
url= "/get_crossdomain_data?url=" + encodeURIComponent(url);
$.get(url, function(str_data) {
    data = jQuery.parseJSON(str_data);
    //process data;
    process(data);
});

With all these approaches, you can choose the one works in your scenario.

Advertisements

How to create executable jar file?

Method 1: I used to create a executable jar file by eclipse export “Runnable JAR file”

It’s very handy, and it has 3 options to handle libraries:

  • Extract required libraries into generated JAR
  • Package required libraries into generated JAR
  • Copy required libraries into sub folder next to the generated JAR.

I really like it. You don’t need to configure anything but just a few clicks. I think it’s the best way if it’s just for test.It works well except when I use the 1st option. I run into this problem :

org.xml.sax.SAXParseException: schema_reference.4: Failed to read schema document 'http://www.springframework.org/schema/beans/spring-beans-4.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.

This is because the xsd file is missed/overwroten when eclipse extract the libs, and my runtime host cannot connect to Internet that makes it impossible to get it. While this is  not a real problem for eclipse, it can be overcome by using the 2nd option. It didn’t extract the lib, but includes them as they’re.

I wonder why eclipse takes a lot of efforts to extract libs (it’s obviously slower than other 2 options in the list). At first, I thought it may want to build small jar packages(it’s obviously smaller ). Later, I found out the real reason is: java can NOT load classes from a Jar inside a Jar! (In real world, how can you put an even larger jar inside a jar? You are kidding!) Java(class loader) is designed to like this.

Now the question is, how eclipse overcome this jar inside jar problem? I unzipped the generated jar file, and found there’s a jarinjarloader class which did a trick behind. See more below.

Method 2: Using onejar-maven-plugin
Because the network between my laptop and the target machine is slow. It takes times to transfer the runnable jar file, even it’s not big. So I need to build it remotely without eclipse, use maven instead. This plugin did the similar trick jarinjarloder did, the 2nd option in eclipse. It basically let java call its main function, and it help you load all dependency jar files including yourselves. Here is the configuration clip:

		  	<plugin>
		  	    <groupId>com.jolira</groupId>
		  	    <artifactId>onejar-maven-plugin</artifactId>
		  	    <version>1.4.4</version>
		  	    <executions>
		  	        <execution>
		  	            <configuration>
		  	                <attachToBuild>true</attachToBuild>
		  	                <classifier>onejar</classifier>
		  	            </configuration>
		  	            <goals>
		  	                <goal>one-jar</goal>
		  	            </goals>
		  	        </execution>
		  	    </executions>
		  	</plugin>

The associated plugins may also needed:

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>2.3.2</version>
				<configuration>
					<source>1.6</source>
					<target>1.6</target>
					<encoding>UTF-8</encoding>
				</configuration>
			</plugin>
		   	<plugin>
		  	    <groupId>org.apache.maven.plugins</groupId>
		  	    <artifactId>maven-jar-plugin</artifactId>
		  	    <configuration>
		  	        <archive>
		  	            <manifest>
		  	                <mainClass>com.wordpress.alexzeng.automation</mainClass>
		  	            </manifest>
		  	        </archive>
		  	    </configuration>
		  	</plugin>

It works well as long as you get the dependency configured well in pom.xml. I missed an ojdbc6.jar at first.

Method 3: Using maven-dependency-plugin, corresponding to the 3rd option in eclipse.
I tried this in my project, but failed somehow. I have 23 jar files in final lib directory. But somehow it only includes 19 jar files in the Class-Path of META-INF/MANIFEST.MF. It refused to work even after I added the jar files to classpath variable in environment. I think this method should work, and this is the best one for big projects (you don’t want to have a very big single jar file). You can reference http://www.mkyong.com/maven/how-to-create-a-jar-file-with-maven/ for a simple case.

Last but not least, their’s also a corresponding method in maven to match the eclipse 1st option, use maven-assembly-plugin pre-defined descriptor jar-with-dependencies. It’s really simple and handy if extract dependency jars didn’t cause troubles. 

		    <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-assembly-plugin</artifactId>
                      <version>2.4</version>
		      <configuration>
		          <descriptorRefs>
		             <descriptorRef>jar-with-dependencies</descriptorRef>
		          </descriptorRefs>
		        <archive>
		          <manifest>
		            <mainClass>com.wordpress.alexzeng.automation</mainClass>
		          </manifest>
		        </archive>
		      </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
		    </plugin>

As a newbie, I spent a lot of time to get it clear. I hope it can save you sometime to if you need to do the same.

how to setup HTTPS for apache

I’m working on setup authentication for an internal website recently.To protect usernames and passwords, setting up HTTPS is very necessary. Here is my first trial on HTTPS/SSL:

Step 1. Check if apache has ssl module compiled 

$ grep mod_ssl apache/conf/httpd.conf
If below line is there, and you can start apache without problem, you don't need to recompile apache
"LoadModule ssl_module modules/mod_ssl.so"

The other way is to check whether the module is there.
If it's there, but it's not in the httpd.conf file, you can add it and try start apache
$ ls modules/mod_ssl.so

Step 2. Compile apache if it didn’t have ssl module

--download apache from http://httpd.apache.org/, and unzip it
cd /home/alexzeng/httpd-2.2.24
--config
#./configure --prefix=/alexzeng/apache --with-config-file-path=/alexzeng/apache/conf --enable-ssl --enable-http --enable-rewrite --enable-track-vars --enable-cgi --with-config-file-path=/opt/apache/conf --enable-modules=all --enable-mods-shared=all --enable-file-cache --enable-disk-cache --enable-cache --enable-mem-cache --enable-dumpio --enable-logio --enable-mime-magic --enable-headers --enable-usertrack --enable-version --enable-proxy --enable-proxy-connect --enable-proxy-http --enable-proxy-ftp --enable-proxy-ajp --enable-proxy-balancer --enable-so

--make
#make

--make install
#make install

Step 3. Config apache SSL
A. Load ssl module, it should already have these lines:

LoadModule ssl_module modules/mod_ssl.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule dir_module modules/mod_dir.so
...

B. Enable httpd-ssl.conf, remove # from httpd-ssl.conf line:

# Secure (SSL/TLS) connections
Include conf/extra/httpd-ssl.conf

C. Config pages that need force using https at VirtualHost part:

NameVirtualHost *:80

   ServerName alexzeng.vip.wordpress.com
   DocumentRoot "/alexzeng/apache/htdocs"
   Redirect permanent /signin https://alexzeng.vip.wordpress.com/signin

D. Config extra/httpd-ssl.conf

Listen 443

#   General setup for the virtual host
DocumentRoot "/alexzeng/apache/htdocs"
ServerName alexzeng.vip.wordpress.com:443
...
#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   Server Certificate:
SSLCertificateFile "/alexzeng/apache/conf/server.crt"

#   Server Private Key:
SSLCertificateKeyFile "/alexzeng/apache/conf/server.key"
#   Server Certificate Chain:

Until now, the HTTPS is setup for the site. But we need to have 2 files: server cert server.crt, and its private key server.key.

The official way it to request it from public SSL certificate companies, such as VeriSign, because their cert is accepted by all browser by default. But it costs a few hundred dollar per year at least.

So many companies have their self-signed cert to reduce the cost. Especially for internal sites, a company can have their root cert installed for their employees OS image by default. In that way, access its signed certs site will be recognized as safe, no security alert, nor https crossed out in red. That’s the case in my company.

I got our IT team signed cert, but it’s in pfx format. I need to convert it to the 2 files server.crt and server.key. The processes are as follows:

$ openssl pkcs12 -in it.pfx  -nocerts -nodes -passin pass:"<password_from_IT>" | openssl rsa -out server.key
MAC verified OK
writing RSA key
$ openssl pkcs12 -in it.pfx  -clcerts -nokeys -nodes -passin pass:"<password_from_IT>" | openssl x509 -out server.crt
MAC verified OK
-- The pfx is created by Windows tools, so I use openssl rsa/x509 to remove some "Bag Attributes" lines.
-- Otherwise, you can just use -out option at the first without the sencond command in pipeline

--copy key to apache directory if needed
$ cp server.crt  server.key /alexzeng/apache/conf

--restart apache
$ sudo ./apachectl stop
$ sudo ./apachectl start

If a browser didn’t installed the companies’ root cert, it’ll get security alert when access the site. It can be avoided by import their root cert.

Besides that, we need to test the HTTPS by ourselves even without an internal team to sign the cert for us. We’ll make ourselves a certificate authority (CA) 🙂

How-to create self-signed cert for test:
a. Create a server private key: server.key

$ openssl genrsa -out server.key 1024
Generating RSA private key, 1024 bit long modulus
...................++++++
......++++++
e is 65537 (0x10001)

b. Create certificate signing request (CSR) : server.csr (using server.key)

$ openssl req -new -out server.csr -key server.key
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:aa
Locality Name (eg, city) [Default City]:aa
Organization Name (eg, company) [Default Company Ltd]:aa
Organizational Unit Name (eg, section) []:aa
Common Name (eg, your name or your server's hostname) []:alexzeng.vip.wordpress.com
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
$

--Note:
"Country Name" must be a valid name
"Common Name" must be the same as ServerName in httpd.conf
"State or Province Name" must has some value

c. Create a certificate authority (CA) private key: ca.key

$ openssl genrsa  -out ca.key 1024
Generating RSA private key, 1024 bit long modulus
.......++++++
........++++++
e is 65537 (0x10001)

d. Create CA certificate : ca.crt (using ca.key)

$ openssl req  -new -x509 -days 365 -key ca.key -out ca.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:aa
Locality Name (eg, city) [Default City]:aa
Organization Name (eg, company) [Default Company Ltd]:aa
Organizational Unit Name (eg, section) []:aa
Common Name (eg, your name or your server's hostname) []:alexzeng.vip.wordpress.com
Email Address []:
$

e. Sign a certificate by my own CA: server.crt (using certificate signing request server.csr, CA private key ca.key, and CA certificate ca.crt)

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Dec 26 07:20:02 2013 GMT
            Not After : Dec 26 07:20:02 2014 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = aa
            organizationName          = aa
            organizationalUnitName    = aa
            commonName                = alexzeng.vip.wordpress.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                70:45:AB:98:23:51:BB:88:23:20:EA:21:21:3C:6A:8A:E2:0A:97:B8
            X509v3 Authority Key Identifier:
                keyid:46:73:F6:1F:85:74:10:D6:B4:5B:AB:B6:2E:1C:5D:A8:97:08:55:4C
Certificate is to be certified until Dec 26 07:20:02 2014 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ ls -lt
total 140
-rw-r--r-- 1 root  root   3048 Dec 26 00:20 server.crt
-rw-rw-r-- 1 dbbox dbbox   948 Dec 26 00:19 ca.crt
-rw-rw-r-- 1 dbbox dbbox   887 Dec 26 00:18 ca.key
-rw-rw-r-- 1 dbbox dbbox   643 Dec 26 00:18 server.csr
-rw-rw-r-- 1 dbbox dbbox   887 Dec 26 00:10 server.key

Usage of these files:

2 files are used in httpd-ssl.conf:
server.key (Server private key) -> this one will keep at server side
server.crt (Server certificate) -> this one will send to client when users access HTTPS

The other 3 files owned by CA (Certificate Authority) are used only during sign processes:
server.csr (certificate signing request)
ca.key (certificate authority private key)
ca.crt (certificate authority certificate)

Problems I got during these processes:
1. index.txt and serial file are missing when sign the cert

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/index.txt: No such file or directory
unable to open '/etc/pki/CA/index.txt'
140528819345224:error:02001002:system library:fopen:No such file or directory:bss_file.c:355:fopen('/etc/pki/CA/index.txt','r')
140528819345224:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:357:

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/serial: No such file or directory
error while loading serial number
140414051186504:error:02001002:system library:fopen:No such file or directory:bss_file.c:355:fopen('/etc/pki/CA/serial','r')
140414051186504:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:357:

--create them to avoid the issue
$ sudo su -
# touch /etc/pki/CA/index.txt
# echo 01 > /etc/pki/CA/serial

2. openssl permission issue

$ openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
I am unable to access the /etc/pki/CA/newcerts directory
/etc/pki/CA/newcerts: Permission denied

--solution
My openssl is installed as root, so run the command with sudo
If openssl is created by the login account itself, you don't need to sudo, and the configure file of openssl will located at your installation place.

3. Error “libtool: install: invalid libtool wrapper script xxx” when “make install” apache.

--error message
libtool: install: invalid libtool wrapper script `htpasswd'
libtool: install: invalid libtool wrapper script `htdigest'
libtool: install: invalid libtool wrapper script `rotatelogs'
libtool: install: invalid libtool wrapper script `logresolve'
libtool: install: invalid libtool wrapper script `ab'
libtool: install: invalid libtool wrapper script `htdbm'
libtool: install: invalid libtool wrapper script `htcacheclean'
libtool: install: invalid libtool wrapper script `httxt2dbm'
libtool: install: invalid libtool wrapper script `checkgid'
make[2]: *** [program-install] Error 1

--It's caused by libtool is not installed at this host.
--Here is fix steps;
$ rpm -qa | grep libtool
$ sudo yum search libtool
$ sudo yum install libtool

--remove all installed files, and redo make, and make install
$ rm -rf /alexzeng/apache/
$ make
$ make install

This is a basic HTTPS setting for newbies like me 🙂

If you need more advanced features, you can reference more options and how-to at apache site:
http://httpd.apache.org/docs/2.2/ssl/ssl_faq.html
http://httpd.apache.org/docs/current/ssl/ssl_howto.html

A Perl script template

I wrote lots of Perl scripts recently. I feel it’s quit good to have a template. When I need to create a new one, I can start with the template, so I can focus on the real logical. Here is the template perl script:

#!/usr/bin/perl -w

format HEADER =
/ ------------------------------------------------------------------------------------------------------------------
| Purpose : update a file by querying DB
| 
|  Options:
|    -v    verbose mode
|    -d    debug mode
|    -t    test mode
|    -h    Give help screen
|  
|  Example:
|    update_vip_file.pl   --silent mode
\ ------------------------------------------------------------------------------------------------------
.

#
# History
# -------
# Alex Zeng      05/15/13  create it

use strict;
use DBI;
use Data::Dumper;
use Getopt::Long;

#DB inventory source
my $dbh = DBI->connect("DBI:mysql:host=alexzeng.wordpress.com;database=mydb","mydb","mydb",{ RaiseError => 1, AutoCommit => 1 });
my $target_file = "/repository/vips";
my $target_file_test = "/tmp/vips";

# configuration
my $home = "/export/home/oracle/admin/cron";
my $loc_file = "$home/log/update_vips_list.loc";
my $log_file = "$home/log/update_vips_list.log";

my ($list, @final_list);
my ($verbose, $debug_on, $istest);
my $finished = 'n';
my $mailwho = 'alexzeng\@wordpress.com';

# -------------------------------------------------------------------
# Main functions
# -------------------------------------------------------------------
&init;

&get_data;
&format_data;
&write_data;

$finished = 'y';
&quit(0);

# -------------------------------------------------------------------
# Sub functions
# -------------------------------------------------------------------
sub get_data {
# Get vip list from inventory 
  my $get_list_sql = qq{select vip, target_fqdn
                        from dblist_dbvip 
                        order by target_fqdn, vip
  };
  my $update_sql = qq{update dblist_dbvip
                         set vip=trim(vip)
                           , target_fqdn=trim(target_fqdn)
                     };
  $dbh->do($update_sql);

  &prtit("Get data start\n");
  $list = $dbh->selectall_arrayref($get_list_sql);
  &debug($list);
  &prtit("Get data done\n");
}

sub format_data {
# -------------------------------------------------------------------
# format vip list
# -------------------------------------------------------------------
  my $i = 0;
  &prtit("Format data start\n");

  foreach my $rowRef (@$list) {
    my ($vip, $target_fqdn)=@$rowRef; 
    
	&debug("$vip, $target_fqdn");
	$final_list[$i]->{vip} = $vip;
	$final_list[$i]->{target_fqdn} = $target_fqdn;
	$i++;  
  }
  &debug(\@final_list);
  &prtit("Format data done\n");
}

sub write_data {
# -------------------------------------------------------------------
# write data to the target file:
#@ VIP # HOST
# world.vip.wordpress.com#db01.wordpress.com
# -------------------------------------------------------------------
  my $target_file_tmp = "${target_file}.tmp";
  my $target_file_bak = "${target_file}.bak";
  
  my $fmstr = "%-50s #%-40s \n";
  &prtit("Write data start\n");
  open (TFT, ">$target_file_tmp") or die "Cannot open $target_file_tmp to write";
  print TFT "#This file is updated by cron job automatically\n";
  print TFT "#If you want to modify the data, please update it from mydb. Manually updates will be overwrote!\n";
  printf TFT $fmstr, "\@VIP", "HOST";
  #printf TFT "\n#\n";
  foreach my $row ( @final_list ){
    printf TFT $fmstr, $row->{vip}, $row->{target_fqdn};
  }
  close TFT;
  chmod 0444, $target_file_tmp or die "Can't chmod: $!";
  rename($target_file,$target_file_bak);
  rename($target_file_tmp,$target_file) or die "Can't rename: $!";
  &debug($target_file);
  &prtit("Write data done\n");
}

sub init {
  my ($opt_h, $rc);

  GetOptions ("h"  => \$opt_h,
			  "v"  => \$verbose,
              "d"  => \$debug_on,
              "t"  => \$istest
  );
  
  &do_help if  (defined $opt_h);
  if($istest) {
    $target_file = $target_file_test; 
  }

  $rc = &flock_lockfile($loc_file); 
  if ($rc) {
  # This means we couldn't get a lock on the file.  Lets do more checking and see whats going on...
    if ($verbose) {
      print ("\n")                  if ($verbose);
      print ("flock NOT obtained on $loc_file\n")   if ($verbose);
    }
    exit 100;
  }

  open (LOG,">$log_file") || die ("\n\nCan't open file to write : $log_file");
}

sub flock_lockfile {
  #--------------------------------------------------------------------------
  # Try to obtain a lock on a file name $file.  Returns non-zero return code
  # if file lock cannot be obtained.
  #--------------------------------------------------------------------------
  my ($lockfile) = @_;

  my $LOCK_EX = 2;

  open (LOC,">$lockfile") ||
    die ("\n\nCan't write to file: $lockfile\n");

  eval {
    local $SIG{ALRM} = sub { die "flock timeout" };
    alarm 5;
    flock(LOC, $LOCK_EX);
    alarm 0;
  };
  alarm 0;

  return ($@);
}

sub do_help {
  #--------------------------------------------------------------------------
  # Give help screen and exit
  #--------------------------------------------------------------------------
  $~ = "HEADER";
  write;
  exit;
}

sub debug {
  # -------------------------------------------------------------------
  # Print debug info to screen
  # -------------------------------------------------------------------
  my ($line) = @_;
  print Dumper($line) if($debug_on);
}

sub prtit {
  # -------------------------------------------------------------------
  # Print lines to screen and log
  # -------------------------------------------------------------------
  my ($line) = @_;

  my $date = localtime;
  my $str = "$date : $line";
  #my $str = "$line";

  print LOG $str;
  print     $str  if ( $verbose);
}

sub quit {
  # -------------------------------------------------------------------------
  # Clean up our mess.
  # -------------------------------------------------------------------------
  my ($retcode) = @_;
  
  $dbh->disconnect if defined($dbh);
  close LOG;
  if($finished eq 'n' && !$verbose) {
    `mailx -s "Cron job $0 failed" $mailwho < $log_file`;
  }
  exit($retcode);
}

As always, it’s not perfect, but a good start.

A script to format pseudo code

To format the pseudo code in previous blog, I wrote a small perl script to do that:

#!/usr/bin/perl -w

#2/3/2013  Alex Zeng  format code with if, else, loop

my $ifile = shift;
my $blank = 0;
my $indent = 2;
open(IFILE, "$ifile") or die "Cannot open $ifile\n";
while(my $line = <IFILE>) {
  if($line =~ /^\s*else/i
     or $line =~ /^\s*end/i
    ) {  #reduce indent before xxx
    $blank = $blank - 2;
  }
  if($blank > 0) {
    print " " x $blank;  #print several blanks
    print "$line";
  } else {
    print "$line";
    $blank = 0;
  }
  if($line =~ /^\s*if/i
     or $line =~ /^\s*else/i
     or $line =~ /^\s*loop/i
    ) {  #increase indent after xxx
    $blank = $blank + 2;
  }
}

As always, it’s not perfect, but a start. Maybe sometimes it’s helpful for you.

How-to avoid ssh prompt for password

Sometimes, we don’t want to use password authentication when using ssh. For example, I wrapped ssh in my script. I want to it always uses key authentication. If there is no key setup, just exit.

I googled it. Most of them are given the option “PasswordAuthentication no”. But in my test, it didn’t work somehow (probably add it the ssh config file will work, but that’s not what I want):

 ssh dbox036 date
The authenticity of host 'dbox036(10.1.1.1)' can't be established.
RSA key fingerprint is 50:ca:af:6c:27:b3:8d:74:e7:27:97:46:36:43:67:78.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'dbox036,10.1.1.1' (RSA) to the list of known hosts.
oracle@dbox036's password: 
Permission denied, please try again.
oracle@dbox036's password: 

I read ssh document and found parameter “NumberOfPasswordPrompts” can be used:

$ ssh  -o "NumberOfPasswordPrompts 0" dbox036
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

If you want to avoid the host key checking, add option “StrictHostKeyChecking no”

$ ssh -o "StrictHostKeyChecking no" -o "NumberOfPasswordPrompts 0" dbox036
Warning: Permanently added 'dbox036,10.1.1.1' (RSA) to the list of known hosts.
Permission denied (publickey,gssapi-keyex,gssapi-with-mic,password).

If you want to avoid the “Warning” message, add option “LogLevel=quiet”, you will get nothing if it didn’t work

$ ssh -o "StrictHostKeyChecking no" -o "NumberOfPasswordPrompts 0"  -o LogLevel=quiet dbox039 date
$

How-to Find & Replace a string in all files within a directory

I run into this several times. I’d like to share how I solve it with shell and perl. Here are the steps:

Step 1. Get a list

for f in `find . -type f`
do
  r=`grep SOURCE_STRING $f`
  if [ "alex${r}" != "alex" ] ; then
    echo $f >> update.lst
  fi
done;

Step 2. Check the list

wc update.lst
cat update.lst
for f in `cat update.lst`
do
  r=`grep SOURCE_STRING $f`
  echo $f
  echo $r
done;

Step 3. Update files

for f in `cat update.lst`
do
    echo $f
    perl -i -pe 's/SOURCE_STRING/TARGET_STRING/g' $f
done;

Step 4. Check it again run Step 2 again