How-to make cross-domain URL call work in javascript

I was quite busy in the past 2014. I only wrote one blog post in the whole year. Busy is a good excuse for lazy :). I hope I can post more in 2015 to help others and help me as well.

In my first post this year, I’ll share my experiences with cross domain URL call in JavaScript.

Case 1. Normal URL call with JQuery

url = "https://alexzeng.wordpress.com/api/report/";    
$.get(url, function(data) {
    //process data;
    process(data);
});

Case 2. Cross-domain URL call with JQuery
Just add a parameter “callback” with value “?” in your URL, JQuery will handle everything else.

url = "http://alezeng.blogspot.com/api/report/";
$.getJSON(url + "?callback=?", function (result) {
    // process data
    process(data);
}); 

Note: if your url already have some prarameters, use “&” to connect callback parameter like “&callback=?”

There’re some pre-requirements on the API at server side:
1. The API should return the data with “Content-Type:text/javascript”. Otherwise this error’ll show in javascript call:

Resource interpreted as Script but transferred with MIME type text/html

You can check the “Content-Type” in Response Headers by manually call the url in Chrome debug mode.

2. The API should handle the callback parameter, and wrapper the data in callback function. Otherwise you’ll get error because it cannot run as a javascript function
Many RESTful API using modern framework can support by default, for example Django, Spring MVC. But if you use an old framework or home-grown API, or even just a statistic text, it won’t work by default.
However, if you understand how it works, you can mimic what it does and make it work for any URL. Let’s practice it in Case 3.

Case 3.  Manually call Cross-domain URL
Scenario:
I have data at http://alezeng.blogspot.com/api/report.dat.
I want to use the data it in my another site https://alexzeng.wordpress.com/
Step 1. Wrapper data in a callback function
http://alezeng.blogspot.com/api/report.dat, content:

my_callback({
   'key1' : 'value1',
   'key2' : 'value2'
})

Note: you can enclose any value in my_callback as long as it’s a valid javascript parameter.

Step 2. Make your server response the URL as javascript.
In my case, apache is my web server. It’ll return the data as “Content-Type:text/html” by default.
Add below line in apache config file to let it response “.dat” as javascript:

AddType text/javascript .dat 

Note: if the URL is a RESTful API, wrote by programming language, like java or python, you can set the response content type in code directly.

Step 3. Write the callback function
In another site https://alexzeng.wordpress.com/, write javascript codes to handle the data:

function my_callback(data)
{
    //process data
    process(data);
}

function cross_domain_call() {
    url = 'http://alezeng.blogspot.com/api/report.dat'    
    var script = document.createElement('script');
    script.src = url +'?callback=my_callback'
    document.getElementsByTagName('head')[0].appendChild(script);
}

With this, we can successfully get the data from http://alezeng.blogspot.com in https://alexzeng.wordpress.com.
But it’s kind of troublesome if you have any URL that need to change at server side. The other way is, we can use our server to call cross-domain URL on behalf of client.

Case 4. Use server to call Cross-domain URL
Step 1. Write a generic URL call function in server side.
Let’s use Django server as example:
In views.py

from django.views.decorators.gzip import gzip_page

#gzip the response content, this's not a must
@gzip_page
def get_crossdomain_data(request):
    try:
        url = request.GET['url']
        url = urllib.unquote(url).decode('utf8')
        jsonData = urllib2.urlopen(url).readlines()
        return HttpResponse(jsonData)
    except Exception,e:
        result={};
        result['success']= 'false'
        result['message']= str(e)
        return HttpResponse(result)

In urls.py, map the URL

(r'^get_crossdomain_data',views.get_crossdomain_data),

Step 2. Use the server URL to call Cross-domain URL
We need to encode the original URL and transfer it as an parameter to server, and then parse the response data

url = "http://alezeng.blogspot.com/api/report.dat";
url= "/get_crossdomain_data?url=" + encodeURIComponent(url);
$.get(url, function(str_data) {
    data = jQuery.parseJSON(str_data);
    //process data;
    process(data);
});

With all these approaches, you can choose the one works in your scenario.

How to create executable jar file?

Method 1: I used to create a executable jar file by eclipse export “Runnable JAR file”

It’s very handy, and it has 3 options to handle libraries:

  • Extract required libraries into generated JAR
  • Package required libraries into generated JAR
  • Copy required libraries into sub folder next to the generated JAR.

I really like it. You don’t need to configure anything but just a few clicks. I think it’s the best way if it’s just for test.It works well except when I use the 1st option. I run into this problem :

org.xml.sax.SAXParseException: schema_reference.4: Failed to read schema document 'http://www.springframework.org/schema/beans/spring-beans-4.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.

This is because the xsd file is missed/overwroten when eclipse extract the libs, and my runtime host cannot connect to Internet that makes it impossible to get it. While this is  not a real problem for eclipse, it can be overcome by using the 2nd option. It didn’t extract the lib, but includes them as they’re.

I wonder why eclipse takes a lot of efforts to extract libs (it’s obviously slower than other 2 options in the list). At first, I thought it may want to build small jar packages(it’s obviously smaller ). Later, I found out the real reason is: java can NOT load classes from a Jar inside a Jar! (In real world, how can you put an even larger jar inside a jar? You are kidding!) Java(class loader) is designed to like this.

Now the question is, how eclipse overcome this jar inside jar problem? I unzipped the generated jar file, and found there’s a jarinjarloader class which did a trick behind. See more below.

Method 2: Using onejar-maven-plugin
Because the network between my laptop and the target machine is slow. It takes times to transfer the runnable jar file, even it’s not big. So I need to build it remotely without eclipse, use maven instead. This plugin did the similar trick jarinjarloder did, the 2nd option in eclipse. It basically let java call its main function, and it help you load all dependency jar files including yourselves. Here is the configuration clip:

		  	<plugin>
		  	    <groupId>com.jolira</groupId>
		  	    <artifactId>onejar-maven-plugin</artifactId>
		  	    <version>1.4.4</version>
		  	    <executions>
		  	        <execution>
		  	            <configuration>
		  	                <attachToBuild>true</attachToBuild>
		  	                <classifier>onejar</classifier>
		  	            </configuration>
		  	            <goals>
		  	                <goal>one-jar</goal>
		  	            </goals>
		  	        </execution>
		  	    </executions>
		  	</plugin>

The associated plugins may also needed:

			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-compiler-plugin</artifactId>
				<version>2.3.2</version>
				<configuration>
					<source>1.6</source>
					<target>1.6</target>
					<encoding>UTF-8</encoding>
				</configuration>
			</plugin>
		   	<plugin>
		  	    <groupId>org.apache.maven.plugins</groupId>
		  	    <artifactId>maven-jar-plugin</artifactId>
		  	    <configuration>
		  	        <archive>
		  	            <manifest>
		  	                <mainClass>com.wordpress.alexzeng.automation</mainClass>
		  	            </manifest>
		  	        </archive>
		  	    </configuration>
		  	</plugin>

It works well as long as you get the dependency configured well in pom.xml. I missed an ojdbc6.jar at first.

Method 3: Using maven-dependency-plugin, corresponding to the 3rd option in eclipse.
I tried this in my project, but failed somehow. I have 23 jar files in final lib directory. But somehow it only includes 19 jar files in the Class-Path of META-INF/MANIFEST.MF. It refused to work even after I added the jar files to classpath variable in environment. I think this method should work, and this is the best one for big projects (you don’t want to have a very big single jar file). You can reference http://www.mkyong.com/maven/how-to-create-a-jar-file-with-maven/ for a simple case.

Last but not least, their’s also a corresponding method in maven to match the eclipse 1st option, use maven-assembly-plugin pre-defined descriptor jar-with-dependencies. It’s really simple and handy if extract dependency jars didn’t cause troubles. 

		    <plugin>
                      <groupId>org.apache.maven.plugins</groupId>
                      <artifactId>maven-assembly-plugin</artifactId>
                      <version>2.4</version>
		      <configuration>
		          <descriptorRefs>
		             <descriptorRef>jar-with-dependencies</descriptorRef>
		          </descriptorRefs>
		        <archive>
		          <manifest>
		            <mainClass>com.wordpress.alexzeng.automation</mainClass>
		          </manifest>
		        </archive>
		      </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
		    </plugin>

As a newbie, I spent a lot of time to get it clear. I hope it can save you sometime to if you need to do the same.

how to setup HTTPS for apache

I’m working on setup authentication for an internal website recently.To protect usernames and passwords, setting up HTTPS is very necessary. Here is my first trial on HTTPS/SSL:

Step 1. Check if apache has ssl module compiled 

$ grep mod_ssl apache/conf/httpd.conf
If below line is there, and you can start apache without problem, you don't need to recompile apache
"LoadModule ssl_module modules/mod_ssl.so"

The other way is to check whether the module is there.
If it's there, but it's not in the httpd.conf file, you can add it and try start apache
$ ls modules/mod_ssl.so

Step 2. Compile apache if it didn’t have ssl module

--download apache from http://httpd.apache.org/, and unzip it
cd /home/alexzeng/httpd-2.2.24
--config
#./configure --prefix=/alexzeng/apache --with-config-file-path=/alexzeng/apache/conf --enable-ssl --enable-http --enable-rewrite --enable-track-vars --enable-cgi --with-config-file-path=/opt/apache/conf --enable-modules=all --enable-mods-shared=all --enable-file-cache --enable-disk-cache --enable-cache --enable-mem-cache --enable-dumpio --enable-logio --enable-mime-magic --enable-headers --enable-usertrack --enable-version --enable-proxy --enable-proxy-connect --enable-proxy-http --enable-proxy-ftp --enable-proxy-ajp --enable-proxy-balancer --enable-so

--make
#make

--make install
#make install

Step 3. Config apache SSL
A. Load ssl module, it should already have these lines:

LoadModule ssl_module modules/mod_ssl.so
LoadModule alias_module modules/mod_alias.so
LoadModule rewrite_module modules/mod_rewrite.so
LoadModule dir_module modules/mod_dir.so
...

B. Enable httpd-ssl.conf, remove # from httpd-ssl.conf line:

# Secure (SSL/TLS) connections
Include conf/extra/httpd-ssl.conf

C. Config pages that need force using https at VirtualHost part:

NameVirtualHost *:80

   ServerName alexzeng.vip.wordpress.com
   DocumentRoot "/alexzeng/apache/htdocs"
   Redirect permanent /signin https://alexzeng.vip.wordpress.com/signin

D. Config extra/httpd-ssl.conf

Listen 443

#   General setup for the virtual host
DocumentRoot "/alexzeng/apache/htdocs"
ServerName alexzeng.vip.wordpress.com:443
...
#   SSL Engine Switch:
#   Enable/Disable SSL for this virtual host.
SSLEngine on

#   Server Certificate:
SSLCertificateFile "/alexzeng/apache/conf/server.crt"

#   Server Private Key:
SSLCertificateKeyFile "/alexzeng/apache/conf/server.key"
#   Server Certificate Chain:

Until now, the HTTPS is setup for the site. But we need to have 2 files: server cert server.crt, and its private key server.key.

The official way it to request it from public SSL certificate companies, such as VeriSign, because their cert is accepted by all browser by default. But it costs a few hundred dollar per year at least.

So many companies have their self-signed cert to reduce the cost. Especially for internal sites, a company can have their root cert installed for their employees OS image by default. In that way, access its signed certs site will be recognized as safe, no security alert, nor https crossed out in red. That’s the case in my company.

I got our IT team signed cert, but it’s in pfx format. I need to convert it to the 2 files server.crt and server.key. The processes are as follows:

$ openssl pkcs12 -in it.pfx  -nocerts -nodes -passin pass:"<password_from_IT>" | openssl rsa -out server.key
MAC verified OK
writing RSA key
$ openssl pkcs12 -in it.pfx  -clcerts -nokeys -nodes -passin pass:"<password_from_IT>" | openssl x509 -out server.crt
MAC verified OK
-- The pfx is created by Windows tools, so I use openssl rsa/x509 to remove some "Bag Attributes" lines.
-- Otherwise, you can just use -out option at the first without the sencond command in pipeline

--copy key to apache directory if needed
$ cp server.crt  server.key /alexzeng/apache/conf

--restart apache
$ sudo ./apachectl stop
$ sudo ./apachectl start

If a browser didn’t installed the companies’ root cert, it’ll get security alert when access the site. It can be avoided by import their root cert.

Besides that, we need to test the HTTPS by ourselves even without an internal team to sign the cert for us. We’ll make ourselves a certificate authority (CA) 🙂

How-to create self-signed cert for test:
a. Create a server private key: server.key

$ openssl genrsa -out server.key 1024
Generating RSA private key, 1024 bit long modulus
...................++++++
......++++++
e is 65537 (0x10001)

b. Create certificate signing request (CSR) : server.csr (using server.key)

$ openssl req -new -out server.csr -key server.key
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:aa
Locality Name (eg, city) [Default City]:aa
Organization Name (eg, company) [Default Company Ltd]:aa
Organizational Unit Name (eg, section) []:aa
Common Name (eg, your name or your server's hostname) []:alexzeng.vip.wordpress.com
Email Address []:
Please enter the following 'extra' attributes
to be sent with your certificate request
A challenge password []:
An optional company name []:
$

--Note:
"Country Name" must be a valid name
"Common Name" must be the same as ServerName in httpd.conf
"State or Province Name" must has some value

c. Create a certificate authority (CA) private key: ca.key

$ openssl genrsa  -out ca.key 1024
Generating RSA private key, 1024 bit long modulus
.......++++++
........++++++
e is 65537 (0x10001)

d. Create CA certificate : ca.crt (using ca.key)

$ openssl req  -new -x509 -days 365 -key ca.key -out ca.crt
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:US
State or Province Name (full name) []:aa
Locality Name (eg, city) [Default City]:aa
Organization Name (eg, company) [Default Company Ltd]:aa
Organizational Unit Name (eg, section) []:aa
Common Name (eg, your name or your server's hostname) []:alexzeng.vip.wordpress.com
Email Address []:
$

e. Sign a certificate by my own CA: server.crt (using certificate signing request server.csr, CA private key ca.key, and CA certificate ca.crt)

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 1 (0x1)
        Validity
            Not Before: Dec 26 07:20:02 2013 GMT
            Not After : Dec 26 07:20:02 2014 GMT
        Subject:
            countryName               = US
            stateOrProvinceName       = aa
            organizationName          = aa
            organizationalUnitName    = aa
            commonName                = alexzeng.vip.wordpress.com
        X509v3 extensions:
            X509v3 Basic Constraints:
                CA:FALSE
            Netscape Comment:
                OpenSSL Generated Certificate
            X509v3 Subject Key Identifier:
                70:45:AB:98:23:51:BB:88:23:20:EA:21:21:3C:6A:8A:E2:0A:97:B8
            X509v3 Authority Key Identifier:
                keyid:46:73:F6:1F:85:74:10:D6:B4:5B:AB:B6:2E:1C:5D:A8:97:08:55:4C
Certificate is to be certified until Dec 26 07:20:02 2014 GMT (365 days)
Sign the certificate? [y/n]:y

1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
$ ls -lt
total 140
-rw-r--r-- 1 root  root   3048 Dec 26 00:20 server.crt
-rw-rw-r-- 1 dbbox dbbox   948 Dec 26 00:19 ca.crt
-rw-rw-r-- 1 dbbox dbbox   887 Dec 26 00:18 ca.key
-rw-rw-r-- 1 dbbox dbbox   643 Dec 26 00:18 server.csr
-rw-rw-r-- 1 dbbox dbbox   887 Dec 26 00:10 server.key

Usage of these files:

2 files are used in httpd-ssl.conf:
server.key (Server private key) -> this one will keep at server side
server.crt (Server certificate) -> this one will send to client when users access HTTPS

The other 3 files owned by CA (Certificate Authority) are used only during sign processes:
server.csr (certificate signing request)
ca.key (certificate authority private key)
ca.crt (certificate authority certificate)

Problems I got during these processes:
1. index.txt and serial file are missing when sign the cert

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/index.txt: No such file or directory
unable to open '/etc/pki/CA/index.txt'
140528819345224:error:02001002:system library:fopen:No such file or directory:bss_file.c:355:fopen('/etc/pki/CA/index.txt','r')
140528819345224:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:357:

$ sudo openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
/etc/pki/CA/serial: No such file or directory
error while loading serial number
140414051186504:error:02001002:system library:fopen:No such file or directory:bss_file.c:355:fopen('/etc/pki/CA/serial','r')
140414051186504:error:20074002:BIO routines:FILE_CTRL:system lib:bss_file.c:357:

--create them to avoid the issue
$ sudo su -
# touch /etc/pki/CA/index.txt
# echo 01 > /etc/pki/CA/serial

2. openssl permission issue

$ openssl ca -in server.csr -out server.crt -cert ca.crt -keyfile ca.key
Using configuration from /etc/pki/tls/openssl.cnf
I am unable to access the /etc/pki/CA/newcerts directory
/etc/pki/CA/newcerts: Permission denied

--solution
My openssl is installed as root, so run the command with sudo
If openssl is created by the login account itself, you don't need to sudo, and the configure file of openssl will located at your installation place.

3. Error “libtool: install: invalid libtool wrapper script xxx” when “make install” apache.

--error message
libtool: install: invalid libtool wrapper script `htpasswd'
libtool: install: invalid libtool wrapper script `htdigest'
libtool: install: invalid libtool wrapper script `rotatelogs'
libtool: install: invalid libtool wrapper script `logresolve'
libtool: install: invalid libtool wrapper script `ab'
libtool: install: invalid libtool wrapper script `htdbm'
libtool: install: invalid libtool wrapper script `htcacheclean'
libtool: install: invalid libtool wrapper script `httxt2dbm'
libtool: install: invalid libtool wrapper script `checkgid'
make[2]: *** [program-install] Error 1

--It's caused by libtool is not installed at this host.
--Here is fix steps;
$ rpm -qa | grep libtool
$ sudo yum search libtool
$ sudo yum install libtool

--remove all installed files, and redo make, and make install
$ rm -rf /alexzeng/apache/
$ make
$ make install

This is a basic HTTPS setting for newbies like me 🙂

If you need more advanced features, you can reference more options and how-to at apache site:
http://httpd.apache.org/docs/2.2/ssl/ssl_faq.html
http://httpd.apache.org/docs/current/ssl/ssl_howto.html