libspatialite problem with Django on OS X

I have no idea how it comes, using GeoDjango on OS X is a kind of pain in ass, especially when it's about spatialite. It has got better and better - it's better at least rather than the time you should install all related package manually. Finally, our awesome friend Homebrew became to be able to handle most of these cumbersome works, but I should say - it's very close.

You can face this error message when you're trying to make GIS Django application on OS X:

Unable to load the SpatiaLite library extension "/usr/local/lib/libspatialite.dylib"

If you install libspatialite 4.2.0 package by using Homebrew, then it installs libspatialite.dylib and mod_spatialite.dylib into its library path. When django.contrib.gis.db.backends.spatialite is used as a database backend, I'm not sure what happens on extension loading logic of sqlite3, it tries to find `sqlite3_spatialite_init` symbol from libspatialite.dylib. But that symbol dwells in mod_spatialite.dylib, so it fails. You can adjust this by adding SPATIALITE_LIBRARY_PATH to settings.py file:

SPATIALITE_LIBRARY_PATH='/usr/local/lib/mod_spatialite.dylib'

For sure, it can vary with the library path. Real path would be /usr/local/Cellar/libspatialite/x.x.x/lib/. (x.x.x is version number)

Testing C10M on a Host with Tornado

Here's my code that I've used, wanna share it with pleasure :)
https://github.com/kenial/tornado-test-c10m. 

Recently, I've tried to make out to test >1M concurrent connections on my PC, by using Tornado (FYI, Tornado is Python based web framework that supports both TCP / WebSocket connections). As you know, there has been a lot of C10K problem discussions, so you can easily find out articles about tuning of network and performance. (just try to google 'linux c10k tuning,' then you gonna get) Here are few useful links for me to get comprehension about C10K tuning :

- The C10K problem
http://www.kegel.com/c10k.html

- Performance Tuning the Network Stack on Mac OS X Part 2
https://rolande.wordpress.com/2014/05/17/performance-tuning-the-network-stack-on-mac-os-x-part-2/ 

- Linux TCP/IP tuning for scalability
http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/

- RED HAT ENTERPRISE LINUX 7 PERFORMANCE TUNING GUIDE (CentOS compatible)
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/index.html 

- The C10M(!) problem
http://c10m.robertgraham.com/p/manifesto.html

A caveat here, it requires really vast memory. There is also a trick for >65K connections on loopback network. By using local address and remote address, OS identifies a socket. If you have multiple clients connect to your server that listens 8080 port, created sockets will be as followings:

local address	remote address
192.168.0.10:20001	192.168.0.10:8080
192.168.0.10:20002	192.168.0.10:8080
192.168.0.10:20003	192.168.0.10:8080
192.168.0.10:20004	192.168.0.10:8080

But range of local ports are between 0~65,535, so you can make 65K connections at maximum with one server port, even in an ideal case. In fact, the number could be less.

This could be resolved by letting server listens to multiple ports, like this:

local address	remote address
192.168.0.10:20001	192.168.0.10:8080
192.168.0.10:20001	192.168.0.10:8081
192.168.0.10:20002	192.168.0.10:8080
192.168.0.10:20002	192.168.0.10:8081

As above, if your server listens to multiple ports, then we can have multiple sockets that has a same local address and different remote addresses. With 200 listening ports and 50K local ports, it ends up with 200 * 50,000 = 10,000,000 (10M) connections! (FYI, you can assign multiple IPs instead of multiple listening ports, but it's little cumbersome - you have to do it manually, and need to restore them on some day)

One caveat more: this works on Ubuntu Server 14.10 (my development environment is) for sure, but not for OS X Yosemite. I'm not sure for the reason, but according to result of netstat, sockets on Yosemite seem to be identified by only local address. There no duplicates amongst local addresses as long as I looked into.

Okay, last but not least, here's the result:

- The host that runs test code is Ubuntu Server 14.10, which hosted by Amazon EC2's r3.4xlarge instance that has 122GiB memory.

- 10.5KB per TCP conn, 15.5KB per WebSocket conn.

- If 1M connections, it will be 21GiB for TCP connections and 31GiB for WebSocket connections.
  (211GiB for 10M!)

- PyPy is helpful for performance, especially in establishing connections.

Idea: location-based minisite for smartphone

This is an idea about the application of Mug and Sam.

While "Minisite" refers to a sort of websites which contains information about one specific product or subject, but in this context, minisite represents just a site that contains little amount of contents. If trying to write "location-based minisite" more detail, that means "location-based discoverable minisite in proximity". Mug and Sam are a sort of file sharing tool that is location-based, Sam has a role of host. Mug can discover nearby Sam, and access to files that are shared by Sam.



I am imaging that local businesses such as stores and restaurants use Sam as this location-based minisite. For example, a little photo exhibition at a cafe, and photos are accessible by customers' smartphones or tablets when they visit the cafe only. I know a cafe's owner who loves a travel, and had ever seen photos are on his cafe's wall. A little space was needed for even this little exhibition, if he utilized Sam then the space might not be required.

How would it works? Look at this (they're from my album, at Lake Louise) :

Above one is relevant to images, but audios or videos can be provided with:

If a document is needed, that is, at a information center or a conference, PDF would be fit:

As you see, Mug is basically designed for sharing media files, but you can configure it accessing any web resources:

On above, Mug is accessing Wikipedia page, but it is possible to configured to redirect a certain web site of a store. The benefit is, customers that visit the store can access to a webpage without inputting URL into web browser directly.



Currently, mega internet services are doing business based on their proprietary information that is collected only for itself and by itself. Lots of services - search engine, portal, map service, review site and so forth - have influenced to local business unilaterally.

But I believe IT technology can also help local business to face to this situation in some ways, like providing location-based minisite with unique user experience. And for that purpose, I'm in considering what features are needed additionally to these applications.

Any suggestions?



- Refer to http://www.fountainproject.com/info for detail information.

Mug and Sam, implementations from Fountain Project are released.

link: http://blog.fountainproject.com/post/54057159608/ready-to-go-with

I have been working on a sort of file/media sharing tool, which is location-based service. File sharing & location, yes, they're applications that similar to Bump. But basically, this applications don't require to 'Bump' at least. Once you run it, it just works to find out shared file nearby.

If you're interested in location based file sharing, just look over them. URL of web site is http://www.fountainproject.com


Thanks!

Portable Django & Python for Mac OS X

Recently, I’m working on python-host program. Under the hood, that program hosts python binary, and the django development server works as HTTP API server on python environment. Although it’s not optimized, but useful enough for my purpose.

There’s other projects about this (like as DjangoOnAStick and Portable Python), however, they don’t provide for OS X version and < Python 2.7 versions. So I made up my mind to create something new. (this is based on python 2.7 and django 1.4)

Okay, refer to this URL : https://bitbucket.org/kenial/portable-django

There is shell script(also known batch file on Windows) ready. Use it like this:



Easy, huh?

Bookshelf template: Personal book list management on Google Apps Spreadsheet

* “Bookshelf template” shared link:

https://docs.google.com/spreadsheets/d/1pVoT6l1HdRkN3aPMKAmuNVEqJg4izxTW_vzECIOzavo/edit?usp=sharing 

A friend of me wrote on SNS that she bought a desktop app managing personal book list. As I saw it then thought that ‘Oh I need this one too’ and was considering to buy one in a minute. But, well, I decided to implement it on my own. Features are simple. Input ISBN, book title, or author’s name, and then just run the script. So that the script will find information of the books from Google Books API, and paste them on a sheet. If you need additional fields, just add them as well as need.



How to use: input a part of book information like this and run the script (click 'Get book information from Google Books API' menu), the sheet will be filled out with information of the books.

I was just trying to enhance my Google Apps Script skill, but if you need this, use freely : )

Execute a terminal command on a Cocoa application

* refer to http://stackoverflow.com/questions/412562/execute-a-terminal-command-from-a-cocoa-app/12310154#12310154

In order to write an cocoa application that can start/stop webserver embedded within, I needed some method could run terminal commands. One user(thanks kent!) suggested to use /bin/sh to run shell script file – and you might know, /bin/sh utility can run a line of command instead of script file with –c option. With NSTask and /bin/sh, I made some method to run a shell command by line:

NSString *runCommand(NSString *commandToRun)
{
    NSTask *task;
    task = [[NSTask alloc] init];
    [task setLaunchPath: @"/bin/sh"];

    NSArray *arguments = [NSArray arrayWithObjects:
                          @"-c" ,
                          [NSString stringWithFormat:@"%@", commandToRun],
                          nil];
    NSLog(@"run command: %@",commandToRun);
    [task setArguments: arguments];

    NSPipe *pipe;
    pipe = [NSPipe pipe];
    [task setStandardOutput: pipe];

    NSFileHandle *file;
    file = [pipe fileHandleForReading];

    [task launch];

    NSData *data;
    data = [file readDataToEndOfFile];

    NSString *output;
    output = [[NSString alloc] initWithData: data encoding: NSUTF8StringEncoding];
    return output;
}

Using above method, you can write a code to run a line of terminal shell command like this:

NSString *output = runCommand(@"ps -A | grep mysql");

What is a DEVELOPER to App Store, Google Play, and Windows Phone Marketplace?

I found out something interesting, when I was adding vary app stores' developer links(show app list of certain developer) on my resume. It was a piece of URL path of the link - my app store developer links are following:

- iTunes App Store apps : http://itunes.apple.com/us/artist/kenial/id319803312
- Google Play apps : https://play.google.com/store/apps/developer?id=Kenial
- WP Marketplace apps : http://www.windowsphone.com/en-US/publishers/Kenial

You see what, Prior to 'kenial', and come up with something as I do. Yeah, there is artist prior to 'kenial' on App Store, in case of Google Play there is developer, in case of Windows Phone Marketplace, publishers.

Okay, now you have some idea about what a developer is for, in terms of each app store master : )

Amazon EC2 network latency test

Last March, I've tried to upload BITNA service(sort of web service for smartphone app) on world-wide Amazon EC2 servers. In that process, for checking latency, I wrote a memo regards ping latency and HTTP latency(specific page that returns kind of 'hello world'). Certainly, it's not pretty precise time and based on Korea's network environment, but I believe that you might need this data for your reference.

GAE(HTTP) is HTTP request/response latency, and others are ping request/response latency. (unit is ms) Surely, GAE means Google App Engine. This result shows HTTP latency is pretty much, as compared to ping latency.

to from	GAE	GAE(HTTP)	Site A*	Site B**
EC2(Oregon)	20~30	118~	228~	179~
EC2(Singapore)	9~15	261~	116~	122~
EC2(Tokyo)	32~34	254~	48~	52~
EC2(Ireland)	21	117~	380~	313~
EC2(Virginia)	2~4	16~	200~	240~
*** KT wibro	130~190	320~	88~	95~
*** KT 3G	300~	380~
Cable Internet****	45~52	280~

* web-hosting type site in Korea

** virtual host type site in Korea

*** KT wibro and KT 3G are mobile internet service of Korean company KT.

**** Cable network ISP in Seoul, Korea

Django 1.4 for Android (only dev. runserver!)

 

I’m writing this post because of this interesting article:
(I’m sorry that it’s written in Korean)

[기고]약정 벗은 안드로이드, 서버가 되다…'서품폰'
http://www.zdnet.co.kr/column/column_view.asp?artice_id=20120518070549

This article, which explains how to make a old android phone into simple server with web2py and ubuntu blahblah, concluded that the post’s author couldn’t set up Django on Android. I can’t understand why – you know, Django doesn’t depend on a bunch of ‘native’ python packages as I know(in other words, Django consists of pure python source files). If python is configured correctly, Django goes well as well, I guess.

Yes, my Django doesn’t go such a fastidious way.

 

 

… So spent half a day, finally I got working Django 1.4 on Android – the working feature is mere runserver thing yet. Whatsoever, with more efforts, I believe that it’s going to be able to set up Django with other web server program. Will mod_wsgi be excutable on Android? hmm…

However, here’s my work:

※ Note: Target device is Motorola MOTOGLAM, rooted and Android Froyo (2.2) installed.

※ Note: The Android is unlike a typical Linux environment, the packages that can be installed in are limited. Django on Android  has such limitation as well. It’s hard to install the packages for Django(e.g. south, celery …), even sometimes impossible it is. So, just for fun, okay?

 

 

---

Django 1.4 for Android
(only dev. runserver!)

 

First, you have to install few apps.

Some can be installed through Google Play, and others sholud be installed manually by using apk package. Heck, you don’t know about apk? DON’T DO THIS. Headache might overwhelm you.

The following apps being on Google Play:

 

- File Manager

This is for installing apk or in case deleting some file manually. I’m using ES File Explorer as file manager.

 

- Terminal Emulator

Virtual terminal for Android. If you don’t wanna root your phone, I guess there are some ways to do this, can you type all of tedious strings with Android’s virtual keyboard? Forget it. Use SSH, for your precious soul.

 

- SSHDroid

A sort of ssh deamon for Android. It makes basic unix workspace for you with some utilities which don’t exist on Android.

Launching SSHDroid, you can check the device’s IP address. And password for connection is configurable on SSHDroid’s menu. (You might change this)

 

 

 

The following apps are what you should install using apk manually:

 

 

- Rooting tool

This work might be done without rooting, but I have no idea on this. I guess it’s real hell without rooting, just do this. You can find vary tools for your Android – and it depends on what device is and what Android OS version is. So I omitted rooting things here.

If your device is rooted, run SSHDroid on your device,

confirm that you could connect to your device through ssh. If you’re mac user, you could use default terminal program. In case of windows user, you could use ssh client program such a putty. According to above image, 10.0.0.14 is IP address of Android device. If ssh connected, you see messages like these:

Kenials-MBA:~ kenial$ ssh root@10.0.0.14

The authenticity of host '10.0.0.14 (10.0.0.14)' can't be established.
RSA key fingerprint is f7:97:44:c3:ab:49:42:14:db:ec:f2:e2:b8:ae:62:74.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.0.0.14' (RSA) to the list of known hosts.
SSHDroid
Use 'root' on rooted devices otherwise any username works
Default password is 'admin'
root@10.0.0.14's password:

/data/data/berserker.android.apps.sshdroid/home #

 

 

- SL4A (Scripting Layer for Android) http://code.google.com/p/android-scripting/
- P4A (Python for Android) http://code.google.com/p/python-for-android/

You guess what, from these names? These guys make your Android device to execute Python script on it, easily. Django is a web framework written in python, so python environment to run is needed.

Connecting to your device through ssh, install them(I downloaded thme at /mnt/sdcard/_apk)

cd /mnt/sdcard/_apk
wget http://android-scripting.googlecode.com/files/sl4a_r6.apk
wget http://python-for-android.googlecode.com/files/PythonForAndroid_r6.apk
su
pm install sl4a_r6.apk
pm install PythonForAndroid_r6.apk

 

Then, check your apps installed, SL4A and Python for Android.

 

Runs Python for Android app, then tap Install button to install modules:

 

 

If the installation done, run SL4A  app and execute helloworld.py script to check python works correctly.

 

- Edit .profile

This .profile script is for configuration of python environment. In terminal(or ssh client), type following commands to edit .profile:

cd
vi .profile

 

The content of .profile file:

PATH=/data/data/berserker.android.apps.sshdroid/home/bin:$PATH
export PATH
export EXTERNAL_STORAGE=/mnt/sdcard                                                                                                                                                                                                 
export LANG=en
PYTHONPATH=/mnt/sdcard/com.googlecode.pythonforandroid/extras/python
PYTHONPATH=$PYTHONPATH:/data/data/com.googlecode.pythonforandroid/files/python/lib
PYTHONPATH=$PYTHONPATH:/data/data/com.googlecode.pythonforandroid/files/python/lib/python2.6/lib-dynload
export PYTHONPATH
export TEMP=/mnt/sdcard/com.googlecode.pythonforandroid/extras/python/tmp
export PYTHON_EGG_CACHE=$TEMP
export PYTHONHOME=/data/data/com.googlecode.pythonforandroid/files/python
export LD_LIBRARY_PATH=/data/data/com.googlecode.pythonforandroid/files/python/lib

# create /usr path for django-admin.py
if [ ! -d /usr/bin ];
then
     mount -o remount,rw /
     mkdir /usr
     ln /data/data/berserker.android.apps.sshdroid/home/bin /usr/bin –s
fi
sh 

 

Save the file and quit. Then type (or just reconnect to device):

sh .profile

 

Link python to sshdroid’s default path:

ln /data/data/com.googlecode.pythonforandroid/files/python/bin/python /data/data/berserker.android.apps.sshdroid/home/bin/python
ln /data/data/com.googlecode.pythonforandroid/files/python/bin/python /data/data/berserker.android.apps.sshdroid/home/bin/python2.6

 

Try to run python interpreter. If works, you see the result:

 

 

- Django installation

Now (finally!!), it’s time to install Django.

Go /mnt/sdcard/_apk path, download Django 1.4 package file, unzip it, copy the ‘django’ folder to python’s library folder. (Time to unzip could take a long time, just copy unzipped folder from PC directly into sdcard, if you want)

cd /mnt/sdcard/_apk
wget http://www.djangoproject.com/m/releases/1.4/Django-1.4.tar.gz
tar xzvf Django-1.4.tar.gz
cd Django-1.4
cp -r django /data/data/com.googlecode.pythonforandroid/files/python/lib/python2.6/lib-dynload/django     # instead of 'python setup.py install'
ln /data/data/com.googlecode.pythonforandroid/files/python/lib/python2.6/lib-dynload/django/bin/django-admin.py /data/data/berserker.android.apps.sshdroid/home/bin/django-admin.py

Here is one what you remember: In most cases, you can’t install python package using general setup.py script on Android. Android OS doesn’t have various developer’s tools, for example, like gcc, so you need to copy django folder into lib-dynload folder by hand. Then python could load Django modules from lib-dynload folder, using by p4a.

* You can also let this Django folder go to site-packages folder, which is more general for pythonic package. But site-packages of P4A is on sdcard path, might cause slow performance(it depends on what your device is). So I copied it to lib-dynload folder under /data path.

ln command will link django-admin.py to SSHDroid’s PATH enviroment variable, and you can django-admin.py on anywhere path.

 

- Create django project

Wanna take a bite? Go ahead:

cd /mnt/sdcard
django-admin.py startproject django_first
cd django_first
python manage.py runserver 0.0.0.0:8000

 

Check it out running:

 

And bring it on:

 

Done! : )

First post, anyway

It would be good, as possible, to write a flawless post for the very first one. But the first thing is not for perfection.

Anyway, it could be the foundation for it. I wish it could be.