libspatialite problem with Django on OS X

I have no idea how it comes, using GeoDjango on OS X is a kind of pain in ass, especially when it's about spatialite. It has got better and better - it's better at least rather than the time you should install all related package manually. Finally, our awesome friend Homebrew became to be able to handle most of these cumbersome works, but I should say - it's very close.

You can face this error message when you're trying to make GIS Django application on OS X:

Unable to load the SpatiaLite library extension "/usr/local/lib/libspatialite.dylib"

If you install libspatialite 4.2.0 package by using Homebrew, then it installs libspatialite.dylib and mod_spatialite.dylib into its library path. When django.contrib.gis.db.backends.spatialite is used as a database backend, I'm not sure what happens on extension loading logic of sqlite3, it tries to find `sqlite3_spatialite_init` symbol from libspatialite.dylib. But that symbol dwells in mod_spatialite.dylib, so it fails. You can adjust this by adding SPATIALITE_LIBRARY_PATH to settings.py file:

SPATIALITE_LIBRARY_PATH='/usr/local/lib/mod_spatialite.dylib'

For sure, it can vary with the library path. Real path would be /usr/local/Cellar/libspatialite/x.x.x/lib/. (x.x.x is version number)

Testing C10M on a Host with Tornado

Here's my code that I've used, wanna share it with pleasure :)
https://github.com/kenial/tornado-test-c10m. 

Recently, I've tried to make out to test >1M concurrent connections on my PC, by using Tornado (FYI, Tornado is Python based web framework that supports both TCP / WebSocket connections). As you know, there has been a lot of C10K problem discussions, so you can easily find out articles about tuning of network and performance. (just try to google 'linux c10k tuning,' then you gonna get) Here are few useful links for me to get comprehension about C10K tuning :

- The C10K problem
http://www.kegel.com/c10k.html

- Performance Tuning the Network Stack on Mac OS X Part 2
https://rolande.wordpress.com/2014/05/17/performance-tuning-the-network-stack-on-mac-os-x-part-2/ 

- Linux TCP/IP tuning for scalability
http://www.lognormal.com/blog/2012/09/27/linux-tcpip-tuning/

- RED HAT ENTERPRISE LINUX 7 PERFORMANCE TUNING GUIDE (CentOS compatible)
https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Performance_Tuning_Guide/index.html 

- The C10M(!) problem
http://c10m.robertgraham.com/p/manifesto.html

A caveat here, it requires really vast memory. There is also a trick for >65K connections on loopback network. By using local address and remote address, OS identifies a socket. If you have multiple clients connect to your server that listens 8080 port, created sockets will be as followings:

local address	remote address
192.168.0.10:20001	192.168.0.10:8080
192.168.0.10:20002	192.168.0.10:8080
192.168.0.10:20003	192.168.0.10:8080
192.168.0.10:20004	192.168.0.10:8080

But range of local ports are between 0~65,535, so you can make 65K connections at maximum with one server port, even in an ideal case. In fact, the number could be less.

This could be resolved by letting server listens to multiple ports, like this:

local address	remote address
192.168.0.10:20001	192.168.0.10:8080
192.168.0.10:20001	192.168.0.10:8081
192.168.0.10:20002	192.168.0.10:8080
192.168.0.10:20002	192.168.0.10:8081

As above, if your server listens to multiple ports, then we can have multiple sockets that has a same local address and different remote addresses. With 200 listening ports and 50K local ports, it ends up with 200 * 50,000 = 10,000,000 (10M) connections! (FYI, you can assign multiple IPs instead of multiple listening ports, but it's little cumbersome - you have to do it manually, and need to restore them on some day)

One caveat more: this works on Ubuntu Server 14.10 (my development environment is) for sure, but not for OS X Yosemite. I'm not sure for the reason, but according to result of netstat, sockets on Yosemite seem to be identified by only local address. There no duplicates amongst local addresses as long as I looked into.

Okay, last but not least, here's the result:

- The host that runs test code is Ubuntu Server 14.10, which hosted by Amazon EC2's r3.4xlarge instance that has 122GiB memory.

- 10.5KB per TCP conn, 15.5KB per WebSocket conn.

- If 1M connections, it will be 21GiB for TCP connections and 31GiB for WebSocket connections.
  (211GiB for 10M!)

- PyPy is helpful for performance, especially in establishing connections.

Idea: location-based minisite for smartphone

This is an idea about the application of Mug and Sam.

While "Minisite" refers to a sort of websites which contains information about one specific product or subject, but in this context, minisite represents just a site that contains little amount of contents. If trying to write "location-based minisite" more detail, that means "location-based discoverable minisite in proximity". Mug and Sam are a sort of file sharing tool that is location-based, Sam has a role of host. Mug can discover nearby Sam, and access to files that are shared by Sam.



I am imaging that local businesses such as stores and restaurants use Sam as this location-based minisite. For example, a little photo exhibition at a cafe, and photos are accessible by customers' smartphones or tablets when they visit the cafe only. I know a cafe's owner who loves a travel, and had ever seen photos are on his cafe's wall. A little space was needed for even this little exhibition, if he utilized Sam then the space might not be required.

How would it works? Look at this (they're from my album, at Lake Louise) :

Above one is relevant to images, but audios or videos can be provided with:

If a document is needed, that is, at a information center or a conference, PDF would be fit:

As you see, Mug is basically designed for sharing media files, but you can configure it accessing any web resources:

On above, Mug is accessing Wikipedia page, but it is possible to configured to redirect a certain web site of a store. The benefit is, customers that visit the store can access to a webpage without inputting URL into web browser directly.



Currently, mega internet services are doing business based on their proprietary information that is collected only for itself and by itself. Lots of services - search engine, portal, map service, review site and so forth - have influenced to local business unilaterally.

But I believe IT technology can also help local business to face to this situation in some ways, like providing location-based minisite with unique user experience. And for that purpose, I'm in considering what features are needed additionally to these applications.

Any suggestions?



- Refer to http://www.fountainproject.com/info for detail information.

Mug and Sam, implementations from Fountain Project are released.

link: http://blog.fountainproject.com/post/54057159608/ready-to-go-with

I have been working on a sort of file/media sharing tool, which is location-based service. File sharing & location, yes, they're applications that similar to Bump. But basically, this applications don't require to 'Bump' at least. Once you run it, it just works to find out shared file nearby.

If you're interested in location based file sharing, just look over them. URL of web site is http://www.fountainproject.com


Thanks!

Portable Django & Python for Mac OS X

Recently, I’m working on python-host program. Under the hood, that program hosts python binary, and the django development server works as HTTP API server on python environment. Although it’s not optimized, but useful enough for my purpose.

There’s other projects about this (like as DjangoOnAStick and Portable Python), however, they don’t provide for OS X version and < Python 2.7 versions. So I made up my mind to create something new. (this is based on python 2.7 and django 1.4)

Okay, refer to this URL : https://bitbucket.org/kenial/portable-django

There is shell script(also known batch file on Windows) ready. Use it like this:



Easy, huh?

Bookshelf template: Personal book list management on Google Apps Spreadsheet

* “Bookshelf template” shared link:

https://docs.google.com/spreadsheets/d/1pVoT6l1HdRkN3aPMKAmuNVEqJg4izxTW_vzECIOzavo/edit?usp=sharing 

A friend of me wrote on SNS that she bought a desktop app managing personal book list. As I saw it then thought that ‘Oh I need this one too’ and was considering to buy one in a minute. But, well, I decided to implement it on my own. Features are simple. Input ISBN, book title, or author’s name, and then just run the script. So that the script will find information of the books from Google Books API, and paste them on a sheet. If you need additional fields, just add them as well as need.



How to use: input a part of book information like this and run the script (click 'Get book information from Google Books API' menu), the sheet will be filled out with information of the books.

I was just trying to enhance my Google Apps Script skill, but if you need this, use freely : )

Execute a terminal command on a Cocoa application

* refer to http://stackoverflow.com/questions/412562/execute-a-terminal-command-from-a-cocoa-app/12310154#12310154

In order to write an cocoa application that can start/stop webserver embedded within, I needed some method could run terminal commands. One user(thanks kent!) suggested to use /bin/sh to run shell script file – and you might know, /bin/sh utility can run a line of command instead of script file with –c option. With NSTask and /bin/sh, I made some method to run a shell command by line:

NSString *runCommand(NSString *commandToRun)
{
    NSTask *task;
    task = [[NSTask alloc] init];
    [task setLaunchPath: @"/bin/sh"];

    NSArray *arguments = [NSArray arrayWithObjects:
                          @"-c" ,
                          [NSString stringWithFormat:@"%@", commandToRun],
                          nil];
    NSLog(@"run command: %@",commandToRun);
    [task setArguments: arguments];

    NSPipe *pipe;
    pipe = [NSPipe pipe];
    [task setStandardOutput: pipe];

    NSFileHandle *file;
    file = [pipe fileHandleForReading];

    [task launch];

    NSData *data;
    data = [file readDataToEndOfFile];

    NSString *output;
    output = [[NSString alloc] initWithData: data encoding: NSUTF8StringEncoding];
    return output;
}

Using above method, you can write a code to run a line of terminal shell command like this:

NSString *output = runCommand(@"ps -A | grep mysql");