In my last post I demonstrated the use of JSON data type of PostgreSQL. But I skipped a crucial part, Indexes.
All my queries in last post ran on a table with about 24k rows with the JSON tweet data. That import script of mine is still running and I’ve crossed 100k records. I’ll run the same queries again, but this time I’ll create indexes for the appropriate columns before querying.
When working on side projects that deal with 3rd party data, how much time do you spend on figuring out the database schema? Not an insignificant amount. You don’t use a DBMS you say? Good luck parsing the data files and querying them.
I’m not going to start a DBMS comparison. I’ve spent a lot of time fiddling with structred data and PostgreSQL has been really kind to me. It’s stable, really feaure rich and it can literally do anything. You want to setup a cluster? Replication? Crunch geo spatial data? parse Gigabytes of CSV files? And want to do all of these really fast? Then PostgreSQL is for you, it can do all that and more out of the box. Just today a friend told me that PostgreSQL parsed and imported his 4.3GB CSV file in under 10 minutes. I’m happy with that performance.
Holi as a festival kind of sucks for me. I guess that’s mostly because I don’t like the idea of having fun while bathing in artifical colors. So, I have to stay home for at least 3 days when everybody is “spreading the joy” outside. And if I go outside, they will share their “joy” with me too regardless of if I want to be a part of it or not. And they are not very considerate either. We Indians are really strict about that.
This year, the bookfair is happening at Milam Mela ground beside Science city like the last year. Here is the map of Kolkata Book Fair 2014 for your use -
Some more images are in my Flickr gallery.
If you’re new to Kolkata, here is the location of the book fair - OpenStreetMap | Google Maps
Go visit today. Buy some books. :)
And just like that, PyCon India 2013 is over. A lot of hard work and dedications comes to an end for another year. Yet extremely happy because all those hard works yielded really awesome feedback. People have enjoyed those 3 days. That’s all we needed to make sure would happen. :)
I reached B’lore on 28th, changed from the original plan of going on 29th. I thought it’d better to reach with two days of time in hand so that I can help prepare for the event.
This August has been special for all the Python lovers in India for it’s being celebrated as “Python Month” by PyCon India. Volunteers all around the country have been visiting colleges and organizing workshops to promote and spread the awesomeness of Python. There has been few workshops in West Bengal as well. I attended and helped organize two of them. One at the B.P.Poddar Institute of Management and Technology and Netaji Subhash Engineering College.
From the documentation itself -
Tornado is a Python web framework and asynchronous networking library, originally developed at FriendFeed. By using non-blocking network I/O, Tornado can scale to tens of thousands of open connections, making it ideal for long polling, WebSockets, and other applications that require a long-lived connection to each user.
It’s basically a minimalistic web framework which supports non-blocking network operations. If you go through the list of modules it supports, you’ll see that it has a lot of modules built in. But there is no proper tutorial of how to achieve most of the things. All the reference to the API is there though. So you must go ahead, read the API and figure out things yourself.
Just wrote this to someone -
I closed all the other tabs and started working on it, you can’t ask for more focus than that from me.
Why #1? Because there is going to be #2 and maybe more. Most probably not in near future, but someday.
Gist is, I’m heading back to Kolkata for now. Consider this post as a commit message to my life, for the changes in last few months.
First two months here were the worst. The entrepreneur who hired me and convinced me to move here, betting a lot of savings of my parents, kept me sitting idle for a month and a half, and then declared that his idea isn’t working so he’s going after another idea and abandoning ship, without paying me or the my friend who was also on the same boat. That’s two months of no income for me. Of course I couldn’t tell my parents anything. They were already depending a lot on me and their health isn’t that good either. Hence I had to borrow money from some friends and manage somehow. And I’m still under a lot of debt. Thanks Mr Entrepreneur. You’re a real hero.
If you’re here from some search engine, that means you’ve been working with Tornado and WTForms, and you’ve faced the same error -
(Pdb) form = SomeForm(self.request.arguments)
*** TypeError: formdata should be a multidict-type wrapper that supports the 'getlist' method
The thing is, Tornado does provide a dictionary that contains the data like a MultiDict should do for GET/POST/others request arguments -
{'_xsrf': ['b2c0d0bebbf84516a8aa0462bb41ea0f'], 'password': ['password'], 'email': ['mail@gmail.com']}
But it sends it as a dictionary -