Barack Obama's Inaugural Speech

With the inauguration of, and speech from Barack Hussein Obama, i could not resist the urge to once again play with my (very incomplete) tag cloud generator.

The speech transcript contained 893 unique words, filtered to remove the most common ones with little meaning, and sorted alphabetically.

Size and Color determine the frequency with which words were used overall in the entire speech.

So without any more beating around the Bush (har har):

A clear indication on what his main concerns and topics for future conversation are going to be, with almost all of his most frequently used words projecting a positive connotation.
An interesting observation that the word war was only used twice, as opposed to peace appearing 4 times.

Here follows a list of the highest recurring words and their frequency:

• (4 times) crisis, come, through, power, words, seek, women, peace, up, things, before, whether, greater, men, long, end, meet.
• (5 times) know, nor, generation, spirit, only, day, more.
• (6 times) cannot, time, work, world, too, now, common.
• (7 times) less, people, no, today, america.
• (8 times) because, been, every, must.
• (9 times) these, all
• (11 times) new
• (12 times) nation

GIS, Online Mapping, and Google's Reverse Geocoding

Having recently undergone some training in GIS, online mapping, geospatial databases, and the tools available to create web based map and routing solutions, I researched some free alternatives to achieving embeddable maps online.

I was surprised to find a large community of open source map developers out there.

Google has also recently provided a nice addition to their mapping API, allowing reverse address searches (click for example) using a longitude and latitude, as can be seen on GPS devices, and online maps.

To Quote Google's Documentation:

The term geocoding generally refers to translating a human-readable address into a point on the map. The process of doing the converse, translating a point into a human-readable address, is known as reverse geocoding.

This is very handy for creating functionality where users can interact, returning the exact nearest street address, or any kind of address resolution using only coordinates provided by some other data source. In my opinion a great addition to an already powerful API.

To see more examples of what can be achieved with the free Google Maps API Click Here.

Another powerful mapping solution purely based in javascript can be found at OpenLayers.org

100% open source, the framework is fully extendable, the interface is fully customizable using css and it already supports most major mapping API's such as the: Google Maps API, Yahoo Maps API, Microsoft's Live Virtual Earth API, as well as other server hosted data sources such as the enterprise ArcGIS solution, and UMN's free MapServer alternative.

The tools are there, all they need are someone to put them to good use.

Text Analysis - Continued

To further on my previous post, follows a new screenshot, of a tag cloud sorted alphabetically.
Still using the text from Alice in Wonderland, we now get a much clearer view of which words repeat most often.

Here is the Alphabetically sorted cloud

I have also started eliminating occurrences of very common words, by building up an array with those providing little or no associative meaning. Even so, many words still fall into the low frequency threshold as can be seen by the red tags.

Lastly, no points for style, i am definitely no designer.

Next on the table is some research into psychological concepts, affect in text, emotion portrayed through words, and how it can be determined what feeling is being carried over.

Data + Context = Information

Lately I have found myself gaining a renewed interest in data, how it can be analyzed, and how one can apply context to a data set and extract useful information... as a developer online user driven text content especially tickles my brain.

So much information lies hidden underneath mountains of ascii, where reading between the lines turns out more interesting facts than the text contents itself.

So as step #1 into the info analysis game I decided to code a basic tag cloud generator.
Input a wad of text, split it into words placed in an array, and count the frequency of each word in the array, which is then placed into a new array of unique words along with its frequency.

So now we have a new set of data about the text.... hardly useful at this point.

I then assign a color and size weighting based on the frequency of each word and print the contents to the screen, now I'm seeing something my brain can make some sense of.

Here's what i have so far, based on the plain text version of Alice in Wonderland.

As can be seen, I have not filtered out common words in this screenshot, though it is there I disabled it to get the screen filled nicely. Words below the viewing threshold are those with a frequency less than half the average of all the word frequencies put together.

This average frequency is also used as the weighting for color and size on each word.
I kept the weighting dynamic, so that smaller text bodies will produce a similar output compared to a larger text.

Sorting this list alphabetically rather than by frequency as seen above, makes for a very interesting visual experience more akin to tag clouds.

My next post will likely be of more screens with different interpretations on the same data set, and where to move forward with extracting further useful knowledge from a plaintext.

Breaking The Ice

It's late, i'm tired and i'm not entirely sure what to write as of yet.

Blog post #1.  No clue as to which direction my blog will take or what specifically will be blogged about.  I hazard a guess at seeing this mirror my interests and hobbies.

Perhaps this will be a blog about nothing? Everything? No. People wont go for that.

When next motivation, and the compelling urge to write about something take hold, i will find direction, and hopefully shed some light on what is to come.

Until then.