god_writes_good_code.py

# Copyright (c) 2010 DBW.
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
 
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.

# pygmentize -f html -o god_writes_good_code.html god_writes_good_code.py
# More info @ http://burstcoding.blogspot.com/2010/04/rendering-code-via-pygments-in-your.html

import nltk
from nltk.corpus import PlaintextCorpusReader as PtCr

import numpy
import os


def cache_url(url, gunzip=True):
    """fetch the url locally if not already local.
    gunzip - {True,False} - pass through gunzip or not
    """

    filename = os.path.split(url)[-1]
    ext = os.path.splitext(url)[-1]

    # will we be unzipping?
    if gunzip and ext==".gz":
        unzipped = filename[:-3]
        if not os.path.exists(unzipped):
            err = os.system("gunzip -f %s" % filename)
            if err!=0:
                raise OSError, "gunzip error on file: %s" % filename
    
        return unzipped


    elif not os.path.exists(filename):
        err = os.system('wget %s' % url)
        if err!=0:
            raise OSError, "wget error on url: %s" % url

    return filename


def draw_from_cfdist(cfdist):
    
    values = numpy.array(cfdist.values())
    keys = cfdist.keys()
    a = numpy.add.accumulate(values)
    # draw random integer upto end of a+1
    rndi = numpy.random.randint(a[-1]+1)
    idx = a.searchsorted(rndi)
    return keys[idx]
    
def generate_model(cfdist, word, num=15):

    for i in range(num):
        print word, 
        word = draw_from_cfdist(cfdist[word])


if __name__=="__main__":

    # King James bible and New Hacker's Dictionary.
    sources = {'kjv':'http://www.ccim.org/~bible/kjv.rawtxt.gz',
               'nhd':'http://catb.org/jargon/oldversions/jarg2912.txt'}

    urls = sources.values()

    # cache the urls and get filenames
    filenames = []
    for url in urls:
        filenames.append(cache_url(url))

    source_files = dict(zip(sources.keys(),filenames))

    # Make a corpus out of local cache
    local_corpus = PtCr('.',source_files.values())

    words = reduce(lambda x,y: x+y, [local_corpus.words(source_files[src]) for src in source_files])

    print "Generating bigrams."

    bigrams = nltk.bigrams(words)

    print "Generating cfd."

    cfd = nltk.ConditionalFreqDist(bigrams)


    for i in range(20):
        generate_model(cfd,'God')
        print "\n"

Utterings of god_writes_good_code.py

God. It is a PDP - paired keyboard macros or parents, that sin
God rather limited their Xerox PARC
God hath put them a sagan
God be willing to construct
God, my roaring lion, and how the universal constructor
God Righteous featurectomies are only if the sour wonton soup is shed innocent
God of mercy on. A pastime is no pleasure in this naming convention
God doth not till all unwanted notoriety
God is cruft: and would cleave unto the LORD, and see the city
God. Female hackers at midnight, and praying, lest my son
God of Solomon gave him: black art not profit margin
God is usually generalizations of a reply, and hast paid the congregation
God, saying and forty years of socks: I am that
God of God our uncomely parts of language
God of rocks I am God, the Chinese Army technique, I conceived
God; and none other; and the knees to his son of Waterloo
God hath been with thee, it, and came to thee ? for science
God of the book of a cloudy pillar stand upon the next day: 9
God in hackish sense 2 Go not unto Taanathshiloh, and his feet.
God: so I will; and with her suburbs, so, whatsoever is.
God, / n. Hackers are essentially dead ? And Moses
God is hardened, Fear not himself for his neighbour in all his mercy
God made loops with us
God: I gather it into a procedure to keep under
God of electric service attack with the floppy disk



-- (c) 2010 DBW, all rights reserved.

Foray into Natural Language Processing

Python 2.6.4 (r264:75706, Dec 7 2009, 18:43:55)
[GCC 4.4.1] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import nltk
>>> nltk.chat.chatbots()

Which chatbot would you like to talk to?
1: Eliza (psycho-babble)
2: Iesha (teen anime junky)
3: Rude (abusive bot)
4: Suntsu (Chinese sayings)
5: Zen (gems of wisdom)

Enter a number in the range 1-5: 4

Talk to the program by typing in plain English, using normal upper-
and lower-case letters and punctuation. Enter "quit" when done.
========================================================================
You seek enlightenment?
>indeed
One may know how to conquer without being able to do it.
>why conquer?
Ask me no questions!
>I seek enlightenment
Heaven signifies night and day, cold and heat, times and seasons.
>And conquer
Attack him where he is unprepared, appear where you are not expected.
>is it worthwhile?
Please consider whether you can answer your own question.
>I cannot
The good fighters of old first put themselves beyond the possibility of defeat, and then waited for an opportunity of defeating the enemy.
>
>>> nltk.chat.chatbots()
Which chatbot would you like to talk to?
1: Eliza (psycho-babble)
2: Iesha (teen anime junky)
3: Rude (abusive bot)
4: Suntsu (Chinese sayings)
5: Zen (gems of wisdom)

Enter a number in the range 1-5: 5
***************************************************************************
Zen Chatbot!
***************************************************************************
"Look beyond mere words and letters - look into your mind"
* Talk your way to truth with Zen Chatbot.
* Type 'quit' when you have had enough.
***************************************************************************
Welcome, my child.
>Hello
Greetings. I sense your mind is troubled. Tell me of your troubles.
>Why conquer?
Why do you think conquer?
>One school might argue that it is a necessity of life
Form is emptiness, and emptiness is form.
>Conquer life, and life conquers us
The reverse side also has a reverse side.
>Submit to life, and life submits to us
When you're enlightened, every word is wisdom.
>Amen
Random talk is useless.
>Amen
The reverse side also has a reverse side.
>Random talk is useless
Form is emptiness, and emptiness is form.
>And so it is revealed that you are only the shadow of a zen master
I pour out a cup of water. Is the cup empty?

What would be awesome?

ksnapshot -> tesseract -> klipper

with programming language syntax hinting support.

Use case: Navigate to a book on google books,



press "print screen", highlight code and it appears in your clipboard.

Update:
In fact, this book is online here.

OCR in Polish on Linux

A mental note to myself on getting text from images, and using a non-standard dictionary.

Software of choice: tesseract, gimagereader, tesseract-polish, Python, emacs.

Installation:

- kubuntu 9.04
- sudo apt-get install tesseract-ocr tesseract-ocr-eng tesseract-ocr-deu tesseract-ocr-fra
- get tesseract-polish source and copy pol.* to /usr/share/tesseract-ocr/tessdata/

gImageReader:

is a quick and dirty GUI frontend to tesseract, but it does the job very well. I got the source ... careful, the tar does not include a main truck "gimagereader" so create one and extract it in there. Edit src/main.py to add Polish dictionary @ lines 239 and 256 add ("pol","Polish") and ("pol","Polish","pl_PL") respectively. Edit src/config.py @ lines 111-116: following the if ... !="/" change outer get_text to set_text, then "make install".

$ gimagereader

In the dialog asking for dictionary files, enter /usr/share/tesseract-ocr/tessdata/, apply.

My images were generated from a pdf using gimp pdf import at 200 DPI, higher resolutions seem to have caused tesseract some problems, such that gimagereader "recognize" found no text.

Process is as follows:

1. Open image

2. In the langauge dropdown in the toolbar, select Polish, the new dictionary we added.

3. Select the column of text to be recognized, and click "recognize". Repeat for additional columns. To the right, in the text box that appears, the OCR'd text will appear. Though the "Insert At cursor" is a sane default, allowing regions to be recognized in sequence, other options are available. Save.