Author: Blair C. Armstrong, September 2008


THE PROGRAMS:

Two pieces of code are used to generate the final summary output for words in the Wordsmyth dictionary: a spider and a parser.
Both were written in python.





WORDSMYTH SPIDER:
This piece of code crawls the wordsmyth site and downloads the entries for the words in the dictionary.   All of the retrieving was done on Sept 3, 2008.  At that time, the database at Wordsmyth appeared to max out at 49361 entries.  Some of the entries do not appear to be valid however, in that there is no word in the Dictionary at that index.  A couple of these turned out to be html errors, but the vast majority clearly are not.  Perhaps some empty entries in their sql database?





WORDSMYTH PARSER:

This parses the pages downloaded by the spider into cleaned definition files and also creates a summary database file suitable for import into Excel or other similar spreadsheet programs.  Simply run the script to create a directory in which each definition is saved in both separate files fore each meaning and part of speech and in a file that merges the definitions across all meanings and parts of speech (*.all.*).  NOTE: The definitions in these files are supposedly rank ordered on the basis of their frequency of use by the Wordsmyth lexographers, which Armstrong, Tokowicz, & Plaut (2012) showed to be correct in basic terms.

These files are named based on <wordname>, <single word or phrase>, <meaning number>, <part of speech>.  


A file which lists various properties of the words is also generated.  This file contains multiple sources of information.  HEADER.txt describes what each column in this file represents.

While the program runs you can collect lots of useful information about how the execution is proceeeding based on the standard output. You may wish to pipe this to a file to keep on track of what is going on.  

An error log is also generated that lists the errors that are encountered and the files that caused the errors.  


=====A note on some of the errors that are output to the log.======

A small number of files (~1500) have problems, in that no meanings are detected by the parser.  The vast majority of these are phrases that are not actually defined in wordsmyth.  I tried recovering a few of them and perhaps a small portion were html errors, but most of the ones I checked were just bad entries in the dictionary.  Note that in light of this, it is best to manually inspect the parsed definition files upon which the summary statistics are generated if the word set of interest is relatively small.




PARSER OUTPUT:
The main output file that you'll want to look at is in:

WOrd_meaning_sense_POS_cfRodd2002.txt

THis file contains the output of the processed text.  Headers for this file are located in HEADER.txt.

