Autonomously crawling through DICE job postings

I am currently working on a book that requires me to search through thousands of job advertisements. For the last couple of days I have been looking at the various websites, collecting data and looking for patterns in employment listings. Even if you are not working on a book, I am sure at some point in time you will be looking for a new job online. I love searching for jobs, and if you don’t love it too… you are probably doing it wrong.
First of all don’t manually search for jobs! It is a waste of time and it will drive you insane. Instead use a scripting language, such as PERL, that mines website databases and outlines the best matches. In fact I wrote a post a few days ago about mining employment postings on craigslist. If you are new to this entire field of data-mining, I recommend the book “mining the social web” by Russell… Nice chap…. Met him at Harvard Square a couple of years ago.



Here I outline the steps I took to extract all job postings from DICE. First of all you have to know how everything is stored in the database. Make any random search on the initial screen (e.g. embedded).

"Embedded" search on dice.com

This particular search generated the following very-long URL… so long that I had to include spaces:

http://seeker.dice.com/jobsearch/servlet/JobSearch?op=300&N=0&Hf=0&NUM_PER_PAGE=30&Ntk=JobSearchRanking&Ntx=mode+matchall &AREA_CODES=&AC_COUNTRY=1525&QUICK=1&ZIPCODE=&RADIUS=64.37376 &ZC_COUNTRY=0&COUNTRY=1525&STAT_PROV=0&METRO_AREA=33.78715899%2C-84.39164034&TRAVEL=0&TAXTERM=0&SORTSPEC=0&FRMT=0 &DAYSBACK=30&LOCATION_OPTION=2&FREE_TEXT=embedded&WHERE=

Since this particular search detected 1689 job postings, we just have to change NUM_PER_PAGE=30 from 30 to 1689, in order to see every single job post on a single page. Save that file into your hard-disk in the HTML format. For completeness, here is the file with all 1689 postings I just downloaded. The following PERL script parses the contents of this file and looks for the associated URL for each job posting.

Save the file (e.g. dice.pl) and execute it with the following command:

perl dice.pl embedded_Jobs_at_Dice.html > embedded_url.txt

This will store every single URL, one per line, in the file embedded_url.txt. Once again… for completeness, here is my generated file.

The next step is to download every single job posting onto a separate file. Since, I am a macsox user, I need to download a the contents of each of the URLs from the web via the OS X command line. This is easy accomplished with the following bash script:

On the same directory as the output of the previous PERL script (e.g. embedded_url.txt), save this bash scrip (e.g. download_all_jobs.sh) and execute it with the following commands:

chmod u=+rwx download_all_jobs.sh
./download_all_jobs.sh embedded_url.txt

The compressed outcome of this last step is a file of 27 MBs.

Now that I have all this data, I need to extract the skills required for each advertised position. So, I placed all the compressed files in the sub-directory dice_jobs and ran the following script:

The skill extraction is actually done on the following PERL script (extract_data.pl).

I then feed the extracted data into a mathematica script; a (readable) pdf version of the Mathematica script is here, and the source is here. In this script, I combined all found skills, ignored skills that were required in less than 30 distinct advertisement (e.g. COBOL and Pascal). Below is the resulting piechart.

Most requested skills in embedded computing jobs.

As expected the most sought after skills in “embedded computing” jobs are C,C++ and Linux. Java, mysql and kernel development is also very strong in demand these days. Surprisingly I saw lots of mobile computing and networking skill requests. However the most surprisingly requested skills is databases (mysql)!

Finally, I am aware that I could have done everything on this post on a single PERL script. However, writing a post about a single script would get tedious very quickly. I also wanted to save the outcome of every single step in my hard-disk so I could perform some additional data tests, without having to connect to dice.com each time.



Automated craigslist job search with Perl and Bash

Most of my students are in the job market and after suggesting them websites where they could look for jobs, I took a peak at craigslist. I like craigslist; its a simple, bare-bones website with pure text. However, the search functionality is a bit awkward, and it is hard to find a good match between the candidate skills and a particular job posting. If you are seriously looking at every single “filtered” post, it may still take you over an hour to look for the best skill-to-job matches. So I created two scripts, one in Perl and the other one in Bash, that scavenge all the job postings for skill matches, and create a new webpage with all the appropriate positions and matched skills in a ready to click link. Data mining at its best!

There are some “limitations” of these scripts. First of all they were only tested in macOSX and Linux, however I am sure you can convert them quite easily to Windows. Secondly, I’ve focused all the craigslist searches around New England. You may add other craigslist locations quite easily by following the instructions on the perl script.

This automated job search requires two files: search_jobs.sh and craigslist.pl. Both can be found below, or at my github repository. To run the code place both files in the same directory, and edit the search_jobs.sh, shown below, with a text editor (after emacs, my second favorite text editor is TextWrangler). In this file modify the appropriate keywords that are being assigned to the variable SEARCH_SKILLS. Currently the search skills are the standard qualifications for an engineering graduate.

This script runs with the following command line:

chmod u=+rwx search_jobs.sh
./search_jobs.sh

Running the scripts on a macosx terminal window.

After it is done executing it will create two files engineering.html and finance.html, where the candidate can see his best job matches.

Two html files are created with the best job matches

Generated HTML file with the best job matches


Below is the Perl script that parses the craigslist job postings.


BeagleBone tutorial: Blinking LEDs using bash

While what I am demonstrating this post has already been accomplished by others (e.g. Matt Richardson, Akademii, Dan Watts and Brian Jepson), I found it useful for me to go through the motions of writing my own version of the “Blinking LED” project in a BeagleBone board. The following script switches between high and low signals, on several pins, and uses that information to drive some connected LEDs.

Without further ado, here is the code.

To run this code, save it into a file (e.g. pin_io.sh), and type the following commands on your BeagleBone terminal.

chmod u=+rwx pin_io.sh
./pin_io.sh

To actually blink LEDs, connect them to the following pins: P8_3, P8_4, P8_5, P8_11, P8_12, P8_14, P8_15, P8_16, P8_17 P8_20 and P8_21. If you do not know where these pins are, make sure you read the BeagleBone system reference manual. Keep in mind that P8, or Port 8, is the expansion port next to the small user LEDs, and the following digit is the respective pin number.

If you would like to add more LEDs and pins to my source code, you must look into the BeagleBone system reference manual, and search for Table 8: Expansion Header P8 Pinout. Search for the signal names that start with gpio and in my script, insert another line of code after:

Replace P8_21 with a new pin identifier (e.g. P8_26), assign the associated GPIOID (e.g. gpio1[29] for P8_26 ), assign the associated GPIONUMBER (e.g. 61 for P8_26, see footnotes [1] for more information about this) and finally assign the associated GPIOMUX (e.g. gpmc_csn0, which is the MODE0 “name” as described on Table 9-P8 Mux Options Modes 0-3.

[1] GPIONUMBER comes directly from the GPIOID. Consider gpioXX[YY], so the GPIONUMBER is XX*32 + YY. If we have gpio1[29], then the GPIONUMBER is 1*32+29 = 61. Yes, I could have included a function that could calculate the number on the spot, but that would have made the code a lot harder to understand. If you want to include a function that does this, you may use the following code:


File related Linux bash snippets

Here are some of extremely useful Linux bash snippets I use all the time to parse experimental data from my simulations.


How to extract the top (insert number here) lines from a file
Consider a file named test-file.txt. You can extract the top 14 lines from that file using the following:


How to extract specific lines in a file using regular expressions.
Consider a file, named test-file.txt, with the following lines:
_N37_:0:_N262_:1:_N696_:0
_N37_:0:_N233_:0
_N37_:0:_N263_:0:_N694_:0
_N37_:1:_N113_:0

To extract the lines that have 5 elements we can type:

To extract the other lines, we can simply negate that regular expression:


How to search and replace text on a file

Where string1 is what you are searching for, string2 is what you want it to be replaced with and file.txt is the file you want to perform this operation on.


How to print a specific line number inside a file using a variable

Where $LINENBR is the number of the line you want to print.


How to append 2 files, column by column, keeping particular columns


How to transfer files across computers with ssh
The scp command copies files to a remote Linux system.

To copy files from a remote system to your local system:


How to remove a file extension


How to remove redundant lines inside a file
In the terminal type:


How to delete the last line of a file
In the terminal type:


How to count the number of lines in a file and write that number into another file
In the terminal type:

Alternatively you can also store the number of lines into a shell variable.


How to perform the same operation over several files
In a bash script type:

this will print out all filenames via an echo command, that will have the regular expression FAULTY*


How to write a loop inside a bash file


How to split a string inside a bash file


How to do input parameter error testing
The variable ${CIRCUIT} is the first command line argument (${1})


How to perform a particular operation on each line of a file


How to ensure that two files have the same number of lines

Where file1.txt and file2.txt are the filenames you wish to compare.


How to count the number of characters in file
In this particular example, the character I am counting is the 0.


How to replace characters in a file
This particular command will replace all “:” with the newline character.


How to delete all instances of a particular character from a file
This particular command will delete all “:” from the file longString.txt and it will write it on the file readableString.txt.

Read the rest of this entry »