Autonomously crawling through DICE job postings
Posted: April 13, 2012 Filed under: Operating Systems, Programming Languages | Tags: Bash, Mac OS X, Mathematica, Perl Leave a comment »I am currently working on a book that requires me to search through thousands of job advertisements. For the last couple of days I have been looking at the various websites, collecting data and looking for patterns in employment listings. Even if you are not working on a book, I am sure at some point in time you will be looking for a new job online. I love searching for jobs, and if you don’t love it too… you are probably doing it wrong.
First of all don’t manually search for jobs! It is a waste of time and it will drive you insane. Instead use a scripting language, such as PERL, that mines website databases and outlines the best matches. In fact I wrote a post a few days ago about mining employment postings on craigslist. If you are new to this entire field of data-mining, I recommend the book “mining the social web” by Russell… Nice chap…. Met him at Harvard Square a couple of years ago.
Here I outline the steps I took to extract all job postings from DICE. First of all you have to know how everything is stored in the database. Make any random search on the initial screen (e.g. embedded).
"Embedded" search on dice.com
This particular search generated the following very-long URL… so long that I had to include spaces:
Since this particular search detected 1689 job postings, we just have to change NUM_PER_PAGE=30 from 30 to 1689, in order to see every single job post on a single page. Save that file into your hard-disk in the HTML format. For completeness, here is the file with all 1689 postings I just downloaded. The following PERL script parses the contents of this file and looks for the associated URL for each job posting.
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 |
#neat function that helps catching scripting errors like undefined variables... use strict; #use the generated HTML as an input argument my $input_file=$ARGV[0]; #parse through every single line in the file open (MYFILE, "<$input_file") or die $!; while (<MYFILE>) { chomp; #each job posting starts with a well defined pattern... #search for <div><a href="/jobsearch/servlet/Jo if (/<div><a href=\"\/jobsearch\/servlet\/Jo/) { #...the following line: #print "$_\n"; #...will print for example: #<div><a href="/jobsearch/servlet/JobSearch?op=302&dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&source=19&FREE_TEXT=embedded&rating=99">ASIC/FPGA Verification Engineer</a></div> #get the job title my $job_title; my @tmp_data; @tmp_data=split(/">|<\/a><\/div>/,$_); $job_title=$tmp_data[1]; #...the following line: #print $job_title . "\n"; #...will print for example: #ASIC/FPGA Verification Engineer @tmp_data=split(/">|div><a href="\//,$_); #...the following line: #print $tmp_data[1] . "\n"; #...will print for example: #jobsearch/servlet/JobSearch?op=302&dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&source=19&FREE_TEXT=embedded&rating=99 #but that is not the correct URL... instead we want something like: #http://seeker.dice.com/jobsearch/servlet/JobSearch?op=302&dockey=xml/b/4/b4ba4b9ed60baf2cf7a3397f336e451e@endecaindex&source=19&FREE_TEXT=embedded&rating=0 #in essence, we must replace all & with & $tmp_data[1] =~ s/\&/\&/g; $tmp_data[1] =~ s/\/jobsearch\/servlet//g; print "http://seeker.dice.com/" . $tmp_data[1] . "\n"; } } close (MYFILE); |
Save the file (e.g. dice.pl) and execute it with the following command:
This will store every single URL, one per line, in the file embedded_url.txt. Once again… for completeness, here is my generated file.
The next step is to download every single job posting onto a separate file. Since, I am a macsox user, I need to download a the contents of each of the URLs from the web via the OS X command line. This is easy accomplished with the following bash script:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 |
FILENAME=embedded_url.txt NUMBERLINES=`wc -l < ${FILENAME}` echo ${NUMBERLINES} > xxx.tmp PNUMBERLINES=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp` rm -f xxx.tmp echo ${PNUMBERLINES} RUN=0 until [ ${RUN} -eq ${PNUMBERLINES} ] do RUN=$(( $RUN + 1 )) LCONTENTS=`sed -n $RUN'p' ${FILENAME}` #echo "line # ${RUN} with contents : ${LCONTENTS}" wget "${LCONTENTS}" done |
On the same directory as the output of the previous PERL script (e.g. embedded_url.txt), save this bash scrip (e.g. download_all_jobs.sh) and execute it with the following commands:
./download_all_jobs.sh embedded_url.txt
The compressed outcome of this last step is a file of 27 MBs.
Now that I have all this data, I need to extract the skills required for each advertised position. So, I placed all the compressed files in the sub-directory dice_jobs and ran the following script:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 |
#!/bin/bash JOBS_DIRECTORY=dice_jobs ls ${JOBS_DIRECTORY}/* > all_jobs.txt #count the number of lines on each file impcount=`wc -l < all_jobs.txt` echo ${impcount} > xxx.tmp NIMPS=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`; rm -f xxx.tmp #rename jobs postings into something more readable RUN=1 until [ $RUN -gt ${NIMPS} ] do FNAME=`sed -n $RUN'p' all_jobs.txt` mv ${FNAME} ${JOBS_DIRECTORY}/${RUN}.txt RUN=$(( $RUN + 1 )) done #extract the job title and necessary skills rm -f parsed_job_data.txt RUN=1 until [ $RUN -gt ${NIMPS} ] do FNAME=`sed -n $RUN'p' all_jobs.txt` perl extract_data.pl ${JOBS_DIRECTORY}/${RUN}.txt >> parsed_job_data.txt RUN=$(( $RUN + 1 )) done rm -f all_jobs.txt echo "...parsed job data is @ parsed_job_data.txt" |
The skill extraction is actually done on the following PERL script (extract_data.pl).
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 |
use strict; #use the generated HTML as an input argument my $input_file=$ARGV[0]; my $next_line_is_job_title=0; my $jobTitle=""; my $next_line_is_area_code=0; my $areaCode=""; my $next_line_is_skills=0; my $skills=""; my $next_line_is_company_name=0; my $companyName=""; #parse through every single line in the file open (MYFILE, "<$input_file") or die $!; while (<MYFILE>) { #chomp; if ($next_line_is_skills==1) { $skills=$_; $skills =~ s/<dd>|<dt>|<\/dt>|<\/dd>| |\t|\n//g; $skills =~ s/\/assets\/images\/detail\/default\/highlite.gif//g; $skills =~ s/<b style="background:url\(//g; $skills =~ s/\); font-weight: bold;">|<\/b>//g; #print "skills=" . $skills . "\n"; $next_line_is_skills=0; } if ($next_line_is_company_name==1) { $companyName=$_; $companyName =~ s/<dd>|<dt>|<\/dt>|<\/dd>| |\t|\n//g; my @splitString=split(/>|</,$companyName); $companyName = $splitString[2]; $next_line_is_company_name=0; #print "companyName = " . $companyName . "\n"; } if ($next_line_is_area_code==1) { $areaCode=$_; $areaCode =~ s/<dd>|<\/dd>|<dt>|<\/dt>| |\t|\n//g; $next_line_is_area_code=0; #print "areaCode=" . $areaCode . "\n"; } if (/<h1 id="jobTitle">/) { my @split_tmp=split(/>|</,$_); $jobTitle=@split_tmp[2]; #print "jobTitle=" . $jobTitle . "\n"; } if (/<dt>Skills:<\/dt>|<dd>Skills<\/dd>/) { $next_line_is_skills=1; } if (/<dt>Company:|<dd>Company/) { $next_line_is_company_name=1; } if (/<dt>Area Code:|<dd>Area Code/) { $next_line_is_area_code=1; } } close (MYFILE); print "$input_file\t$companyName\t$jobTitle\t$areaCode\t$skills\n"; |
I then feed the extracted data into a mathematica script; a (readable) pdf version of the Mathematica script is here, and the source is here. In this script, I combined all found skills, ignored skills that were required in less than 30 distinct advertisement (e.g. COBOL and Pascal). Below is the resulting piechart.
Most requested skills in embedded computing jobs.
As expected the most sought after skills in “embedded computing” jobs are C,C++ and Linux. Java, mysql and kernel development is also very strong in demand these days. Surprisingly I saw lots of mobile computing and networking skill requests. However the most surprisingly requested skills is databases (mysql)!
Finally, I am aware that I could have done everything on this post on a single PERL script. However, writing a post about a single script would get tedious very quickly. I also wanted to save the outcome of every single step in my hard-disk so I could perform some additional data tests, without having to connect to dice.com each time.
