Password protecting entire directories in MacOSX

Posted: June 26, 2012 | Author: Nuno Alves | Filed under: Operating Systems, Programming Languages | Tags: Expect, Mac OS X | Leave a comment »

Every once in a while I have some files in a directory that need a password. I am not looking for a fancy encryption mechanism like PGP… I just want to compress the entire directory, put in a strong password and forget about it. Anyway, there are several applications that do this on the mac store… but none provide the flexibility that I was looking for. Plus, I realized this can be done in a couple of lines in the terminal.

Lets say you have a directory tmp with a bunch of files you want to compress and password protect it. You can go the standard route using the zip utility. Just type the following:

zip storage.zip -R tmp/* -e

This will compress the directory tmp/ and store all the files in the storage.zip. It will also ask you for the password you would like to use. Unfortunately you cannot give the password as an argument. So, you are stuck typing the password every time you want to compress a new directory.

There is a hack you can use to bypass the manually password submission… It involves using expect. This program is a really neat utility. It basically allows you to create interactive dialogues with your terminal programs, which makes task automation a walk in the park. You can check its manual pages here.

Step #1: Go into the mac terminal and find out where the expect utility is located. This is done using the whereais command. Type the following on your terminal window.

whereis expect

In my macosx version, the program expect is located at /usr/bin/expect.
Step #2: Create a file (e.g. protect_directory.sh) with the following lines. Make sure you modify the location of the expect program in the first line of the script.

#!/usr/bin/expect -f
eval spawn zip [lindex $argv 0].zip -R [lindex $argv 1]/* -e
expect "password:"
send "[lindex $argv 2]\r"
expect "password:"
send "[lindex $argv 2]\r"
interact

#!/usr/bin/expect -f

eval spawn zip [lindex $argv 0].zip -R [lindex $argv 1]/* -e

expect "password:"

send "[lindex $argv 2]\r"

expect "password:"

send "[lindex $argv 2]\r"

interact

Step #3: Make sure your script is executable and run it.

chmod u=+rwx protect_directory.sh
./protect_directory.sh file.zip tmp hello

The first argument will be the output filename (file.zip), the second argument is the directory you wish to compress (tmp) and the last argument is the password you wish to use (hello). Neat no? Here is a screenshot of all the steps.

Autonomously crawling through DICE job postings

Posted: April 13, 2012 | Author: Nuno Alves | Filed under: Operating Systems, Programming Languages | Tags: Bash, Mac OS X, Mathematica, Perl | Leave a comment »

I am currently working on a book that requires me to search through thousands of job advertisements. For the last couple of days I have been looking at the various websites, collecting data and looking for patterns in employment listings. Even if you are not working on a book, I am sure at some point in time you will be looking for a new job online. I love searching for jobs, and if you don’t love it too… you are probably doing it wrong.
First of all don’t manually search for jobs! It is a waste of time and it will drive you insane. Instead use a scripting language, such as PERL, that mines website databases and outlines the best matches. In fact I wrote a post a few days ago about mining employment postings on craigslist. If you are new to this entire field of data-mining, I recommend the book “mining the social web” by Russell… Nice chap…. Met him at Harvard Square a couple of years ago.

Here I outline the steps I took to extract all job postings from DICE. First of all you have to know how everything is stored in the database. Make any random search on the initial screen (e.g. embedded).

"Embedded" search on dice.com

This particular search generated the following very-long URL… so long that I had to include spaces:

http://seeker.dice.com/jobsearch/servlet/JobSearch?op=300&N=0&Hf=0&NUM_PER_PAGE=30&Ntk=JobSearchRanking&Ntx=mode+matchall &AREA_CODES=&AC_COUNTRY=1525&QUICK=1&ZIPCODE=&RADIUS=64.37376 &ZC_COUNTRY=0&COUNTRY=1525&STAT_PROV=0&METRO_AREA=33.78715899%2C-84.39164034&TRAVEL=0&TAXTERM=0&SORTSPEC=0&FRMT=0 &DAYSBACK=30&LOCATION_OPTION=2&FREE_TEXT=embedded&WHERE=

Since this particular search detected 1689 job postings, we just have to change NUM_PER_PAGE=30 from 30 to 1689, in order to see every single job post on a single page. Save that file into your hard-disk in the HTML format. For completeness, here is the file with all 1689 postings I just downloaded. The following PERL script parses the contents of this file and looks for the associated URL for each job posting.

#neat function that helps catching scripting errors like undefined variables...
use strict;

#use the generated HTML as an input argument
my $input_file=$ARGV[0];

#parse through every single line in the file
open (MYFILE, "<$input_file") or die $!;
 while (<MYFILE>) {
 	chomp;
 	#each job posting starts with a well defined pattern...
	#search for <div><a href="/jobsearch/servlet/Jo
	if (/<div><a href=\"\/jobsearch\/servlet\/Jo/)
	{
		#...the following line:
		#print "$_\n";
		#...will print for example:
		#<div><a href="/jobsearch/servlet/JobSearch?op=302&amp;dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&amp;source=19&amp;FREE_TEXT=embedded&amp;rating=99">ASIC/FPGA Verification Engineer</a></div>

		#get the job title
		my $job_title;
		my @tmp_data;

		@tmp_data=split(/">|<\/a><\/div>/,$_);		
		$job_title=$tmp_data[1];
		#...the following line:
		#print $job_title . "\n";
		#...will print for example:
		#ASIC/FPGA Verification Engineer


		@tmp_data=split(/">|div><a href="\//,$_);
		#...the following line:
		#print $tmp_data[1] . "\n";	
		#...will print for example:	#jobsearch/servlet/JobSearch?op=302&amp;dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&amp;source=19&amp;FREE_TEXT=embedded&amp;rating=99

		#but that is not the correct URL... instead we want something like:	#http://seeker.dice.com/jobsearch/servlet/JobSearch?op=302&dockey=xml/b/4/b4ba4b9ed60baf2cf7a3397f336e451e@endecaindex&source=19&FREE_TEXT=embedded&rating=0

		#in essence, we must replace all &amp; with &
		$tmp_data[1] =~ s/\&amp;/\&/g;
		$tmp_data[1] =~ s/\/jobsearch\/servlet//g;
		print "http://seeker.dice.com/" . $tmp_data[1] . "\n";
	}
 }
 close (MYFILE);

#neat function that helps catching scripting errors like undefined variables...

use strict;

#use the generated HTML as an input argument

my $input_file=$ARGV[0];

#parse through every single line in the file

open (MYFILE, "<$input_file") or die $!;

while (<MYFILE>) {

chomp;

#each job posting starts with a well defined pattern...

#search for <div><a href="/jobsearch/servlet/Jo

if (/<div><a href=\"\/jobsearch\/servlet\/Jo/)

{

#...the following line:

#print "$_\n";

#...will print for example:

#<div><a href="/jobsearch/servlet/JobSearch?op=302&dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&source=19&FREE_TEXT=embedded&rating=99">ASIC/FPGA Verification Engineer</a></div>

#get the job title

my $job_title;

my @tmp_data;

@tmp_data=split(/">|<\/a><\/div>/,$_);

$job_title=$tmp_data[1];

#...the following line:

#print $job_title . "\n";

#...will print for example:

#ASIC/FPGA Verification Engineer

@tmp_data=split(/">|div><a href="\//,$_);

#...the following line:

#print $tmp_data[1] . "\n";

#...will print for example: #jobsearch/servlet/JobSearch?op=302&dockey=xml/7/0/70b58be12a872e8268939a525389b927@endecaindex&source=19&FREE_TEXT=embedded&rating=99

#but that is not the correct URL... instead we want something like: #http://seeker.dice.com/jobsearch/servlet/JobSearch?op=302&dockey=xml/b/4/b4ba4b9ed60baf2cf7a3397f336e451e@endecaindex&source=19&FREE_TEXT=embedded&rating=0

#in essence, we must replace all & with &

$tmp_data[1] =~ s/\&/\&/g;

$tmp_data[1] =~ s/\/jobsearch\/servlet//g;

print "http://seeker.dice.com/" . $tmp_data[1] . "\n";

}

close (MYFILE);

Save the file (e.g. dice.pl) and execute it with the following command:

perl dice.pl embedded_Jobs_at_Dice.html > embedded_url.txt

This will store every single URL, one per line, in the file embedded_url.txt. Once again… for completeness, here is my generated file.

The next step is to download every single job posting onto a separate file. Since, I am a macsox user, I need to download a the contents of each of the URLs from the web via the OS X command line. This is easy accomplished with the following bash script:

FILENAME=embedded_url.txt

NUMBERLINES=`wc -l <  ${FILENAME}`
echo ${NUMBERLINES} > xxx.tmp
PNUMBERLINES=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`
rm -f xxx.tmp
echo ${PNUMBERLINES}

RUN=0
until [ ${RUN} -eq ${PNUMBERLINES} ]
do
    RUN=$(( $RUN + 1 ))
    LCONTENTS=`sed -n $RUN'p' ${FILENAME}`
    #echo "line # ${RUN} with contents : ${LCONTENTS}"
    wget "${LCONTENTS}"
done

FILENAME=embedded_url.txt

NUMBERLINES=`wc -l < ${FILENAME}`

echo ${NUMBERLINES} > xxx.tmp

PNUMBERLINES=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`

rm -f xxx.tmp

echo ${PNUMBERLINES}

RUN=0

until [ ${RUN} -eq ${PNUMBERLINES} ]

RUN=$(( $RUN + 1 ))

LCONTENTS=`sed -n $RUN'p' ${FILENAME}`

#echo "line # ${RUN} with contents : ${LCONTENTS}"

wget "${LCONTENTS}"

done

On the same directory as the output of the previous PERL script (e.g. embedded_url.txt), save this bash scrip (e.g. download_all_jobs.sh) and execute it with the following commands:

chmod u=+rwx download_all_jobs.sh
./download_all_jobs.sh embedded_url.txt

The compressed outcome of this last step is a file of 27 MBs.

Now that I have all this data, I need to extract the skills required for each advertised position. So, I placed all the compressed files in the sub-directory dice_jobs and ran the following script:

#!/bin/bash
JOBS_DIRECTORY=dice_jobs

ls ${JOBS_DIRECTORY}/* > all_jobs.txt

#count the number of lines on each file
impcount=`wc -l <  all_jobs.txt`
echo ${impcount} > xxx.tmp
NIMPS=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`;
rm -f xxx.tmp

#rename jobs postings into something more readable
RUN=1
until [ $RUN -gt ${NIMPS} ]
do
	FNAME=`sed -n $RUN'p' all_jobs.txt`	
	mv ${FNAME} ${JOBS_DIRECTORY}/${RUN}.txt
	RUN=$(( $RUN + 1 ))
done

#extract the job title and necessary skills
rm -f parsed_job_data.txt
RUN=1
until [ $RUN -gt ${NIMPS} ]
do
	FNAME=`sed -n $RUN'p' all_jobs.txt`	
	perl extract_data.pl ${JOBS_DIRECTORY}/${RUN}.txt >> parsed_job_data.txt
	RUN=$(( $RUN + 1 ))
done

rm -f all_jobs.txt
echo "...parsed job data is @ parsed_job_data.txt"

#!/bin/bash

JOBS_DIRECTORY=dice_jobs

ls ${JOBS_DIRECTORY}/* > all_jobs.txt

#count the number of lines on each file

impcount=`wc -l < all_jobs.txt`

echo ${impcount} > xxx.tmp

NIMPS=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`;

rm -f xxx.tmp

#rename jobs postings into something more readable

RUN=1

until [ $RUN -gt ${NIMPS} ]

FNAME=`sed -n $RUN'p' all_jobs.txt`

mv ${FNAME} ${JOBS_DIRECTORY}/${RUN}.txt

RUN=$(( $RUN + 1 ))

done

#extract the job title and necessary skills

rm -f parsed_job_data.txt

RUN=1

until [ $RUN -gt ${NIMPS} ]

FNAME=`sed -n $RUN'p' all_jobs.txt`

perl extract_data.pl ${JOBS_DIRECTORY}/${RUN}.txt >> parsed_job_data.txt

RUN=$(( $RUN + 1 ))

done

rm -f all_jobs.txt

echo "...parsed job data is @ parsed_job_data.txt"

The skill extraction is actually done on the following PERL script (extract_data.pl).

use strict;

#use the generated HTML as an input argument
my $input_file=$ARGV[0];

my $next_line_is_job_title=0;
my $jobTitle="";

my $next_line_is_area_code=0;
my $areaCode="";

my $next_line_is_skills=0;
my $skills="";

my $next_line_is_company_name=0;
my $companyName="";

#parse through every single line in the file
open (MYFILE, "<$input_file") or die $!;
 while (<MYFILE>) {
 	#chomp;
	
	if ($next_line_is_skills==1)
	{
		$skills=$_;
		$skills =~ s/<dd>|<dt>|<\/dt>|<\/dd>|&nbsp;|\t|\n//g;
		$skills =~ s/\/assets\/images\/detail\/default\/highlite.gif//g;
		$skills =~ s/<b style="background:url\(//g;
		$skills =~ s/\); font-weight: bold;">|<\/b>//g;
		#print "skills=" . $skills . "\n";
		$next_line_is_skills=0;
	}
	
	if ($next_line_is_company_name==1)
	{
		$companyName=$_;
		$companyName =~ s/<dd>|<dt>|<\/dt>|<\/dd>|&nbsp;|\t|\n//g;
		my @splitString=split(/>|</,$companyName);
		$companyName = $splitString[2];
		$next_line_is_company_name=0;

		#print "companyName = " . $companyName . "\n";
	}

	if ($next_line_is_area_code==1)
	{
		$areaCode=$_;
		$areaCode =~ s/<dd>|<\/dd>|<dt>|<\/dt>|&nbsp;|\t|\n//g;
		$next_line_is_area_code=0;

		#print "areaCode=" . $areaCode . "\n";
	}

	if (/<h1 id="jobTitle">/)
	{
		my @split_tmp=split(/>|</,$_);
		$jobTitle=@split_tmp[2];

		#print "jobTitle=" . $jobTitle . "\n";
	}
	
	if (/<dt>Skills:<\/dt>|<dd>Skills<\/dd>/) { $next_line_is_skills=1; }
	
	if (/<dt>Company:|<dd>Company/) { $next_line_is_company_name=1; }
	
	if (/<dt>Area Code:|<dd>Area Code/) { $next_line_is_area_code=1; }
	
 }
 close (MYFILE); 
 
 print "$input_file\t$companyName\t$jobTitle\t$areaCode\t$skills\n";

use strict;

#use the generated HTML as an input argument

my $input_file=$ARGV[0];

my $next_line_is_job_title=0;

my $jobTitle="";

my $next_line_is_area_code=0;

my $areaCode="";

my $next_line_is_skills=0;

my $skills="";

my $next_line_is_company_name=0;

my $companyName="";

#parse through every single line in the file

open (MYFILE, "<$input_file") or die $!;

while (<MYFILE>) {

#chomp;

if ($next_line_is_skills==1)

{

$skills=$_;

$skills =~ s/<dd>|<dt>|<\/dt>|<\/dd>| |\t|\n//g;

$skills =~ s/\/assets\/images\/detail\/default\/highlite.gif//g;

$skills =~ s/<b style="background:url\(//g;

$skills =~ s/\); font-weight: bold;">|<\/b>//g;

#print "skills=" . $skills . "\n";

$next_line_is_skills=0;

}

if ($next_line_is_company_name==1)

{

$companyName=$_;

$companyName =~ s/<dd>|<dt>|<\/dt>|<\/dd>| |\t|\n//g;

my @splitString=split(/>|</,$companyName);

$companyName = $splitString[2];

$next_line_is_company_name=0;

#print "companyName = " . $companyName . "\n";

}

if ($next_line_is_area_code==1)

{

$areaCode=$_;

$areaCode =~ s/<dd>|<\/dd>|<dt>|<\/dt>| |\t|\n//g;

$next_line_is_area_code=0;

#print "areaCode=" . $areaCode . "\n";

}

if (/<h1 id="jobTitle">/)

{

my @split_tmp=split(/>|</,$_);

$jobTitle=@split_tmp[2];

#print "jobTitle=" . $jobTitle . "\n";

}

if (/<dt>Skills:<\/dt>|<dd>Skills<\/dd>/) { $next_line_is_skills=1; }

if (/<dt>Company:|<dd>Company/) { $next_line_is_company_name=1; }

if (/<dt>Area Code:|<dd>Area Code/) { $next_line_is_area_code=1; }

}

close (MYFILE);

print "$input_file\t$companyName\t$jobTitle\t$areaCode\t$skills\n";

I then feed the extracted data into a mathematica script; a (readable) pdf version of the Mathematica script is here, and the source is here. In this script, I combined all found skills, ignored skills that were required in less than 30 distinct advertisement (e.g. COBOL and Pascal). Below is the resulting piechart.

Most requested skills in embedded computing jobs.

As expected the most sought after skills in “embedded computing” jobs are C,C++ and Linux. Java, mysql and kernel development is also very strong in demand these days. Surprisingly I saw lots of mobile computing and networking skill requests. However the most surprisingly requested skills is databases (mysql)!

Finally, I am aware that I could have done everything on this post on a single PERL script. However, writing a post about a single script would get tedious very quickly. I also wanted to save the outcome of every single step in my hard-disk so I could perform some additional data tests, without having to connect to dice.com each time.

Automated craigslist job search with Perl and Bash

Posted: March 30, 2012 | Author: Nuno Alves | Filed under: Operating Systems, Programming Languages | Tags: Bash, Linux, Mac OS X, Perl | Leave a comment »

Most of my students are in the job market and after suggesting them websites where they could look for jobs, I took a peak at craigslist. I like craigslist; its a simple, bare-bones website with pure text. However, the search functionality is a bit awkward, and it is hard to find a good match between the candidate skills and a particular job posting. If you are seriously looking at every single “filtered” post, it may still take you over an hour to look for the best skill-to-job matches. So I created two scripts, one in Perl and the other one in Bash, that scavenge all the job postings for skill matches, and create a new webpage with all the appropriate positions and matched skills in a ready to click link. Data mining at its best!

There are some “limitations” of these scripts. First of all they were only tested in macOSX and Linux, however I am sure you can convert them quite easily to Windows. Secondly, I’ve focused all the craigslist searches around New England. You may add other craigslist locations quite easily by following the instructions on the perl script.

This automated job search requires two files: search_jobs.sh and craigslist.pl. Both can be found below, or at my github repository. To run the code place both files in the same directory, and edit the search_jobs.sh, shown below, with a text editor (after emacs, my second favorite text editor is TextWrangler). In this file modify the appropriate keywords that are being assigned to the variable SEARCH_SKILLS. Currently the search skills are the standard qualifications for an engineering graduate.

#!/bin/bash
#################################################################################
#Functions (do not modify anything here)
#################################################################################
SEARCH_SKILLS=""

function search_jobs
{
	SEARCH_NAME=${1}
	perl craigslist.pl ${SEARCH_SKILLS}> raw_data.txt

	#create the header of an html file
	echo "<html><title>Job Search Results</title><body>" > job_data.html

	#sort the entire file contents and make sure the best matches are on top
	sort -t! -n -r -k3 raw_data.txt >> job_data.html

	#clean up the file
	perl -p -i -e "s/!/\&nbsp;\&nbsp;\&nbsp;\&nbsp;\&nbsp;\&nbsp;/g" job_data.html

	#terminate the html file
	echo "</body></html>" >> job_data.html
	
	mv job_data.html ${SEARCH_NAME}.html
	rm -f raw_data.txt	
}

#################################################################################
#You may modify your skills below
#################################################################################
SEARCH_SKILLS="embedded, circuit, transistor, VLSI, firmware, RTOS, kernel, MacOSX, JTAG, oscilloscope, HDL, FPGA, Arduino, MSP430, OMAP3540, micro-controllers, microcontrollers, SVN, programmer, Perl, linux, Mathematica, LabVIEW, schematics, Verilog, VHDL"
SEARCH_NAME="engineering"
search_jobs ${SEARCH_NAME}

SEARCH_SKILLS="quantitative, mathematica, finance, programmer, developer, high-frequency, fpga, microcontroller"
SEARCH_NAME="finance"
search_jobs ${SEARCH_NAME}

#!/bin/bash

#################################################################################

#Functions (do not modify anything here)

#################################################################################

SEARCH_SKILLS=""

function search_jobs

{

SEARCH_NAME=${1}

perl craigslist.pl ${SEARCH_SKILLS}> raw_data.txt

#create the header of an html file

echo "<html><title>Job Search Results</title><body>" > job_data.html

#sort the entire file contents and make sure the best matches are on top

sort -t! -n -r -k3 raw_data.txt >> job_data.html

#clean up the file

perl -p -i -e "s/!/\ \ \ \ \ \ /g" job_data.html

#terminate the html file

echo "</body></html>" >> job_data.html

mv job_data.html ${SEARCH_NAME}.html

rm -f raw_data.txt

}

#################################################################################

#You may modify your skills below

#################################################################################

SEARCH_SKILLS="embedded, circuit, transistor, VLSI, firmware, RTOS, kernel, MacOSX, JTAG, oscilloscope, HDL, FPGA, Arduino, MSP430, OMAP3540, micro-controllers, microcontrollers, SVN, programmer, Perl, linux, Mathematica, LabVIEW, schematics, Verilog, VHDL"

SEARCH_NAME="engineering"

search_jobs ${SEARCH_NAME}

SEARCH_SKILLS="quantitative, mathematica, finance, programmer, developer, high-frequency, fpga, microcontroller"

SEARCH_NAME="finance"

search_jobs ${SEARCH_NAME}

This script runs with the following command line:

chmod u=+rwx search_jobs.sh
./search_jobs.sh

Running the scripts on a macosx terminal window.

After it is done executing it will create two files engineering.html and finance.html, where the candidate can see his best job matches.

Two html files are created with the best job matches

Generated HTML file with the best job matches

Below is the Perl script that parses the craigslist job postings.

#This script fetches the last 2 days new job postings from craigslist that match  
#a specific criteria and reports the URLs that correspond to that match.
#The search criteria comes from the input arguments. The cragislist sites
#are hardwired to the New England area. You may change them by manually 
#altering the variables in Section #3.
#
#Version 0.2 30/march/2012
#Author: Nuno Alves
#
#############################################################################
#Section #1 - load libraries
#############################################################################
use strict;
use POSIX;
use LWP::Simple;

#############################################################################
#Section #2 - input arguments are your skillsets
#############################################################################
my $num_args = $#ARGV + 1;
if ($num_args == 1) { print "You must add some skills as arguments\n"; exit; }

#############################################################################
#Section #3 - defining variables
#############################################################################
#what cragislist sites 
my @search_site=("http://boston.craigslist.org","http://nh.craigslist.org","http://maine.craigslist.org","http://burlington.craigslist.org","http://westernmass.craigslist.org","http://worcester.craigslist.org");

#type what positions you are looking for (egr = engineering, sof = software)
my @positions=("egr","sof","bus","acc");

#this array contains the arguments which are your resume skills
my @skills=@ARGV ;

#############################################################################
#Section #4 - debug code
#############################################################################
#instead of work on every single URL, setting $debug=1, will just scan
#two webpages
my $debug=0; 
my @debug_urls=("http://boston.craigslist.org/gbs/egr/2902012136.html","http://boston.craigslist.org/bmw/egr/2929181526.html","http://boston.craigslist.org/gbs/egr/2926742528.html");

#############################################################################
#Section #5 - subroutines for collecting craigslist data
#############################################################################
sub collect_job_posting_http
{
	my $url=$_[0];
	my $content = get $url;
	
	#print $content . "\n";
	my @splitcontents=split(/<h4 class=\"ban\"/,$content);
	my $size_splitcontents=@splitcontents;
	
	my @url_data=();
	for (my $i=1 ; $i<$size_splitcontents ; $i++)
	{
		#just want the last 2 days of postings
		if ($i<3) 
		{
			#print "============\n\n\n";
			#print $splitcontents[$i] . "\n";
		
			#get all the posting urls for this particular day
			my @postingdata=split(/<p><a href=\"|\">/,$splitcontents[$i]);
	
			for (my $j=0; $j<@postingdata ; $j++)
			{	
				#print ">>[$j]>>" . $postingdata[$j] . "<<<\n";
				if ($postingdata[$j]=~m/^http/) 
				{
					push(@url_data,$postingdata[$j]);
				}
			}
		}
	}

return(@url_data);	
}


sub extract_date
{	
	my @url_data=$_[0];
	my @date_data=split(/Date: 2012-|EDT<br>/,$url_data[0]);	

	return("2012-" . $date_data[1]);
}

#############################################################################
#Section #6 - main program: collecting http data for each job posting
#############################################################################
my @urls=();


if ($debug == 0)
{
	for (my $k=0;$k<@search_site;$k++)
	{
		for (my $z=0;$z<@positions;$z++)
		{
			my $base_url=$search_site[$k]."/".$positions[$z];
			my @tmp_data=collect_job_posting_http $base_url;
			push(@urls,@tmp_data);
		}
	}
}
else
{
	@urls=@debug_urls;
}

#foreach (@urls)
#{
#	print $_ . "\n";
#}


#############################################################################
#Section #7 - check if each posting matches at least one skill
#############################################################################
my @matched_skills=();
my @skill_type=();
my @post_date=();

for (my $i=0 ; $i<@urls ; $i++)
{
	my $url=$urls[$i];
	my $content = get $url;
	my $counter=0;
	my $date;
	
#	print $url . "\n";
#	print $content . "\n";
	
	my $skill_type_desc="";

	for (my $k=0; $k<@skills ; $k++)
	{
		if ($content =~ m/$skills[$k]/i) 
		{ 
			$counter++; 
			$skill_type_desc = $skill_type_desc . $skills[$k] . " ";
		}	
	}
	push(@matched_skills,$counter);
	push(@skill_type,$skill_type_desc);
	push(@post_date,extract_date($content));
}

#############################################################################
#Section #8 - print results to the screen
#############################################################################
for (my $i=0; $i < @matched_skills ; $i++)
{
	if ($matched_skills[$i]>0)
	{ 
		print "<li><a href=\"$urls[$i]\">site #$i\<\/a\>" . "!" . $post_date[$i] . "!" . $matched_skills[$i] . "!" . $skill_type[$i] . "\n"; 
	}
}

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

#This script fetches the last 2 days new job postings from craigslist that match

#a specific criteria and reports the URLs that correspond to that match.

#The search criteria comes from the input arguments. The cragislist sites

#are hardwired to the New England area. You may change them by manually

#altering the variables in Section #3.

#Version 0.2 30/march/2012

#Author: Nuno Alves

#############################################################################

#Section #1 - load libraries

#############################################################################

use strict;

use POSIX;

use LWP::Simple;

#############################################################################

#Section #2 - input arguments are your skillsets

#############################################################################

my $num_args = $#ARGV + 1;

if ($num_args == 1) { print "You must add some skills as arguments\n"; exit; }

#############################################################################

#Section #3 - defining variables

#############################################################################

#what cragislist sites

my @search_site=("http://boston.craigslist.org","http://nh.craigslist.org","http://maine.craigslist.org","http://burlington.craigslist.org","http://westernmass.craigslist.org","http://worcester.craigslist.org");

#type what positions you are looking for (egr = engineering, sof = software)

my @positions=("egr","sof","bus","acc");

#this array contains the arguments which are your resume skills

my @skills=@ARGV ;

#############################################################################

#Section #4 - debug code

#############################################################################

#instead of work on every single URL, setting $debug=1, will just scan

#two webpages

my $debug=0;

my @debug_urls=("http://boston.craigslist.org/gbs/egr/2902012136.html","http://boston.craigslist.org/bmw/egr/2929181526.html","http://boston.craigslist.org/gbs/egr/2926742528.html");

#############################################################################

#Section #5 - subroutines for collecting craigslist data

#############################################################################

sub collect_job_posting_http

{

my $url=$_[0];

my $content = get $url;

#print $content . "\n";

my @splitcontents=split(/<h4 class=\"ban\"/,$content);

my $size_splitcontents=@splitcontents;

my @url_data=();

for (my $i=1 ; $i<$size_splitcontents ; $i++)

{

#just want the last 2 days of postings

if ($i<3)

{

#print "============\n\n\n";

#print $splitcontents[$i] . "\n";

#get all the posting urls for this particular day

my @postingdata=split(/<p><a href=\"|\">/,$splitcontents[$i]);

for (my $j=0; $j<@postingdata ; $j++)

{

#print ">>[$j]>>" . $postingdata[$j] . "<<<\n";

if ($postingdata[$j]=~m/^http/)

{

push(@url_data,$postingdata[$j]);

}

return(@url_data);

}

sub extract_date

{

my @url_data=$_[0];

my @date_data=split(/Date: 2012-|EDT<br>/,$url_data[0]);

return("2012-" . $date_data[1]);

}

#############################################################################

#Section #6 - main program: collecting http data for each job posting

#############################################################################

my @urls=();

if ($debug == 0)

{

for (my $k=0;$k<@search_site;$k++)

{

for (my $z=0;$z<@positions;$z++)

{

my $base_url=$search_site[$k]."/".$positions[$z];

my @tmp_data=collect_job_posting_http $base_url;

push(@urls,@tmp_data);

}

else

{

@urls=@debug_urls;

}

#foreach (@urls)

# print $_ . "\n";

#############################################################################

#Section #7 - check if each posting matches at least one skill

#############################################################################

my @matched_skills=();

my @skill_type=();

my @post_date=();

for (my $i=0 ; $i<@urls ; $i++)

{

my $url=$urls[$i];

my $content = get $url;

my $counter=0;

my $date;

# print $url . "\n";

# print $content . "\n";

my $skill_type_desc="";

for (my $k=0; $k<@skills ; $k++)

{

if ($content =~ m/$skills[$k]/i)

{

$counter++;

$skill_type_desc = $skill_type_desc . $skills[$k] . " ";

}

push(@matched_skills,$counter);

push(@skill_type,$skill_type_desc);

push(@post_date,extract_date($content));

}

#############################################################################

#Section #8 - print results to the screen

#############################################################################

for (my $i=0; $i < @matched_skills ; $i++)

{

if ($matched_skills[$i]>0)

{

print "<li><a href=\"$urls[$i]\">site #$i\<\/a\>" . "!" . $post_date[$i] . "!" . $matched_skills[$i] . "!" . $skill_type[$i] . "\n";

}

Using a SD card in Mac OS X terminal

Posted: March 11, 2012 | Author: Nuno Alves | Filed under: Operating Systems | Tags: Mac OS X | 1 Comment »

Since all Macbook Pros, come with a SD card drive, the other day I started to work on a set of automated scripts that would backup all my data in a SD card. This means I had to learn how to read and write data into a SD card using the Mac OS X terminal.

Step #1- When you insert a card into the system, you need to find out what is the system identifier for that particular SD card. So go to the console and type:

diskutil list

And you will get something like:

In my case, the SD card is identified as disk1.

Step #2- In order for you to write anything directly into the SD card, you need to know where it is mounted. Go to the console and type:

mount

And you will get something like:

In my case, the mount point for the SD card is /Volumes/BEAGLE_BONE.

Step #3- To actually copy something into the SD card, just type any unix command considering the mount point for the SD card as a normal directory. For example:

ls > /Volumes/BEAGLE_BONE/tmp.txt

This command will create a new file (tmp.txt) in the SD card with a listing of all files in the current directory.

Step #4- When you are done, make sure to unmount the SD card by typing, something like:

diskutil umountDisk /dev/disk1

File related Linux bash snippets

Posted: March 9, 2012 | Author: Nuno Alves | Filed under: Operating Systems, Programming Languages | Tags: Bash, Linux | Leave a comment »

Here are some of extremely useful Linux bash snippets I use all the time to parse experimental data from my simulations.

How to extract the top (insert number here) lines from a file
Consider a file named test-file.txt. You can extract the top 14 lines from that file using the following:

head -14 test-file.txt

1	head -14 test-file.txt

How to extract specific lines in a file using regular expressions.
Consider a file, named test-file.txt, with the following lines:
_N37_:0:_N262_:1:_N696_:0
_N37_:0:_N233_:0
_N37_:0:_N263_:0:_N694_:0
_N37_:1:_N113_:0
To extract the lines that have 5 elements we can type:

grep \:.*.\:.*.\:.*.\:.*.\: test-file.txt

1	grep \:..\:..\:..\:..\: test-file.txt

To extract the other lines, we can simply negate that regular expression:

grep -v \:.*.\:.*.\:.*.\:.*.\: test-file.txt

1	grep -v \:..\:..\:..\:..\: test-file.txt

How to search and replace text on a file

perl -p -i -e "s/string1/string2/g" file.txt

1	perl -p -i -e "s/string1/string2/g" file.txt

Where string1 is what you are searching for, string2 is what you want it to be replaced with and file.txt is the file you want to perform this operation on.

How to print a specific line number inside a file using a variable

LINENBR=3
sed -n $LINENBR'p' filename.txt

1 2	LINENBR=3 sed -n $LINENBR'p' filename.txt

Where $LINENBR is the number of the line you want to print.

How to append 2 files, column by column, keeping particular columns

paste -d"_" file1.txt file2.txt | cut -d"_" -f1,4 > file3.txt
This will paste file1.txt into file2.txt separating them by the delimiter "_". Then, it will extract column 1 and 4, delimited by "_", and place it into file3.txt.

1 2	paste -d"_" file1.txt file2.txt \| cut -d"_" -f1,4 > file3.txt This will paste file1.txt into file2.txt separating them by the delimiter "_". Then, it will extract column 1 and 4, delimited by "_", and place it into file3.txt.

How to transfer files across computers with ssh
The scp command copies files to a remote Linux system.

scp (file) (user)@(host.domain):(path)

1	scp (file) (user)@(host.domain):(path)

To copy files from a remote system to your local system:

scp (user)@(host.domain):(path) (destination)

1	scp (user)@(host.domain):(path) (destination)

How to remove a file extension

FILENAME="hello.cpp"
FNOEXTENSION=`echo ${FILENAME} | cut -d "." -f1`;

1 2	FILENAME="hello.cpp" FNOEXTENSION=`echo ${FILENAME} \| cut -d "." -f1`;

How to remove redundant lines inside a file
In the terminal type:

sort (filename) | uniq

1	sort (filename) \| uniq

How to delete the last line of a file
In the terminal type:

sed -n '$!p' (file)

1	sed -n '$!p' (file)

How to count the number of lines in a file and write that number into another file
In the terminal type:

impcount=`wc -l <  ${CIRCUIT}-impID.dat`
echo ${impcount} >> (filename)

1 2	impcount=`wc -l < ${CIRCUIT}-impID.dat` echo ${impcount} >> (filename)

Alternatively you can also store the number of lines into a shell variable.

impcount=`wc -l <  ${CIRCUIT}-impID.dat`
echo ${impcount} > xxx.tmp
NIMPS=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`;
rm -f xxx.tmp
echo ${NIMPS}

impcount=`wc -l < ${CIRCUIT}-impID.dat`

echo ${impcount} > xxx.tmp

NIMPS=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`;

rm -f xxx.tmp

echo ${NIMPS}

How to perform the same operation over several files
In a bash script type:

for file in $( ls FAULTY* )
do
	echo "$file"
done

for file in $( ls FAULTY* )

echo "$file"

done

this will print out all filenames via an echo command, that will have the regular expression FAULTY*

How to write a loop inside a bash file

#!/bin/bash
RUN=1
until [ $RUN -eq 10 ]
do
	echo "the current loop index is ${RUN}"
	RUN=$(( $RUN + 1 ))
done

#!/bin/bash

RUN=1

until [ $RUN -eq 10 ]

echo "the current loop index is ${RUN}"

RUN=$(( $RUN + 1 ))

done

How to split a string inside a bash file

x=a:b:c
#printing a:b
echo ${x%:*}
#printing b:c
echo ${x#*:}

x=a:b:c

#printing a:b

echo ${x%:*}

#printing b:c

echo ${x#*:}

How to do input parameter error testing
The variable ${CIRCUIT} is the first command line argument (${1})

if test -z ${CIRCUIT}; then
    echo ""
    echo "ERROR: First Parameter is missing. Add the bechmark name (eg c17.v)."
    echo ""
    exit
fi

if test ! -e ${CIRCUIT}; then
    echo ""
    echo "ERROR: Specified benchmark file does not exist."
    echo ""
    exit
fi

if test -z ${CIRCUIT}; then

echo ""

echo "ERROR: First Parameter is missing. Add the bechmark name (eg c17.v)."

echo ""

exit

if test ! -e ${CIRCUIT}; then

echo ""

echo "ERROR: Specified benchmark file does not exist."

echo ""

exit

How to perform a particular operation on each line of a file

FILENAME=${CIRCUIT}-sites_in_path.txt

NUMBERLINES=`wc -l <  ${FILENAME}`
echo ${NUMBERLINES} > xxx.tmp
PNUMBERLINES=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`
rm -f xxx.tmp
echo ${PNUMBERLINES}

RUN=0
until [ ${RUN} -eq ${PNUMBERLINES} ]
do
      RUN=$(( $RUN + 1 ))
      LCONTENTS=`sed -n $RUN'p' ${FILENAME}`
      echo "line # ${RUN} with contents : ${LCONTENTS}"
done

FILENAME=${CIRCUIT}-sites_in_path.txt

NUMBERLINES=`wc -l < ${FILENAME}`

echo ${NUMBERLINES} > xxx.tmp

PNUMBERLINES=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`

rm -f xxx.tmp

echo ${PNUMBERLINES}

RUN=0

until [ ${RUN} -eq ${PNUMBERLINES} ]

RUN=$(( $RUN + 1 ))

LCONTENTS=`sed -n $RUN'p' ${FILENAME}`

echo "line # ${RUN} with contents : ${LCONTENTS}"

done

How to ensure that two files have the same number of lines

#!/bin/bash
function compareNlines
{
  fileA=${1}
  fileB=${2}

  impcount=`wc -l <  ${fileA}`
  echo ${impcount} > xxx.tmp
  A=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`
  rm -f xxx.tmp
  impcount=`wc -l <  ${fileB}`
  echo ${impcount} > xxx.tmp
  B=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`
  rm -f xxx.tmp

  if [ ${A} -ne ${B} ]; then
    echo "ERROR: both files MUST have the same number of lines"
    exit;
  fi
}

compareNlines file1.txt file2.txt

#!/bin/bash

function compareNlines

{

fileA=${1}

fileB=${2}

impcount=`wc -l < ${fileA}`

echo ${impcount} > xxx.tmp

A=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`

rm -f xxx.tmp

impcount=`wc -l < ${fileB}`

echo ${impcount} > xxx.tmp

B=`perl -n -e '@splitline=split(/ /,$_); $splitline[1]=~s/ //g; print $splitline[0] ."\n"; ' xxx.tmp`

rm -f xxx.tmp

if [ ${A} -ne ${B} ]; then

echo "ERROR: both files MUST have the same number of lines"

exit;

}

compareNlines file1.txt file2.txt

Where file1.txt and file2.txt are the filenames you wish to compare.

How to count the number of characters in file
In this particular example, the character I am counting is the 0.

res=`tr -dc '0' < file.in | wc -c`
echo {res}

1 2	res=`tr -dc '0' < file.in \| wc -c` echo {res}

How to replace characters in a file
This particular command will replace all “:” with the newline character.

tr ':' '\n' < longString.txt > readableString.txt

1	tr ':' '\n' < longString.txt > readableString.txt

How to delete all instances of a particular character from a file
This particular command will delete all “:” from the file longString.txt and it will write it on the file readableString.txt.

tr -d ':' < longString.txt > readableString.txt

1	tr -d ':' < longString.txt > readableString.txt

Read the rest of this entry »

Wunderkammer

Embedded systems, coding and electrical circuits

Password protecting entire directories in MacOSX

Autonomously crawling through DICE job postings

Automated craigslist job search with Perl and Bash

Using a SD card in Mac OS X terminal

File related Linux bash snippets

Recent Posts

Categories

Latest Tweets

Wunderkammer

Embedded systems, coding and electrical circuits

Password protecting entire directories in MacOSX

Autonomously crawling through DICE job postings

Automated craigslist job search with Perl and Bash

Using a SD card in Mac OS X terminal

File related Linux bash snippets

Recent Posts

Categories

Topics

Latest Tweets