Converting base-16 roman numbers to arabic numbers (and vice-versa)

Here is neat python programming challenge.

A hex roman numeral is very much like the standard roman numeral, except with different values. In normal roman numerals, I = 1, V = 5, X = 10 and so on. In hex roman numerals, I = 1, V = 8, X = 16, L = 128, C = 256, D = 2048 and M = 4096. So for example:

VIIII = 8 + 1 + 1 + 1 + 1 = 12
IX = 16 – 1 = 15
XV = 16 + 8 = 24
XL = 128 – 16 = 112

The goal is to write a program in python that converts it in either direction. If given a decimal number, it should return the hex roman numeral version of the number and if given a hex roman numeral, it should return the decimal version of the number.

I started this by creating a program that performs a normal roman to arabic conversion. This wasn’t too hard, especially since python has a ton of neat features such as as dictionaries and solid string parsing methods. Since I am using unittest to test my code, I’ve named this file roman_numerals.py.

The package unittest provides a great way to test your programs. I love it. You can pretty much run a another script and it will perform all the necessary assertions as it tests the proper package. Here is my unittest code, which I named test_roman_numerals.py.

Here are some screenshots of the program in action: first testing through the command line, and then testing it with unittest.

Taking this base code and making it compatible with base-16 numerals was trivial. All that I needed to do was make a minor modification to the dictionary roman_dict and adding extra elements to the lists units, tens, hundreds and thousands. Of course I had to perform a base conversion with the hex2dec function each time I wanted to access a position in the list.
Here is the code that converts base-16 roman numbers to arabic numbers. I saved this file as roman_numerals_base16.py.

And here are my test cases, taken directly from the problem statement and saved as test_roman_numerals_base16.py.

Finally, here is a screenshot of the program in action.





Password protecting entire directories in MacOSX

Every once in a while I have some files in a directory that need a password. I am not looking for a fancy encryption mechanism like PGP… I just want to compress the entire directory, put in a strong password and forget about it. Anyway, there are several applications that do this on the mac store… but none provide the flexibility that I was looking for. Plus, I realized this can be done in a couple of lines in the terminal.

Lets say you have a directory tmp with a bunch of files you want to compress and password protect it. You can go the standard route using the zip utility. Just type the following:

zip storage.zip -R tmp/* -e

This will compress the directory tmp/ and store all the files in the storage.zip. It will also ask you for the password you would like to use. Unfortunately you cannot give the password as an argument. So, you are stuck typing the password every time you want to compress a new directory.

There is a hack you can use to bypass the manually password submission… It involves using expect. This program is a really neat utility. It basically allows you to create interactive dialogues with your terminal programs, which makes task automation a walk in the park. You can check its manual pages here.

Step #1: Go into the mac terminal and find out where the expect utility is located. This is done using the whereais command. Type the following on your terminal window.

whereis expect

In my macosx version, the program expect is located at /usr/bin/expect.
Step #2: Create a file (e.g. protect_directory.sh) with the following lines. Make sure you modify the location of the expect program in the first line of the script.

Step #3: Make sure your script is executable and run it.

chmod u=+rwx protect_directory.sh
./protect_directory.sh file.zip tmp hello

The first argument will be the output filename (file.zip), the second argument is the directory you wish to compress (tmp) and the last argument is the password you wish to use (hello). Neat no? Here is a screenshot of all the steps.





Automatically leveling your Castlevania character with an Arduino

I love playing video-games. Rarely play more than 1 hour a week these days… but when possible I put some time into old school PSone games… in particular Castlevania: Symphony of the Night. Phenomenal game! The only problem… is that there is some annoying grinding component to it.

Anyway, on the weekends I also love spending time with my son. Unfortunately, my son is 1.3 years old which means i can’t play with both at the same time. Time to optimize! I created a very simple system that repeatedly presses the same button over an over again. When the character is in an adequate location, it will repeatedly kill any enemies that are walking in his direction.

The point? I want to play the hard bosses and skip all the grinding. This simple contraption does the job. It is not pretty (all materials used were scavenged from here and there) but it works.

Here are two videos showing the machine in action…

And of course the Arduino source code… Everything was made, assembled and implemented in about 15 minutes. Automation FTW!





Python and duck curses

Today at lunch I realized that life would be much easier if I could check the status of my simulations in some sort of interactive menu over ssh. When I got up to my office, someone told me about Python and curses. Curses is pretty much a module that allows you to deal with terminal-independent screen-painting and keyboard-handling over text-based terminals.
So… 10 minutes after reading this neat tutorial, and inspired by the craziness happening at 38 studios, I wrote my first game playable via ssh. You control a duck that collects worms… yes, how vapid.

Moving a duck with Python and curses






Choose your own adventure… in audio

One of the things that bores me the most is driving. Ten minutes into my daily commute and I am checking out my email and reading my twitter feed. Anyway, the other day when I was coming from Boston I had this interesting idea; why not create a choose your own adventure game, which could be played while driving. Instead of reading a book, the book would be read to us by a speech synthesizer. The choices would be done by pressing buttons instead of manually flipping pages. The idea was so neat that I proceeded to create a prototype, so I could send it to a magazine, but the outcome was so bad that I pushed it aside until I find a better technology. For my failed prototype I used the linux open-source flite speech synthesizer and a beagleboard XM. As the driving engine I created a simple script in python that read a particular story and used buttons to control the flow of the adventure. The first problem is that beagleboard requires a pretty clean 5V power supply which is a mess to get in a car. Also, the beagleboard is fairly expensive ($120) to use as a dedicated game engine. Finally, creating a customized OS that is fast enough to launch the a particular application on the beagleboard is not trivial.

Anyway, I still think that audio based interactive entertainment systems have potential, but my technical solution I chose was not the best. Most definitely I will revisit this idea soon with Android phones and tablets. Implementing this in android seems to be very easy, thanks to the IOIO connectors. For those interested, this connector is available for purchase through sparkfun. Regardless, here is a very simple code for my choose your own adventure. It is done in python and if you have a macosx it will read out the entries and choices using the built in speech synthesizer.

The story itself (hardwired on the previous file as test_story.txt) is pretty self-explanatory.

<1>
You are in the top of a very tall building.
[choices]
- Jump <2>
- Yell <3>
- Do nothing <1>
[end choices]
<2>
You decided to jump… thats too bad.
[end]
<3>
You yell something. No one replies.
[choices]
- Jump <2>
- Do nothing <1>
[end choices]

Finally, here is a screenshot of the program in action.

Running the python script...

Enjoy.





Autonomously crawling through DICE job postings

I am currently working on a book that requires me to search through thousands of job advertisements. For the last couple of days I have been looking at the various websites, collecting data and looking for patterns in employment listings. Even if you are not working on a book, I am sure at some point in time you will be looking for a new job online. I love searching for jobs, and if you don’t love it too… you are probably doing it wrong.
First of all don’t manually search for jobs! It is a waste of time and it will drive you insane. Instead use a scripting language, such as PERL, that mines website databases and outlines the best matches. In fact I wrote a post a few days ago about mining employment postings on craigslist. If you are new to this entire field of data-mining, I recommend the book “mining the social web” by Russell… Nice chap…. Met him at Harvard Square a couple of years ago.



Here I outline the steps I took to extract all job postings from DICE. First of all you have to know how everything is stored in the database. Make any random search on the initial screen (e.g. embedded).

"Embedded" search on dice.com

This particular search generated the following very-long URL… so long that I had to include spaces:

http://seeker.dice.com/jobsearch/servlet/JobSearch?op=300&N=0&Hf=0&NUM_PER_PAGE=30&Ntk=JobSearchRanking&Ntx=mode+matchall &AREA_CODES=&AC_COUNTRY=1525&QUICK=1&ZIPCODE=&RADIUS=64.37376 &ZC_COUNTRY=0&COUNTRY=1525&STAT_PROV=0&METRO_AREA=33.78715899%2C-84.39164034&TRAVEL=0&TAXTERM=0&SORTSPEC=0&FRMT=0 &DAYSBACK=30&LOCATION_OPTION=2&FREE_TEXT=embedded&WHERE=

Since this particular search detected 1689 job postings, we just have to change NUM_PER_PAGE=30 from 30 to 1689, in order to see every single job post on a single page. Save that file into your hard-disk in the HTML format. For completeness, here is the file with all 1689 postings I just downloaded. The following PERL script parses the contents of this file and looks for the associated URL for each job posting.

Save the file (e.g. dice.pl) and execute it with the following command:

perl dice.pl embedded_Jobs_at_Dice.html > embedded_url.txt

This will store every single URL, one per line, in the file embedded_url.txt. Once again… for completeness, here is my generated file.

The next step is to download every single job posting onto a separate file. Since, I am a macsox user, I need to download a the contents of each of the URLs from the web via the OS X command line. This is easy accomplished with the following bash script:

On the same directory as the output of the previous PERL script (e.g. embedded_url.txt), save this bash scrip (e.g. download_all_jobs.sh) and execute it with the following commands:

chmod u=+rwx download_all_jobs.sh
./download_all_jobs.sh embedded_url.txt

The compressed outcome of this last step is a file of 27 MBs.

Now that I have all this data, I need to extract the skills required for each advertised position. So, I placed all the compressed files in the sub-directory dice_jobs and ran the following script:

The skill extraction is actually done on the following PERL script (extract_data.pl).

I then feed the extracted data into a mathematica script; a (readable) pdf version of the Mathematica script is here, and the source is here. In this script, I combined all found skills, ignored skills that were required in less than 30 distinct advertisement (e.g. COBOL and Pascal). Below is the resulting piechart.

Most requested skills in embedded computing jobs.

As expected the most sought after skills in “embedded computing” jobs are C,C++ and Linux. Java, mysql and kernel development is also very strong in demand these days. Surprisingly I saw lots of mobile computing and networking skill requests. However the most surprisingly requested skills is databases (mysql)!

Finally, I am aware that I could have done everything on this post on a single PERL script. However, writing a post about a single script would get tedious very quickly. I also wanted to save the outcome of every single step in my hard-disk so I could perform some additional data tests, without having to connect to dice.com each time.



Start a C program when BeagleBone boots

I’ve been spending some time lately trying to turn the BeagleBone into a “single-serving” system for my prototypes. Most of these prototypes require running some program whenever the system gets power. Before you continue, I strongly recommend you read my previous BeagleBone posts, if you start feeling a bit lost.

So here I have a very simple circuit… 5 LEDs connected to 5 different pins (P8_3,P8_4,P8_5,P8_11,P8_12). I would like these LEDs to display a perform a simple pattern whenever the BeagleBone is turned ON. Something like the following video.

The code for this circuit is pretty straight forward and looks like this:

To compile and run this code, download my BeagleBone_IO library, save the above code as main.c and type the following commands:

gcc -c BeagleBone_gpio.c
gcc BeagleBone_gpio.o main.c -o main
./main

What is neat about my code, is that it is able to trap a signal whenever the user types CONTROL+C and ends the program normally… after performing some generic BeagleBone GPIO cleanup.

Anyway, I would like this program to run whenever my BeagleBone starts up. BeagleBone’s default OS is Angstrom Linux which uses systemd as the initialization scheduler. I am still learning about this, but according to the wikipedia entry and also to Kezhong’s Weblog, this daemon is meant to efficiently express services dependencies, by allowing more services to be launched in parallel at system startup while reducing the need to mess around with many scripts. I used to do this on ubuntu… and it was a mess.

Anyway, I would like to declare my LED blinking program as a service. So I need to execute the following steps:

Step #1: Create a script that will call the program that you wish you launch at boot time
This means we need to create a file in the /usr/bin/ directory with any name. For example, assuming that our compiled LED program is in /home/root/main we can create an executable simple script called myScript.sh with the following lines of code.

echo “#!/bin/bash” > /usr/bin/myScript.sh
echo “/home/root/main” >> /usr/bin/myScript.sh
chmod u+x /usr/bin/myScript.sh

Step #2: Create the service file
Create the file /lib/systemd/myScript.service . Actually your service name can be anything you want… not restrained to myScript.service.

nano /lib/systemd/myScript.service

Step #3: Write the appropriate service settings
Your /lib/systemd/myScript.service can be for example:

[Unit]
Description=Run the LED pattern script
After=syslog.target network.target

[Service]
Type=simple
ExecStart=/usr/bin/myScript.sh

[Install]
WantedBy=multi-user.target

Step #4: Make a symbolic link

cd /etc/systemd/system/
ln /lib/systemd/myScript.service myScript.service

Step #5: Make systemd take notice of it, activate the service immediately and enable the service to be started on boot-up

systemctl daemon-reload
systemctl start myScript.service
systemctl enable myScript.service

After executing all commands on step 5, the LED program should immediately start. Try switching the BeagleBone power OFF & ON… and you will see the LED program.

How to terminate the script that is constantly running in the background?
If everything went okay, your script in constantly running in the background. To terminate it, you need to kill it. Use the top command and find out the process id (pid) of your looping program. As shown on the screenshot, my main pid is 13.

Listing all jobs using the command "top".


The following command will take care of terminating the job.
kill 113

How disable the script from starting up at the boot time?
Just type:

systemctl disable myScript.service

Thanks to Piranha at the official BeagleBoard forum for guidance.



Automated craigslist job search with Perl and Bash

Most of my students are in the job market and after suggesting them websites where they could look for jobs, I took a peak at craigslist. I like craigslist; its a simple, bare-bones website with pure text. However, the search functionality is a bit awkward, and it is hard to find a good match between the candidate skills and a particular job posting. If you are seriously looking at every single “filtered” post, it may still take you over an hour to look for the best skill-to-job matches. So I created two scripts, one in Perl and the other one in Bash, that scavenge all the job postings for skill matches, and create a new webpage with all the appropriate positions and matched skills in a ready to click link. Data mining at its best!

There are some “limitations” of these scripts. First of all they were only tested in macOSX and Linux, however I am sure you can convert them quite easily to Windows. Secondly, I’ve focused all the craigslist searches around New England. You may add other craigslist locations quite easily by following the instructions on the perl script.

This automated job search requires two files: search_jobs.sh and craigslist.pl. Both can be found below, or at my github repository. To run the code place both files in the same directory, and edit the search_jobs.sh, shown below, with a text editor (after emacs, my second favorite text editor is TextWrangler). In this file modify the appropriate keywords that are being assigned to the variable SEARCH_SKILLS. Currently the search skills are the standard qualifications for an engineering graduate.

This script runs with the following command line:

chmod u=+rwx search_jobs.sh
./search_jobs.sh

Running the scripts on a macosx terminal window.

After it is done executing it will create two files engineering.html and finance.html, where the candidate can see his best job matches.

Two html files are created with the best job matches

Generated HTML file with the best job matches


Below is the Perl script that parses the craigslist job postings.


Controlling a HD44780 LCD on a BeagleBone (Part 2)

On the previous post I described how a HD44780 can be integrated with a BeagleBone. For this post, I created a small HD44780 library for the BeagleBone that makes the process of writing text to a LCD as easy as it was on the Arduino.

First of all you need some files. Go to the github for this project (https://github.com/nunoalves/BeagleBone-HD44780) and place the following files into your BeagleBone:

  • beagle_gpio.c
  • beagle_gpio.h
  • beagle_hd44780.c
  • beagle_hd44780.h

    These files provide all the necessary LCD I/O. In this post we will be discussing the example_03.c and example_04.c files that are also available on that same repository.

    Once the files are on your BeagleBone, make all the appropriate connections, exactly as described on the previous post.

    Instead of describing every single function in detail, its probably easier if you look at the following code (example_03.c), which performs some very basic LCD operations.

    You can compile, and run, this code with the following commands:

    gcc -c beagle_gpio.c
    gcc -c beagle_hd44780.c
    gcc beagle_gpio.o beagle_hd44780.o example_03.c
    ./a.out

    This next sample code (example_04.c) shows how to write full strings and how to address a particular LCD position.

    Running the code from example_04.c on a BeagleBone with a HD44780 compatible LCD


    You can compile, and run, this code with the following commands:
    gcc -c beagle_gpio.c
    gcc -c beagle_hd44780.c
    gcc beagle_gpio.o beagle_hd44780.o example_04.c
    ./a.out

    Have fun!


  • Controlling a HD44780 LCD on a BeagleBone (Part 1)

    Hitachi HD44780 compatible LCD screens are those common displays we see all over the place. Getting them to display some information is fairly easy, as all you have to do is send the appropriate bits to each of the LCD screen inputs.

    Basic 16x2 Character LCD - HD44780 compatible

    In this post I will describe the steps I took to integrate a Hitachi HD44780 LCD with the BeagleBone board. First of all you need a LCD display. From my Arduino days, I had a couple of them laying around my lab. In this tutorial, I used a fairly inexpensive Microtivity LCD Module 1602. Because this LCD module requires a 5V source, and the BeagleBone GPIO pins only provide 3.3V, you need to get power from other sources: either connect a 5V DC adaptor to the BeagleBone and use P9_5 which provides +5V, or use any adapter that you have in your house to deliver the appropriate voltage with a voltage regulator. If you opt for the voltage regulator route, I recommend you get a 9V adaptor and use this neat sparkfun kit, just make sure the ground of this external voltage supply is connected to the ground pin of the BeagleBone. Of course you can bypass all these problems, by getting a +3.3V LCD display, but they will be more expensive.

    BeagleBone and a HD44780 compatible LCD

    Once you have acquired the LCD and connected to an appropriate power source, use the following connections.

    LCD Pin # LCD Pin Description LCD Pin Connects to
    1 VSS Ground
    2 VDD +5V
    3 VO Attach a potentiometer here to adjust contrast
    4 Register Select (RS) BeagleBone pin P8_4
    5 Read/Write (R/W) Ground
    6 Clock Enable (E) BeagleBone pin P8_3
    7 Data Bit 0 Unconnected
    8 Data Bit 1 Unconnected
    9 Data Bit 2 Unconnected
    10 Data Bit 3 Unconnected
    11 Data Bit 4 BeagleBone pin P8_5
    12 Data Bit 5 BeagleBone pin P8_11
    13 Data Bit 6 BeagleBone pin P8_12
    14 Data Bit 7 BeagleBone pin P8_14
    15 Backlight Anode (+) +5V (or lower)
    16 Backlight Cathode (-) Ground

    Source-Code
    The code below will prompt you to enter a bitstream into the LCD screen. Keep in mind that you need the library beagle_gpio.h, or else the program will not compile. You can find all the necessary files to run the program in the following github site (https://github.com/nunoalves/BeagleBone-HD44780). You can compile, and run, this code by typing the following commands in the BeagleBone terminal:

    gcc -c beagle_gpio.c
    gcc beagle_gpio.o example_01.c
    ./a.out
    rm -f a.out

    Once the code is compiled you will be asked to type the appropriate bitstream. If you type the following bits, you will type the word “hi” on the LCD.

    Type the following bitstream to display "hi" in the LCD

    If you would like to manually type different characters look at the following table:

    HD44780 LCD character table

    … And here is a screen shot of the program output.

    Screenshot of the program manually sending a bitstream to the HD44780.

    Finally, if you would like to know more about controlling these LCD devices I suggest this website, and the article “How to use intelligent L.C.D.s – Part One” by Julyan llett, which can be found here.

    In a future post, I will release another library that will allow us to automatically display text without manually typing the entire bitstream.