• Lexical Analysis – convert the input text into a list of tokens that distinguishes language features from whitespace, comments, pre-processor commands, etc. (“Unknown character on line 2″ errors are produced by this stage).
• Syntax Analysis – convert the list of tokens into a syntax tree which shows the entire file hierarchically as perfectly valid expressions (this is where “Missing semicolon on line 4″ type errors come from).
• Semantic Analysis – process the syntax tree to make sure that it makes sense according to the rules of the language. Type checking falls under this category for instance, so this is where “warning, assigning ‘int’ to ’short’ – possible loss of data” type errors come from.

SimpleHLST only needs a very basic language and I’m not planning to have it support complex data, so I don’t think we’ll need a semantic analysis stage at the moment.

An example

Suppose we have the following mathematical expression, f(t) = 2t + k, and we want to implement it in some program; we could write something like

f(t) = 2*t + k.

Before any more advanced stages can take place, the compiler must extract the information from this expression. This is exactly the same principle as we humans extracting information from text. We have languages that each have a vocabulary of words with specific definitions which we can use to convey detailed information. Each word only represents a small piece of information; the context and sentence structure contains the rest.

Programming languages are exactly the same and this is where Lexical Analysis (or, colloquially, Tokenisation) gets its name from. Programming languages tend to use symbols, or tokens, instead of words for brevity. The output of this stage would be an array that looks somewhat like the following table.

Now the parser can operate on this set of tokens instead of having to wade through the text itself. This example is, obviously, very simplistic and having a separate tokenising and parsing stages may even complicate things, but it is incredibly useful for handling more complex languages such as C and Verilog.

Finite Automata

It’s very tempting to try and define all the jobs of the parser as regular expressions (regex) to begin with, but there’s a very commonly used construct that they cannot cope with: parenthesis. It’s easy to define a regex that can handle say “()” but that can’t adapt to “(())” and so on. We need something more general.

Finite Automata (FA) are abstract state machines that operate on sets of data. Despite being very abstract we can actually use regular expression to define a simple FA but we may not be able to do the reverse of this. Let’s consider a compiler reading in a keyword or a variable name.

This diagram shows a Deterministic Finite Automata (DFA) that will transition into the terminal state when it encounters a letter and will stay in that state for every alphanumeric character or underscore it encounters thereafter. Every state transition is clearly defined therefore it is said to be deterministic but this may not always be the case.

One of the state transitions in this diagram is marked with ε which means that it is taken if no other transitions match. This allows us to make very complex state transitions with optional states, possibly even bypassing large sections of the state diagram (much like the * and ? operators allow complex regular expressions). Unfortunately every state transition is no longer rigidly defined and hence they are called Non-deterministic Finite Automata (NFA).

ε-edges should be used with caution. It is perfectly acceptable (at least, in the theory) to have several ε-edges transitioning out of a single state but this means that all of these edges must be taken at once. The first deterministic edge taken on any of these branches then defines the next state of the NFA hence all other edges must be discarded. In practice this makes NFAs unattractive for compiler front-ends because they potentially need large amounts of memory and computation. It is possible to convert NFAs into DFAs but I think I’ve rambled on long enough about the theory now.

From Theory to Code

Theory is useful and worth understanding but it can sometimes be difficult to map onto actual code. I learned a lot from Tommy Carlier’s blog series before I wrote my own lexer so it might be worth looking there as well to fill in any gaps I leave. There are several ways to implement finite automata in code and the final decision will probably depend on the language you’re using. I’ve decided to write SimpleHLST in C++ which might not be the best decision but it’s the language I feel most comfortable with.

output = 15*in_1 + 39*in_2

If we look at the above expression there are only actually three types of token (words, integers and symbols) so the finite automaton only needs three states. The SimpleHLST tokeniser implements each state as a function call inside the main worker function, readNextToken, as shown below.

int Tokeniser::readNextToken ()
{
    // Skip over all whitespace
    munchWhitespace ();
 
    // Make sure we don't advance beyond the end of file
    if (atEnd ())
        m_curTokType = T_EOF;
    else if (isalpha (m_char) || m_char == '_')
        readWordToken ();
    else if (isdigit (m_char))
        readNumericalToken ();
    else // Assume it must be a symbol
        readSymbolToken ();
 
    // Return the token type
    return m_curTokType;
}

Before attempting to read a token, this function skips over all whitespace using the function shown below. The obvious improvement this function needs is to be able to skip over comments.

void Tokeniser::munchWhitespace ()
{
    // Just keep reading until there's no whitespace left.
    while (isspace (m_char) && !m_eof)
        readNextChar ();
}

Finally, readWordToken is listed below which is capable of extracting “output” and “in_1″ from the expression given earlier. The pushBuffer and extractBuffer functions operate on a std::string member variable that is used to store words and integers. An accessor function, tokenData, can then be used to find out what the token data is.

void Tokeniser::readWordToken ()
{
    // Keep reading alphanumeric characters into the buffer.
    while (isalnum (m_char) || m_char == '_')
        pushBuffer ();
 
    // Store the token details
    extractBuffer ();
    m_curTokType = T_WORD;
}

You can download the source code for the tokeniser in the form of a Visual Studio 2010 solution. I haven’t used any features particular to VS2010 so it should be possible to build with any prior version or g++ but I can’t confirm this. I’ve also included some sample code that I hope will become the HDL we’ll feed into SimpleHLST at a later stage.

To try it out: build the project and open a command prompt in the same directory as the SimpleHLST executable. Then type SimpleHLST <expression or filename> on the command line to see the output. The picture above shows some sample output.

T_EOF

Now we can read a stream in terms of tokens instead of characters we can get started on the parser! Next time, of course…

One of my minor duties on this dual-job is to assemble slides from about twelve people into a large presentation, with cover slides for each speaker, every Friday for a progress meeting we all have. Naturally the first Friday I just did it by hand by importing each one in turn into PowerPoint. However it is a fairly tedious job, and to paraphrase a certain member of staff: why do something by hand when I have a powerful computer under the desk?

So I began to investigate automating the process.

Turns out that Python has an IMAP module in its standard library, which isn’t too surprising I suppose as the Python standard library is enormous. After some playing I managed to write a program that logged into my university email account and downloaded the appropriate PowerPoint attachments.

from imaplib import IMAP4_SSL
from email import message_from_string
from datetime import datetime, timedelta
import re
 
M = IMAP4_SSL ('imapserver')
M.login ('user', 'pass')
M.select ()
 
since = (datetime.today () - timedelta (days=5)).strftime ("%d-%b-%Y")
 
for user in users:
    typ, data = M.search (None, '(HEADER FROM "%s" SINCE %s)' % (user, since))
 
    for id in data[0].split ():
        typ, data = M.fetch (id, 'RFC822')
        msg = message_from_string (data[0][1])
        if not msg.is_multipart ():
            continue
 
        for part in msg.walk ():
            fn  = part.get_filename ()
            ext = re.match (r'.*\.(pptx?)$', fn):
            if not ext
                continue
 
            fp = file ('%s.%s.temp' % (user, ext.group (1)), 'w')
            fp.write (part.get_payload ())
            fp.close ()
 
M.close ()

This code would download any .ppt or .pptx attachment from anyone with an email address in the users list and save the file as <user>.ppt.temp or <user>.pptx.temp. It should be possible to decode the message part with msg.get_payload (decode=True) but it seemed to introduce a fair number of errors into the file. I think this is because it seems to convert line-by-line instead of as one large block. So I used the following code to fix this.

raw      = part.get_payload ()
raw      = re.sub (r'[\r\n]+', '', raw)
encoding = part['Content-Transfer-Encoding'].lower ()
if encoding == 'base64':
    raw = b64decode (raw)
elif encoding == 'quoted-printable':
    raw = quopri.decodestring (raw)
else:
    print "ERROR: Unknown coding strategy - ignoring."
    continue

Not particularly pretty, but it does the job.

So now I had all the emails downloaded, but I still needed to merge them all into a single file. At first I was thinking of making a C++ program to exploit the PowerPoint COM interface, but then I found Python for Windows Extensions which fully supports COM!

pythoncom.CoInitializeEx (pythoncom.COINIT_APARTMENTTHREADED)
gencache.EnsureModule ('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 4)
gencache.EnsureDispatch ("PowerPoint.Application.12")
 
# Create an instance of PowerPoint and presentation
pp = Dispatch ("PowerPoint.Application.12")
pp.Activate ()
pres = pp.Presentations.Add ()
 
# Insert all the downloaded presentations
count = 1
for filename in os.listdir ('presentations'):
    pres.Slides.InsertFromFile (os.path.realpath (filename), count)
    count = pres.Slides.Count
 
# Save and exit
pres.SaveAs ('compiled.pptx')
pres.Close ()
pp.Quit ()
pythoncom.CoUninitialize ()

I was quite shocked at how straightforward it was to automate PowerPoint from Python, but this solution wasn’t quite good enough yet. Using InsertFromFile means that the imported presentation acquires the formatting of pres which is not what I wanted. Also, there appears to be a bug in PowerPoint 2007 which causes image references to be broken when importing from a .pptx into a .pptx with the COM interface.

Searching for a solution to the import with formatting issue lead me to this awesome VBA function which has been referenced many, many times. I ported this to Python and it worked perfectly! There was still the weird image problem, but I used a really crude fix for that:

for filename in [f for f in os.listdir ('presentations') if f[-4:] == 'pptx']:
    pres = pp.Presentations.Open (path.realpath (filename))
    pres.SaveAs (path.realpath (filename[:-1]))
    pres.Close ()

Yes. I converted all the .pptx’s into .ppt’s. Nothing intelligent here. Finally, I wrote a function to replace all the fonts added by the import with Arial. Job done.

fonts = {}
for i in range (1, pres.Fonts.Count + 1):
    fonts[pres.Fonts.Item (i).Name] = 1
fonts = fonts.keys ()
fonts.remove (u'Arial')
for font in fonts:
    pres.Fonts.Replace (font, u'Arial')

So after all that I have a Python program that logs into my email, downloads a load of PowerPoint files, converts them all to .ppt format, inserts them into a blank presentation, and then normalises the font to Arial. Not bad for a few hundred lines of code!

I did change the program a bit so that it would copy a template with title slides in and add the presentations to that instead of a blank file, and then set the date appropriately on the main title slide. But the key point here is that I’ve replaced my tedious Friday-morning activity with a single command: makepres.

Victory.

Yesterday I managed to download the front page of my website using libcurl. As good as that was as a learning experience, it wasn’t interesting or useful in the slightest. Today however, I decided to see if I could fetch my status updates from Twitter and display them in a program. So I had a look at the API documentation and it looks quite easy to use, with the exception of OAuth which I’m yet to get my head around. Thankfully, for now, basic authentication is still supported.

The Twitter API uses the REST (REpresentational State Transfer) paradigm which means there’s no concept of a state on the server; i.e. each transaction is considered separately. It also means that it uses HTTP, which is pretty simple to understand. Basically in a REST protocol the URI’s are objects in the system, and the HTTP verbs are how you interact with them. So a GET on a http://server/article?name=REST object would download an article named REST. Simple eh? Check this article if you’re interested.

Anyway, onto the meat ‘n’ taters. Data in a REST transaction is typically stored as XML or JSON. I considered downloading LibYAML and taking the JSON route but a) I already had Xerces, b) I understand XML more than JSON, and c) I couldn’t be bothered to learn yet another new thing.

Xerces is incredibly well written. If you look at the class listings of Xerces or Xalan you’ll appreciate they’re both enormous and support basically everything. In fact, right out of the box Xerces supports fetching XML documents over the internet using HTTP GET. I chose not to use this purely because I wanted to use libcurl. Thankfully libcurl is surprisingly easy to use:

static size_t _CurlWriteCB (void* ptr, size_t nLen, size_t cbElem,
                            CMemFile* pFile)
{
    size_t cbSizeAtStart;
    size_t cbSizeAtEnd;
 
    // Write data to file, but measure buffer size before and after.
    cbSizeAtStart = (size_t)pFile->GetLength ();
    pFile->Write (ptr, (UINT)(nLen * cbElem));
    cbSizeAtEnd   = (size_t)pFile->GetLength ();
 
    // Return the difference in buffer size, i.e. number of bytes written.
    return (cbSizeAtEnd - cbSizeAtStart);
}
 
BYTE* GetStatusesFromTwitter (char* szUserName, UINT& uiSize)
{
    // Attempt to initialise curl
    CURL* curl = curl_easy_init ();
    if (curl != NULL) {
        // Set up the http target
        CString strFmt;
        strFmt.Format (IDS_TWITTER_STATUS, szUserName);
        curl_easy_setopt (curl, CURLOPT_URL, strFmt.GetString ());
 
        // Save the result into memory for now.
        CMemFile buffer;
        curl_easy_setopt (curl, CURLOPT_WRITEFUNCTION, _CurlWriteCB);
        curl_easy_setopt (curl, CURLOPT_WRITEDATA,     &buffer);
 
        // Attempt to grab the data from Twitter.
        CURLcode res = curl_easy_perform (curl);
        curl_easy_cleanup (curl);
 
        // Return the data.
        if (res == 0) {
            uiSize = (UINT)buffer.GetLength ();
            return buffer.Detach ();
        }
    }
 
    return NULL;
}

The above listing will download a users tweets and store them in a growable buffer (see CMemFile). But now we have to present this to Xerces in a way that it will understand. Thankfully we can supply an arbitrary InputSource to a DOMParser, including one that will wrap a piece of memory.

bool DoGetStatuses (char* szUserName)
{
    // Query Twitter
    UINT  uiSize;
    BYTE* pbData = GetStatusesFromTwitter (szUserName, uiSize)
    if (pbData == NULL)
        return false;
 
    // Move the memory into an object Xerces understands.
    MemBufInputSource* pDataSrc = new MemBufInputSource
        (pbData, uiSize, L"TwitterXML", true);
 
    // Parse the data
    XercesDOMParser parser;
    parser.setValidationScheme (XercesDOMParser::Val_Never);
    parser.setDoNamespaces (false);
    parser.setDoSchema (false);
    parser.setDoValidation (false);
    parser.parse (*(InputSource*)pDataSrc);
 
    // Get the root node
    DOMDocument* pDoc = parser.getDocument ();
 
    //...
 
    // Free memory.
    delete pDataSrc;
    return true;
}

I was especially lazy in that last listing actually, because I told Xerces to adopt my buffer which means it’ll free it for me when it’s finished with it. Curiously though, even a memory object needs a system id which is the purpose L”TwitterXML” serves. With a DOMDocument in memory it was trivial to add the statuses to a list box.

I was quite surprised at how complex a task I’d achieved given the effort I’d put in; hats off to both Xerces and libcurl. Now that I’d managed to list my tweets, naturally the next step is to try and submit one! So I made a new dialog for the occasion:

Clicking OK causes some magic to happen:

int CPostStatusDlg::DoStatusUpdate ()
{
    static const char* cszUrl =
        "http://api.twitter.com/1/statuses/update.xml";
 
    // Attempt to initialise cURL.
    CURL* curl = curl_easy_init ();
    if (curl == NULL)
        return 0;
 
    // Configure the authentication
    CString strFmt;
    strFmt.Format (_T("%s:%s"), m_strUserName, m_strPassword);
    curl_easy_setopt (curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
    curl_easy_setopt (curl, CURLOPT_USERPWD,  (char*)strFmt.GetString ());
 
    // Format the string entire in C form.
    char* szStatus = curl_easy_escape (curl, m_strStatus.GetString (),
        m_strStatus.GetLength ());
    char szPostBody[BUFSIZ];
    sprintf (szPostBody, "status=%s", szStatus);
    curl_free (szStatus);
 
    // Set up the HTTP connection and use the POST method
    curl_easy_setopt (curl, CURLOPT_POST,           1L);
    curl_easy_setopt (curl, CURLOPT_POSTFIELDSIZE,  strlen (szPostBody));
    curl_easy_setopt (curl, CURLOPT_POSTFIELDS,     szPostBody);
 
    // Finally, set the callback function and the URL.
    CMemFile buffer;
    curl_easy_setopt (curl, CURLOPT_WRITEFUNCTION, _CurlWriteCB);
    curl_easy_setopt (curl, CURLOPT_WRITEDATA,     &buffer);
    curl_easy_setopt (curl, CURLOPT_URL,           cszUrl);
 
    // Now we can execute at last!
    int nResponse = 0;
    CURLcode res = curl_easy_perform (curl);
    curl_easy_getinfo (curl, CURLINFO_RESPONSE_CODE, &nResponse);
    curl_easy_cleanup (curl);
 
    // Check for success
    return nResponse;
}

Boom! Tweet submitted!

In the above listing, the growable buffer and the callback are largely to just eat the output from libcurl because we don’t really care about it. CMemFile will free the memory it allocated when the function returns too, which saves hassle. I originally wrote all the code listings with Unicode in mind which is why they might appear to be a bit odd. libcurl is an ANSI C library so you may need to convert your strings for it to work. Thankfully Xerces includes some basic support because it uses Unicode internally.

Little victory indeed.

I can’t remember what I was doing now, but for some reason I needed to copy the full path of some file into some program to do some… thing. I was finding it increasingly annoying that I had to copy the path from the Explorer window, then either hand-transcribe (complete with mistakes) or rename, select all, copy the file name.

“Why can’t I just right click and select Copy Filename or something?” I said to myself, “I wonder…”

So I searched CodeProject for some information on how to write my own damn Shell Extension (with Blackjack and Hookers) and stumbled across this fantastic article by Michael Dunn. In the space of an hour I had managed to learn enough to make half of my extension: I added a context menu item to Explorer!

Now over to MSDN to learn about the Clipboard and history was made. After around an hour and a half I’d gone from knowing nothing about writing Shell Extensions or using the Clipboard to having a working Shell Extension that used the Clipboard. I love the internet.

This is one of the smallest pieces of code I’ve ever written and, ironically, one of the few projects I consider myself to have finished. For more information, including how to download it, go to the CopyExt page.

An unfortunate side effect of this atrocity is that you must now register to post comments. Every cloud has a silver lining however: now you shameless, friendless people with an increased sense of self-worth can’t post plugs to your boring blog that would bring a sloth to tears.

End rant, moving on…

Finally, D4 (our big project of the year) is done and dusted! It was a pretty interesting couple of weeks, despite being filled with stress, aggrovation and late nights. We had to design a fire detection system that could accurately measure temperature, detect smoke and the presence of people from up to four sensors and display that information on a web interface.

The basic design consisted of up to four Sensor nodes that contained a thermistor (measures temperature), smoke sensor from a butchered smoke alarm, and a PIR (Passive InfraRed) detector, and the Central Node which then connected to a PC. Each node had to have a memory to store data in the event of a power or communications failure, and the PC was used to control and monitor the whole system. A database (well, CSV file…) was used to store data from all sensors on the PC and that same file was used by the web interface to display tables and graphs of what was going down.

All in a week…

Regardless of the crazy timescale and annoying setbacks, we managed to get a fairly good prototype up and running. It was my job to build the Central Node and the software on the PC (named Monitor, see how imaginative I am?) which taught me a lot.

I spent the first day trying to get my unconventional approach to a USB interface working correctly. The documentation is a bit sparse, but fairly good nonetheless. I managed to use Command Endpoint 0 to control the state of some LED’s with relative ease, but I had a small amount of difficulty using Interrupt-In Endpoint 1 to send data to the PC when I flicked some switches. I got it down eventually though, so I was happy with my first day of work.

Shame this didn’t continue… I have no idea what happened to Tuesday and Wednesday. I just remember taking a particularly long walk home on Wednesday evening thinking “What have I actually achieved?” I was walking with my housemate, and the weather was good, so it was a good excuse to escape from stress and work.

Thursday was eventful as hell! We arrived in the morning with very small parts of the system working but nothing major going on. Matt and I joined forces. In a single day we had a fully working datapath that could accurately measure temperature to within half a degree Celsius! It also supported the necessary stuff to allow Smoke and PIR warnings to be sent to the computer. I had half written a piece of software to read the data from the Central Node and save it to a file, but it wasn’t good enough for the specification.

Thursday night was a long bastard. I completely rewrote my broken software into a multithreaded slice of moist cake that handled more than it needed to, just in case. I made it detect the Central Node being unplugged from the computer so it could ASK for lost data! I was, and still am, pretty pleased with it.

Friday, deadline day. First thing in the morning I tested my newly written software, and much to my surprised it worked fairly well. Needed a few changes and tweaks but it performed nicely. By this time we had most of the components we needed, just a case of making the firmware a bit more capable, but we needed to give all the nodes some memory. Our chosen device was a Philips I²C EEPROM because we were rocking the serial technologies in this lab. We used

• USB as the interface between the Central Node and PC,
• UART to link the Central Node and Sensor Nodes,
• SPI for the Sensor Nodes to control an ADC, and
• I²C to give each node some memory.

Which turned out to be awesome, grand total of seven pins used on a 28 pin AVR microcontroller!

Matt wrote a memory controller which we spent the morning trying to debug but, unfortunately, couldn’t get it to work despite using recommended code from Atmel themselves! By lunch it still wasn’t working so we decided to cut our losses and get a system together.

We had a few hiccups when getting the Sensor to send two bytes of data, but that was my sloppy programming. I tried to compensate for a single error in two places resulting in the same byte being recieved twice. Never mind eh? After that crease was ironed out successfully we had a working data path from the Sensor (didn’t have time to make another to test our awesome bespoke communications protocol) to the PC software. The web guys brought their software over and the other guys bought the smoke and PIR sensors over and both interfaced nicely. A few small errors, but nothing major.

And that was it! We demonstrated our end-to-end system successfully… until Brad melted our thermistor! Only a minor setback however, a quick dash to stores by John sorted that out. Now the system was back into it’s (unfortunately memory lacking) glory and we got some nice ticks in boxes. Job done. How anticlimactic.

Well that was longer than I expected. Easter is upon me now, so I intend to get into the swing of updating this often. Bet that won’t last long.