• Lexical Analysis – convert the input text into a list of tokens that distinguishes language features from whitespace, comments, pre-processor commands, etc. (“Unknown character on line 2″ errors are produced by this stage).
• Syntax Analysis – convert the list of tokens into a syntax tree which shows the entire file hierarchically as perfectly valid expressions (this is where “Missing semicolon on line 4″ type errors come from).
• Semantic Analysis – process the syntax tree to make sure that it makes sense according to the rules of the language. Type checking falls under this category for instance, so this is where “warning, assigning ‘int’ to ’short’ – possible loss of data” type errors come from.

SimpleHLST only needs a very basic language and I’m not planning to have it support complex data, so I don’t think we’ll need a semantic analysis stage at the moment.

An example

Suppose we have the following mathematical expression, f(t) = 2t + k, and we want to implement it in some program; we could write something like

f(t) = 2*t + k.

Before any more advanced stages can take place, the compiler must extract the information from this expression. This is exactly the same principle as we humans extracting information from text. We have languages that each have a vocabulary of words with specific definitions which we can use to convey detailed information. Each word only represents a small piece of information; the context and sentence structure contains the rest.

Programming languages are exactly the same and this is where Lexical Analysis (or, colloquially, Tokenisation) gets its name from. Programming languages tend to use symbols, or tokens, instead of words for brevity. The output of this stage would be an array that looks somewhat like the following table.

Now the parser can operate on this set of tokens instead of having to wade through the text itself. This example is, obviously, very simplistic and having a separate tokenising and parsing stages may even complicate things, but it is incredibly useful for handling more complex languages such as C and Verilog.

Finite Automata

It’s very tempting to try and define all the jobs of the parser as regular expressions (regex) to begin with, but there’s a very commonly used construct that they cannot cope with: parenthesis. It’s easy to define a regex that can handle say “()” but that can’t adapt to “(())” and so on. We need something more general.

Finite Automata (FA) are abstract state machines that operate on sets of data. Despite being very abstract we can actually use regular expression to define a simple FA but we may not be able to do the reverse of this. Let’s consider a compiler reading in a keyword or a variable name.

This diagram shows a Deterministic Finite Automata (DFA) that will transition into the terminal state when it encounters a letter and will stay in that state for every alphanumeric character or underscore it encounters thereafter. Every state transition is clearly defined therefore it is said to be deterministic but this may not always be the case.

One of the state transitions in this diagram is marked with ε which means that it is taken if no other transitions match. This allows us to make very complex state transitions with optional states, possibly even bypassing large sections of the state diagram (much like the * and ? operators allow complex regular expressions). Unfortunately every state transition is no longer rigidly defined and hence they are called Non-deterministic Finite Automata (NFA).

ε-edges should be used with caution. It is perfectly acceptable (at least, in the theory) to have several ε-edges transitioning out of a single state but this means that all of these edges must be taken at once. The first deterministic edge taken on any of these branches then defines the next state of the NFA hence all other edges must be discarded. In practice this makes NFAs unattractive for compiler front-ends because they potentially need large amounts of memory and computation. It is possible to convert NFAs into DFAs but I think I’ve rambled on long enough about the theory now.

From Theory to Code

Theory is useful and worth understanding but it can sometimes be difficult to map onto actual code. I learned a lot from Tommy Carlier’s blog series before I wrote my own lexer so it might be worth looking there as well to fill in any gaps I leave. There are several ways to implement finite automata in code and the final decision will probably depend on the language you’re using. I’ve decided to write SimpleHLST in C++ which might not be the best decision but it’s the language I feel most comfortable with.

output = 15*in_1 + 39*in_2

If we look at the above expression there are only actually three types of token (words, integers and symbols) so the finite automaton only needs three states. The SimpleHLST tokeniser implements each state as a function call inside the main worker function, readNextToken, as shown below.

int Tokeniser::readNextToken ()
{
    // Skip over all whitespace
    munchWhitespace ();
 
    // Make sure we don't advance beyond the end of file
    if (atEnd ())
        m_curTokType = T_EOF;
    else if (isalpha (m_char) || m_char == '_')
        readWordToken ();
    else if (isdigit (m_char))
        readNumericalToken ();
    else // Assume it must be a symbol
        readSymbolToken ();
 
    // Return the token type
    return m_curTokType;
}

Before attempting to read a token, this function skips over all whitespace using the function shown below. The obvious improvement this function needs is to be able to skip over comments.

void Tokeniser::munchWhitespace ()
{
    // Just keep reading until there's no whitespace left.
    while (isspace (m_char) && !m_eof)
        readNextChar ();
}

Finally, readWordToken is listed below which is capable of extracting “output” and “in_1″ from the expression given earlier. The pushBuffer and extractBuffer functions operate on a std::string member variable that is used to store words and integers. An accessor function, tokenData, can then be used to find out what the token data is.

void Tokeniser::readWordToken ()
{
    // Keep reading alphanumeric characters into the buffer.
    while (isalnum (m_char) || m_char == '_')
        pushBuffer ();
 
    // Store the token details
    extractBuffer ();
    m_curTokType = T_WORD;
}

You can download the source code for the tokeniser in the form of a Visual Studio 2010 solution. I haven’t used any features particular to VS2010 so it should be possible to build with any prior version or g++ but I can’t confirm this. I’ve also included some sample code that I hope will become the HDL we’ll feed into SimpleHLST at a later stage.

To try it out: build the project and open a command prompt in the same directory as the SimpleHLST executable. Then type SimpleHLST <expression or filename> on the command line to see the output. The picture above shows some sample output.

T_EOF

Now we can read a stream in terms of tokens instead of characters we can get started on the parser! Next time, of course…

I’m not sure how long it’ll take me to complete so I’ve decided to split my progress over multiple blog posts. I’ll start by briefly describing what high level synthesis actually is and what elements I’m expecting my simple program to have.

Digital Design Flow

Chip design is immensely difficult because we can fit so many transistors in them (around 30 million on a pin head!) so we can’t think of it as a single design. It is broken up into many smaller modules that are designed under various “levels of abstraction” which basically means a set of interdependent tasks.

The figure above shows the process used for digital chip design and it may be surprising just how far removed the physical transistors are. There are so many in modern designs that it just becomes impractical to consider each one. Hardware Description Languages (like Verilog, VHDL and SystemC) are used for every level apart from the physical layout and look very similar to programming languages but there is a difference that will become apparent later.

You may be wondering if this means that modern chips are actually designed by programs. Well they certainly play a very large part in a lot of widespread designs. Hardware description language (HDL) code is fed into compilers in exactly the same way that programming language code is, but there is a specialist compiler for each dashed line in the figure. High level synthesis tools produce RTL, RTL compilers produce netlists, and layout tools produce silicon layouts. It does take care and skill to drive these tools effectively and often some hand tweaking may be required to make the most of the design, but ultimately this is how it all works.

I’m going to focus on the thing at the very top. I’m planning to make a very basic HDL that is capable of describing equations and algorithms, and then feed that into SimpleHLST which will generate Verilog code. In an ideal situation I would synthesise the output of this onto an FPGA but that is a long way off just yet.

The Plan

There is a lot to consider for this project, and I think the tasks will break down as follows.

• Read in language and generate data flow graph (DFG)
• Run ASAP/ALAP algorithms on the DFG to work out what functional units are required
• Use constrained algorithms on the DFG to get an optimal design for our purpose
• Join the functional units together using data-path synthesis techniques
• Finally, synthesise a controller to drive the data-path

After this lengthy process SimpleHLST should produce some Verilog code that we can simulate to make sure it works.

Onwards!

In the next part we’ll design a very basic HDL that will fit our needs and, hopefully, start a program that may even understand it!

One of my minor duties on this dual-job is to assemble slides from about twelve people into a large presentation, with cover slides for each speaker, every Friday for a progress meeting we all have. Naturally the first Friday I just did it by hand by importing each one in turn into PowerPoint. However it is a fairly tedious job, and to paraphrase a certain member of staff: why do something by hand when I have a powerful computer under the desk?

So I began to investigate automating the process.

Turns out that Python has an IMAP module in its standard library, which isn’t too surprising I suppose as the Python standard library is enormous. After some playing I managed to write a program that logged into my university email account and downloaded the appropriate PowerPoint attachments.

from imaplib import IMAP4_SSL
from email import message_from_string
from datetime import datetime, timedelta
import re
 
M = IMAP4_SSL ('imapserver')
M.login ('user', 'pass')
M.select ()
 
since = (datetime.today () - timedelta (days=5)).strftime ("%d-%b-%Y")
 
for user in users:
    typ, data = M.search (None, '(HEADER FROM "%s" SINCE %s)' % (user, since))
 
    for id in data[0].split ():
        typ, data = M.fetch (id, 'RFC822')
        msg = message_from_string (data[0][1])
        if not msg.is_multipart ():
            continue
 
        for part in msg.walk ():
            fn  = part.get_filename ()
            ext = re.match (r'.*\.(pptx?)$', fn):
            if not ext
                continue
 
            fp = file ('%s.%s.temp' % (user, ext.group (1)), 'w')
            fp.write (part.get_payload ())
            fp.close ()
 
M.close ()

This code would download any .ppt or .pptx attachment from anyone with an email address in the users list and save the file as <user>.ppt.temp or <user>.pptx.temp. It should be possible to decode the message part with msg.get_payload (decode=True) but it seemed to introduce a fair number of errors into the file. I think this is because it seems to convert line-by-line instead of as one large block. So I used the following code to fix this.

raw      = part.get_payload ()
raw      = re.sub (r'[\r\n]+', '', raw)
encoding = part['Content-Transfer-Encoding'].lower ()
if encoding == 'base64':
    raw = b64decode (raw)
elif encoding == 'quoted-printable':
    raw = quopri.decodestring (raw)
else:
    print "ERROR: Unknown coding strategy - ignoring."
    continue

Not particularly pretty, but it does the job.

So now I had all the emails downloaded, but I still needed to merge them all into a single file. At first I was thinking of making a C++ program to exploit the PowerPoint COM interface, but then I found Python for Windows Extensions which fully supports COM!

pythoncom.CoInitializeEx (pythoncom.COINIT_APARTMENTTHREADED)
gencache.EnsureModule ('{2DF8D04C-5BFA-101B-BDE5-00AA0044DE52}', 0, 2, 4)
gencache.EnsureDispatch ("PowerPoint.Application.12")
 
# Create an instance of PowerPoint and presentation
pp = Dispatch ("PowerPoint.Application.12")
pp.Activate ()
pres = pp.Presentations.Add ()
 
# Insert all the downloaded presentations
count = 1
for filename in os.listdir ('presentations'):
    pres.Slides.InsertFromFile (os.path.realpath (filename), count)
    count = pres.Slides.Count
 
# Save and exit
pres.SaveAs ('compiled.pptx')
pres.Close ()
pp.Quit ()
pythoncom.CoUninitialize ()

I was quite shocked at how straightforward it was to automate PowerPoint from Python, but this solution wasn’t quite good enough yet. Using InsertFromFile means that the imported presentation acquires the formatting of pres which is not what I wanted. Also, there appears to be a bug in PowerPoint 2007 which causes image references to be broken when importing from a .pptx into a .pptx with the COM interface.

Searching for a solution to the import with formatting issue lead me to this awesome VBA function which has been referenced many, many times. I ported this to Python and it worked perfectly! There was still the weird image problem, but I used a really crude fix for that:

for filename in [f for f in os.listdir ('presentations') if f[-4:] == 'pptx']:
    pres = pp.Presentations.Open (path.realpath (filename))
    pres.SaveAs (path.realpath (filename[:-1]))
    pres.Close ()

Yes. I converted all the .pptx’s into .ppt’s. Nothing intelligent here. Finally, I wrote a function to replace all the fonts added by the import with Arial. Job done.

fonts = {}
for i in range (1, pres.Fonts.Count + 1):
    fonts[pres.Fonts.Item (i).Name] = 1
fonts = fonts.keys ()
fonts.remove (u'Arial')
for font in fonts:
    pres.Fonts.Replace (font, u'Arial')

So after all that I have a Python program that logs into my email, downloads a load of PowerPoint files, converts them all to .ppt format, inserts them into a blank presentation, and then normalises the font to Arial. Not bad for a few hundred lines of code!

I did change the program a bit so that it would copy a template with title slides in and add the presentations to that instead of a blank file, and then set the date appropriately on the main title slide. But the key point here is that I’ve replaced my tedious Friday-morning activity with a single command: makepres.

Victory.

An unfortunate side effect of this atrocity is that you must now register to post comments. Every cloud has a silver lining however: now you shameless, friendless people with an increased sense of self-worth can’t post plugs to your boring blog that would bring a sloth to tears.

End rant, moving on…

Finally, D4 (our big project of the year) is done and dusted! It was a pretty interesting couple of weeks, despite being filled with stress, aggrovation and late nights. We had to design a fire detection system that could accurately measure temperature, detect smoke and the presence of people from up to four sensors and display that information on a web interface.

The basic design consisted of up to four Sensor nodes that contained a thermistor (measures temperature), smoke sensor from a butchered smoke alarm, and a PIR (Passive InfraRed) detector, and the Central Node which then connected to a PC. Each node had to have a memory to store data in the event of a power or communications failure, and the PC was used to control and monitor the whole system. A database (well, CSV file…) was used to store data from all sensors on the PC and that same file was used by the web interface to display tables and graphs of what was going down.

All in a week…

Regardless of the crazy timescale and annoying setbacks, we managed to get a fairly good prototype up and running. It was my job to build the Central Node and the software on the PC (named Monitor, see how imaginative I am?) which taught me a lot.

I spent the first day trying to get my unconventional approach to a USB interface working correctly. The documentation is a bit sparse, but fairly good nonetheless. I managed to use Command Endpoint 0 to control the state of some LED’s with relative ease, but I had a small amount of difficulty using Interrupt-In Endpoint 1 to send data to the PC when I flicked some switches. I got it down eventually though, so I was happy with my first day of work.

Shame this didn’t continue… I have no idea what happened to Tuesday and Wednesday. I just remember taking a particularly long walk home on Wednesday evening thinking “What have I actually achieved?” I was walking with my housemate, and the weather was good, so it was a good excuse to escape from stress and work.

Thursday was eventful as hell! We arrived in the morning with very small parts of the system working but nothing major going on. Matt and I joined forces. In a single day we had a fully working datapath that could accurately measure temperature to within half a degree Celsius! It also supported the necessary stuff to allow Smoke and PIR warnings to be sent to the computer. I had half written a piece of software to read the data from the Central Node and save it to a file, but it wasn’t good enough for the specification.

Thursday night was a long bastard. I completely rewrote my broken software into a multithreaded slice of moist cake that handled more than it needed to, just in case. I made it detect the Central Node being unplugged from the computer so it could ASK for lost data! I was, and still am, pretty pleased with it.

Friday, deadline day. First thing in the morning I tested my newly written software, and much to my surprised it worked fairly well. Needed a few changes and tweaks but it performed nicely. By this time we had most of the components we needed, just a case of making the firmware a bit more capable, but we needed to give all the nodes some memory. Our chosen device was a Philips I²C EEPROM because we were rocking the serial technologies in this lab. We used

• USB as the interface between the Central Node and PC,
• UART to link the Central Node and Sensor Nodes,
• SPI for the Sensor Nodes to control an ADC, and
• I²C to give each node some memory.

Which turned out to be awesome, grand total of seven pins used on a 28 pin AVR microcontroller!

Matt wrote a memory controller which we spent the morning trying to debug but, unfortunately, couldn’t get it to work despite using recommended code from Atmel themselves! By lunch it still wasn’t working so we decided to cut our losses and get a system together.

We had a few hiccups when getting the Sensor to send two bytes of data, but that was my sloppy programming. I tried to compensate for a single error in two places resulting in the same byte being recieved twice. Never mind eh? After that crease was ironed out successfully we had a working data path from the Sensor (didn’t have time to make another to test our awesome bespoke communications protocol) to the PC software. The web guys brought their software over and the other guys bought the smoke and PIR sensors over and both interfaced nicely. A few small errors, but nothing major.

And that was it! We demonstrated our end-to-end system successfully… until Brad melted our thermistor! Only a minor setback however, a quick dash to stores by John sorted that out. Now the system was back into it’s (unfortunately memory lacking) glory and we got some nice ticks in boxes. Job done. How anticlimactic.

Well that was longer than I expected. Easter is upon me now, so I intend to get into the swing of updating this often. Bet that won’t last long.