NIST challenge

February 15, 2011

Can an amateur compete in the NIST challenges?

Absolutely! And have fun doing it as well.

The National Institute of Standards and Technology (NIST) sponsors a large number of ongoing technical challenges. Currently this is the domain of think tanks and colleges. This is my encouragement to those that are not.

Can a person with limited resources really compete with success against these giants? Yes and here is why.

  • The need to get published limits options
  • Over investment in one approach or theory
  • Lack of resources

In many cases the need to get published can lead to a not invented here syndrome. Theories and algorithms that are not brand new tend not to get published. Even though an established or classical approach can be superior to what is already out there. This almost brutal need to find the next new thing can leave untapped gems laying at your feet ready to be utilized. So you don’t have to come up with a fancy new idea or approach survey the literature and resurrect a promising approach.

In many cases researchers in a single field tend to stay in their field of research. This is tends to generate a rather limited view of other things going on in other areas. Also a university may be pursuing one area of resource with an established code base and ready resources. They need to prove the value of these resources and thus are more inclined to lean on them in repeated attempts to improve their results.  Since you are not a “professional” in this field you can apply a wide variety of approaches to the problem you have no preconceived notions or emotional investment in prior work. In this case knowing or having too much experience is a hindrance.

Also you have one resources most of these folks do not have.  You have time. In many cases students and companies have alloted a limited amount of time to work on these challenge problems. This provides you a distinct advantage to play with ideas and more importantly fail more often then the established competitors. And if you are anything like me you can spend a little more money on that new computer you’ve been looking at. And for the most part that may be my actual motivation for competing. (Just don’t tell my wife)

I’m not saying anyone can compete. You do have to have some basic skills. I am saying if you have some basic knowledge you can in fact compete in these challenges. I have found the NIST personnel to be professional and accommodating. After all they already fund a large body of basic resource. They don’t need results on things they are already funding. They are looking for that new disruptive approach that you may develop.

Ok what did you really do?

It is easy to spout off on what someone can do.  Here is my practical experience with the NIST folks.

You can find current challenges at the NIST.GOV site by simply searching for challenge or benchmark.
The challenges I gravitate to can be found here:

http://www.nist.gov/itl/iad/mig/bmt.cfm

I have a background in Natural Language Processing so this challenge was not too much of a stretch. But keep in mind I am not part of a university and do not have PhD.  My practical experience is in Machine Translation and the integration of these technologies. Does that make me an expert of renowned authority? Not by a long shot.

The competition I signed up for was the Machine Translation Metrics Challenge.

In essence you need to write a program to rate the output of a machine translation system. How well does Google translate? Is essentially the question you are trying to answer.

How hard was it? Actually the schedule was pretty short and I didn’t get everything done that I would have liked. I did in essence research and formulate an approach, generated new data and developed the code base within the time frame provided.  In reality much more work then a university would need to do as they probably already have code available or at least some framework to work within. So it is doable even starting from nothing.

The one lesson I would take away from this was that in their first challenge I provided an automated installation program. They promptly ran into security issues with my installation program. I also provided a Graphical User Interface (GUI) for the system. Again this did nothing for them as they wanted to automate a large amount of data processing to test my system. I ended up simply sending them a single java jar file that they could invoke on the command line. So my initial lesson was don’t try to do too much to make you system fancy. Create a stand alone binary that can be invoked on the command line. This will save you and the evaluators time in the long run. Think research quality not commercial quality.

After some back and forth with the evaluators on how to setup the program they were finally able to run my data through the test set.

Oh crap I found a bug after reviewing the results. I notified the evaluators and sent them a new copy. They were able to rerun the  results and didn’t even give me a hard time about wasting their valuable time. I think this may happen pretty regularly :)

So after the evaluation results are done how did I do.

My Metric name is Badger. (An inside joke with some of my old colleges)

2008 – After fully expecting to be in the middle of the pack after reviewing the sample data, I was pretty depressed to find my performance was almost at the bottom of the rankings.

http://www.itl.nist.gov/iad/mig//tests/metricsmatr/2008/results/metrics.html

2010- Much better now I have a place in the middle of the pack and I actually scored the highest in one data set!

http://www.itl.nist.gov/iad/mig/tests/metricsmatr/2010/results/correlation_French_Fluency5_segment.html

But the most impressive thing for me was the fact that I was actually cited for my 2008 work by a team competing in the 2010 competition. The NCD (Normalized Compression Distance) was very similar to my earlier work of 2008. So much so that they cited me in their paper. Not bad for an amateur :)

Am I an amateur? Ok not really. Did I think I was going to win, no way. Can someone with limited resources and no university affiliations have fun working with other folks in a field of personal interests? Absolutely. Did I have fun? Yes. And it was a great way to challenge myself intellectually.

So check on the current challenges and see if you can find one that you may think is fun. Form a team if you need one and challenge yourself!

Comments are closed.