Looks like another company is losing to a patent troll. I keep thinking that Congress or the courts need to put a stop to this practice, but it’s so hard to define. Where do you draw the line between patent troll behavior and a legitimate inventor protecting his creation before he can bring it to market? No one seems to have an answer yet, but we need one soon. Perhaps patent lawsuits should have a stricter requirement for ‘standing’ - the patent owner must have a product that is damaged by the defendant’s actions? It would eliminate patent trolling, but it would also limit the usefulness of patents in protecting new products before they release.

Unfortunately, there is little emphasis on patent reform - it’s a complicated issue with a headline-generation-quotient of 0.0 for politicians. Plenty of reason there for them to stay away.

A few years ago, I took Amtrak trains between Washington, D.C. and New York. Never went through a metal detector. Never sent my bags through an X-ray machine. And, I never felt insecure, simply because trains are a terrible target. You can’t do anything with them if you hijack them, and they’re always on the ground. We need to either make planes the same type of terrible target, or we need to go back to trains. That’s the only way to actually secure our travel.

So, NSTX is awesome. I started playing with this just before I left to come home, but my DNS changes didn't have time to propagate until tonight. It's actually quite easy to set up, and surprisingly fast - at least, from where I'm currently testing it (my home in Midland). Definitely usable. There's also an IP-over-ICMP program, I may look into it next. That system seems to have some advantages over the DNS system, and could be an interesting project to hack on. It looks easier to set up, too - no DNS changes that require propagation.

Netflix update: still creating the users index. I think creating the index while inserting data may have been a better idea after all. Top reports 34:54.75 runtime so far. I'll let it keep running and check it again tomorrow.

Importing the ratings took about 2 hours. That was with a little tuning, too. Now I'm trying to generate those indices. It's going to take just as long. Ugh. Too much data.

So, I took another look at the training data for this contest. Absolutely freaking enormous. 100 Million ratings from 17000 movies and nearly 500K users. Unfortunately, the user ids run from 1 to 2.6M and have lots of gaps. So, importing the ratings as a matrix in a C app is not an option - the matrix would be over 8.5 GB, even using just chars. So, I thought SQL might be good - the biggest requirements here are being able to index in like a matrix. SQL allows that well enough, especially with indexes on both of those columns. I started with SQLite3, and it seemed slow. So now I'm trying MySQL. It's slower. A lot slower. Either I suck at configuring it (possible, but it shouldn't be this bad out of the box) or it's just slower than Christmas for lots of INSERTs. I may go back to SQLite. Either way, I need an easy read-only DB for this part. CDB seemed like a good option - they're really quick, read-only, and even fast to create. The downside is that is a hash-table only. Since I need a matrix-style system, that seems like a bad plan. I could just use "userid.movieid" as the key, but then I can't read in all the ratings from a user or ratings on a movie. Even duplicating the data in different views doesn't help. This needs to be either SQL or a matrix - there are things I'll need to do that only work with those access methods. The rest of this program can be done in C/C++ without databases, but this really needs to be a DB. Definitely going back to SQLite though. And then I'm going to let this run overnight. Because it's going to take that freaking long to import this data.

So, the Netflix Prize has me intrigued. Really, really intrigued. I'm not good enough at machine learning, stats, or whatever to win, but I'm going to be playing with it in the future. Code to follow. I'll be implementing something like this at first.