Darren Herman gets it:

This is important.  Data alone does not make decisions.

An organization built for the next century is one who has to be able to wonk through large datasets, find insights and action them.  Just having data alone is not a winning proposition.  It’s the application of data, the extrapolation, and understanding that will lead to competitive differentiation.

(Emphasis his.)

Here's a brilliant post by Andrew Gelman, highlighting a tutorial that will actually destroy information in 25 steps, allowing you (yes, you!) to create this anti-masterpiece:

People who treat chartjunk infographics as real data visualizations should be redirected to a GeoCities archive every time they access Wikipedia.

The NYT's Bits section, which up until now I thought was doing a wonderful job of evolving technology reporting to a higher, "post-blog" level, has left me stunned with a bizarre editorial in which the author requests compensation for his contribution to Facebook's success.

Is it just a tongue-in-cheek opinion designed to attract eyeballs and -- yes -- goad bloggers into responding? Probably. But it inadvertently highlights how seriously people are taking the alien business of social networking -- so seriously, in fact, that as usual they appear unable to understand it at all.

Take, for example, this opinion:

“The idea that a business benefits from social interaction is not so strange or new. A lot of cafes and small restaurants will let people hang out because they attract other people,” said Yannis M. Ioannides, a professor of economics at Tufts University. “What is unusual and new is that Facebook takes access to information about these people to make its business more powerful.” He added: “The proprietor of a cafe doesn’t use personal information about me and my friends to make money.”

Really? Cafes don't use your personal information to make money? Find me a cafe that doesn't tailor its menu, prices, inventory, background music, specials, wifi, tables arrangement, etc. etc. based on the interests of its customers. Tell me that Starbucks has never once run a focus group (yes, Internet, there actually was A/B testing before there were computers). My preferred cafes in New York are those where the people behind the counter greet me by name and have my order ready before I have a chance to ask for it.

The difference between Facebook and your average cafe isn't that the experience (and more importantly, the key inventory) is tailored to the customer; it's that Facebook's inventory is virtual (unlimited) and their profit margins are insane. The fact that user-created content draws people in is no different than a cafe hanging paintings by local artists, or people being attracted to busy establishments. The social aspect, as the professor does point out, is not new. And therefore (among other reasons), the op-ed's entire point -- that because the author contributed to the social aspect, he deserves compensation -- is absolutely ridiculous.

The NYT's Bits blog has a new post on "high tech’s hottest calling:" statistical analysis. The article isn't just about the jobs market, focusing as well on students' increased demand for statistics classes at top universities.

The opening anecdote will be familiar to anyone in the field:

“Most of my life I went to parties and heard a little groan when people heard what I did,” says Robert Tibshirani, a statistics professor at Stanford University. “Now they’re all excited to meet me.”

But the observation that follows it is quite serious:

Computing has become cheap and available enough to process any number of formulas.... What no one has are enough people to figure out the valuable patterns that lie inside the data.

Roger Ehrenberg gets it:

Every so often a term becomes so beloved by media that it moves from “instructive” to “hackneyed” to “worthless,” and Big Data is one of those terms....

Every business generates data, but it is a far smaller number that view data as a strategic asset that is actively managed for the benefit of their customers and the bottom line....

Whether the data is big, small, fast, slow, structured or unstructured, everything that is going on now is attempting to do one thing: making data smart and actionable.

[Emphasis his.]

Here's a fascinating essay by Mike Kaplan, who oversaw marketing for the movies 2001 and A Clockwork Orange, which explains how Stanley Kubrick became one of the first commercial data scientists. In 1971, as Kaplan and Kubrick were trying to determine which theaters should show the new movie, they realized that Variety published box office totals for individual cinemas in every city. The data would potentially reveal people's preferences for different theaters, if only the two men could come up with a way to access it. In the days of manual databases, that meant spending six weeks collecting volumes of carefully curated notebooks to tally 18 months of back data. The payoff, however, was enormous: the studio was able to show the new film exclusively in theaters with the perfect mix of margins and demographics.

Rumors even presaged today's computer-driven industry:

Word quickly spread that Stanley had a computerized system to track theaters and grosses based on technical information he had acquired while developing HAL 9000, the all-knowing computer in 2001. For months these stories persisted in the trades as the roster of Clockwork cinemas was refined. They were neither confirmed nor denied.

(via Daring Fireball)

Anderson Cooper, on air, referring to CNN's nonsensical "Social Media Screen":

The social media screen, again with the social media screen. My Lord. This is the third hit, I still don't understand what the hell this thing shows!

It's a shame that it takes a 3am broadcast for someone to let the emperor know his clothes are missing.

(via The Huffington Post)

Bryce Roberts gets it:

Here’s the thing. Data, big, medium or small, has no value in and of itself. The value of data is unlocked through context and presentation.

I really love the latest post on Lessons from my Twenties, called Stat Is Magic.

Sometimes, things are better left as magic.

The other day, I was unexpectedly asked, "What's the easiest way to make a sepia-toned image in WordPress?"

The questioner has a blog with an "antique" theme, and wanted to use the sepia images inline. However, the blog is quite image-heavy and she (understandably) didn't want to dive into Photoshop for every single post. She wanted a "plugin" approach: just tell WordPress to make the image sepia-toned, and let the software work its magic.

Unfortunately, that plugin doesn't exist... so I hacked one up. I'm posting the code here for future reference and in case anyone finds this useful.

Objective:

A drop-dead simple way to create sepia-toned images for a WordPress blog, performing three actions:

1. Resize the source image to a standard size
2. Apply the sepia tone
3. Add a CSS class
4. Remove any "width" and "height" settings from the original HTML

Solution:

I started with the TimThumb PHP script. TimThumb was originally conceived as an idiot-proof image resizer for WordPress (to automatically generate post thumbnails). It does so by dynamically generating a new image from a source image based on provided dimensions. More recently -- and more interestingly, for my purposes -- a hook was added for PHP's imagefilter function. Imagefilter can apply a variety of simple effects to an image, including grayscale, colorize, blur, etc.

With TimThumb in hand, I needed an easy way to apply it. For this, I created a WordPress shortcode. Shortcodes allow bloggers to execute PHP functions (with arguments!) without ever leaving the text editor or seeing a word of code. To make the execution as smooth as possible, I wanted to pass a standard WordPress-inserted image to TimThumb, but this meant I would have to do some parsing, because WordPress wraps the image with all the required HTML and TimThumb accepts just the source URL.

The following function takes in the HTML code that WordPress automatically generates for an image, replaces the source image with one dynamically-generated by TimThumb, and parses the HTML to add a new class (sepia) and remove width and height parameters. The sepia-toning is accomplished by combining imagefilter's contrast, grayscale, and colorize functions.

function sepia($attr, $content = null) {
    preg_match('/< *img[^>]*src *= *["\']?([^"\']*)/i', $content, $matches);
    $old_src = $matches[1];
    $new_src =  home_url("/wp-content/timthumb/timthumb.php?src=").$old_src."&w=375&h=275&q=100&f=2|4,-10|5,50,20,0";
    $content = str_replace($old_src, $new_src, $content);

    preg_match('/class[ \t]*=[ \t]*"[^"]+"/', $content, $matches);
    $old_class = substr($matches[0], 0, -1);
    $new_class = $old_class . " sepia\"";
    $content = str_replace($old_class, $new_class, $content);

    preg_match('/width[ \t]*=[ \t]*"[^"]+"/', $content, $matches);
    $old_width = $matches[0];
    $new_width = "";
    $content = str_replace($old_width, $new_width, $content);

    preg_match('/height[ \t]*=[ \t]*"[^"]+"/', $content, $matches);
    $old_height = $matches[0];
    $new_height = "";
    $content = str_replace($old_height, $new_height, $content);

    return $content;
}
add_shortcode('sepia', 'sepia');

And the following line creates the WordPress shortcode:

add_shortcode('sepia', 'sepia');


By adding these two bits of code to the blog's functions.php file, a blogger can create a perfect sepia-toned image simply by placing a normal WordPress image in between two tags, like this:

[sepia] (the image goes here) [/sepia].

And voila: a foolproof way to create sepia-toned images without ever leaving the post editor! Moreover, the applied "sepia" class allows these images to be easily styled with CSS.

Here's an example, from the beach in Tel Aviv. Original:

And with the sepia shortcode (note that I'm not running the code on TGR, so this is just a screenshot!):

Via my friend Will Gaybrick (@gaybrick), I discovered an excellent programming tutorial called Eloquent JavaScript. Not only is it extremely well-written, clear and friendly, but it features a completely interactive console allowing readers to run and experiment with every single example. You'll never have to struggle to decipher what a piece of code is doing -- just hit the arrow next to the code block, and it jumps into the console and prints its output.

Obviously, the guide is about JavaScript but the concepts apply almost universally to any language, making this by far the best interactive tutorial I've seen. (That's from a completeness standpoint -- for user-friendliness the hands-down winner is tryruby.org!)

I actually found myself enjoying reading the text despite covering topics I was intimately familiar with, and certainly learned a thing or two in some of the more advanced chapters. I can't recommend this enough for anyone looking for a gentle introduction to programming.

Also, I want to mention that Will will be teaching HackYale, a semester-long course at Yale focused on new web technologies like HTML5, JavaScript and Ruby. Unfortunately, it isn't an open enrollment like Stanford's new AI class, so the rest of us will have to watch enviously from afar.

I've been observing an interesting phenomenon here in Tel Aviv: people who lie about their cars. I'm not talking about lying in conversation; I mean physically altering their vehicles to make them appear to be other (generally more expensive) cars. The practice is known as "rebadging", since it involves manipulating the model numbers typically found on the back of a vehicle. This is hardly a behavior unique to Tel Aviv -- I noted it regularly in New York -- but I think that because I drive regularly here, I'm more attuned to driving behavior.

Anyway, I was motivated to write this after seeing two different examples on a single drive: an E39 BMW 5 series rebadged as an M5 and a Mercedes SL 500 rebadged as an SL55 AMG. Both cars were masquerading as their "racing spec" big brothers. To someone passionate about cars like myself, the lie was obvious.

In both cases, the badge was simply wrong: the M5 badge was on the left side of the car (it should be on the right) and, more obviously, the owner left the other badge on the car. In other words, the car proclaimed that it was simultaneously an M5 and a 530i.

The SL's lie was more subtle -- the letters "AMG" had been added to the right side of the car, as Mercedes does with their true racing models, and the original badge was left on the left. However, on its AMG models Mercedes removes the last digit of the identifier -- instead of an SL500, there is an SL55 AMG.

Anyway, what I really started wondering was, "What's the point!?" This type of rebadging targets an extremely narrow audience: people who know enough about cars to be impressed by an M or AMG badge, but not so much that they would see through the ruse. That's a very slim intersection on which to focus, especially since with very little effort the lie could be made so much stronger (just get rid of the other badge!), leaving only cues like the number of exhausts to the true nature of the cars.

In other words, people who know a bit about cars will see that this driver is lying; everyone else won't even care in the first place. So why bother?

It made me think of Donald Rumsfeld's famous quote, which I like very much as a characterization of knowledge:

There are known knowns; there are things we know we know.
We also know there are known unknowns; that is to say we know there are some things we do not know.
But there are also unknown unknowns – the ones we don't know we don't know.

Cars are an excellent domain for studying the implications of these characterizations. Even someone with basic knowledge can recite what they know: at the least, some idea of the basic interaction between the gas pedal and the steering wheel, or just the fact that gas is required to travel. And those same people can quickly volunteer the things they don't know, even if the information is readily available: how the engine works; how many gallons the tank holds; what is "horsepower", exactly? Finally, there are things we don't know and it would never even occur to us to find out (these vary more strongly with experience): how far can you drive on empty? What is unique about a BMW's C-pillar?

Back to the point, sloppily rebadging a car indicates a few things. First, the fact that there are different tiers of cars is one of the owner's known knowns, and he (or she) presumes that is also true of other drivers. However, he must also believe that other drivers don't know how to tell those cars apart except to look for an explicit declaration (that is their known unknown).

The problem is that the owner himself must have a giant unknown unknown regarding other distinguishing features of those cars. So he didn't set out to target those few individuals who respect nice cars but haven't bothered to learn much about them -- he's one of those people himself!

And, because I know my audience, I known I need to tie this back to finance. Fortunately, that's pretty easy. Assume that everything is an unknown unknown.

More specifically, everything you think you know is irrelevant because the market already knows it.

Everything you know you don't know is incredibly important -- how can you possibly make an investment when you're aware that you don't have all the information about it?

And everything else is impossible to find out because you wouldn't know where to start.

The investing game is all about minimizing the things that could remain in your unknown unknowns -- even if you can't know what they are. That's why the "information advantage" of a trader at a large bank (rogue or otherwise) is so critical, and that's why every article that tells you "how to pick stocks!" is ludicrous. The exploration of an investment idea -- turning it around, looking at it in different light -- illuminates facets you would never have seen otherwise.

If you go in with a checklist, you'll never reduce the unknown unknowns: the idiosyncratic risks, the ones you get paid to hold.

More good news for data scientists, this time from Fortune:

The unemployment rate in the U.S. continues to be abysmal (9.1% in July), but the tech world has spawned a new kind of highly skilled, nerdy-cool job that companies are scrambling to fill: data scientist.

Please note this is intended for a Mac audience (all Unix users may of course apply... but I'm sure they've all cooked up something more sophisticated than this already). Windows users, I'm afraid you may be out in the cold.

One problem I've had while traveling (especially since my first laptop died and has been replaced) is needing to keep files and settings synced across computers. When I return home in a few months, I'll have a computer that is not only running a different operating system (Lion) from my main computer (Snow Leopard), but has a host of different tools and settings stored on it. I'm not talking about typical files like images and documents, I'm talking about system-level files, settings, configurations, and tools like Python which lack friendly icons.

I thought about a few ways to tackle this problem, ultimately settling on a Dropbox-based solution which I'm quite happy with. I'm detailing it here in case anyone is interested, or has any suggestions for improvement. This will work for any Unix-based system (including any Mac)... I'm not too sure about Windows. I imagine some adjustments might be required. The only caveat is that I think .bash_profile would be .bash_rc on a Linux box, but the spirit here is the same regardless: stick the global settings in a shared synced directory, and symlink them to wherever the computer expects to find them.

Before I begin, I should note that I considered using a version-control system, like GitHub, but decided not to because I don't want all of my files to be publicly available and I don't want to have to waste a private repository just for this purpose. Furthermore, I'm more interested in syncing the files; I don't care too much about retaining their history. As I become more comfortable with git, I can foresee my opinion changing.

The default Dropbox folder is at ~/Dropbox. Since this is the same across my computers, that suits me fine.

First, I need a repository for my configuration files. I created a hidden directory at ~/Dropbox/.system and moved the following files into it (for example): .bash_profile, .vimrc, .gvimrc, the entire .vim folder, etc.

Next, I need to actually do something with those files. I created another directory at ~/Dropbox/Scripts and wrote a few shell scripts to symlink the computer's configurations to the Dropbox ones. For example, my script for Terminal configuration looks something like this:

#!/bin/bash

# Symlink .bash_profile and other profiles
echo "Making Terminal symlinks..."
cd ~; rm .bash_profile; ln -s ~/Dropbox/.system/.bash_profile .bash_profile
cd ~; rm .bash_ps1; ln -s ~/Dropbox/.system/.bash_ps1 .bash_ps1

echo ""
echo "Finished!"
read -p "Press any key to exit."

It's important to note that only two lines of this script actually do anything -- the rest is just user-friendly filler (except the shebang in the first line). The line beginning with a # is a comment. The lines starting with echo are telling the system to print something to the screen, as is the line beginning read -p, which simply waits for a keypress before continuing.

The two lines beginning cd are doing the heavy lifting. First, they navigate to the home folder (cd stands for "Change Directory" and ~ represents the home folder). Then, they delete their target files (rm stands for "ReMove"). Finally, they create the symlinks. To create a symlink, use the ln ("LiNk") command with a -s argument, standing for "Symbolic link". Next, provide the symlink target in the Dropbox folder and give the link a name in the current directory.

By the way, Mac users, you may recognize the new symlinks as alias files -- in fact, that's all they are. You could accomplish this as easily with Apple-L, but what's the fun in that? (It's also terribly non-algorithmic, wouldn't save you time at all!)

If you save the script as a .command file (i.e. Terminal_symlinks.command), your Mac will execute it when you double-click it, no need to muck around in the command line. For me, this means I can execute all my scripts just by opening them all, like any other file. Some of them have dependencies (i.e. the SciPy script needs NumPy, which needs Python, which needs Homebrew), so I put numbers in front of those to indicate they should be run in order.

Now, when I move to a new computer, I just run the main script (that nests all the other scripts) and it links the current computer's settings to the stored ones. I never need to run the scripts again as long as the location of the files doesn't change. Any updates are saved and synced automatically, no need for commits. If I make a change on computer A, the file is automatically updated and computer B reflects it immediately.

As another example, the following script walks through the Homebrew installation process (or see my post here) -- note that this gives root access to /usr/local, which I'm fine with but you may not be!

#!/bin/bash

# Install Homebrew
echo "Installing Homebrew..."
/usr/bin/ruby -e "$(curl -fsSL https://raw.github.com/gist/323731)"
export PATH=/usr/local/bin:$PATH

# Get root access to /usr/local
echo ""
echo "Getting root access to /usr/local..."
sudo chown -R `whoami` /usr/local

# Warn about PATH
echo ""
echo "Add the following to ~/.bash_profile to use Homebrew:"
echo ""
echo "# Homebrew path"
echo "export PATH=/usr/local/bin:\$PATH"

echo ""
echo "Finished installing Homebrew!"
read -p "Press any key to exit."

I've seen many different people with as many different opinions on how to do this "best" -- what are your suggestions?

My friend Darren Herman recently tweeted a statement I couldn't agree more with (I'm linking to his blog post rather than the tweet itself; as we all know, attempting to take advantage of Twitter's disastrous data model is like trying to catch water in a sieve):

”The data itself isn’t overly interesting.  The application of data is what is fascinating.”

Data is just a bunch of numbers -- the raw material. It has to be refined in order to extract value and prepare it for consumption. I've written along those lines here.

However, I will confess that I'm a little worried about how the "big data" buzzword is increasingly tossed out by the media when describing predictive analytics, as during the spate of data company fundraising and acquisition announcements over the last week. As I've noted, the two are not the same, though they are of course related.