From Cartogrammar, an absolutely brilliant application of the Flickr API produces this map of the colors of Harvard Square:
The map was created by taking geocoded photos from Flickr and calculating the average hue of the photograph, then plotting that color on the map and interpolating between all the resulting points. Astoundingly, this image shows the colors (on average) that people in Harvard Square were looking at! Lots of red along the streets come from the abundance of brick buildings, but spots of green shine through just where you'd expect them to - though Harvard Yard is a bit muted (probably because its ringed by so many bricks). My guess is that the blue in the upper right corner comes from sky captured in pictures of Annenberg's roof.
To take this image to the next level, one would need geographic information on not just where the person stood, but of each object in his or her image. That's a tall order for a single photo, but with enough images you could solve for the locations of each object in space. A blurred point cloud from Microsoft's Photosynth would do the trick, or see for example this recreation of Dubrovnik, created purely from tourist's geolocated photos: