The iPhone - A Pointing Device for the Real World?

Will Google use the snapshots you take with the iPhone for automatically adding 3D building detail and for increasing the resolution of Google Earth imagery?

(Image - Courtesy of Apple)

- What on Earth makes you think that?

I started thinking about virtual annotation and information overlays on the real world while writing my previous post "Pulling Answers from Thin Air".

I wrote about sending geotagged images from a phone for automatic image recognition and data lookup services. A technology that the company Neven Vision have been developing. They were aquired by Google in 2006, which I find interesting.

You can use the geographic position of an image to help identify known buildings depicted in that image. Then you also can calculate the exact position and the exact orientation of the phone in relation to the real world if you know the focal length and sensor specifications of the camera.

You will have the exact orientation at that instant, at least...

But doesn't the iPhone have an orientation sensor implemented using accelerometers? This means that the iPhone could keep track of its position and orientation in relation to the real world, even after the image was snapped, when you are moving around!

Maybe my mind is easily boggled, but this makes a lot of interesting things possible. You could envisage:

• Extreme resolution enhancement for Google Earth - Use the images snapped by iPhone users all over the world to automatically get high resolution up-to-date map images and extremely good 3D building detail.
• Information overlays on top of a live image of the real world on your iPhone.
  
• Accessing mobile services and data by using the phone as a pointing device for the real world.
• Orientation tagged images. Generate virtual tours where you seamlessly zoom into an image, blending with the virtual landscape. Build Quicklime VR panoramas of the globe. Orientation tagged movies and web cams. Moving images directly in the 3D view of Google Earth.
• Historical overlays. Scrub a slider on the iPhone screen to change the point in time and see the scene in front of you change accordingly.
• Scrub the same time slider and change the accumulated digital images that are mapping the area and you have instant animated history.

I just wish that this kind of device already had been around for about four billion years. I would love to be able to see the same kind of animation for an area using a geological time scale.

Highly accurate GPS and orientation data is needed to make something like this work. To make it useful, I guess you roughly would need to have a position that is accurate within a few meters and an angular error that not much higher than one degree. Is that possible?

That would depend on the accuracy of the accelerometer in the iPhone. It could be the same type as the accelerometer in the MacBook, which is pretty accurate, but I have no idea if that is accurate enough for this kind of application. An electronic magnetic compass combined with accelerometers and image recognition of known landmarks sounds like a workable solution.

I'll admit that this is a little speculative. It relies on the premise that the iPhone has or will have a GPS chip and a method for acquiring accurate 3D orientation data. But, is it difficult to put together such a contraption?

No. It has already been done (pdf).

What will Google have to gain from something like this? Besides getting ahead of competitors in the ongoing battle of more resolution and 3D detail, the possibilities for very effective advertising can not be ignored. I would not pass on something like this. Would you, Google?

The Google Phone is Not a Phone. It is a Location Aware Browser for Mobiles

And it is going to be used in the iPhone.

"Google appears to be building software for Web search on cell phones and location-finding services to work with Apple Inc.'s iPhone and other cell phones." - Piper Jaffray analyst Gene Munster.

This is starting to get interesting...

If you have been following my speculations about the Apple and Google collaboration, you know that I have claimed that geo-browsing is the secret sauce that brings it all together.

The idea of browsing and searching geolocated or geotagged information using a GPS-enabled mobile phone is what Google have been planning to implement all along. Throw in well targeted localized ads that take into account your position and all the data that Google has gathered about you and you will see an ad revenue explosion for Google. It is a whole new field of advertising, with tons of ad money just waiting to be mined all over the globe.

Now, Computerworld is running a well written article about the same thing

They are connecting the same dots together as I did earlier. Andy Rubin and his secretive company Android is the key to understanding what Google has been working on. And it is location-aware mobile phone software. Not a Google phone.

Allowing users to add geolocated information using the phone is a natural extension of the idea. You would be able to post information at a location, hanging in the air, ready to be browsed by people passing by.

This is revolutionary software and a revolutionary application. "Geo-browsing", "Blogging the Globe", whatever you want to call it, is going to change the way we look at the world and the way we organize information. And it is going to generate huge profits for Google.

Now we know why Steve Jobs called the iPhone a "magical" device that will "Change the world" when it ships in June.

Now we know why the Apple logo at the MacWorld keynote presentation looked like the Apple Logo was floating in space, the sun about to go up behind it.

Some previous posts on the topic:

Apple and Google to Blog the Globe?
The Location Aware iPhone
Google and Apple in Your TV?

Video of the iPhone Shows Hidden Video Camera?

The following is an excerpt from a longer post – "Google and Apple in Your TV".
[...]
Google Talk is compatible with Apples iChat AV messaging and video conferencing application and there has been rumors of video calls or video conferencing abilities in an incarnation of the iPhone.

(Image - Courtesy of Apple)

- But where's the video camera? The iPhone only has a camera lens at the back. That won't work.

Some have pointed to Apple's patent for a technique that allows a camera to be hidden behind a TFT screen, others have claimed that the home button looks suspiciously similar to a video camera lens and that the button may be both a button and a camera.

In this video, reflections can be seen in the home button (at 1:37 and 1:50) that looks strange for a simple button, but makes more sense if it is a combined camera lens and button: (Rex Crumb from MarketWatch interviews Apple's Vice President of iPod marketing, Greg Joswiak)

Google and Apple in Your TV?

Blog the Globe – A Fringe Interest?

- How many really want a system for sharing information stored at geographic locations and a way of browsing this info using location-aware mobile phones? This cannot generate a lot of profit. Isn't it more likely that Googles recent investments in internet bandwidth points to future online film, TV and video distribution? The really big bucks must be in TV advertising and well targeted TV Ads.

Well, Google is definitely going in that direction (Google CEO wants $74 billion TV ad market), but they have also stated their interest in expanding their advertising into mobiles. Why? There is a lot of mobile phones out there and it is a technology that is very personal. Mobile phones go where their users go and it is very natural for a mobile phone to have personal location-based services.

"Users would be able to post information at a location, hanging in the air, ready to be browsed by people passing by."

It is a perfect match for Google to implement a system for "geobrowsing" with intermixed targeted ad content. All they need to do is put Google Earth or Google map in location-aware phones, toss in some useful georeferenced data they already have, add possibilities for rating, personalization and categorization and enable users to supply georeferenced information directly from the phone. Users would be able to post information at a location, hanging in the air, ready to be browsed by people passing by. The users preferences combined with rating information and other techniques will make sure mostly useful information pops up, along with well targeted ads.

Are Google contemplating doing this? Hard to say, but they are certainly going into mobile advertising and location-based services. There is very rudimentary support for sharing KML files with friends by saving the KML file and mailing it to someone for viewing in Google Earth. You can also use a small subset of KML for sharing favorite locations by uploading the KML file to a website and then either view it in Google Maps or Google Maps for Mobile. You can share information by publishing layers for others to download or post placemarks in Google Earth Community. That's pretty good, but it is all a very manual process. Automate and streamline posting and browsing to make it more user-friendly and add rating and categorization - and Bob's your uncle.

And I cannot see how Google are going to pass on this opportunity. They have already forged ahead in this territory and made Google Earth and Google maps into runaway hits, but Google has not yet seen any massive revenue streams from these areas. That must change.

Something of this sort will eventually have to emerge from the company’s advanced research arm, Google Labs. They already need to implement better techniques for filtering out junk and presenting mostly useful or high quality information in Google Earth. The solution to that problem can be used to tune a mobile location-based service of this sort to the users needs.

- Most people want location-awareness and maps in their phone but they will be hesitant about the idea of location-based ads!

"...this is a whole new field of advertising"

But it is not so much about what we as users think we want. It's about what Google wants - ad revenue. And sure, there is going to be a lot of that in combination with online film, TV and video distribution, but this is a whole new field of advertising that is waiting to be mined by someone and if you are the only one mining it, it is going to be rich vein of gold you can keep for yourself.

Imagine a small business owner in somewhere in USA. A restaurant perhaps. Will he spend his hard earned dollars on internet ads? No. Will he spend a few bucks on ads that target possible customers as they drive by his restaurant? Yes, especially if they are well targeted ads, that match the interests and eating habits of these possible customers.

There are millions small business owners like this all over the globe. Billions of ad dollars out there for Google or someone else to to collect. The technology needed to implement it is already here.

So, will anyone agree to having geolocated ads pop up in this fashion? Yes, if it is not too disruptive and if it is part of a free mobile phone service or if it comes with free maps, a large directory of free points of interest and free turn-by-turn directions.

It will be also be more easily tolerated if the ads come as a part of a service that presents other information that the user finds interesting and practical. It will be accepted if the ads are well targeted and the user feels it's more like information he needs rather than ads. Google can deliver this.

But that's not all they can deliver. They want to move cable television and telephony onto the internet. IM, VOIP, IPTV, Video, film, mobile communication, office applications, maps, location-based services, mail, search – they want to bring all of this together. For the most part free for the user and funded by ads. In this scenario, Google is not going to have any problems finding acceptance for a few well targeted ads that pop up here and there.

What has been called Google sprawl and has been viewed as a problem for Google, is going to come together beautifully when when all this is tied together in an information gathering giant ad-serving network. Google will be everywhere.

- But there isn't enough bandwidth on the internet to handle all of that video traffic!

This bring us to Googles recent huge investments in data server farms, telecom and fiber. It appears as if Google is building a separate high capacity internet.

What is all that for? Only reducing operating costs for Google as they claim themselves? Hardly. Google has seen the future and are investing to prepare for when the future arrives. When the average user hooks up his TV and starts using 40 times more bandwidth, regular ISPs are not going to cope anymore. They already are struggling with handling P2P downloads and YouTube video. Google will come to the rescue. "You need more bandwidth? Ah, how convenient that we can supply you with this. Who could [[cough]] dream of this kind of explosion in network traffic?"

So, is the delivery of video, films and TV the central theme of the Google-Apple collaboration? A collaboration both for content and technology for delivery – Apple's Akamai together with Googles recent massive investments will enable truckloads of information to be shifted.

Google Talk, VOIP and video conferencing is another possibility. Google Talk is compatible with Apples iChat AV messaging and video conferencing application and there has been rumors of video calls or video conferencing abilities in an incarnation of the iPhone.

(Image - Courtesy of Apple)

- But where's the video camera? The iPhone only has a camera lens at the back. That won't work.

Some have pointed to Apple's patent for a technique that allows a camera to be hidden behind a TFT screen, others have claimed that the home button looks suspiciously similar to a video camera lens and that the button may be both a button and a camera.

In this video, reflections can be seen in the home button (at 1:37 and 1:50) that looks strange for a simple button, but makes more sense if it is a combined camera lens and button: (Rex Crumb from MarketWatch interviews Apple's Vice President of iPod marketing, Greg Joswiak)


Oh, and if you are delivering TV, video and film through the internet, why not watch it on the iPhone? No need to use the small amount of memory available on the iPhone for a traditional iTunes-iPod downloading-uploading scheme.

- I thought you said "Video from YouTube through Apple TV is not something revolutionary. It is something grainy and mildly entertaining."

Well, it is, but it can be a lot more. If you change the very nature of how you watch television and go from mass entertainment with little or no variation, which is what we have today, and instead get a very wide variety of content, rated and categorized, enabling you to easily find your favorite content through search and recommendations from friends, and if you get this delivered exactly when you want it, and regardless of where you are, that's not such a bad thing. It might even be called revolutionary. If Apple is going to take a big part in transforming the TV-landscape in collaboration with Google, I don't mind calling it revolutionary, even though some of the content still is going to grainy and mildly interesting.

So my favorite subject, the accumulation of rated, classified and geolocated information, can be seen as just a small tiny part of business, if it ever sees the light of day, compared to the video, film and TV-distribution?

"Geo-ads" is more in line with Googles core business of search engine technology and information indexing coupled with ad delivery. It becomes a powerful mix if Google would have information about your interests gathered from internet searches, your movie or TV watching habits and your geographic location over time.

"Special Crater Pumpkin Pie today. Eat with a view of the Arizona Crater."

"He stops regularly and eats mainly at Asian restaurants, most often in the evening, but appears to eat lunch at home. Watched TV-shows indicates he is interested in geology, meteorites, gemstones and he regularly searches for pumpkin pie recipes."

Serve him this ad when he passes by: "Special Crater Pumpkin Pie today. Eat with a view of the Arizona Crater." Or why not send information about his favorite foods and interests to the restaurants along his normal routes, enabling them to hook him as a regular customer with an irresistible offer as he passes by.

- Hey! Do we really want this?

If you can chose from a life infested with loads of badly targeted ads everywhere you look - or, much much fewer but well targeted ads, and on top of this, you don't have to pay for using your mobile phone, your TV-shows are free etc etc, then I think I would let Google get away with a lot of this information gathering, but you may want to see some very clearly defined limits to it.

So, TV, video and movie distribution will go the same way as music. Will it be Google and Apple or someone else who will end up as the biggest player?

What about Joost? It is free ad-funded internet TV delivered via P2P technologies similar to Skype. It's from the guys behind Kazaa and Skype and is a contender with similar goals for advertising – fewer but better targeted ads.

Joost is not about user supplied video at all. It more like ordinary TV, concentrating on popular content which works best for delivery via P2P-technologies. Oddball, niche or fringe content will not work well in a P2P-system and having a lot of that would require more investments in server technology. Having user supplied video also creates a lot of legal headaches and the Joost team are very familiar with these type of problems from the Kazaa experience.

Google has been very aggressive in the online video arena. They bought YouTube, possibly to protect its investments in Google Video. YouTube was beating Google to the market it wanted and they had to buy YouTube. The same thing may happen again if someone manages to get ahead. Now, with Googles enormous investments in internet bandwidth and telecom, Zennström and Friis, responsible for Joost and Skype, must feel a little squished by the giant Google octopus.

Who will win? Google, Apple, Amazon, Microsoft, Joost? I am betting on Google, perhaps in collaboration with Apple. If they do win, it seems likely we will have TV with a lot wider spectrum of programming, not just the same old mass entertainment. I wouldn't mind that.

User supplied TV-shows? Sure, bring it on. If rating, categorization and personalization will enable me to effortlessly find what I want to watch, I am all for it.

Everything is going to be personalized. Personalized ads, search, TV-channels, music – any information regardless if it pops up in a geographic location on your phone or if it is your own personalized news page or news TV channel.

And that is a good thing.

But Google will be everywhere – and that might not be such a good thing. We can only hope that Googles company motto "Do no evil" prevails and that there are not too many gray areas in their definition of evil.

Some links:

Google's Own Private Internet
When Being a Verb is Not Enough: Google wants to be YOUR Internet.
Why Joost Is Good for TV
Google Talks the Talk
Google: Dark Fiber Story Not So Dark
GOOGANOIA: Why Everyone is so Scared of Google
Winner-Take-All: Google and the Third Age of Computing
Google CEO wants $74 billion TV ad market

The Location Aware iPhone

"First 30 years were just the beginning". This phrase could be seen at www.apple.com shortly before MacWorld. That sort of statement raises the expectations of what is to come. Now, did we see something earth-shattering at MacWorld?

Not really. The iPhone looks great and has some very innovative features, I seriously want one, but it seems it's mostly about doing the same old things – surf the web, email, call people, although it does so with considerable style and elegance.

(Image - Courtesy of Apple)

Isn't that good enough? Not if you set up the stage by saying "First 30 years were just the beginning." Then I expect something more than a phone.

Something big is missing from the picture.

There were a lot of things left out from the presentation at MacWorld – iWork 07, iLife 07 and Leopard. What is the common denominator here? They have all been rumored to feature GPS functionality or use geographic metadata.

The big thing missing is the Earth.

Why have Google Maps in the iPhone and not make it location aware? I think it already is. And if it isn't location aware, it will be. Or there will be a another version that is equipped with a GPS-chip. The iPhone is still only a working prototype. We haven't seen the final product yet.

But why didn't they say anything at all about GPS functionality at MacWorld? Why are they leaving out the whole juicy part of location awareness in Leopard and iLife including iPhoto, iWork, iCal, Address Book and possibly the iPhone?

Because they are saving it for a special event?

Someone might say:

- Hey! Wake up! What makes you think location awareness has to be an important part of the Google - Apple collaboration? It may have very little to do with location based services and geotagging. Google YouTube Video and Apple TV may be what it is all about.

Video from YouTube through Apple TV is not something revolutionary. It is something grainy and mildly entertaining.

- Couldn't it be the case that GPS-chips and location awareness was excluded to reduce costs and complexity of a mobile phone that is primarily aimed at capturing a large part of the market, and not intended to satisfy the tech-nerds of the world with functionality that isn't going to be used by most consumers?

That is entirely possible, but if so, I still would expect to see a GPS-equipped version of the iPhone introduced at a later stage.

(Image - Courtesy of Apple)

- Why? Is GPS functionality in a mobile phone really something that is going to be used by most consumers?

I'd like to think that the functionality resulting from position awareness is going to be heavily used - partly unobtrusively behind the scenes when searching and also for normal positioning and turn-by-turn directions. And it does not have to be difficult to use. There are a lot of navigation devices available that are easy to use. These functions can be easily accessible through a flick of a finger at a button in a search result listing or in an address book. Browsing location based information simply by walking around in the real world, is not going to be terribly complicated either.

Navigation and positioning is one of the most highly requested features in a mobile phone if you look at market polls of consumer wishes.

Google knows this, and they are happy about it because it fits beautifully with their mobile strategy:

"The second big category we are focusing on is location-based services. People take their cell phones with them everywhere, and they generally are looking for information in the context of a location." -- Deep Nishar, director of product management for Google. From CNet News.com

There is going to be big money in location based advertising. Google interest in mobile advertisements is perfectly natural. Google wants to serve users with ads and sponsored search results that are well targeted to the user. They are already in a good position to do so, but expanding their presence into your location aware mobile phone is going to be huge for Google.

Google has already partnered with Orange (Google Phone) and Helio (the Drift) to crank out location aware phones with this in mind.

Google CEO Eric Schmidt, who is on Apple's Board of Directors, has even said that your mobile phone should be free and that it just makes sense that subsidies should increase as advertising rises on mobile phones.

What do we have?

• Googles Orange phone: GPS, Location based services,
• Google and Helio Mobile Phone: GPS, Location based services, Buddy Beacons
• Google and Apple iPhone: No GPS, No Location based services?

Not very likely.

If you look for connections between some of the people working for Apple and Google, you will find Andy Rubin (co-founder of Danger) at Google, hard at work, you might assume, implementing Googles mobile strategy after selling his company Android to Google.

What Android is working on has been a well guarded secret, but Rubin shares his interest in location based applications of technology with Apple's co-founder Steve Wozniak, who was on the Board of Directors at Danger.

In a 2003 interview with BusinessWeek, Rubin said there was tremendous potential in developing smarter mobile devices that are more aware of its owner’s location and preferences. “If people are smart, that information starts getting aggregated into consumer products,” said Rubin.

Travis Geiselbrecht, working together with Rubin at Danger, can now be found working for Apple.

Coincidence? Perhaps. But I still have to say:

There is a reason for the geotagging support in iPhoto and links to Google maps.

There is a reason for not mentioning location awareness with regard to the iPhone in the presentation at MacWorld.

iWork 07 and iLife 07 were conspicuously absent for a reason.

The reason may very well be revealed later at a special event when all of this will be connected together. The true nature of the collaboration with Google will be revealed and it will become evident why Apple has been hard at work at implementing OS-level integration of geographical mapping technology in Leopard.

Google and Apple would not want the common enemy, Microsoft, to snatch the world away from them.

Did you notice that Steve Jobs called the iPhone a "magical" device that will "change the world" when it ships in June?

Why did he say "change the world"?

Apple and Google to Blog the Globe?

What are they up to?

Will there be a Apple phone with GPS capabilities that includes automatic georeferencing of photos, videos, audio, text messages and web pages presented at MacWorld on Tuesday?

And a clever way of browsing this info automatically using your location aware iPhone or MacBook?

Users would be able to post information at a location, hanging in the air, ready to be browsed by people passing by. Imagine getting highly relevant messages, without even pressing a button, simply because you are in the vicinity and your preferences match the content of the post.

The idea of placing information on a location, making it accessible for anyone passing by, has been around for quite some time now – but nothing much has happened yet.

It is about time someone connected the dots and implemented this idea in a way that is so simple to use that not only geeks adopt and use the technology.

Apple and Google would be the perfect team to pull this off. And they are collaborating. iPhoto 6.0.5 shows evidence of this collaboration, with support for GPS coordinates and a MapURL that points to Google Maps.

AppleInsider has reported that Apple has been working on OS-level integration of an geographical mapping technology as an integral part of Leopard, its next-generation OS. It was also rumored, according to Appleinsider, to employ GPS functionality.

And Google CEO Eric Schmidt recently joined Apple's board. What is going on?

The new iWork and iLife apps may well be GPS enabled, as well as new MacBooks and of course the iPhone. This opens up endless possibilities and would put an end to the sad state of affairs that is GPS and GIS on the Mac.

If something along these lines where to happen, would Apple and Google go further than the obvious photo geotagging, turn-by-turn directions and the normal location aware services that already exists? Will they enable us to "blog the globe" and cover the earth with deep strata of information, available to be automatically sampled just by being close to it?

Perhaps not, at least not starting on Tuesday. But I hope it has been up for discussion and that Steve Jobs presents something that is at least a small step in this direction. I wouldn't want Microsoft to be the first to implement the idea.

The implementation would have to take into account how to deal with spam, bad quality information and only deliver information that the recipient finds interesting or relevant. And it has to be easy to use.

I imagine that rating and some form of classification have to be employed to enable the users to find only the hidden gems that are useful for them.

By rating the content as well as the poster and the classification, it would be possible to create a database that can be mined according to the preferences of the user. Automated forms of collaborative filtering could also be used.

"I am only interested in posts rated over 6, classified as non-commercial and they have to be in at least one of my defined areas of interest, but I will also allow any posts having a well established humour-rating higher than 8 and any automatic proximity dating/matchmaking offers and all notes about geology regardless of rating."

While travelling, you get messages like. "Hey, there is an interesting geological site nearby, a meteor crater where small pieces of lunar material still is being found from time to time."

Oh, and while they're at it, they might as well make the new iPod-video device show informational overlays over the real world using a heads-up display as suggested by IBM's J. C. Spohrer.

Other links:

Urban Tapestries
Weaving the net into your world
Geocorder
Location Linked Information
Local Interest GPS
Position-Dependent Services Using Metadata Profile Matching
Net Sticky Notes All Over London - Slashdot