How the Amazon Fire Phone Works


Take a picture of a thing, and you’re instantly able to purchase it online.
Take a picture of a thing, and you’re instantly able to purchase it online.
© JASON REDMOND/Reuters/Corbis

The other unique feature is the "Firefly" app. To use it, you press and hold a physical Firefly button (which also serves as the camera button) on the side of the phone while pointing the camera at an item. It lets you scan a wide variety of things with the rear-facing camera, including barcodes, QR codes, URLs, email addresses, phone numbers and real world objects such as games, DVDs, CDs and works of art. The phone sends the gathered information to Amazon's cloud servers to find a match, and if it recognizes the item, it will bring up a link that provides information and lets you take various actions, including purchasing the item from Amazon or sharing it on social media.

Not only does it recognize physical items, but it can also use the microphone to identify songs, movies and TV shows that are playing. It can identify URLs, phone numbers and emails in printed text. Firefly uses image, text and audio recognition tools along with a massive catalog of more than 100 million items in the cloud to identify the things people scan.

Rather than sending an entire picture to Amazon's cloud, it picks out and sends partial images or information that can help identify an object, such as its outline (which can convey shape and size), color and product logo. For music and video, it will send a sound sample. Algorithms go to work to try to quickly match the object with something in Amazon's huge database of items.

With sound, the software can tell what type of thing it is listening to, for instance a song versus dialogue. If it's a TV show, it will identify the episode and the scene or timestamp. It will even recognize a lot of broadcast television that isn't yet for sale at Amazon.

Firefly keeps a viewable history of things you've scanned in the form of a list of image links. The links allow you to perform various actions, including viewing item information, buying the items at Amazon, saving them to watch lists or wish lists, or, in the case of scanned text, calling or texting a phone number, sending an email, adding a contact or going to a website.

Plug-ins created by developers can extend the functionality and allow things like bringing up book reviews for books, starting an iHeartRadio station based on a song or buying concert tickets through StubHub for a group whose song you scanned.

Firefly will also use the items you look up to add to its data to help it learn to identify similar objects more quickly and accurately.