Results tagged “information shadows”

Tish Shute of Ugotrade generously invited me to present at Augmented Reality Event 2011 yesterday in a session on augmented reality user experience. My time slot was relatively short, and she challenged me to talk outside of the usual topics, so I chose to talk about something that's been interesting me for a long time: the use of non-visual senses for communicating information about the information shadows around us. In the process, I humbly decided to rename "augmented reality" (because I'm somewhat obsessed with terminology). My suggested replacement term is somatic data perception. Also, as an intro to my argument, I decided to do a back of the envelope calculation for the bandwidth of foveal vision, which turns out to be pretty low.

Here is the Slideshare version:

Scribd, with note:
Somatic Data Perception: Sensing Information Shadows

You can download the PDF(530K).

Here's the transcript:

Good afternoon!

First, let me tell you a bit about myself. I'm a user experience designer and entrepreneur. I was one of the first professional Web designers in 1993. Since then I've worked on the user experience design of hundreds of web sites. I also consult on the design of digital consumer products, and I've helped a number of consumer electronics and appliance manufacturers create better user experiences and more user centered design cultures.

In 2003 I wrote a how-to book of user research methods for technology design. It has proven to be somewhat popular, as such books go.

Around the same time as I was writing that book, I co-founded a design and consulting company called Adaptive Path.

I wanted to get more hands-on with technology development, so I founded ThingM with Tod E. Kurt about five years ago.

We're a micro-OEM. We design and manufactures a range of smart LEDs for architects, industrial designers and hackers. We also make prototypes of finished objects that use cutting edge technology, such as our RFID wine rack.

I have a new startup called Crowdlight.

[Roughly speaking, since we filed our IP, Crowdlight is a lightweight hardware networking technology that divides a space into small sub-networks. This can be used in AR to provide precise location information for registering location-based data onto the world, but it's also useful in many other ways for layering information in precise ways onto the world. We think it's particularly appropriate for The Internet of Things, for entertainment for lots of people, and for infusing information shadows into the world.]

This talk is based on a chapter from my new book. It's called "Smart Things" and it came out it September. In the book, I describe an approach for designing digital devices that combine software, hardware, physical and virtual components.

Augmented reality has a name problem. It sets the bar very high and implies that you need to fundamentally alter reality or you're not doing your job.
This in turn implies that you have to capture as much reality as possible, that you have immerse people as much as possible.

This leads naturally to try to take over vision, since it's how we most perceive the world around us. If we were bats, we would have started with hearing, if we were dogs, smell, but we're humans, so for us reality is vision.

The problem is that vision is a pretty low bandwidth sense. Yes. It's possibly the highest bandwidth sense we have, but it's still low bandwidth.

This morning I decided to do a back of the envelope estimate of how much bandwidth we have in our vision. This is a back of the envelope estimate by a non-scientist, so excuse it if it's way off. Anyway, I started with the fovea, which typically has between 30,000 and 200,000 cones in it.

To compensate, our eyes move in saccades which last between 20ms and 200ms, or 5 to 50 times per second.

So this leads to a back of the envelope calculation of eye bandwidth between 100 bits per second and 10K bits per second

That's around 4 orders of magnitude slower than a modern front-side bus.

The brain deals with this through a series of ingenious filters and adaptations to create the illusion of an experience of all reality, but at the core there's only a limited amount of bandwidth available and our visual senses are easily overwhelmed.

In the late 70s and early 80s a number of prominent cognitive scientists measured all of this and showed that, roughly speaking, you can perceive and act on about four things per second. That's four things period. Not four novel things that just appeared in your vision system--that takes much longer--or the output of four new apps that you just downloaded. It's four things, total.

This is a long digression to my main point, which is that augmented reality is the experience of contextually appropriate data in the environment. And that experience not only can, but MUST, use every sense available.

Six years ago I proposed using actual heat to display data heat maps. This is a sketch from my blog at the time I wrote about it. The basic idea is to use a peltier junction in an arm band to create a peripheral sense of data as you move through the world. So that you can have it hooked up to Wifi signal strength, or housing prices, or crime rate, or Craig's List apartment listings, and as you move through the world, you can feel if you're getting warmer to what you're looking for because you arm actually gets warmer. This allows you to use your natural sense filters to determine whether it's important. If it's hot, it will naturalliy pop into your consciousness, but it'll only be there otherwise if you want it, and you can check in while doing something else, just as when you're gauging which direction the wind is going by which side of your face is cold, and you're not adding additional information to your already overstuffed primary sense channels.

If AR is the experience of any kind of data by any sense then we have the options to associate secondary data with secondary senses to create hierarchies of information that match our cognitive abilities.

For me, augmented reality is the extension of our senses into the realm of information shadows where physical objects have data representations that can be manipulated digitally as we manipulate objects physically. To me this goes further than putting a layer of information over the world, like a veil. It's about enhancing the direct experience of the world, not to replace it, and to do it in a way that's not about being completely in the background, like ambient data weather, or about taking over our attention.

So what I'm advocating for is a change in language away from "augmented reality" to something that's more representative of the whole experience of data in the environment. I'm calling it "Somatic Data Perception" and I close on a challenge to you. As you're designing, think about what IS secondary data and what are secondary, and how can the two be brought together?

Thank you.

This is Part 6 of a pre-print draft of Chapter 6 from Smart Things: Ubiquitous Computing User Experience Design, my upcoming book. (Part 1) (Part 2) The final book will be different and this is no substitute for it, but it's a taste of what the book is about.

Earlier chapters in this series: Chapter 3, Chapter 1

Citations to references can be found here.

Chapter 6: Information shadows

Part 6: WineM, an example of design with information shadow

Figure 6-10. ThingM's WineM smart wine rack (Photo by Tod E. Kurt)

In 2007, my company, ThingM, used these ideas to design a smart wine rack (Figure 6-10). It was created to demonstrate one endpoint of a service based on wine information shadows. Every bottle of wine in it has an RFID tag (Figure 6-4), with an RFID reader in every cell of the rack. The rack, in turn, connects to an online information shadow service. This service aggregates wine information from the Internet and all of the racks that are connected to it. [Footnote: As this was a proof of concept, we implemented only bare-bones functionality and worked with a wine data aggregator (Inertia Beverage) to verify that we could get appropriate information in a production environment.]

When someone associates an RFID with a specific wine bottle, the service would connect it to all of the other wines of the same type. Many wine management services can already analyze a wine collection and recommend wines resembling those in the collection. Our service went one step further: every bottle could serve as a subscription to a data feed from the winery and to a social network of enthusiasts with similar interests. A winery could have a sale for existing owners, or recent drinkers, of its wines. Then, every rack that contained a bottle of that wine would get a message, and light up the wine in a specific color, or send a text message that said "your 2004 Domaine Roger Perrin Chateauneuf-du-Pape has mail!"

WineM is an experiment to understand how the potential of information shadows can be expressed in a good user experience. It is designed to keep the user experience focused on the experience of choosing and drinking wine. It minimizes the presence of a full-purpose computer while still providing the full power of Internet information exchange.

Figure 6-11. WineM control panel (prototyped on a Nokia 770 tablet)

For example, the interface is a faceted classification browser (Figure 6-12). Every click adds another constraint to the search set and lights up the appropriate bottles in the rack. Thus, it's possible to organize the wine not just by year or grape, but also by current market price or number of bottles in stock (or all of the above).

Hallmarks took the difficult process of identifying the manufacturer of a given piece of flatware by its style and instead made it a matter of matching a small stamp to pictures in a catalogue. Similarly, this kind of information exploration would have been very difficult with a traditional wine rack. However, once the bottles had their information shadows stitched to them using RFIDs and a simple database connected to the massive amounts of wine information online, it was relatively easy.

Next month: Chapter 8, Service Avatars

This is Part 5 of a pre-print draft of Chapter 6 from Smart Things: Ubiquitous Computing User Experience Design, my upcoming book. (Part 1) (Part 2) The final book will be different and this is no substitute for it, but it's a taste of what the book is about.

Earlier chapters in this series: Chapter 3, Chapter 1

Citations to references can be found here.

Chapter 6: Information shadows

Part 5: Design with Information Shadows

Designing with information shadows means using devices, such as RFIDs, that may have specific, limited functionality and capabilities. However, as with the FedEx example, designing with information shadows often requires global service design. Information shadow user experience design must simultaneously consider (1) what happens when every object is automatically tracked and (2) how to associate those objects with all available digital information about them.

A systematic approach to user experience design can reduce the possibility vertigo of multiplying two such nearly infinite sets. Despite the speed and novelty of changing technologies, people's underlying needs and desires change slowly. What has changed is that a new powerful tool is now available to address those needs.

The use of information shadows is still in its infancy, but several interesting design properties of information shadows have emerged:

  • They simplify the design of certain kinds of devices.
  • They allow designers to treat dedicated devices like physical embodiments of Web services and create mashups.
  • They allow mass customization of experiences without mass customization of objects.
  • They allow devices to be self-disclosing for disposal and recycling.
  • They blur the line between devices and services.
  • They create novel, pleasurable, entertaining experiences.

These are described in more detail below.

Information shadows simplify devices

When an object no longer has to display all of the human-readable metadata needed by users, its design can be simpler. The labels on bags of chocolate chips only have room for one or two recipe suggestions. Now, the chocolate chips can have their own cookbook, and the label only to point to it. Similarly, devices can be simplified down to the single thing they do best. You might want to use a pedometer to track miles walked each day for a week. The pedometer interface can be quite minimal if devices—such as a mobile phone—can access that pedometer's information shadow. The pedometer itself just needs a power button, status indicator, and walking progress display. Other devices—with larger screens and more computing power—can focus on helping users make sense of information about their exercise plans.

Physical/Network mashups

Ubicomp mashups attempt to move computation off the desktop and integrate it with the artifacts of everyday life. They extend beyond the Web and combine the functionality of both software and hardware components. (Hartmann et al, 2008)

Many Web-based services have published API (application programming interfaces) that allow other services to use their information and computational capabilities in novel ways. Google Maps, the classic of the genre, allows developers to layer information over map images that Google provides. Physical/network mashups create novel experiences that merge the power of simple, lightweight devices with the power of existing Web services.

Figure 6-9. Tweet-a-watt (Fried, 2009)

Fried and Torrone's 2009 Tweet-a-Watt project (Figure 6-9) is one such mashup. It posts electricity use to Twitter, using the same API that's normally carries people's Twitter posts about their own activities. But Twitter can easily broadcast information about devices, making hour-by-hour updates about energy use accessible to humans and readable by software.

Ubicomp device user experience designers can hook up information about objects to data sources on the Internet using the same APIs and protocols used by web site mashups. [Footnote: Bleecker (2005) takes it further, and defines the term blogject to describe devices that act like people on the Internet. They can blog, they can post to Twitter, they can reply to human conversation. For Bleecker, "blogjects become first-class a-list producers of conversations in the same way that human bloggers do—by starting, maintaining and being critical attractors in conversations around topics that have relevance and meaning to others who have a stake in that discussion."] These physical/network mashups build on existing web design methods and provide a familiar set of web concepts to describe how physical objects and online information can interact.

Mass customization

The digital information shadow associated with an object is much easier to change at whim than that object's physical form. Mass customization of experiences gets much easier when the majority of the customization happens digitally. For example, the WebKinz toy line (See Chapter 7) connects toys that physically differ only slightly (like Cabbage Patch dolls of an earlier generation), but have rich online personalities.

Conversely, merging information shadows with rapid manufacturing techniques such as 3D printing allows for the instantiation of data in a physical, purchased object. Materialise, a Belgian 3D printing firm, sells a line of intricate designer lamps for the high-end furniture market (under the .MGX brand). Each lamp is individually printed. When first introduced, every lamp came with a disk containing a CAD file describing how to recreate that lamp. It's the lamp's DNA and part of its information shadow. Since Materialise keeps copies of the files, the lamps are, in effect, immortal: if a lamp is broken, they can print another one. Each lamp can be unique, or replicated as often as a buyer wants.

What happens to "mass production" when an object's physical form is based on a unique digital file? In a sense, we can now go back to a pre-Industrial Revolution era of unique objects. But now the uniqueness stems not from the imperfections and unpredictability of hand craft processes, but from a manufactured object’s relationship with its informational shadow.

Smart disposal and recycling

Because information shadows can contain any kind of information, they can contain instructions for how to dispose of the object they shadow. [Footnote: I first heard this idea in a lecture by Bruce Sterling. In Sterling (1999) he writes "Smart garbage doesn't fester in darkness, ignorance and denial. It becomes a resource."] They can self-disclose not just what information they collect and use (as per Greenfield, 2006), but how to fix, disassembled and recycle them.

For example, information about the materials from which the object is made can be mashed up with a database of municipal recycling rules to generate instructions for how to locally recycle the object. San Francisco—where I live—has an advanced recycling program that automatically distinguishes between many materials. However, I still don't know if I can put a steel car part, Styrofoam or shoes with a "recycle" logo on the sole in my plastic recycling bin. The rules of what is acceptable, and how to prepare it, change regularly. An information shadow mashup linked to each object could clarify that question instantly, directing me to take my esoteric recyclable to a specific location or to treat it in a specific way.

Similarly, complex items, such as consumer electronics or robotic toys, are difficult to recycle because they require too much disassembly and contain unknown materials. For the municipality, these objects' information shadows could contain disassembly instructions and complete materials lists. With more information, city systems could know what do to with old toys besides sending them to the dump.

Information shadows enable new kinds of services

Note: See Chapter 8 for a more general examination of this topic.

By giving objects unique identifiers, shadows allow those objects to become the subjects of services that track them and interact with them. Everyday objects can become subscription services. [Footnote: Again Sterling got here first. In Sterling (2001) he describes a furniture subscription system that creates one-off customized furniture on demand.]

In the days before the breakup of AT&T, Americans didn't own their own telephones. They leased them from the phone company. Although "Ma Bell" limited the range of phone choices, the phone company was required to repair broken equipment. The company could arrange update the whole system systematically and thoroughly, whenever it wanted. Though not ideal, the system had its benefits. It was also nearly impossible to replicate without the resources of an enormous company like AT&T. Information shadows could facilitate similar—but not so resource-intensive—for many other kinds of products and consumers.

For example, a shoe company could sell sustainable shoes by subscription. The shoe easily disassembles, yet is sturdy and comfortable. Buying the shoe means buying into a subscription for that shoe. As one part wears out, or as fashions change, the shoe can be disassembled, and mailed to a central warehouse, which mails back a replacement part. The shoe's information shadow says exactly which replacement it requires.

More directly, unique item-level identification allows for services that determine authenticity and trace provenance. In some parts of Africa, 30% of pharmaceuticals are counterfeit. mPedigree is using unique identifiers, printed under a scratch-off material, to identify authentic drugs (Schenker, 2008). Sending the number by text message to a trusted central location checks the authenticity—and then the expiration date—of the pharmaceutical. If the identification number is valid and the medicine's shelf-life has not expired, the system sends back another text message with a simple affirmation. Similarly, a purchaser can use the information shadow of a grocery item to trace its progress back to the farm where it was made and verify whether their farming practices are sustainable and humane. Similarly, an expensive designer handbag can be quickly authenticated.

The service possibilities of information shadows are enormous.


Once an object is identifiable and trackable, it can become a token in a game. One of the earliest such games was "Where's George?" (, which traces the passage of one dollar bills across the world using the bills' serial numbers. Many people find it fascinating to see where the bill they're holding had been and where bills they entered into the system have gone.

That is, of course, just the tip of the iceberg. Mediamatic, a technology design organization, challenged a group of designers at the PICNIC 08 conference in Amsterdam to create social games using RFID tags that every conference participant was given (Mediamatic, 2008). After a week of hacking, the 30 people in the workshop had created ten functioning games (Table 6-1).

Table 6-1. Games developed usink ikTag RFID tags in a week of hacking by groups at the PICNIC 08 conference. The text is from a flyer printed by Mediamatic, the organizer of the hacking week.

  • ikRun. Run from the conference to the PICNIC Club and record your fastest time and finishing photo. Scan your ikTag at the start next to the E-Art dome, scan again to finish and win!
  • Friend Drink Station. Free drinks for new friends! Mediamatic offers a free drink and a new friendship in the network. Just swipe your ikTag and push the button.
  • Department. Use the ikTag to see what the Department of Information Security & Privacy knows about you. The DISP is buying privacy and selling security.
  • ikCam. Swipe your ikTag to add your portrait to your profile. Or gather up to 20 friends with ikTags and make shapshots!
  • Breathalyzer. Use the ikTag, blow into the straw and test your alcohol intake. The outcome will be published on your profile. Compare your drinking skills with others at PICNIC.
  • ikWin! Use your ikTag to challenge someone in a battle for Google ranking. Two scissor lifts will go up, the more hits, the higher you go.
  • Mobile Massage Couch. Sit down on the two seater with a new friend, use your ikTag to get a free massage. You can win bonus time as a gift from the crowd.
  • DuckRace. 2 players start their race cars with their ikTag. The race track is based on your profile and network. The audience will influence your race car with their ikTag.
  • Breedrs. Drop your ikTag in the Breedrs Pond and see it evolve into a creature, with DNA based on your profile. Is this love or war?
  • Vbird. Contact the Vbird with your ikTag, help it fly, meet new friends and find the interactive film in your profile.

These games represent a completely new genre of play, one that mixes physical objects (like scissor lifts, couches and breathalyzers) with online information (profiles and Google ranks) and computation. The possibilities implicit in this one week exercise are fascinating and exciting. They imply that information shadows can touch all aspects of everyday life.

Tomorrow: Chapter 3, Part 6: WineM

This is Part 4 of a pre-print draft of Chapter 6 from Smart Things: Ubiquitous Computing User Experience Design, my upcoming book. (Part 1) (Part 2) The final book will be different and this is no substitute for it, but it's a taste of what the book is about.

Earlier chapters in this series: Chapter 3, Chapter 1

Citations to references can be found here.

Chapter 6: Information shadows

Part 4: The Internet of Things

The Center's mission is to create an "Internet of Things" that will: merge the centuries old "network of atoms" (the production, distribution, sale, use & disposal of products) with the "network of bits" (the Internet).

- Kevin Ashton, executive director of the MIT Auto-ID Center, 1999

The concept of information shadows is intertwined with the "The Internet of Things," a term coined by the staff of the MIT Auto-Id Center in 1999. The familiar Internet of bits is of course still made of things, but these things are primarily computational devices (routers, modems, etc.) whose purpose is to store, manipulate, and transmit data. These things generate the form of the Internet, not its content. People experience the Internet, however, through its content. This content, for now, exists for most people only through general-purpose digital devices such as laptops and mobile phones. [Footnote: Even specialized net-aware devices (Game consoles, ATMs, home security systems, etc.) are clearly presented as digital products whose connection to an electronic network does not come as a surprise.]

In 1999, MIT Auto-Id Center's vision was that non-electronic things should also have digital identities. The Center's main focused was automatic identification [Footnote: Leading to a number of visionary proposals for how such identification would work (Brock, 2001-1, 2001-2, etc.).], rather than the digital social life such identification creates for things. Information shadows, however, enable more symmetrical experiences than just identifying non-digital objects, as economically and socially powerful as that is. The Internet of Things has by now taken on a broader meaning. It describes the collection of all objects with information shadows, whether those objects' relationship to the Internet is asymmetrical (as in the case of identification and tracking) or symmetrical.

Figure 6-8. FedEx SenseAware (Courtesy FedEx)

The possibilities created by feedback between an object and its information shadow are immense. A shipped object can conceal its actual destination, only revealing the next step in its path to a shipper and dynamically adjusting its route if diverted. It can validate its authenticity and refuse to function if the person holding it is unauthorized, with the list stored in its information shadow. A FedEx's SenseAware smart tag (Figure 6-8), for example, reports not just the location of its package, but also the environmental conditions of its transfer. As it travels, it adds its location, temperature, pressure, humidity, and whether (and when) its box has been opened to its information shadow. In a more speculative scenario, a SenseAware-equipped box could theoretically reroute itself if storage conditions were likely to have caused its contents to spoil. [Footnote: FedEx does not currently offer this service. This scenario is speculative only.]

Despite its roots in shipping, an Internet of Things could use information shadows in a wide variety of ways. A motorcycle could have a strain gauge built into its chain, and store both the status of the chain, and the specifications for a replacement, in its information shadow. When the chain was stretched to the point where it needed replacement (an important part of motorcycle maintenance that can lead to very dangerous conditions if not performed), the motorcycle could log that state in its information shadow. This, in turn, could trigger an alert on the motorcycle owners' phone and on the motorcycle's dashboard. The repair shop could interrogate the motorcycle for the kind of chain last used and the chain performance characteristics indicated based on sensor data taken since the last replacement.

Tomorrow: Chapter 3, Part 5: Design with Information Shadows

The blog of future think design consultancy PSFK interviewed me by email.

In the interview I talk about the book (of course) and ThingM's upcoming products. I also took the opportunity to think about how I've noticed the trend of services that provide data streams, rather than just units of data:

I think that there’s a really interesting trend in opening up data sources. Pachube works as a free data stream brokerage that sits on top of TCP/IP and HTTP to provide a kind of semantic resource location technology for small net-enabled devices that has been missing. This kind of data openness is being matched by things such as the US Governement’s open data initiative at

The trend I see here is a combination of openly sharing data sources and streams and creating business models around making technology layers that make those data streams meaningful and valuable. Both Pachube and are a kind of search engine for data streams, rather than documents, which I think is a very powerful concept.

This is definitely related to the discussions around syndication that have been going on for years (since the launch of RSS), to micro-content, and to various services that add structured semantic information to Web-accessible data. However, I think what we're seeing now goes beyond those largely abstract discussions to create a more pragmatic understanding of what it means to create meaningful sources of data, rather than just meaningful units of data.

It means, as my last sentence implies, that there are enough data sources--whether it's sensor data automatically collected, organized and tagged by Pachube or the human-created sources of data presented by we can start having search services for such data. The conversation becomes again about "wrangling" information shadows, as I discussed in my NASIG keynote two years ago.

In that discussion I talked about how journal subscriptions--which are a kind of knowledge white hole, wellsprings of specific kinds of information--represent a model for how information shadows can be organized and managed in the future. Well, it looks like we may be closer to that, and that the wrangling may be a combination of automated tagging and human curation.

Does this mean that Google will soon be automatically cataloging data streams? I'd be surprised if they're not already.

In writing my book, I've been trying to keep track of companies that are creating consumer-facing information shadows for various kinds of products (as opposed to the other kind of item-level identification technologies that are primarily for use by businesses in their logistics operations).

Here are several of them, and the products they're tracking:

- High Fashion, CertiLogo (a client of mine)
- High Design, ThingLink
- Food, TraceTracker (and a BusinessWeek story about them)
- Food and high technology, YottaMark
- Food, FoodLogiQ
- Pharmaceuticals, mPedigree (another BusinessWeek story about them)
- Goods in general, Sproxil, which appears to be a for-profit venture by the founders of mPedigree, a nonprofit.

There are many other companies that are doing other kinds of identification (for example, tesa scribos, but

I gave a presentation at the Web 2.0 Expo today where I tried to tie together the basic tenets of Web 2.0-style thinking about sharing data through open APIs and the promise of embedded information processing and networking distributed through the environment (i.e. ubicomp).

Here's the description:

Ubiquitous computing has been here since at least 2005, but we may not have noticed it. Computers are rapidly fragmenting from expensive general-purpose devices to cheaper specialized networked tools (phones, netbooks, desktop RFID readers, MP3 players, running shoe sensors, etc.). These tools bridge the physical world and the Internet in new ways, often using Web 2.0-style interaction to create unexpected ways to work and play in the real world while simultaneously having the power of the Net available to us. This talk will discuss how mashups between meatspace and the Net have already happened, what the emerging patterns are, and how widgetization is about to jump from social networks to devices and then disappear altogether.

The presentation with full text is available here (1.2MB PDF).

I also realized belatedly that I never once mentioned "The Internet of Things" as a unifying concept, but it's definitely what I was talking about. My apologies.

I presented a talk at ETech today. It links the capabilities of ubiquitous computing and intersects it with service design to come up with a justification for creating subscription-based services out of (certain) everyday objects.

The original description is

Things have long had identifying marks, from silversmiths’ hallmarks to barcodes, but mating machine-readable identification with pervasive networking greatly increases the value of the marks.

For example, when a machine-readable identification method such as an RFID or a high-density visual code is combined with the wireless networking of a mobile phone, a new way of interacting with everyday objects is created. Once you have the capability uniquely identify anything immediately, you can attach meta information to it. Any meta-information. How much is this worth on eBay? Which of my friends has one? Will this go with my Mom’s china? Will it make me sick if I eat it? Was it made by children?

I call this digital representation as accessed through a unique ID, an object’s “information shadow” and I now see them attached to just about everything. Beyond getting meta information, however, lies an even more powerful concept: changing the physical object to a service, for which the thing you’re looking at is but a single instantiation of that agreement. It’s already happened to media, and to car-shared cars and shared bicycles in urban areas.

When this happens, the objects have to change at a fundamental level. They have to be designed differently and they have to be described and discussed differently. The “owner’s” relationship to the object changes. The very idea of ownership changes. The solid object grows a dotted line that is filled-in as-needed, when-needed, and with the features that are needed. This is not the same thing as renting or co-ownership, its anytime/anywhere nature-enabled by the underlying technology makes these new service objects fundamentally new.

This talk will discuss the implications of the social and design changes created by these technologies and give multiple examples of services that already exist.

I've put up a PDF with all of the images and notes(884 PDF), and Slideshare, which is missing many of the images (I think it doesn't know what to do with pictures that have been pasted into Mac Powerpoint 2004), but still has all of the text.

About six months ago, I was invited by the North American Serials Interest Group (NASIG) to keynote their annual conference. After admitting that I didn't know what "serials" were (think periodicals, journals and other similar things), I realized that this was a perfect opportunity to address a unique group, so I jumped at the opportunity. Serials librarians, and publishers of serialized works, have been at the forefront of understanding the relationship between physical objects, digital objects and how the two relate to each other. The music industry, as we know, tried to fight the conversion to digital of the physical objects with which they had traditionally made their money . As soon as Internet search engines appeared, and I suspect as soon as Google started indexing PDFs, the serials world realized that digitization of their content made it simultaneously more visible (since now the most obscure journal could now be found and cited) and possibly less valuable (since it could not be copied easily). Moreover, libraries quickly realized that online access changed their understanding of what it means to the products of their subscriptions. Most people no longer look at the physical journal in the library, which the library clearly owns outright, but look at articles through online services. The questions then become: how is that paid for? What is owned? What happens to the "owned" content when publication goes out of business?

These are profound questions at the core of nearly all modern digital products, and instead of hiding from the problem, the serials world has quietly and methodically tried to articulate the questions and negotiate answers. Publishers, libraries and information brokers all participate in the NASIG conferences and discussions, and they've collectively come up with a range of fascinating solutions ranging from license models to technologies that preserve access to owned resources when the original provider disappears. In short, they've spent years trying to answer questions that are only starting to occur to folks in other disciplines.

My presentation, called Information Shadows: How ubiquitous computing serializes everyday things (1.2MB PDF) is my attempt at showing how ubiquitous computing technology is, in essence, turning whole classes of everyday objects into serials, or services, by creating pervasive digital access to the objects' metainformation, their information shadows. In the process, I talk about blenders, timeshares, Cuddle Chimps, City Carshare, and Exactitudes. I think it's a fun talk, and I'm really happy to have had the opportunity to articulate these ideas in this forum.

Luxist reports on a new service to help track the provenance of wine. When Tod and I were at NextFest we spoke to some folks at Hitachi who had contemplated using their RFID technology to do the same thing, but just recently two new services have come online that are designed to track wine by the bottle (the other is Great Wall of Wine, a Chinese merchant, who are using RFIDs to stop counterfeit wine).

This seems to be an unnecessarily narrow use of the technology. Counterfeit wine is a problem, but (in my understanding) it's primarily it's a problem with a few very old, very expensive bottles. Old wine is not going to have RFID stickers on it, and if one is applied in such a was as to not damage the expensive bottle, it'll probably be easily moved to another bottle. Moreover, the business plan (as I understand it) is dependent on Metcalfe's Law network effects to work. In other words, it becomes useful if lots of wine has the stickers (the old "if only you have a fax machine, then faxing isn't a useful technology" argument), but initially very little wine is going to have stickers on it. For systems like that to be successful, the technology needs to be useful to just a single user. That's exactly the situation where a rich information shadow becomes valuable, and if you excuse the self-promotion, that's why we made WineM focused on the richness of wine's information shadow rather than just a single application. But maybe I'm misunderstanding the business model. Maybe the people who really need these systems are not end consumers, but dealers, distributors and regulators. Regardless, between these projects, Smartcorq and the Queen's ISETAN experiment, it looks like wine and RFID are going to be mated at some point in the very near future. The question is whether the power of all of the information about the wine will be exposed to end users (to everyone's great benefit, in my opinion) or whether the technology will remain stuck in the realms of logistics and security.

Over the years there have been many projects that use mobile phones to associate physical objects with their information shadows (YellowArrow, QR Code, ScanLife, etc.). There have also been many projects that use phones in social setting as a way for people to find out about the people around them (Nokia Sensor, Dodgeball, etc.), usually with the promise of dating. None have been particularly successful, in either category (though I gather that QR Coded stuff is pretty common in Japan).

Now comes a company that's trying to combine the two unsuccessful ideas to make one successful one. Wickd is betting that there will be enough people who want to snap shotcode barcodes on t-shirts to find out about someone, and that there will be enough people who wants to wear those (kinda expensive, kinda plain) t-shirts to create a critical mass of urban singles willing to pay for the privilege to make their business model work. I'm pretty dubious of the underlying interaction model (it's more related to first person shooters--where you run up to a target, fire your weapon while they're not looking, and run away--than dating, where you get close, interact face-to-face and ideally stay close), but for me it's another example that people are the most popular object to unify with its information shadow. Having information about the people with you at brunch or at a conference (which is where an adult version of Disney Clickables could be very valuable), and having your intersection in physical space mapped to social networking space opens up huge possibilities for maintaining your social network.

All of these projects point to the inevitability of that unification, it's just that no one has found the right vehicle to move the penetration of the technology to the appropriate place on the Metcalfe's Law curve.

Oh, and two final words of advice to the Wickd people: Hanky Code. Think about it. ;-)

[UPDATE: Several people suggested I should be more clear about why I think that this is a questionable idea. OK, here it is: I think that no woman in her right mind wants to wear a shirt that gives random people behind her back personal information. It's not the quality of the information or the content, it's the coupling of that with anonymity and immediate physical presence.]

There are two projects I've become aware of recently that represent the explicit linking of physical objects to their information shadows, both in children's products. This kind of thing has existed before, but its prevalence seems to be on the rise.

The first (found by Liz) is WebKinz which are plush toys that each have a unique code on their tag that brings up a unique play space that's just for that specific toy (randomly generated of course). The idea is "Beanie Babies meets NeoPets meets Cabbage Patch Kids" and although I think the execution of the concept leaves much to be desired (why buy furniture for your toy basset's online "room"?), it's an interesting example of how toy companies are merging offline and online conceptual play spaces in a very direct way.

The second is Disney's Clickables, which I learned about from CNET's Matter/Antimatter blog.

Clickables that we are launching in connection to our new Disney Fairies virtual world. It's a way for kids to take their online world experience into the real world. The core of it is a magical bracelet. By simply clicking their bracelets together, girls become friends in the online environment.

From the press release:

“The future of toys is about connecting online and offline play,” said Chris Heatherly, vice president of technology and innovation, Disney Consumer Products. “Kids and tweens are quickly embracing virtual worlds and, while there are several Internet-related toys in the market today, the play ends when the computer gets shut down. With our new line of Disney Fairies toys featuring Clickables technology, we're bringing the fun of social networking, collecting, and trading into the real world so that girls can extend the fun of the enchanting online world of Pixie Hollow to school, the park, or wherever they may be.”

This system of course owes a lot to Ruth Kikin-Gil's Buddy Beads project in terms of its use of jewelry to communicate social relationship between BFFs, but it also explicitly links the online world to the physical world using magic as a metaphor. It's not surprising that it's coming from Disney thematically, but what's interesting to me is how much Disney is investing in it. This is a sizable product rollout, which typically means that they have done enough research to believe that it'll be successful on their terms, which typically means hundreds of thousands, if not millions of customers. It's a project, and a genre, to watch.

I spoke last night at Berkeley's School of Information Future of Interaction Design lecture series, presenting the "Sketching Smart Things" talk I gave at BayCHI last month and at CHIFOO the month before. I'm evolving this talk, rather than doing every talk from scratch. There's about 80% overlap with the previous talks, though this time around I've added several slides to explain the origin of the Information Shadow idea by citing Tom Coates' and Ulla-Maaria Mutanen's work, and I've referenced Bag, Borrow or Steal when talking about how digital technology is shifting the nature of everyday objects into subscription services.

You can download the presentation with all the text as a 900K PDF.

(image by MGChan, found on Flickr)

That last reference shows my current interest in the way that digital networked technologies allow for objects to shift from "buy and store" model to a "rent and share" subscription model. Bag, Borrow or Steal, City Carshare and timeshared condos (thanks to Nicholas Nova for reminding me of this) are all occasional use/high price products that technology has changed the ownership model for. What's a high price niche functionality today becomes commodity functionality eventually. Netflix has done it for DVDs. In one of Bruce Sterling's original Viridian speeches from 1999, he brings it all the way don to the most commodified of tools, the hammer:

If all your possessions are network peripherals, then you have a possible LINUX model for objects in the real world. In this world, I don't buy a hammer. What I really want to own is the hammering functionality. I might as well share the hammer with my neighbor == he can't steal it, and if he breaks it, I'll know immediately. A modern hammer in this world comes built around a chip, with a set of strain gauges that determine if it is worn or broke or abused. Let's network that hammer.

Berkeley's famous Tool Lending Library did a low-tech version (the subscription price for it is the cost of owning a house in Berkeley), and Ford, DeWalt and ThingMagic are tantalizingly close with their Tool Link product:

The innovative Ford Work Solutions Tool Link from DeWalt uses RFID technology to track what's in your cargo box and what isn't. Checking Tool Link before heading out to a job site ensures all tools you need are on hand. At the end of the workday, Tool Link guarantees all the gear used at a job site is back onboard.

The Ford version is a kind of personal inventory control system, but once every tool has an embedded RFID tag in it, you can start doing all kinds of things, including the kind of subscription-based resource sharing that Sterling alluded to. Soon, though, more occasional-use products will become dotted outlines that get filled in as we need them.

[Update: Treehugger has an article that says that services like Bag Borrow or Steal are Product Service Systems by the EU. Their definition is "in essence they are a means, by which we get what we want, without needing to own the product that provides that service." I think that the term, and its PSS acronym, sounds too abstract and generic and that the idea would spread if it was called something more informative and evocative--I dunno, "library" for nonprofit ones and maybe something like "thingshare" (riffing off of "carshare" and "timeshare") for the for-profit general class.]

[Update: Phil points to Jeremy Rifkin's Age of Access as a book-length discussion of some of these ideas with the core thesis being "Property [in the age of networked information] continues to exist, but is far less likely to be exchanged in markets. Instead, suppliers hold onto property in the new economy and lease, rent, charge an admission fee, subscription, or membership dues for its short-term use." I haven't read it--it's on order now--but like much of Rifkin's work, it seems like there's an essence of truth to the idea even though the presentation is hyped.]

Last night I presented a version of the Sketching Smart Things talk I gave last month at CHIFOO to BayCHI. It was an honor to be invited to speak there because BayCHI is such an institution in the HCI world and because the talk was in the PARC auditorium, feet from where the core concepts of ubiquitous computing were first formulated. Thank you, BayCHI and Rashmi!

The presentation is available on Slideshare:

And as a 710K PDF where you can see a complete transcript of my talk in the notes.

(photo from Flickr, (cc) dailydog)

CHIFOO, the CHI forum of Oregon, invited me to speak at their January gathering, and it was an honor and a pleasure to accept their invitation. Their lecture theme this year is "From Ideation to Innovation," and I used the theme as an opportunity to describe our recent projects, including our work with the Henry Ford, and our products, and the theoretical framework that we're developing to think about ubiquitous computing user experience design and incorporating the principles of agile software development into design.

The full presentation is available as a 1M PDF.

Here's a highlight, the ThingM theoretical framework:

1. Information Processing is a Material
When a designer can include information processing in a product for very little cost, the calculation becomes not one of engineering complexity, that’s relatively cheap, but one of competitive advantage. What you do with that CPU becomes part of the design of the product and needs to be designed with the same attention to the other parts as any of the materials being used. And just like a material, it creates some new capabilities, and imposes new constraints.
2. Applianceness
Coined by Bill Sharpe of the Appliance Studio, states that applianceness is "the set of properties that guide the design process towards simple, helpful devices that exploit the potential of embedded information technology in everyday things." The core of the idea for me is that focus in functionality is more important than arbitrary flexibility. When computation is cheap, we no longer have to make general purpose computers. (Sharpe and his colleagues at an earlier incarnation of the Appliance Studio also did an excellent set of design principle cards (120K) that I still carry around)
3. Physical Objects Cast Information Shadows
In our modern world, everything exists simultaneously in the physical world and in the world of data. Nearly every object’s information shadow can be examined and manipulated without having to touch the physical object. Think of the Amazon and Google book APIs. Information shadows have lives of their own. Wine has a particularly rich one.
4. Devices are Service Avatars
Networks mean that the same information can be accessed and manipulated through a variety of devices. Most value rests in information, rather in the device that’s communicating it, which means that the devices become secondary. A number of familiar information appliances--cell phones, ATMs--are basically worthless without the networks they’re attached to. They are physical manifestations, avatars, projections into physical space of services, but are not services themselves. This means that when thinking about how to design user experiences for ubiquitous computing, the design of the service becomes as important as the design of the device. (I wrote more about this idea a couple of years ago)
5. Granularity Determines Key Aspects of Experience Design
Ubiquitous computing devices can come in all sorts of sizes and the user experience design for them must take this into account. This has been true since the earliest days at PARC when Weiser defined the tab, pad and board as names for the scales of the devices they were developing. I use a different set of terms, but the key idea is the same: what works at one granularity doesn't necessarily work at another.
6. Magic is a Powerful Interaction Metaphor
The concept of enchanted objects can help generate ideas about interaction and as a way to create user experiences that are easier to explain. People have a tendency to create animist explanations for the behavior of technologies that exhibit unpredictable behaviors. They treat their Roombas like pets, they get mad at their laptops, they think their iPod is obsessed with a band, etc. We can use these natural associations to design ubiquitous computing interactions. (I've written and talked about this idea more extensively before)

[I wrote this for the ThingM newsletter that went out yesterday, but thought it may be of broader interest]

Wine keeps reappearing at the intersection of the digital world and the physical one. Bruce Sterling's pioneering book on the implications of ubiquitous computing, Shaping Things, uses it extensively as an example, but he wasn't the first to discuss it. Wine is the textbook example (literally) in the Information Architecture world, where the problem of organizing is often used to explain an approach known as faceted classification. Virtual Vineyard (arguably the first successful ecommerce website) launched before Amazon did.

Why? Our theory is that wine exists in two worlds: as a physical object and as an informational one. The informational object doesn't just exist as a way to help people select wine to drink, but the information about the wine becomes an important part of the process of collecting wine. Moreover, unlike other collectibles that exist as physical and informational objects (think Magic the Gathering cards), wine is a consumable. You can never get a complete set and what you have is always shrinking, so there's a perpetual pressure to gather new information to gather new wine.

The problem is that wine bottles are terribly difficult to track. As collectibles, there are market pressures to create scarcity, which leads many producers (especially of high-end wines) to avoid using the most common object tracking mechanism, the UPC barcode. Barcodes symbolize mass production to wine producers struggling to create scarcity, so they don't use them, or use them haphazardly. We feel this ends up backfiring on wine producers, creating obscurity instead. Wine is a classic Long Tail product: in other words, there's a huge volume of potential in the obscure end of the market, but despite wine's early entry as objects of cutting-edge technological consideration, it hasn't achieved nearly its potential.

We believe the core problem is that most wine is virtually untrackable in the information space. It's a physical object that has no anchor to which to attach data. There is huge potential in creating such anchors. Ulla-Maaria Mutanen created the Thinglink project to create "social objects" that "make it possible to 1) aggregate online discussion around particular items, 2) track their history, and 3) develop new ways of connecting through particular objects on the web." She's talking about handicrafts, but the same thing can--and should--apply to wine.

However, when I went to the Wine Industry Technology Symposium a couple of weeks ago, there was virtually no discussion of ideas like this, even as the group discussed the power of "Web 2.0" and social networks.

Since we're currently working on an RFID wine rack, we're thinking a lot about these issues. We would like the answer to be RFIDs embedded in wine labels (invisibly) coupled with open, shared communication standards for exchanging wine information. These should look forward toward the capabilities of the technology and the "social life" of objects that bridge the information and physical worlds, rather than trying to copy UPCs or ISBNs, as valuable as those have been. Until then, wine, that most textbook example of hybrid objects, will be frustratingly out of reach for consumers, who then will be themselves frustratingly unavailable to producers. It's a situation that could be much better (i.e. profitable and enjoyable) for everyone involved.





A device studio that lives at the intersections of ubiquitous computing, ambient intelligence, industrial design and materials science.

The Smart Furniture Manifesto

Giant poster, suitable for framing! (300K PDF)
Full text and explanation

Recent Photos (from Flickr)

Smart Things: Ubiquitous Computing User Experience Design

By me!
ISBN: 0123748992
Published in September 2010
Available from Amazon

Observing the User Experience: a practitioner's guide to user research

By me!
ISBN: 1558609237
Published April 2003
Available from Amazon

Find recent content on the main index or look in the archives to find all content.