Ok, I’ll say something about the iPad

I don’t have an iPad, or any immediate plans to get one, but I have been reading the usual articles since it came out. I was reading The iPad in the Eyes of the Digerati and kinda freaked out when I saw the Gelernter piece because it made so little sense to me.

[…] future touch-screens will be designed to show you a slice of time, not (like the iPad) an old-fashioned slice of space.

Really? I mean lifestreams are cool and all, but are we really going to use a visual space to represent a temporal metaphor? Like Time Machine becomes our desktop?

Oh, so he was talking about himself.

One meme that’s annoyed me since the first time I heard it, is this apocalyptic idea that we’ll all stop being creative because we have a new consumption device.

Where to start on this? Ok, so books are consumption devices. Are they bad because they don’t invite you to tinker with them? Hm… maybe it’s more compelling to cite Atlantic’s What I Read to highlight that creative people spend a ton of time as information consumers. There seems to be a huge correlation between the people who read the most and the people who write the most.

I also found the Atlantic essays interesting because many of the authors block out their days into chunks (morning, at work, evenings, etc) and break down their behavior based on those chunks. Generally speaking, the evenings are when the long-form consumption happens, because people have time to focus and inclination to sit in one place. If an iPad is convenient during that time, it means they’ll consume more or better during that time… and then the next day can sit down at their keyboards and write.

I’ve actually been working on a kind of model based on Shirky’s idea that everyone is a media outlet. There are kind of the roles that each person plays, typically cycling from one state to another during the course of a day. Those states are Consume, Evaluate, and Produce. I even drew a pretty picture about how these states kind of bind a social network together into a kind of neural information processing network. It looks like this:

So you can use this model to look at viral spreading. It is related to the network structure (who listens to whom), and the probability, for any given agent, that, having consumed something, he or she will produce something related (whether it’s simply forwarding a link, referencing it in an original work, remixing it, or whatnot).

In other words, a device that improves consumption would lead to more production, which leads to more activity in the whole network, new connections being formed, and so on. Which is good for things like networked journalism.


Random thoughts about hierarchies and networks

In my post about network democracy, I discuss how a group of people might govern itself by forming a social network of trust associations that can be collated to weigh some votes more than others. I’ve been thinking more generically about societal structures a bit, specifically how structures are selected. Ok, I don’t have anything very complete or organized here, just some thoughts.

The first is that, it seems, that both hierarchical and scale-free network structures are stable. Specifically, if you consider each person as a node in a graph, and ask how you can fill in edges to arrive at a structure that achieves maximal agreement between neighboring nodes, deep hierarchies and scale-free networks stabilize, while other configurations do not (Stocker, et al).

This observation seems intuitive: the two ways people seem to get anything done is either by a rigid command structure (firms, military, government, etc), or by loose associations of common interests with community leaders (open source, election campaigns, etc).

The second thought is about Coase’s “The Nature of the Firm”, which states that firms form despite open labor markets because of transaction costs. In other words, it ends up being cheaper to hire a bunch of people rather than contract work on a piecemeal basis.

It also seems intuitive that many transaction costs are technological. Mutual omniscience about who knows what and who to trust would make it trivial to assemble teams with shared goals on-demand. Search engines, social networks, open source software, and other innovations are far from delivering omniscience, but are definitely steps in that direction.

The other cost that Coase cites is information costs. There’s no question that the cost of discovering and sharing information (ie, communication) is approaching zero.

Ok, so the implication is that choices are made, somehow, between hierarchical structures and more free-form scale-free networks, and that a disruption in the costs associated with that choice could favor one structure over another.

I’d propose that this is what we’re seeing happen in journalism. Many of the costs of doing journalism, including The costs of gathering, sharing, vetting, and curating information are becoming so cheap, that the advantages that a firm had in the past are vanishing. As a result, a the equilibrium is shifting away from print and broadcast to scale-free networks like twitter. (Here’s an example that’s recent and local to me.)

A great case study for such a transition is in computing itself (as Jay Rosen wrote about ages ago). Clearly, open source software is based on scale-free networks, while closed-source software comes from hierarchical firms. Consider the web, and all the open protocols and standards that lower the costs of communicating and sharing as being the dominant model.

Also clear, is that closed source hasn’t gone away (Apple, Google, Microsoft, etc), and both models co-exist and have different strengths and weaknesses. In fact, it seems that closed-source thrives when it when interoperates with open source. The NYT piece today about how social network usage actually increases television popularity seems to confirm that this parallel might also appear in media.

I guess this framing serves as a kind of litmus test for “social” technologies, or ones that favor scale-free networks over firms. Their importance will depend on their ability to lower the transaction cost of accomplishing something. Increasingly, an important cost is filtering. This is a place where firms have traditionally had an advantage – eg: Firefox’s problem with too many developers.

And filtering comes back to trust.


The Demise of the Proj89 Warehouse

There was a warehouse here in the Dogpatch district of San Francisco where I live/work. It looked like it had long since been abandoned, and they started tearing it down to make room for a new UCSF hospital. It said “PROJ.89” on the side. That’s where I got the name for this blog (yay!) I almost took the sign, but it was huge and on rotting wood, so I let that go.

Anyway, I was walking by as they took down the last standing bit of the warehouse:

Here’s another view of it: (It’s the big grey box in the middle)


Network Democracy

I was a philosophy major, and so when I’m not hacking I sometimes muse about the underpinnings behind what I’m working on. Well, rarely are they so rich as with my current work. Here’s some recent thoughts:

There are several reasons for choosing representative democracy over direct, or pure democracy. One is about scalability: the larger the population, the more decisions that need to be made, and the more work it is to keep up with those issues and make informed decisions about them. Another is stability: concentrating power in a smaller number of people’s hands means that, by and large, the same kind of values and methods will be applied to making future decisions as past decisions.

Representative democracies solve the major problems of direct democracy. By delegating the right to make decisions, citizens can focus on other aspects of livelihood, and career politicians can specialize, building the background and skills necessary to make governance choices. As representatives are themselves elected by the populace, they must present themselves as good proxies for the decisions their constituents would make, and they must stand accountable to those constituents or risk losing the next election. It all makes sense.

But we can look at representative democracy, also, as an information processing optimization that minimizes the number of decisions that have to be made by each participant. It’s not surprising, when we look at it this way, that representative democracies are hierarchical. Hierarchies (also known as “trees”, since their is always only one path from any branch to the trunk) are the key to scalability–they’re why we can represent one million with seven digits instead of a million, and why databases can find a single record out of millions in a fraction of a second.

Humans form hierarchies to manage the number of decisions they need to make. Hire a good store manager, and you don’t have to worry about keeping change in the cash register. Shop around for a good job, so you don’t have to worry about running a business all by yourself. Or vote for a candidate for president, so that your guy (or gal) is in office, standing for your country.

All of these decisions have the same character: delegation. By delegating someone, you enable them to make decisions on your behalf. In order to be able to delegate, you need to trust that person. Often this comes down to a mix of reputation–what others think of them, and what you think of those who think of them–and individual judgment, whether it be an assessment of their education, a gut feeling about their eyes, or anything else.

So every decision to delegate is simultaneously about scale and stability. If it weren’t about scale, then there’d be no need to delegate in the first place If it weren’t about stability, then it wouldn’t matter who you delegated to make decisions.

With The Hourly Press, we’ve been looking at delegating the choice of who’s worth listening to. Put differently, it lets you curate the curators of information. The idea is that, by building up a medium scale network (say 50,000 people) through a couple of levels of delegation, users can get the information that’s most important to flow to them. This delegation is about scale, because users can leverage the decisions of other users: instead of following 50,000 people, they can follow 20 who they trust to make the right choices about the others. And it’s about stability, since their selections, if made well, can be expected to stay focused on a particular community, topic, and set of values.

This experiment has led me to think about how the loose hierarchy that powers the Hourly Press with simple follow/unfollow gestures in twitter might be applied to the general topic of democracy. The structure of the Hourly Press network (which we sometimes call an Authoritative Social Filter) has some aspects in common with direct democracy, and some in common with representative democracy. I think it’s something new, though, which I’d like to call a network democracy.

In a network democracy, there are two primary decisions that participants make. The first is a decision about whom to trust, or whom to delegate decision-making power to. The second is about issues–whether it be about recommending a link, or spending money, or going to war. Every participant makes both kinds of decisions. Trust delegations can be made at any time, though decisions about issues often will have a deadline.

Now, let’s look at how power is distributed in such a network. The first thing you can say is that the more people who trust you, the more power you have. Sounds reasonable. If you misuse such power, you might alienate those people and lesson your influence–so there are checks and balances at the core.

Power is also a matter of perspective. If you’re trusted by one million people, but I don’t trust you, then you might have no standing from my point of view. This kind of network reflects that fact. Put differently, power projects from some point or set of points. In the case of staying informed, it might project from your own personal perspective: these guys are worth listening to; follow them; leverage their decisions about others. In the case of social order, it might look a little different: power projects from each individual citizen, and therefore the influence someone has is proportional to the total number of people that trust him.

An assumption that I have not yet spelled out is the transitivity of trust. I would argue that this is a common factor in any hierarchical organization as well as in social networks. If you trust someone who trusts untrustworthy people, you’ll probably trust that person a little less (and vice-versa). But how long can such chains be expected to proceed, and at what rate do they attenuate?

With our work on the Hourly Press, we’ve become rather enamored of the number two. Two degrees seems to be a very good spot, as it makes it easy to scale to numbers that are statistically useful, while retaining a very transparent chain of trust. If someone two degrees away from you makes a bad decision, you can see to whom you delegated the choice of that selection and interact with (or unfollow) those people. However, in the case of three degrees, the accounting becomes much more complicated, and the actions needed to respond to a bad decision will also be more complicated.

Likewise, using a PageRank-style iterative algorithm to accumulate trust in a large scale network, while very compelling, does not seem to facilitate personal participation. For example, someone two steps away might have a very high influence due to a global calculation, regardless of your personal opinion of him. There are two problems with this: it’s necessary to communicate this increased influence, and it’s not clear how the user can respond if they disagree. In short, it takes the power out of the individuals hands’ and puts it in the collective.

Another assumption I’ve made is that trust is generic. In other words, if one person trusts another, that trust applies to any kind of decision. It certainly is easy to give examples where this isn’t the case: for example, you might trust your car mechanic with your car but not your hair, and similarly for your hair dresser. Certainly the desire to categorize people along different lines is popular in social networks, for example with twitter lists. Nevertheless, for simplicity, I’m going to assume that trust is generic for now.

Generic trust, I don’t think, is as bad as it might sound. Unless you have a particularly good relationship with your hair dresser, I doubt you’d trust them as a source of information or to make decisions about how to spend tax dollars. And if you do have a good relationship, then I’d say, go ahead and trust them. If their values are quite different then most of the other people you trust, their aggregate power over any particular issue will be very small anyway.

Now, being trusted in not just a thing that happens: it’s a process that evolves over time. The way that people will accumulate power in such a network is by being trustworthy. If they make haphazard decisions, they’ll never get into a position where their decisions matter much. Likewise, once they’ve achieved power in such a system, they are strongly incentivized to make good decisions. After all, many eyes are on them, scrutinizing the decisions that they make, and unfollowing them when they make poor ones. If the highly trusted car mechanic were to make an assertion about hair care products that was frivolous, he would jeopardize his hard-earned standing.

Notice that while the network is clearly flexible and “democratic”, the power will accumulate based on long-earned reputation. Stability is at its core. And possibly more so than representative democracy, where formal structures don’t necessarily match de facto power structures (eg, Cheney’s power as vice-president), and expensive, slow processes like elections can trail far behind political realities.

And of course it’s also scalable. Participants can get as involved or uninvolved as they choose to be. They can delegate power to the popular choices and move on with their life, or they can mine the details, figure out who makes the most sense and most worthy of their trust. And, of course, all of that work is re-usable by anyone else, who can delegate their decisions to those with more time or inclination to scrutinize.

Now, The Hourly Press is about attention, and much of what I’ve been talking about here is collective decision-making. But there’s another step to be made, bringing these ideas together: the same trust network can be used to inform as well as make decisions. By delegating trust to someone, you’re simultaneously giving them some piece of your attention. As issues come up in that person’s sphere of interest, they will show up in your inbox as worthy of your consideration. If an issue arises that you’re not interested in, then you might simply unfollow the person through whom that issue came to your attention: if you don’t care enough to know about it, you also won’t care what kinds of decisions that person makes about it.

So the scalability of the system is not just about delegation of power, it’s also about delegation of attention. If you have a new interested in, say, the use of fresh water in your county, then you’ll start finding people and media outlets that are covering the topic about that topic. As you get to know them, you follow the ones you think are saying the right stuff, and the more you do this, the more informed you will be. As issues come up for vote, you’ll be notified through these very same channels, and be able to cast your vote using the same tools. By letting individuals choose to be as involved they want, one network could scale indefinitely to cover all the kinds of collective decisions we as humans need to make.

So this is a very rough sketch and there are a number of questions that need to be worked out. Where to start? I’ve been picturing some kinds of small scale experiments. Maybe a small town or even a company or school where decisions are made either hierarchically or by voting. One could start doing experiments, letting them arrange themselves and vote using network democracy, and see what happens.

I’d expect such an experiment to start to answer a few of the huge questions in my mind, and many others I haven’t thought of. So, for instance, are the follows public knowledge, or are they secret? What would be the protocols around following and, more importantly, unfollowing someone. (If people are afraid to unfollow people because it will hurt their feelings, that would undermine the value of the network.) Should symmetric follows cancel each other out? (Hmm… that might solve the feelings problem.) Is trust generic, or do people need to classify others based on the kinds of decisions they want to delegate to them? What about transitivity of trust: is our position on two-degree networks right, or will longer lengths provide more value?

If representative democracy emerged, as I suspect, due to considerations about the efficiency of the flow of information, then maybe they’re not actually the ideal way for people to make collective decisions. Maybe we don’t need to live with “the worst form of government except for all the others”? I’d love to play a part in finding out…


Socially Augmenting Twitter Profiles with Lists

After adding lists to the hourly press as low-level abstraction, I had some time to reflect upon lists more, and what their value is. A few examples from chatting with Lyn sprung to mind:

Don’t read his bio – the quickest way to see who this guy is, is by looking at the lists he’s on.

and, jokingly,

You should call her up and ask if she thought anyone was missing from her “smart and talented” list.

This started to remind me of my experiences with person tagging at IBM, and I realized that I had already been down this same path. In fact, my earlier (unpublished) variant of the idea was “fringe folders” — almost exactly twitter lists.

What I didn’t like about lists then (and now), is that they don’t combine nicely with each other. Each is a distinct entity, and when there are many of them you have, well, simply many of them. For example, look at Tim O’Reilly who (at the time of this writing) is about to break 4000 lists. Are you going to page through 4000 lists? No one is. Had they been tags, you’d see that 3000 people had tagged him publishing, 2000 had tagged him technology, etc. This could be condensed into a nice, meaningful, overview.

Whether tags or lists are better, the end result is still important because it means that a relatively small number of people can maintain information about a relatively large number of people. What we saw with tags at IBM was that about 3% of the company was tagging 30% of the company, thereby “augmenting” their profiles. This was a socially efficient way of solving a problem. The same will likely be true of twitter lists.

Now, tags can also be used for person search and discovery. On the discovery side, clicking on a tag (say “db2” in IBM) shows you a ranked list of people, sorted by the frequency with which they’d been tagged. Since others were cautious about the way people were tagged, this was a great way to find someone who was knowledgeable about DB2 and receptive to questions about it.

Lists would be a great corpus of data upon which to build twitter user search. In fact, the same feature that makes them imperfect for discovery — their heterogeneity — will make them more valuable for free-text search.


Lists of lists

We’ve added a new feature to the hourly press—the ability to create lists of twitter lists.

The significance of this feature is that we can leverage even more decisions about who’s worth listening to. For example, compose a bunch of lists tracking a breaking news story (like the Fort Hood Shootings last week), and track all of the lists simultaneously.

We’ve also started tracking changes in the social graph, so you can see who’s getting added to these lists. In the case of breaking news, you could use this feature to see who are the new sources being discovered. Twitter doesn’t provide a streaming API for social graph changes, so we resort to polling for updates hourly.

The addition of lists was a technical hurdle, since we’d built in the assumption that the nodes in our social graph are users with twitter_ids stored as integers. We’ve expanded our definition to be more generic, and thus can now incorporate twitter lists, and potentially other kinds of attention focal points.



The last post on this blog (actually imported from smackman.com) was 3 years ago.  In that time I (a) stopped blogging (b) did a bunch of cool stuff at IBM Research before leaving this past May (c) learned to speak Spanish in Mexico and South America (d) built payyattention and the hourlypress with Christian Gromoll, Matt Gibson, and Lyn Headley.

I’ve been having an awesome time doing some technical work lately.  I’ve been processing twitter streams and distilling them into meaningful chunks.  I wanted to start writing about it, so I’m resuscitating my technical blog and rebranding it as PROJ 89.  The name comes from a huge (apparently) abandoned warehouse I walk by every day in the Dogpatch.  (I presume this is a long defunct name and I can appropriate it freely… if you know otherwise let me know!)


An Old Idea

I’ve been giving some thought to parsing microformats lately. A few threads seem to be converging…

The first is that it’s hard to parse microformats. You can hand-write a parser in a little bit of time that’s 80% right. But getting all of the hcard rules, e.g., encoded is tricky. It’s reasonable to assume, therefore, that there are a lot of 80% parsers out there like the one I wrote for my Ray Ozzie Clipboard example.

The second issue relates to hatom, which uses different class names for the same concept at different scopes. For example, the entry title is called “entry-title” not “title”. I asked Ryan about this when I saw him at www2006, and he told me that they vacillated on this decision, but they settled on “entry-title” because people can nest other microformats inside hatom, and so it would be easier for the parser writers if there were no colliding class names, even in different microformats. In fact, he suggested that they’d probably made a mistake with hcard, since the class names were so likely to collide with other microformats. Ok, so in other words “entry-title” is a hack around the problem of it being hard to parse microformats, and we can expect more of these.

When I bumped into Brian at the same event, I commented that microformats really have a problem with nesting. He agreed. He said it put a burden on the parser writer to potentially have to understand all microformats in order to reliably parse web pages that contain them.


  1. It’s a lot of trouble to write a parser
  2. Bad parsers will proliferate
  3. Microformats are evolving toward being easier to parse, not easier to create
  4. It’s not clear how you can nest microformats w/o knowing how parsers will behave
  5. Users are discouraged from inventing their own specialized microformats, presumably because of the risk of collisions and difficulty others will have in parsing them

My proposal is that we employ a very old solution to this problem: create proper, machine-readable schemas or grammars for each microformat.

The schema…

  1. is a formal specification of the microformat
  2. can be used to generate parsers (like yacc)
  3. can be used to dynamically parse new microformats
  4. is language-neutral

Here’s a fragment of a schema for hcard in a BNF-inspired syntax:

{vcard} ::= {fn} {n} [{org}] [{url}] [{email}] [{photo}] [{tel}]
{n} ::= {fn}
{tel} ::= ({tel-entry})
{tel-entry} ::= [{type}] {value}
{url} ::= a@href
{email} ::= a@href
{photo} ::= img@src | object@data
{fn} ::= body
{org} ::= body
{type} ::= body
{value} ::= body

Note that it has domain-knowledge of HTML (e.g., “img@src”, which means pull the value out of the src attribute of an img tag, and “body” means pull the body of the tag). This syntax doesn’t encode all of the kinds of rules you’ll find in the hcard spec, but it probably could be extended to do so. (Note that a link could be added to the header of web pages pointing to the schema.)

So in addition to making it trivial to generate or find correct parsers for microformats in any language or environment, how does this solve the nesting problem? First, the parser will only “find” data that matches the schema. So if you stick a hcard inside an hatom entry, then the hatom parser wouldn’t be looking for the “title” beneath the “author”, since that’s not in the schema. Second, if you wanted to have a rule like that the DOM-depth were used to disambiguate two “title” properties, then you could enforce this at the parser-generator level, not at the level of every-parser-in-the-world. Third, it’s actually possible to use link tags to refer to every schema inside the web page, making it feasible that the parser would understand all of the microformats contained in the page without any additional work.

The other thing that’s interesting is that this specification actually implies a json-compatible data-model. The “( … )” notation refers to a list, the terminals refer to values, and each of the labels (e.g., “fn”) refer to keys in a name/value pair list. So we’d expect to parse,

<a class="url fn" href="http://smackman.com">Steve</a>


{vcard: {fn: "Steve", url: "http://smackman.com" }}

in JSON-syntax. (Don’t confuse JSON-syntax with JSON-data-model. The latter can be represented in (almost?) any programming language using built-in language constructs while the former is a serialization format).

So this means that the schema spec allows you to parse from HTML to a JSON-data-model. This means that, in contrast to yacc, there isn’t a need to have application-specific instructions in the spec. I’d also point out that the process of going in the opposite direction—from JSON-data-model to HTML—is exactly what microtemplates buy you.

That’s the gist of the idea… a lot more details to be worked out, of course.


Fried pizza, really?

I’m off to Edinburgh, Scotland today for WWW2006. I have a paper in the tagging workshop. Here’s my presentation (done with S5). I used microtemplates to generate a lot of the tag visualizations.


Promoting microtemplates

It was cool to see that Elias blogged about microtemplates and got straight to the point: it’s easy.

My goal now is to try take to make this point to a few of the right people, have them get it and say something about it, and then others will pay attention. It’s, honestly, kind of a funny position to be in. I guess I do promote my ideas, at least inside the four walls of my workplace, but not usually so deliberately.

I’ve started with approaching the microformats folks, since it is, to some extent, derivative, and also the adoption of microtemplates greatly facilitates the adoption of microformats. I’ve gotten a few “very promising” remarks, but not the whoah! that I kinda expected. But I think I was being a little optimistic — it will take some time and some compelling examples for the potential to be apparent. Also, I don’t know if the microformat people are generally as concerned with creating dynamic or ajax web applications as others might be, so there’s a bit of a mismatch.

Ok, so what about the Rails folks? I started to dissect an example I found of rails programming at OnLamp and make some recommendations on the microtemplates wiki. I’ll find the discussion list and forward this to them…. but I probably still need to implement what I describe and show some examples. It would be particularly compelling if I did the same dynamic table as in this example.

The other item on my agenda is the ROCB. One idea is that if you drop a vcard on my web page, I want to be able to create the rendering of that vcard using microtemplates. I met Ray once… maybe I can drop him a note when this demo works?

January 2023