RSS 
Email 
Twitter
FacebookI Need Your Human Aggregated Content
June 29th, 2009 | Posted in Web 2.0 8 Comments »
If you have a way of tagging or marking the good content you read online — such as adding it to a specific category on your blog, bookmarking it through Delicious, or putting the link on some other online site — send me the RSS feed for it, and I’ll add it to the Yahoo Pipes aggregated feed that I have going with Writer River.
Here’s what the Yahoo Pipes feed looks like at the moment.
Writer River Yahoo Pipes feed
It’s simple compared to other Yahoo Pipes feeds. Basically the pipe takes RSS feeds from as many sources as I add here, sorts the posts by the date published, filters out any duplicate titles, and then merges all the information into one RSS feed. Writer River then displays this RSS feed on its home page. When you subscribe to the Writer River RSS feed (or when you subscribe to Writer River’s email delivery or Twitter updates), you’re also subscribing to this same Yahoo Pipes feed.
I’m convinced that human-assisted aggregation and filtering, with the help of such tools as Yahoo Pipes, is the trend for managing the deluge of information online. Since everyone is an author, publishing on separate sites, RSS is the only way to keep up. And people are publishing like mad, pushing out about a million posts a day.
Post titles are often hit and miss in terms of quality, so some human filtering is necessary. We need people to pick and choose the good content from the poor. People are naturally doing this all the time. I’m just trying to leverage those efforts in an effortless way to pull all of this good information into one running feed. This is what Writer River is all about. It attempts to gather all of this worthwhile content and help you find better information more quickly. If enough people participate, the quality of content flowing through Writer River could easily surpass the quality of any print publication.
Here’s an example. I like UXMatters, but I missed the latest articles published on it because I have hundreds of feeds in my feedreader and I don’t sit there watching feeds all day. However, Alistair Christie saw an interesting UX Matters article by Mike Hughes and posted briefly about it in his What I’m Reading category. I saw it on Writer River because Alistair told me about his What I’m Reading feed, and I added it to the Yahoo Pipe that’s feeding Writer River. I checked out the article tonight and immediately felt it was a valuable post. Without this human filtering and aggregation, I would have missed the post.
Now imagine if not just one or two people submitted similar What I’m Reading or What I’m Bookmarking feeds to Writer River, but dozens, even 100 people. It would be like having 100 researchers scouring the Internet for you, looking for the best posts available.
Here’s a little more math. Let’s say on average, the 100 researchers post one article a day to their What I’m Reading feed — one article a day they feel is worthwhile. In one month, that would be 3,000 articles.
Now of course not everyone has the same interests and tastes as you, so let’s say that only about 10% of these “worthwhile” articles are actually interesting to you. That still means that in one month, you’ll have 300 worthwhile articles to read.
Compare that to static print publications like the Tech Comm Journal, Intercom, the Communicator, or other print publications, which only have about 10 articles per issue, and you begin to see how valuable and powerful human aggregated content can be. This is the rationale behind Writer River. We now need more people to add feeds to it.
The manual method of going to the Writer River site and publishing a link to your post is somewhat archaic. It takes time and is slow. It takes effort. But the RSS feed doesn’t take effort. It only asks that you share your what-I’m-reading RSS feed with the Yahoo Pipe (by sending it to me, so I can add it), and then you don’t ever have to return to the site again. Content will just flow through the feed, however you choose to subscribe to it.
It makes sense to somehow mark or tag or bookmark or post or share or tweet good content that you read, right? You want to hang on to that article somehow so that you can find it later. That’s the nature of reading. But for online content, you need a method for keeping track of it, because the World Wide Web is too deep and wide and slippery to find something again after letting it go.
For those people who don’t have a blog or Delicious account, or Identi.ca or some other way of posting or marking content, I recommend starting one. One of the easiest ways to keep track of your good reads is through a WordPress.com blog, which is free, requires no maintenance, and provides you with an easy-posting bookmarklet that allows you to quickly add a link in two clicks from any page you’re reading.
Let’s pull together these efforts. Rather than having everyone run in their own direction, which accomplishes little, let’s harness all these individual efforts (which people are already doing) and turn them into a massive collective effort that dwarfs anything one simple person can do alone. Send me your category-specific RSS feed or links page and we’ll build an information machine that churns out the best content of the web without requiring you to do much at all to find it.
Tags: aggregation, collective efforts, crowdsourcing, feeds, mashups, RSS feeds, Yahoo Pipes
The problem with aggregating content from RSS feeds is that they tend to contain so few metadata. Title, URL, description — and that’s all. You can end up with an enormous database of HTML links, but it will have no categories, no ratings, and no metadata one could browse or search to find truly relevant content. You’re giving up on the advantages of human-aggregated content by adopting an XML standard with too little data (IMHO).
If we’re all going to benefit from assembling a database of links we all wish to use, we’ll need additional metadata. At the TC Library we collect four levels of category, as well as author, publisher, and year published (none of which is contained in most RSS feeds).
We have an XSLT for importing RSS content from diverse feeds into a ‘pending’ database, which humans must tag and categorize before they’re released into the public ‘TC Library’ index. That way, we can implement a carefully-planned content strategy, rather than simply use the limited features offered by the software — such as Yahoo Pipes and the thousands of RSS feeds already online.
Maybe you’d look into developing something like this for Writer River? It really would make your collection more useful for browsing, searching, and later collections of links. And now would be a good time for this, rather than after you’ve added thousands of uncategorized posts?
I could be wrong; I may misunderstand your content strategy for the Writer River database. But I’ve been doing this for years, and I can (and do) very strongly recommend rich tagging of content.
Geoff, you make a good point. I don’t want to try to duplicate or compete with the rich repository you already have going with tc.eserver.org. What if Writer River could be a means of making fresh, new content visible, which could then be checked out, evaluated, and possibly entered into the tc.eserver.org database — by you or your team — with all the structured meta tagging that you’re doing?
I think that if people have to tag the content with so much data to get it onto Writer River, the increased effort will decrease the amount of participation. I want to make this effortless. At the same time, you’re right — a giant repository of links isn’t very usable. If you have any ideas about how these two could work in tandem, I would love to hear it.
Tom, I’ve been wondering the same thing for over a year now. Without any luck. Yours is a great site, and I like ours. We just need to think of a way for the two to complement each other, rather than running parallel on either side of a brick wall, dividing volunteer labor between the two projects.
Perhaps something of the sort you suggest? Your site gets content more quickly, more easily. Ours follows with some selectivity, more detailed metadata, and slower incorporation of detailed content. Perhaps we articulate these in our missions, somehow?
And find a way to link back and forth when there’s overlap? We could parse your aggregated RSS feed. And there’s probably some way you could incorporate our feeds into your aggregator? Doing this without duplicating entries would be tricky, since neither of our feeds include original URLs, but link to our respective sites’ ‘detail’ pages. Hmm.
What do readers think?
Thats a great idea though it has some loopholes and problems which can be worked upon. People sometime just get fed up to find out some real useful information or articles which can help them. But your idea has certainly given me food for thought.If its possible, it will be fantastic.
Nice explanation given with diagram. It is really awesome and i know that its has some drawbacks to work on and surely all the loopholes and drawbacks will be good to use it up. The way of explanation is pretty awesome.
I love Writer River and have got a lot of interesting reading out of it since you set it up.
However I’m a little alarmed by the thought that this change might result in 3000 entries a month. One of the things I like about Writer River is that there’s not too much of it. The entries are well selected and nicely targeted.
I can only spend so much of my time reading up about technical communication and I know that, when I do have some time for reading, I can go to Writer River and be informed or entertained, without wasting time browsing through lots of pages before I find something interesting.
For me, Writer River works as a sort of a “Best Of” site.
The thought of Writer River becoming bloated with entries reminds me of a technical communicator I was following for a while on twitter. At first I was impressed by the volume of links he pumped out into the twittersphere. But pretty soon, although some of what he tweeted about was interesting, I began to find it irritating that I was spending time searching through his links for things that were worth reading. I realised that the trouble was he was not applying much of a filter. After a while I found I was just scrolling past his batches of tweets to find those from people with something interesting to say or to pass on. Eventually I decided this was ridiculous and I unfollowed him, which made me feel bad.
But sometimes less is more.
[...] I Need Your Human Aggregated Content [...]
I use Delicious. My username is eclark131: http://delicious.com/eclark131 .