G'day! Welcome to my blog. I only post intermittently. If you'd like to keep track of what I'm doing, please consider subscribing to my RSS feed. You can also follow me on Twitter (1-20 short posts a day) and Tumblr (a few found objects per month).
Dave Winer has been thinking about ways to preserve the Twitter community even when Twitter is down. His latest effort is a web service to save Twitter feeds and expose their content via another feed.
I tried it out, but couldn’t parse the response:
Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17)
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlrpclib, urllib2
>>> twittergram = xmlrpclib.Server("http://rpc.twittergram.com/RPC2/")
>>> content = urllib2.urlopen("http://twitter.com/statuses/user_timeline/704593.rss").read()
>>> twittergram.saveFeed('garthk', '*****', content)
Traceback (most recent call last):
File "", line 1, in
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
return self.__send(self.__name, args)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request
verbose=self.__verbose
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request
return self._parse_response(h.getfile(), sock)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1335, in _parse_response
p.feed(response)
File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 547, in feed
self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: mismatched tag: line 10, column 7
I’ll investigate later.
I’d love to wade into this kind of problem domain, perhaps using Google App Engine for hosting so I didn’t have to worry about usage spikes. Feed backup is a brilliant start. We also need to helping our followers find the feeds, and I’m sure more thought will yield more areas for funimprovement.
{ 26 comments… read them below or add one }
Lol, yeah, we definitely need something when Twitter is down. Many were moving to Plurk or Friendfeed, but now Twitter seems to be getting it back on track. Let’s hope so, we don’t want to be left in the dark.
Had not heard of Tumblr till I happened upon your post…now I have been looking it over and it looks like fun!
May i know what exactly you mean with “Feed backup”?
Thanks for the very interesting post about Twitter! I find it very useful but that is one of the problems I see with it — the lack of reliability. When the site is down, there’s nothing you can do about it. I know there are some similar services that might compete with Twitter, and I wonder if the deciding factor in who becomes dominant will be who can maintain near-contant uptime? Thanks!
Twitter is a great community portal, but do you really think that creating this new service will change something. I mean twitter feeds don’t just dessapper if twitter is down. And when it’s down it’s only for an hour or so.
Saving twitter feeds would be a good solution when it goes down imho.
maybe if they create mirror sites that could help. Look at what wikileaks.org has done
Don’t you find it an excersise in futility to backup something supposed to be dynamic and ever changing? Having a record of a twitter is akin to saving every sticky note you’ve ever written.
No?
Nice blog,
Josh
x2 on saving twitter feeds. I do it regularly so I’m not screwed when it goes down!
I can see this being a service like archive.org, except for Twitter. Well, not exactly, but I’m sure you can turn it into some big Twitter search engine that lets you do research on users and comments.
I HATE it when twitter is down, esp. when you are in a conversation with someone.
Forget twitter, It’s so over rated -_-. Honestly, whats up with microblogging these days.
hadn’t heard of tumblr — will def check it out. Is twitter for real? With all the down time I feel like they are driving everyone away….
Oh man! Usually with most sites you don’t have to worry. :(
Hi, I agree that twitter can still be a bit “temperamental”, the Google app engine sounds like a good way to go. Look forward to reading updates on this.
I suppose one of the key problems in having feed backups is it would probably turn into a really expensive process, so somewhere there’ll be an account saying it costs less to be down for a bit than to provide a continuous service…
Many people get hooked on certain sites so they should definitely do something to insure the people get their doses when they come for them.
It’s akin to having that cup of coffee in the morning, you want it but you also need it.
Most social site dwellers don’t even care if the cable goes out anymore unless their high speed internet is going through it.
Why not simply backup the feeds onto another domain? When Twitter is down have it redirect all the feeds and users to the backup. The backups could check Twitter by regularly pinging it (or some other method).
Sammy
I wonder if rsync would be a possible solution. we use it all the time to make large mirrors of data.. Look at all the cpan mirrors…
It seems to me that Twitter isn’t that unreliable and as the old adage goes, If it ain’t broke, don’t fix it!
Tumblr was a new one to me. Thanks for the headsup.
rsync won’t work, its meant for standard file structures.
This brings up a good point about using web services in general. What if flickr were to lose all your photos, or tumblr all your blog posts? How many people do you think keep only 1 copy of their data and it’s on a cloud somewhere online.
How often is twitter down these days? It’s worth while if it happens, but nowadays twitter is too big to go down, and it it goes down, it won’t stay down. It’s like saying “Zomg what if google goes down”. Big sites go down, but they usually come back up lightning fast.
interesting post about Twitter! Twitter is a popular posting site and it should not be down. Google App. Engine is nice way to go. Looking forward for updates.
Come on guys, I’ve used twitter for a while now, and the site is BARELY down. It’s reached a point where it’s become almost like an MSN or GOOGLE, and it has like multiple servers online, so if ONE ever goes offline, the backups kick in. How often are ‘big’ sites really down anyway? Not that often, and with the increase in popularity for twitter, the site will certainly grow more and more stable.