Twitter Emergency Backup

by garth on May 5, 2008

Dave Winer has been thinking about ways to preserve the Twitter community even when Twitter is down. His latest effort is a web service to save Twitter feeds and expose their content via another feed.

I tried it out, but couldn’t parse the response:

Python 2.5.1 (r251:54863, Jan 17 2008, 19:35:17) 
[GCC 4.0.1 (Apple Inc. build 5465)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import xmlrpclib, urllib2
>>> twittergram = xmlrpclib.Server("http://rpc.twittergram.com/RPC2/")
>>> content = urllib2.urlopen("http://twitter.com/statuses/user_timeline/704593.rss").read()
>>> twittergram.saveFeed('garthk', '*****', content)
Traceback (most recent call last):
  File "", line 1, in 
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1147, in __call__
    return self.__send(self.__name, args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1437, in __request
    verbose=self.__verbose
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1201, in request
    return self._parse_response(h.getfile(), sock)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 1335, in _parse_response
    p.feed(response)
  File "/System/Library/Frameworks/Python.framework/Versions/2.5/lib/python2.5/xmlrpclib.py", line 547, in feed
    self._parser.Parse(data, 0)
xml.parsers.expat.ExpatError: mismatched tag: line 10, column 7

I’ll investigate later.

I’d love to wade into this kind of problem domain, perhaps using Google App Engine for hosting so I didn’t have to worry about usage spikes. Feed backup is a brilliant start. We also need to helping our followers find the feeds, and I’m sure more thought will yield more areas for funimprovement.

{ 27 comments }

Hjorutr Smarason 09.04.08 at 10:34 pm

Lol, yeah, we definitely need something when Twitter is down. Many were moving to Plurk or Friendfeed, but now Twitter seems to be getting it back on track. Let’s hope so, we don’t want to be left in the dark.

Smiley Girl 09.05.08 at 2:05 pm

Had not heard of Tumblr till I happened upon your post…now I have been looking it over and it looks like fun!

Web Diagnostic 09.07.08 at 2:55 am

May i know what exactly you mean with “Feed backup”?

emergency notification 09.11.08 at 2:32 am

Thanks for the very interesting post about Twitter! I find it very useful but that is one of the problems I see with it — the lack of reliability. When the site is down, there’s nothing you can do about it. I know there are some similar services that might compete with Twitter, and I wonder if the deciding factor in who becomes dominant will be who can maintain near-contant uptime? Thanks!

linkdor 09.13.08 at 6:31 pm

Twitter is a great community portal, but do you really think that creating this new service will change something. I mean twitter feeds don’t just dessapper if twitter is down. And when it’s down it’s only for an hour or so.

Omer 09.14.08 at 12:02 pm

Saving twitter feeds would be a good solution when it goes down imho.

College Betting Fan 09.18.08 at 10:49 am

maybe if they create mirror sites that could help. Look at what wikileaks.org has done

Tempe Real Estate 09.19.08 at 3:01 pm

Don’t you find it an excersise in futility to backup something supposed to be dynamic and ever changing? Having a record of a twitter is akin to saving every sticky note you’ve ever written.

No?

Nice blog,

Josh

Maple Ridge Notary 09.23.08 at 4:32 pm

x2 on saving twitter feeds. I do it regularly so I’m not screwed when it goes down!

Error Codes 09.24.08 at 5:07 am

I can see this being a service like archive.org, except for Twitter. Well, not exactly, but I’m sure you can turn it into some big Twitter search engine that lets you do research on users and comments.

Brad 09.25.08 at 6:06 am

I HATE it when twitter is down, esp. when you are in a conversation with someone.

mmorpg 09.28.08 at 12:39 pm

Forget twitter, It’s so over rated -_-. Honestly, whats up with microblogging these days.

Ink Cartridge Man 09.30.08 at 5:33 am

hadn’t heard of tumblr — will def check it out. Is twitter for real? With all the down time I feel like they are driving everyone away….

Acai Berry 09.30.08 at 11:40 pm

Oh man! Usually with most sites you don’t have to worry. :(

Ant 10.02.08 at 8:07 pm

Hi, I agree that twitter can still be a bit “temperamental”, the Google app engine sounds like a good way to go. Look forward to reading updates on this.

Mr Finish 10.02.08 at 8:22 pm

I suppose one of the key problems in having feed backups is it would probably turn into a really expensive process, so somewhere there’ll be an account saying it costs less to be down for a bit than to provide a continuous service…

DON 10.14.08 at 8:00 pm

Many people get hooked on certain sites so they should definitely do something to insure the people get their doses when they come for them.

It’s akin to having that cup of coffee in the morning, you want it but you also need it.

Most social site dwellers don’t even care if the cable goes out anymore unless their high speed internet is going through it.

Commercial Vehicle Insurance 10.20.08 at 12:43 pm

Why not simply backup the feeds onto another domain? When Twitter is down have it redirect all the feeds and users to the backup. The backups could check Twitter by regularly pinging it (or some other method).

Sammy

Acai Berry 11.18.08 at 12:29 pm

I wonder if rsync would be a possible solution. we use it all the time to make large mirrors of data.. Look at all the cpan mirrors…

Silver Jewellery 11.22.08 at 1:00 am

It seems to me that Twitter isn’t that unreliable and as the old adage goes, If it ain’t broke, don’t fix it!

Jopart 11.28.08 at 12:55 am

Tumblr was a new one to me. Thanks for the headsup.

Help Prevent Colon Cancer 12.09.08 at 2:39 pm

rsync won’t work, its meant for standard file structures.

dennis 12.18.08 at 12:38 pm

This brings up a good point about using web services in general. What if flickr were to lose all your photos, or tumblr all your blog posts? How many people do you think keep only 1 copy of their data and it’s on a cloud somewhere online.

MMO Gamer 01.23.09 at 5:47 pm

How often is twitter down these days? It’s worth while if it happens, but nowadays twitter is too big to go down, and it it goes down, it won’t stay down. It’s like saying “Zomg what if google goes down”. Big sites go down, but they usually come back up lightning fast.

Gospel Singer 02.12.09 at 8:51 pm

interesting post about Twitter! Twitter is a popular posting site and it should not be down. Google App. Engine is nice way to go. Looking forward for updates.

Omer 02.17.09 at 10:56 am

Come on guys, I’ve used twitter for a while now, and the site is BARELY down. It’s reached a point where it’s become almost like an MSN or GOOGLE, and it has like multiple servers online, so if ONE ever goes offline, the backups kick in. How often are ‘big’ sites really down anyway? Not that often, and with the increase in popularity for twitter, the site will certainly grow more and more stable.

PVC Door 05.10.10 at 5:36 pm

This brings up a good point about using web services in general. What if flickr were to lose all your photos, or tumblr all your blog posts? How many people do you think keep only 1 copy of their data and it’s on a cloud somewhere online.

good

Comments on this entry are closed.