Statistics for Twitter

Filed Under Twitter | 2008-08-28, 18:07

Tweetrush stats for edrabbit

Tweetrush stats for edrabbit

Anyone else out there a statistics junkie? Yes, I’m looking at you, the one with a account who’s adamant about scrobbling every last mp3 you play so you can generate fancy charts. Well if you’re a Twitter user, you’ll be happy to know about Tweetrush. Tweetrush is a site that will spit back statistics on how many twitters/tweets you sent, what time of the day you send them, etc. Currently they’re only showing stats for the last 7 days, but they’re using data from Gnip, which parses Twitter’s full data feed of all the tweets. This means they could theoretically parse more data as long as they have the servers to handle it. Want to see my (weak) stats? If you’re missing out on my low traffic Twitter, you can follow me here.

However TweetRush isn’t just a fun little Twitter site. It’s a test of the new RushHour engine, an analytics system for measuring actions and events as opposed to raw page views and clicks. RushHour might be just the thing that many sites are looking for to get a more detailed analysis of how people use their site. Unfortunately it’s under a closed beta right now, but you can apply to get into that beta over at


Demographic Info From 26,000 Phished MySpace Account

Filed Under Geek | 2006-10-10, 19:10

So a few days ago LoLo tapped me for some quick PHP-ing to analyze a file of phished MySpace passwords that he had found. For those that don’t know what’s been going on with MySpace and the password phishing, check out this article for an explanation.

So armed with this data, I sat down and imported the file into a MySQL database, cleaning up bits and pieces, and then wrote some scripts to pull out useful data. Not content with just info on the emails and passwords, I started playing around with MySpace’s search page, and wrote a screen scraping script to grab the info of all the users by searching for their emails. It took a few hours to gather all the data, but then I was able to run some useful reports on it. I’ll let you check out Lolo’s full write-up on the thing, including the stats over on his article: Demographic Info From 26,000 Phished MySpace Accounts. If there’s any interest/demand, I’ll clean up the scripts and post the code.

1 Comment

Traffic Logging with WRTbwlog

Filed Under Gadgets & Hardware, Hacks and Mods | 2006-10-07, 01:32

I’m a stats junkie, I’ll be the first to admit it. Ever since the first few days at college on a T1 line I’ve wanted to keep an eye on how much bandwidth I could burn through. With a few additions to DD-WRT on the Linksys WRT54G router you can see where all your bandwidth is going. The WRTbwlog page has some great info on setting things up, but we ran into a few snags so we thought it was worth a post. Here’s how to test it out to see how you like it:

First, we’ll assume you’ve already purchased a Linksys WRT54G router and flashed the firmware to install DD-WRT. We’re working with DD-WRT v23 SP1 Final (05/16/06) std.

Next you’ll need to go and get a copy of WRTbwlog. As of writing this the version of WRTbwlog that’s actually available on their site does not work with the latest releases of DD-WRT. But never fear, krikkit over on the DD-WRT forums fixed it. You can read all about it in this thread. So we’ll use this copy of his fixed version. But wait, let’s save us some trouble and just download it straight to the router.

Go ahead and ssh into the DD-WRT router. (how to enable ssh) Copy and paste the following commands:

cd /tmp
tar -xzf wrtbwlog_cust_exp.tgz
rm wrtbwlog_cust_exp.tgz
cd bwlog

This should startup WRTbwlog. If you grabbed krikkit’s version, you’ll probably see a message that says:
ftpget: Unable to connect to remote host ( No route to host
You can either ignore it or you can download our fixed copy.

Now, from a desktop load up a web browser, hit a few of your favorite websites to generate some traffic, and then go to and marvel at the stats. (If your router isn’t, replace it with the appropriate IP address obviously). Go surf some sites, download some things, and then go back and refresh the traffic page. Pretty slick, eh?

Only one problem at this point, you’d have to execute the commands above every single time you rebooted your router. If you’re like me, that’s once every few weeks, but it would still be a pain. So let’s get bwlog installed permanently on the router. It’s painless, I promise.

Log into the web interface for DD-WRT and go to the Administration tab->Diagnostics Tab. Most of you can probably just click here.
We’re going to add something so that WRTbwlog is automatically installed and started, but first a comment. If you can, please download the .tgz file and host it on your own server. It’ll be faster for you (and cheaper for me in bandwidth costs). And then replace the obvious text below

In the Commands box paste the following:

for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20;
sleep 5
cd /tmp
/bin/tar -xzf wrtbwlog_cust_exp.tgz
rm wrtbwlog_cust_exp.tgz
cd bwlog
( ./ & ) &


Click “Save Startup”
Reboot your router, and check and hopefully smile. You’re all set!

If you want to take the extra steps to save your traffic logs across reboots you’ll need to look into setting up an FTP backup. Unfortunately with the latest versions of DD-WRT necessary components (ftpput) are missing and this isn’t possible yet. But when it’s fixed, you can check WRTbwlog’s page for more info on this and the rest of WRTbwlog’s features and settings.

Forum discussions: 1 2


Where’d my Internet go?

Filed Under Geek | 2006-10-06, 23:30

The last week or so my internet connection has been going down seemingly randomly. It’ll go down for awhile and then come back just like normal. What made it odd was that a friend in Texas (I’m in Chicago) was also going down at the same times. After 3 or 4 “coincidences” we decided this wasn’t normal. After narrowing down the similarities, we started looking at our routers, both Linksys WRT54G running the hacked DD-WRT firmware. While it wasn’t a problem with a router, this focused my attention to where I would find the necessary clues.

The next time I lost my connection I noticed the led on my WRT54G router for “Internet” blinking like crazy, indicating a large amount of traffic. I couldn’t even access the router across my lan. I unplugged the router from the cable modem and instantly got access to the web interface. Plugged it back in, no access. Plugged directly into the cable modem with my desktop and got a solid connection again. Whatever is coming through doesn’t affect my Windows machine.

Logging is always good, so I downloaded Link Logger and setup DD-WRT to log to my desktop (see below for how-to) and waited. Sure enough I lost a connection about an hour later. Luckily all the evidence I needed was sitting in Link Logger.

Looks like someone from .se was trying to hammer on my ssh server with multiple machines. Since there are so many machines and so many attempts it was holding me down and not letting anything else through. A typical denial of service. So I unplugged the desktop from the router, plugged directly into the cable modem, power reset the modem to get a new dhcp lease, and started figuring out how to drop the ssh packets. I ssh’ed into DD-WRT while it was not plugged into the internet, and ran the following command line:

iptables -A INPUT --source -j DROP

(This will block all incoming traffic from 213.114.179.*)

Of course when I plugged back in, he was no longer hammering, and I didn’t get a chance to test it. I dropped into the DD-WRT web interface, under Administration->Diagnostics and added the iptables command so it will run on startup. This way when I reboot the router it will still have that rule. Let’s hope that stops this mess.

While discussing this with my friend in Texas, we realized the true link between us and these Swedes was not our router, but rather our use of DynDNS. DynDNS is a quick (free) way to get an easy to remember domain that resolves to your home connection. We both have * addresses. I’d be willing to bet these wankers were just scanning for people with addresses. Another clue to this was the fact that the attacks were still occurring with IP changes. DD-WRT has an option to automatically update your DynDNS name when your IP changes.

IP Addresses logged: (I wasn’t able to capture all of them as the Trial version of Link Logger doesn’t save them)
All of these resolve to *

How to log from your router
(This applies to DD-WRT v23 SP1 final)

  • Log into the web interface for DD-WRT and go to Administration->Services
  • Scroll down and enable “System Log”
  • Save Settings
  • Scroll back down and provide the IP address of the machine you want to log to.
  • Save Settings
  • Go to the Administration->Log tab
  • Enable the Log, set the Log Level to High, and everything to On (this way everything gets logged)
  • Save Settings
  • Download and install Link Logger
  • Point it to your router’s IP if necessary (Edit menu->Setup->Router Tab)

Leave a Comment