PDA

View Full Version : Seeing all downloads



ClockWork
5th November 2007, 08:40 PM
Hi Hipsters,

Does anyone know of a way in which one can see every Download in Mac OS X, and how much it weighed? Kinda like a Download Log ?

Naturally one can see what one has downloaded in Firefox / Safari / Camino / Speed Download etc, yet what about ALL downloads, including streaming audio / video, email, software updates, Lime Wire, BT, and every mouse click that opens a new page - etc ?

(Or would such a thing only exist as a piece of software?)

cheers,

cw

Currawong
5th November 2007, 08:55 PM
Netbarrier will record all bandwidth used. Someone asked about this recently.

ClockWork
5th November 2007, 09:12 PM
Thanks Curra - I notice its 5 user liscence is a tad pricey.

I need some way to explain what it is that teenagers do to their parent's monthly download limit, and how they can easily slice through 2GBs in 5 days - especially when Lime Wiring, BT-ing and visiting 100 dreadful MySpace Sites, each with their own streaming audio, every day.

Volume Usage Charts on the ISP show quite clearly that the major downloads occur just after 4:00pm in the afternoon, yet like all parents, the final word is:

"I'm sorry, but my kids would never lie to me, and I've asked them, so it must be something you've done."

=/

cw

dotnet
5th November 2007, 10:34 PM
That said, if I pinged you with large packets all the time day in day out you'd have to pay for that, too, even if your firewall blocks it. Of course, being on Bigpond and having to pay for uploads I would never do that... ;)

My point being: the current billing system used by ISPs everywhere counts all traffic their router forwarded to you, regardless of whether you (or any of your users) asked for the traffic or not.

Cheers
Steffen.

AcesHoney
5th November 2007, 10:48 PM
2 1/2 months ago Bigpond adjusted my billing cycle, since that date my usage has increased from 4 gig to 12 gig per month... they assure me that they have not changed the usage meters and our browsing habits have not significantly changed. To me it seems like too much of a coincidence for it to not be in some way related Basically they told me that it was my problem, happening at my end and that I must have a virus or something, or that my neighbours are using my network... I explained that we have 5 computers all the windows ones have virus protection and that the same ones are not necessarily on at the same times and that up until 2 weeks ago my network was password protected(cant seem to get it working with the mac thou). Is there anyway I can dispute what they are saying? Does this program you are talking about measure the usage in the same way as Bigpond?

zeddie2001
5th November 2007, 10:55 PM
http://homepage.mac.com/rominar/net.html

That will take a little bit of effort to setup, but it's only $10.

AcesHoney
5th November 2007, 10:58 PM
Thanks :)

Brains
5th November 2007, 11:54 PM
Clockwork: ask what kind of modemrouter they are using (if you don't know already) then check its manual -- many of them can generate logs that will do something pretty close to what you want, and mail the logs to a predefined address regularly (hourly, daily, etc). If the miscreant kiddiwinks have their own computer(s) then getting this kind of data at the router is the only way you're going to get reliable information on who's doing what, as both Netbarrier (severe overkill) and NetMonitor are per-machine.

At the very least, you can do an experiment: if the modemrouter has built-in site blocking smarts (and most do), block youtube.com for 24 hours and then do a comparison on the download quota used ... I know from experience that YouTube is the greatest quota suckers amongst teens, because it is so easy to embed a YouTube video into a webpage (especially Myspace pages).


B.

ClockWork
6th November 2007, 11:10 AM
Brains: Networking is my major weakness in computer knowledge. The modem routers are mainly Netcomms. I've googled: blocking youtube - in all imaginable questions, yet come up empty handed.

In - let's say - a Netcomm NB5Plus4W - one goes to the Advanced Settings > and then into either Port Forwarding or Access Control, yet... I really don't know... what to put into these fields to block YouTube / Google Video - and so forth.

I can google these subjects - but I can't see any tutorials on how to learn how to do these things. Most google results return with answers on how to open those ports.

Help please?

cheers,

cw

all I really know, is YouTube's IP address: 208.65.153.251 - yet I'm barking up the wrong tree here, aren't I...?

gehenna
6th November 2007, 11:15 AM
thats one of youtube's addresses, but when i was working on blocking youtube at work i noticed there were a range of about 10 ip addresses that it appeared users were accessing, and they all resolved to youtube.

if you can find something on your router called QOS or quality of service, this is the area where you can specify how traffic is sent/received from a particular source or destination. for instance you could enter the range of youtube addresses, and tell the router to only allow speeds of 20-50kbps for youtube. people soon get bored of trying when their video takes an age to download. similarly you can assign bandwidth by internal ip address....for example if a particular user is using too much, you can limit their bandwidth so that they are throttled.

as i say that all depends on the router but i believe there are software solutions that can perform the sameish tasks.

conufsed
6th November 2007, 11:16 AM
Actually it has 3 ip addresses:


[alan@Zoidberg]
~ $ dig www.youtube.com IN A

; <<>> DiG 9.3.2 <<>> www.youtube.com IN A
;; global options: printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17280
;; flags: qr rd ra; QUERY: 1, ANSWER: 3, AUTHORITY: 4, ADDITIONAL: 2

;; QUESTION SECTION:
;www.youtube.com. IN A

;; ANSWER SECTION:
www.youtube.com. 12 IN A 208.65.153.253
www.youtube.com. 12 IN A 208.65.153.238
www.youtube.com. 12 IN A 208.65.153.251It could though also be streaming data from other servers as well

ClockWork
6th November 2007, 11:46 AM
The trouble is - Gehenna - that I can't find any tutorials on these subjects, and Google is the master teacher of all things these days. I don't know... anyfing about it... and I don't know where to look, or what specific questions to ask.

Downloaded the PDFs for various Modem / Routers - they're all about 120 pages long - and though they explain what various utilities within the modem router does, they provide no examples to learn from - for instance - Port Fowarding has a preset list of IPs that can be opened or closed, yet no mention of how to create one's own, nor what one should key in.

Seeking further education...

gehenna
6th November 2007, 11:53 AM
understood. can you give a brief rundown on the kind of network you're dealing with? is this a bunch of users on a single computer, or a bunch of computers? are you wanting to prove where traffic is going to/coming from? or just avoid any future occurrences of traffic going astray?


firstly your ISP should be able to provide a report of your usage stats for a particular billing period. this may end up being several hundred pages long but they will provide it if you push. at least then you can skim through and look for patterns of source/destinations of traffic. once you have a better idea of where traffic is going or coming from, then you will have a better plan for how to manage it.

i don't think port forwarding will help you much in this sort of scenario, as most apps and routers support udp now anyway so port forwarding is becoming less and less of a requirement.

is there a section on the router that looks like it is QoS or Quality of Service related? i think it's this kinda area you need to look into. QoS should allow you to specify particular settings per IP address if you have multiple machines on the network. OR if the machines are shared by multiple users you'd need a software monitoring solution on the PC to monitor each users statistics. The latter I probably can't be much help with as I deal more with the hardware solutions.

edit: if you want to know more about port forwarding and how to use it, with examples pertaining to specific router and applications go to www.portforward.com

Brains
6th November 2007, 12:14 PM
Networking is my major weakness in computer knowledge. The modem routers are mainly Netcomms ...

Well, I just hunted down the manuals for the NB5Plus4W and frankly, it's a piece of labrador poopy. It can't do domain-based blocking. It doesn't generate traffic reports. It does do QoS (Quality of Service, aka traffic prioritisation) but it cannot forcibly limit traffic sub-types. On top of that, the onboard web interface is counter-intuitive and the manual hard to decipher unless you know exactly what the hell everything means (and they don't bother to explain most of it either).

The only way I can think of that would let you block YouTube is to manually edit the Hosts file on each computer connected to the internet, by adding a new entry that points 'youtube.com' and 'video.google.com' to 127.0.0.1

A suggestion would be to buy your own Billion 74xx modemrouter (a small, affordable Aussie-designed unit that can do everything the Netcomm cannot), drop it in place of the client's Netcomm, and have it configured to send you logs. As far as the client is concerned, there would ne no change to their internet, but you would have greater control over what it does and does not allow. It can domain-block, its QoS can forcibly limit traffic to any machine by IP number or even the connection-socket on the built in switch, and it can generate comprehensive traffic reports that can be emailed. It also allows for remote access to the built-in web interface from a single external IP number, so you can play with things from home.


Brains

ClockWork
6th November 2007, 12:21 PM
Thanks for listening Gehenna.

All the situations are the same:

Netcomm Wireless Networks of two to three Apple Mac computers plus ADSL2+ Speeds plus 2GB to 4GB to 10GB Download Limits - PLUS... Teenagers (hundreds of MySpace / YouTube / Google Video / Lime Wire / Bit Torrent / iTunes Store sampling / Networked WarCraft etc - Downloads per day).

Simply the ability to stop all those teenage downloads for 24 to 48 hours, in order to show parents those Volume Usage charts - in turn to show them how "much" they've gone down when those particular downloads have been disabled for 48 hours.

Addit: And to Brains - it's not been a problem until the average user was made aware they could give BigPond's measly 256kpbs downloads @ $79.95 per month, the flick, in turn for other ISPs - Internode etc, for 24,000kpbs speeds...

gehenna
6th November 2007, 12:26 PM
hmm, starting to sound like something that would be better served by applying restrictions to the machines themselves, instead of at the router - judging by the difficulty in using your particular router from brains' post. and as i said earlier, software solutions aren't really my forte unfortunately.

as the machines are shared, does each user have their own account? or is it one account for multiple users? could the mac firewall be put to use here? or parental controls?

Brains
6th November 2007, 12:27 PM
Port forwarding isn't going to be your answer -- you need to be able to block sites by name. Netcomms won't let you do that (which is why i would never ever recommend a Netcomm modem to anyone).

BitTorrent and LimeWire will prove almost impossible to block, but if you do the host file trick mentioned above, you can at least forcibly block some of the offending items on each machine.

Without a smart router in the loop somewhere -- be it the Billion I suggested, or a standalone Netgear blue-metal-box router or equivalent -- you're never going to get to the bottom of exactly who is doing what.


Brains

tommelbourne
6th November 2007, 12:27 PM
Ask the kiddies to watch some TV, go for a bike ride, play outside, read a book, play a board game (Carcasonne would be my pick) - anything just so long as they don't use the computer for 2 days. :)

ClockWork
6th November 2007, 12:47 PM
Port forwarding isn't going to be your answer -- you need to be able to block sites by name. Netcomms won't let you do that (which is why i would never ever recommend a Netcomm modem to anyone).

BitTorrent and LimeWire will prove almost impossible to block, but if you do the host file trick mentioned above, you can at least forcibly block some of the offending items on each machine.

Without a smart router in the loop somewhere -- be it the Billion I suggested, or a standalone Netgear blue-metal-box router or equivalent -- you're never going to get to the bottom of exactly who is doing what.


Brains



The only way I can think of that would let you block YouTube is to manually edit the Hosts file on each computer connected to the internet, by adding a new entry that points 'youtube.com' and 'video.google.com' to 127.0.0.1


How do I do this trick, Brains ?

Tommelbourne: ask a hard core smack fiend to quit looking for fresh spaces of skin to reach a vein, and go for a bike ride, play outside, read a book, play a board game... :p

AddIt: Many parents I've encountered are also in therapy, and all over the same offspring problems.

Currawong
6th November 2007, 12:50 PM
Maybe have a look on the computer for music, movies etc then show the parents how many GB of stuff has been accumulated?

Brains
6th November 2007, 01:01 PM
Clockwork: Methinks this page will do the trick:

http://macmerc.com/sections.php?op=viewarticle&artid=83

It even covers how to 'fool' the Mac into more or less ignoring a 127.0.0.1 lookup failure by tweaking the default error message.


B.

dotnet
6th November 2007, 01:44 PM
Parental Controls in Leopard logs visited sites (blocked or allowed) with timestamp and URL, but I don't think it counts bytes. I haven't looked deep into its logging capabilities, I doubt however that it would catch things like BT. You *can* use it to block access to certain apps (like BT clients).

In any case, there might be useful information in those logs, once your client has moved to Leopard and provided that Parental Control is active.

Cheers
Steffen.

ClockWork
6th November 2007, 01:57 PM
Clockwork: Methinks this page will do the trick:

http://macmerc.com/sections.php?op=viewarticle&artid=83

It even covers how to 'fool' the Mac into more or less ignoring a 127.0.0.1 lookup failure by tweaking the default error message.


B.

Okay... followed his instructions to use pico in the Terminal - and at the prompt, typed in:

127.0.0.1 youtube.com

then hit control + o

then hit enter

then control + x

Then open File Sharing and activated: Personal Web Sharing

his article gets a little confusing at this point, as he doesn't tell you what to do with the open Terminal - just to keep it open for the 404 message...

anyway - up-shot is, is : 127.0.0.1 youtube.com
the right thing to enter into the Terminal? Or am I off the mark?

On restart, I can still access youtube...

Please make this: Blocking For Dummies :p

Brains
6th November 2007, 03:04 PM
That is the right command to add, yes, but methinks there's some kind of DNS cache that needs to be 'flushed' as well ... I recall Dana telling me about it once, and there's some bizarre commandline thing to type in to do it ... but from here, I'll have to hand it over to someone else.

(getting a router with some real smarts would be so much easier that fart-arsing around with the bleedin' terminal!)


B.

zeddie2001
6th November 2007, 03:12 PM
I didn't actually look at the instructions, but flushing the DNS cache is as simple as:

lookupd -flushcache

This will force MacOS X to take into acount the changes to the hosts file

tommelbourne
6th November 2007, 03:34 PM
Don't worry CW, i can't get it to work either :(

ClockWork
6th November 2007, 05:01 PM
Parental Controls in Leopard logs visited sites (blocked or allowed) with timestamp and URL, but I don't think it counts bytes. I haven't looked deep into its logging capabilities, I doubt however that it would catch things like BT. You *can* use it to block access to certain apps (like BT clients).

In any case, there might be useful information in those logs, once your client has moved to Leopard and provided that Parental Control is active.

Cheers
Steffen.

I think this is going to be the only true answer for this problem.

I don't have a shred of doubt that there are millions of parents with these problems, forever digging deeper into their back pockets to fuel DSL2+ addicted children, who will have massive screaming tantis if their parents enforce "rules" upon internet use.

If one thinks about it, blocking the ports with another modem, defies the whole purpose of ADSL2+ which is designed to be enjoyed.

Thanks for all your suggestions lads. I reckon Steffen is on the mark there with Parental Control under 10.5.

cw

Currawong
7th November 2007, 02:09 PM
To repeat what I said before, did you search the hard drive for large files that have been downloaded or add up how much music has been accumulated? What's in the browser history, just Youtube?

iPirate
7th November 2007, 03:38 PM
I need some way to explain what it is that teenagers do to their parent's monthly download limit, and how they can easily slice through 2GBs in 5 days - especially when Lime Wiring, BT-ing and visiting 100 dreadful MySpace Sites, each with their own streaming audio, every day.

My simplest solution is having an iStat Pro widget, logging into the account and loading the widget, letting the teen use it, and before they log out, look at the widget and it should give total network traffic on each interface (Total In, Total Out). At uni my Airport interface can sometimes reach ~100MB from browsing youtube alone.

Oh yeah, and make sure the widget has the appropriate network things displayed.

This seems to give a total of web browsing, IMming, Weather Widget loading and even Torrenting.

ClockWork
7th November 2007, 11:05 PM
To repeat what I said before, did you search the hard drive for large files that have been downloaded or add up how much music has been accumulated? What's in the browser history, just Youtube?

The problem there is that they (teenagers) appear to be mostly visiting hundreds of MySpace Pages, containing YouTube Rock Videos, and browsing through lordy knows how many iTunes sample music, so there are very few actual downloads.

The only give away is looking back through browser history and seeing an endless list of:

MySpace
MySpace
MySpace
MySpace
MySpace
MySpace

all the way down the screen.

Jumping from one MySpace Page to another and then back again (to leave comments), appears to devour around 500MB in just one afternoon / evening, from 4:30pm to 10:00pm.

rtc
7th November 2007, 11:13 PM
Perhaps Brad and Marc would consider incorporating a cumulative counter into their next release? (iStat Pro currently has a counter that resets each session. I find it very useful.)

EDIT: Oops just saw iPirate's post two previous... similar idea it would seem!

Brains
8th November 2007, 12:55 AM
The problem there is that they (teenagers) appear to be mostly visiting hundreds of MySpace Pages, containing YouTube Rock Videos, and browsing through lordy knows how many iTunes sample music ... Jumping from one MySpace Page to another and then back again (to leave comments), appears to devour around 500MB in just one afternoon / evening, from 4:30pm to 10:00pm.

I can testify to this -- this is is exactly what happened at our place the school holidays before last, when my housemate's 15 year old son stayed with us. John & I share an 18 GB TPG account, and we both have a comprehensive usage-meter program on our Windows machine. Between the two of us, we rarely exceed 200 MB per day, so the 18 GB plan is more than ample. We had just clicked into a new month when Storm (yes, that's the young lad's name -- hippy parents) rocked up, and in his first three days our usage chart spiked, 7 GB in three days.

The culprit was obvious, young Storm spending several hours on the spare PC before heading out to be social, then another couple hours at night. What was he doing? Myspace. Myspace Myspace and more Myspace, with a few tunes for his iPod. "But I'm not downloading anything, honest!" and he wasn't ... not how he knew downloading, anyway. He didn't see any wrong in being active in his social network's Myspace pages, alll tweaked up the wazoo with Flash, Youtube videos and music.

So, I logged into my trusty Netgear DG834 (four years old!), turned on the logs and set it to mail me once an hour the traffic counts. Sure enough, when Storm was home, traffic use was pegging, when he was out, it slipped back to its trickle. So the next day, I logged back in and added the following to the blocked sites list (a standard feature in Netgear modems):

*video.google.com
*.youtube.*
*.myspace.*

I then turned on the scheduler (another standard Netgear feature) to disable the block-list for one hour, from 7pm to 8pm every evening, told Storm and John (who approved mightily) and then kept an eye on the logs for the next couple of days. Sure enough, traffic was back to about 200 MB a day except during that one hour when 300 MB was added to our quota. The end result was Storm spent more time out with his friends and catching up on DVDs.

I recall telling you about a similar system used by a Melbourne single-mum friend with three teens, back when you were having issues with that tantrum-chucker who stripped paint whenever she was denied MSN access. My friend Webby has a larger quota, but only allowed access to the same three sites between 4pm and 8pm, but had some nanny program on the Windows PCs that would only allow each kid two sessions of one hour each per day. Before I helped her set the system up (she has a Netgear DG814, the older model but still with the log function, block list and scheduler) her 20 GB quota would vanish inside a week, meaning she had to stay awake all night during unshaped time so she could work on her vast web empire (http://www.webgrrl.biz).

If I were you, CW, I would print this out and show it to the affected parents, and as one parent-of-teens to another, the only way you're going to stop this quota-raping is to replace the horrible Netcomm modem with a Netgear modem (or something else that offers blocklists and scheduling) and set up the above, because kids will be kids.

Put it this way. Spending just under $100 to upgrade to a Netgear DG834v4 (a bit more, if wifi is needed) will save money not spent on quota surcharges, avoid the stress of family arguments, and does away with frustrated parents coping with dialup-speed internet. A small price to pay for greater peace of mind, don't you think?


Brains

Currawong
8th November 2007, 07:48 PM
Clockwork: I'd print Brains brilliant post above and show it to the parents, along with the browser history.

arkenstone
8th November 2007, 08:02 PM
but seriously guys, as a first step what's wrong with just editing the hosts file(s) to prove a point?

it's in /etc/hosts. Just add

127.0.0.1 youtube.com
127.0.0.1 myspace.com

save, flushdns, move on to the next computer.

Give it a couple of days and check the usage stats.

It's VERY simple, all you really need is a text editor and with any luck the kids won't be intelligent enough (hey, if they use myspace it's a pretty good indicatino they have NO FUCKING TASTE OR SENSE - Jesus that site has some 'standards' issues) - where was I?

Yeah, give it a go - see what happens.

ClockWork
10th November 2007, 12:30 PM
With you there Arkenstone and Brains.

Okay - I think I have a method that works to utterly block MySpace, per machine without blocking ports in the modem router / replacing routers. Have tested this in both Panther and Tiger - appears to work:

Open Terminal, and type:

sudo nano -w /etc/hosts

press return

type in your Password

press return

using down arrow key, knock flashing block down to under:

::1 localhost

type:

127.0.0.1 myspace.com www.myspace.com

press return

press control + X

press shift + Y

press return

Quit the Terminal and Restart. Then try to get into MySpace. It should return with an Apache Page.

Could someone else who's willing to lose MySpace care to try this out to see if the results are duplicated?

ta,

cw

ClockWork
10th November 2007, 03:05 PM
It appears to work for blocking Tubes also.

Same process - and then after last prompt, type:

127.0.0.1 youtube.com www.youtube.com

and follow the rest of the procedure outline above.

then try it out:

http://forums.mactalk.com.au/showthread.php?t=25089&page=18

appears to block successfully :thumbup:

cw

Currawong
10th November 2007, 09:47 PM
So have you done that on your client's computer yet? Tell us how it goes.

ClockWork
10th November 2007, 10:53 PM
Haven't yet Currawong. They all got shaped back to 64k after 5 days, so I'll wait for the next month, when they get boosted back up to 24,000 kbps, and then block just the MySpace hosts... and see what happens within 2 days.

If I see it peaking again, I'll block the YouTube host... and see how the next 2 days go.

oh... and naturally not a word to the kiddies... and / or possibly their parents - as parents oft give up to tantis and spill the beans

Thanks muchly for all input guys ! :)

Just in the nick of time too - and though I know I hold no real responsibility for this, one can still feel really bad inside, one single mum had herself... umm... committed for a night after her broadband got shaped and child threw a major (and I do mean major) tanti.

She'll be the first one I aide.

Have also noticed numerous articles, like this one (http://crave.cnet.com/8301-1_105-9722963-1.html).

Apple is aptly naming these nomadic teens as "MySpace Loiterers".

Cheers,

cw

ClockWork
11th November 2007, 10:54 AM
What would be even more spiffy - (and now again beyond my fathoming) - would be some way of entering a Terminal command that didn't simply Block MySpace, or Unblock MySpace - which is a black and white situation - yet a command that could shape MySpace down to ... say... 64kbps...

Thus access wouldn't be Blocked. It would just become... far too tiresome to bother with.

Anyone know any hints - or links to learn...?

cheers,

cw

ClockWork
22nd November 2007, 09:15 AM
I'm returning to this subject, and yes - the 127.0.0.1 Host Block in the Terminal works, yet I'm returning with another - as not everyone is willing to whip out, buy Leopard, and go through the expense of rejigging 5 accounts.

Here is a common question from parents:

"Is there a way within the computer to set a limit on the number of hours per day that the kids can play on World of Warcraft? We get so tired of policing them. Our timer has a mind of it's own.
It would be even more fantastic if there were also a way to give them a 5 minute warning that "time was up" to give them a chance to "save" themselves so they won't be killed before they get off."

Outside of Leopard's Parental Control scheduling, is there a way to override this old message in Tiger:


http://www.clockworkoren.com.au/AutoShutdown.jpg

- and remove the option to Cancel ?

Just give them 4 minutes or whatever to Save before it shuts itself orf.

Anyone?

Cheers,

cw

Hamsmyth
22nd November 2007, 10:19 AM
Someone needs to make an app that can limit computer time use, websites etc.

Brains
22nd November 2007, 10:31 AM
Many such apps exist ... but they're all for Windows.

Clockwork (or moderators). this last 'timer issue / control' question might be better served if it were a general plea for a 'nanny' program over in the programming forum.


B.

MacDave
22nd November 2007, 10:51 AM
Clockwork -

You might look into MRTG (http://oss.oetiker.ch/mrtg/) (Multi Router Traffic Grapher.)

It's a five minute setup with MacPorts. (http://www.macports.org/) Here's MRTG analyzing all WAN traffic on my servers' router in a data centre:

http://www.macdave.com/external/analysis.html

Dave