[00:03:28] *** GoussX has joined #bittorrent
[00:16:55] *** MassaRoddel has quit IRC
[00:41:00] *** cosmodad has joined #bittorrent
[00:59:42] <firefly2442> DeHackEd: is this something you were planning to write in PHPBTTracker? function updatePeer($peerid, $hash)
[01:03:02] <DeHackEd> PHPBTTracker was my "teach myself PHP" project. very little planning actually went into it. it probably served a purpose at some point, but I've forgotten what it could be.
[01:03:38] <firefly2442> mkay, I was just curious because the function is empty ^^
[01:03:43] <DeHackEd> yes it is
[01:04:21] <firefly2442> I'll just take it out then
[01:09:52] <firefly2442> Are there any programs that can stress-test a BT tracker and show results?  sort of like the apache benchmark tool?
[01:10:23] <DeHackEd> never heard of one. I don't see any reason you couldn't use any other benchmark tool if it can generate random peer_ids
[01:10:37] <The_8472> tape a celebrity having sex, offer it on your tracker
[01:10:40] <The_8472> instant stress test
[01:10:46] <firefly2442> LOL
[01:10:47] <DeHackEd> point
[01:11:19] <DeHackEd> there was a hilarious image someone linked here a few years back regarding stupid people and bittorrent... it went something like this:
[01:12:08] <DeHackEd> 1)   F = file.open("iplist.txt", "w");   for a=1..255: for b=1..255: for c=1..255: for d=1..255 do F.write("$a.$b.$c.$d\n"); end; end; end; end; F.close();
[01:12:11] <DeHackEd> 2) zip file
[01:12:23] <DeHackEd> 3) Offer on bittorrent. label as "list of IP addresses (hacker tool)"
[01:12:29] <DeHackEd> 4) Idiot list appears on peer list
[01:13:11] <DeHackEd> as I recall there was something like 1 seed, ~65 peers
[01:13:24] <The_8472> it's actually older... was there for edonkey too
[01:13:38] <firefly2442> lol, I'm tempted to try writing the program just to see how big the file is ;)
[01:13:58] <DeHackEd> let's see.. 3+3+3+3+3 = 15
[01:14:01] <DeHackEd> * 4 gigabytes
[01:14:12] <DeHackEd> so somewhere around 25 gigs would be my bet
[01:14:32] *** [diablo] has joined #bittorrent
[01:14:34] <firefly2442> wow, that makes the joke even funnier
[01:14:49] <firefly2442> the fact that they wasted that much bandwidth on a file they could have generated ;)
[01:14:50] <DeHackEd> write the program, but just pipe it to stdout through wc
[01:16:17] <DeHackEd> running...
[01:16:24] <firefly2442> hehehe
[01:16:45] <firefly2442> you could even use it as a benchmark tool ^^
[01:17:10] <DeHackEd> I just brought it up because of The_8472's implication of "the things people will download, just by name"
[01:17:26] <DeHackEd> because I imagined you could just call something "a celebrity having sex" regardless of content, people just grab it.
[01:17:42] <The_8472> yeah, but not enough to stresstest a tracker
[01:18:00] <The_8472> if it were real and hit a few blogs.... then you'd have your test ^^
[01:18:19] <DeHackEd> what's popular on anime these days? find something really popular and you're golden
[01:19:45] <The_8472> http://www.animenewsnetwork.com/encyclopedia/ratings-anime.php#most_viewed
[01:20:53] <firefly2442> well, benchmark bittorrent tracker didn't yield anything good from google... hmm
[01:21:10] <The_8472> because it's rarely done
[01:21:29] <The_8472> most trackers are good enough to scale up to the needs of closed communities
[01:21:53] <The_8472> and there aren't that many public trackers that there'd be specialized stress testing software
[01:22:15] <The_8472> http://www.mininova.org/stats/tor/ <- more interesting stats
[01:23:00] <The_8472> some of those things most likely are fakes due to mediasentry&co... but some of those torrents are authentic
[01:23:49] <The_8472> hehehe "Top 100 Hot-Pop 12-07 Torrent Tatty Featuring Billboard RIAA Stars" ... the irony
[01:23:51] <The_8472> delicious
[01:24:47] *** taube is now known as Taube
[01:27:18] <DeHackEd> program still running
[01:27:35] <DeHackEd> been 11 CPU minutes.... lots of IP address to hack
[01:27:47] <firefly2442> you should have made it print the percentage done
[01:27:54] <DeHackEd> yeah I should
[01:27:59] <DeHackEd> that's what strace is for
[01:28:09] <DeHackEd> 62.72.48.140 oh hell no
[01:28:19] <DeHackEd> I'm only ~24% done
[01:28:52] <The_8472> now... repeat for ipv6
[01:29:00] <DeHackEd> denied
[01:29:03] <firefly2442> haha
[01:29:24] <DeHackEd> there's not enough storage for that many bits, let alone that many strings
[01:29:24] <The_8472> "Populating 128-bit file systems would exceed the quantum limits of earth-based storage. You couldn't fill a 128-bit storage pool without boiling the oceans." - ZFS dev
[01:29:44] <thewird_> hey DeHackEd, any progress?
[01:30:06] <DeHackEd> no, I've been busy. I'd like to see the network topology before I continue any further though.
[01:30:14] <thewird_> how so?
[01:30:19] <DeHackEd> also, a friend't birthday it today, so tonight's not looking good either
[01:30:30] <thewird_> ic
[01:30:39] <DeHackEd> I remember you talking about having some central storage for finished torrents... I don't remember the specifics, and digging through logs is not sounding fun
[01:30:53] <thewird_> well we have nothing yet
[01:31:09] <thewird_> but we were planning on having dedicated storage servers using mogileFS
[01:31:28] <thewird_> storage/uploading servers
[01:31:49] <thewird_> and possibly some squid frontends for serving the files
[01:32:06] <DeHackEd> RAID for whole systems?
[01:32:26] <thewird_> to quote a huge site, "fuck raid" :)
[01:32:32] <DeHackEd> so once a torrent is seeding, you submit it to the mogilefs and user downloads are through that...
[01:32:35] <camrdale> my iplist guess: 65,632,468,992 bytes (61.125 GiB)
[01:32:42] <thewird_> correct
[01:33:02] <DeHackEd> camrdale: sounds close
[01:33:02] <thewird_> mogilefs takes care of having multiple files on different machines as well
[01:33:28] <thewird_> so we can class the files based on how popular they are and mogile will make copies
[01:33:34] <camrdale> DeHackEd: brain power trumps computer power ;)
[01:33:57] <thewird_> cause we all know anime releases like Naruto are going to attract 999999 simulteanous downloads
[01:33:59] <thewird_> lol
[01:34:23] <DeHackEd> I made a rough guess, thinking that 3 digits in numbers will be common, but not THAT common
[01:34:31] <The_8472>  <camrdale> DeHackEd: brain power trumps computer power ;) <- you did use a calculator for the multiplication, didn't you?
[01:34:45] <thewird_> http://www.dattebayo.com/t/
[01:34:47] <camrdale> true, but I didn't have to
[01:34:49] <thewird_> [DB] Naruto Shippuuden 042
[01:35:01] <thewird_> torrent was released 21 hours ago
[01:35:08] <DeHackEd> thewird_: wow...
[01:35:09] <thewird_> 229,850 downloads
[01:35:19] <DeHackEd> the internet serves
[01:35:23] <The_8472> that is nothing
[01:35:25] <thewird_> :D
[01:35:46] <The_8472> lost, prison break and heroes etc. get a million in the first day
[01:35:46] <thewird_> im one of those of course :)
[01:36:03] <thewird_> possibly
[01:36:06] <thewird_> which reminds me
[01:36:13] <thewird_> need to download some stargate atlantis
[01:36:16] <The_8472> and that's only one torrent
[01:36:47] <firefly2442> anybody seen this? it's kinda creative: http://www.welcometothescene.com/
[01:36:49] <The_8472> a new ep airs tonight... so you better live in the US or... use other means
[01:37:18] <The_8472> i have meta-seen it
[01:38:51] <firefly2442> if it was real using a real Linux box it would be much more exciting ^^
[01:39:41] * DeHackEd calls it time to go.
[01:40:26] <thewird_> alright have fun
[01:40:30] <firefly2442> later
[01:46:36] <alus> their torrent link is broken :(
[01:47:16] <alus> uh
[01:47:22] <alus> and all their other links as well
[01:51:35] *** camrdale has quit IRC
[01:52:03] <The_8472> http://bramcohen.livejournal.com/29886.html <- awww, found an old gem :)
[02:00:47] <thewird_> how so?
[02:09:02] *** init0 has joined #bittorrent
[02:14:14] *** camrdale has joined #bittorrent
[02:14:56] *** rcjsuen has quit IRC
[02:23:18] *** init0_ has quit IRC
[02:23:21] *** lioux has joined #bittorrent
[02:27:03] *** The_8472 has quit IRC
[02:35:48] *** Taube is now known as taube
[02:49:44] *** Elperion has joined #bittorrent
[03:21:43] *** Elperion has quit IRC
[03:28:46] *** Tigerplug has joined #bittorrent
[03:44:01] *** slipstream-- has joined #bittorrent
[03:52:17] *** [diablo] has quit IRC
[03:57:18] *** slipstream has quit IRC
[03:59:12] *** Tigerplug has quit IRC
[04:01:58] *** GoussX has quit IRC
[04:37:21] *** cosmodad has quit IRC
[04:37:36] *** cosmodad has joined #bittorrent
[04:48:22] *** thewird_ has quit IRC
[04:49:08] *** thewird has joined #bittorrent
[05:28:31] *** cosmodad has quit IRC
[05:53:51] *** rcjsuen has joined #bittorrent
[06:27:36] *** rcjsuen_ has joined #bittorrent
[06:28:17] *** rcjsuen has quit IRC
[06:32:01] *** rcjsuen_ is now known as rcjsuen
[07:26:51] *** rcjsuen has quit IRC
[08:06:30] *** init0_ has joined #bittorrent
[08:19:05] *** init0 has quit IRC
[09:37:31] *** GoussX has joined #bittorrent
[09:42:05] *** camrdale has quit IRC
[09:52:14] *** ultimusprime has joined #bittorrent
[09:55:04] *** MassaRoddel has joined #bittorrent
[09:58:16] *** ultimusprime has quit IRC
[10:06:40] <TheSHAD0W> http://linux.slashdot.org/article.pl?sid=08/01/12/0152208
[10:12:39] *** MRoddel has joined #bittorrent
[10:12:39] *** MassaRoddel has quit IRC
[10:12:58] *** MRoddel is now known as MassaRoddel
[10:20:39] *** MassaRoddel has quit IRC
[10:20:48] *** lymeca has quit IRC
[10:25:15] *** [diablo] has joined #bittorrent
[10:27:35] *** MassaRoddel has joined #bittorrent
[10:54:36] *** lymeca_ has joined #bittorrent
[11:17:24] *** Elperion has joined #bittorrent
[12:19:28] *** taube is now known as Taube
[12:34:57] *** GoussX has quit IRC
[12:36:12] *** GoussX has joined #bittorrent
[13:23:51] *** rcjsuen has joined #bittorrent
[13:34:23] *** slipstream has joined #bittorrent
[13:47:33] *** slipstream-- has quit IRC
[14:06:36] <TheSHAD0W> http://tech.blorge.com/Structure:%20/2008/01/12/bittorrent-site-the-pirate-bays-operators-could-finally-walk-the-plank/
[14:12:01] *** EvolutionCrazy has quit IRC
[14:16:39] *** The_8472 has joined #bittorrent
[14:28:25] *** charlie126 has joined #bittorrent
[14:36:02] *** charlie126 has quit IRC
[15:08:03] <DeHackEd> yawn. good morning.
[15:08:33] <thewird> morning
[15:18:07] <DeHackEd> 61337501696 bytes is the final result of my "make a listing of how large that list of IP addresses would be" appl
[15:19:16] <thewird> hmm?
[15:19:37] <DeHackEd> yesterday we were discussing a gag torrent listing all 4 billion IP addresses
[15:19:52] <thewird> lol
[15:20:20] <The_8472> DeHackEd, did you include newlines?
[15:20:27] *** cosmodad has joined #bittorrent
[15:20:28] <DeHackEd> yes, for 1 byte
[15:20:34] <The_8472> kk
[15:26:30] *** lioux has quit IRC
[15:27:14] *** lioux has joined #bittorrent
[15:31:15] *** [diablo] has quit IRC
[15:52:03] *** Taube is now known as taube
[15:54:17] *** MassaRoddel has quit IRC
[16:17:34] *** mxs has quit IRC
[16:17:36] *** mxs has joined #bittorrent
[16:37:29] *** EvolutionCrazy has joined #bittorrent
[16:46:38] *** cosmodad has quit IRC
[16:55:03] *** MassaRoddel has joined #bittorrent
[17:01:11] *** cosmodad has joined #bittorrent
[17:01:16] *** cosmodad has quit IRC
[17:07:58] *** CitiGroup has joined #bittorrent
[17:08:12] *** CitiGroup has left #bittorrent
[17:12:18] *** [diablo] has joined #bittorrent
[17:45:55] <firefly2442> it's still going! wow.... :)
[17:46:11] <firefly2442> oh nvmind
[17:47:58] <firefly2442> too early in the morning...
[17:48:09] <firefly2442> ok ok, it's 11 AM
[18:03:31] *** K`Tetch has joined #bittorrent
[18:36:57] *** lioux has quit IRC
[18:37:01] *** camrdale has joined #bittorrent
[18:50:45] *** Elperion has quit IRC
[19:01:14] * DeHackEd can't seem to get mogilefs working...
[19:01:32] <thewird> oh
[19:01:45] <thewird> we didn't need you to do that part
[19:01:48] <DeHackEd> I'm using their vmware virtual appliance...
[19:01:53] <DeHackEd> yeah, I'm just trying to see what I'm up against
[19:02:03] <thewird> ic
[19:02:08] <thewird> you and your vmware
[19:02:09] <thewird> lol
[19:02:17] <thewird> use xen
[19:02:20] <DeHackEd> it was quick and easy
[19:02:20] <thewird> stuff works better
[19:02:20] <DeHackEd> denied
[19:02:40] <thewird> rejected...
[19:02:54] <DeHackEd> you're suggesting that if it was running on Xen, it would be easier to set up and documentation would be more redily available?
[19:03:11] <thewird> yes
[19:03:29] <thewird> its super easy to setup if you dont want custom kernels
[19:03:44] <The_8472> vmware doesn't require custom kernels either
[19:04:05] <DeHackEd> also, mogilefs doesn't care about the kernel. it's not a FUSE driver or anything.
[19:04:14] <The_8472> they support hardware virtualisation these days too
[19:04:19] <thewird> yup
[19:05:09] <DeHackEd> Undefined subroutine &main::cmd_host_del called at /home/mogile/bin/mogadm line 75.
[19:05:10] <DeHackEd> Beautiful
[19:05:20] <DeHackEd> clearly there's a problem with my virtual machine setup
[19:06:08] *** lioux has joined #bittorrent
[19:06:22] <thewird> you know if you wanted to put some more dedicated time into this funky and I could buy you a nice quad-core machine
[19:18:50] <DeHackEd> okay, got it sorta working. their linux installation is incomplete and perl won't run the user tool...
[19:22:32] <DeHackEd> we'll just call this a waste of effort
[19:22:57] <thewird> yah dont worry about mogile
[19:23:08] <thewird> thats  the stuff funky wants to work on anyway
[19:23:30] <DeHackEd> okay, but I need an interface to the data. nearest I can tell it's an HTTP interface, so I'll have to do a pass-thru.
[19:23:56] <thewird> for what?
[19:24:09] <thewird> only the storage servers are going to use mogile
[19:24:20] <thewird> the torrenting part is separate
[19:24:20] <DeHackEd> okay... let's put this another way
[19:24:51] <DeHackEd> nearest I can tell, mogilefs is mostly a write-once, read-many platform. therefore we download, and once it starts seeding we move to mogilefs, for deletion (in the BT directories) once we give up seeding
[19:25:05] <DeHackEd> so what we do is wait for seeding to finish, it goes into mogilefs, and then we serve to users via mogile
[19:25:17] <thewird> yup
[19:25:24] <thewird> thats the idea
[19:25:40] <DeHackEd> and mogilefs is accessed via an HTTP interface, so we'll need to access that
[19:26:18] <thewird> we could use apache to protend its mogile
[19:26:25] <thewird> and then just addin mogile later
[19:26:59] <DeHackEd> but we don't let users access mogile directly. first, it's highly abusable. second, I plan to use a rather cryptic naming scheme and want to mask that.
[19:27:15] <thewird> ic
[19:28:03] <DeHackEd> for example:  website.com/getfile.php/1/25/Season1/Episode1.avi
[19:28:19] <thewird> how do you plan to run the link?
[19:28:31] <DeHackEd> getfile.php would treat 1 as being the torrent id, 25 as being the number of the file in the torrent, and the rest is noise that the user can appreciate and the web browser will save the file as Episode1.avi
[19:28:59] <DeHackEd> I use something vaguely similar in another web script, and it works pretty well
[19:30:21] <thewird> remember this needs to be scalable
[19:30:54] <The_8472> 1 million torrents makes the URL 5 characters longer...
[19:31:04] <The_8472> i'd say that scales well
[19:31:16] <DeHackEd> I thought mogilefs did load balancing. we just do some quick database work to verify authorization to download the file, maybe do some name mapping, close the DB connection, open mogilefs connection and fpassthru
[19:31:31] <DeHackEd> can you fpassthru an HTTP connection? I recall fpassthru doing bad stuff with files on disk...
[19:31:47] <The_8472> php itself should be able to cope with that i think
[19:32:01] <The_8472> i mean i've seen sites serving gigabytes through php
[19:32:33] <DeHackEd> I remember php doing the somewhat stupid thing of mmap()ing the whole freaking file and then doing the mother of all write() calls... which kinda crashed some of my... well...
[19:33:19] <DeHackEd> there's a user mode linux limitation which killed the transfer ~256 MB through
[19:34:16] <The_8472> ah yes... mmap fails for bittorrent... unless you have 64 bit ^^
[19:34:32] <The_8472> think of mapping a 50GB torrent *gg*
[19:34:41] <thewird> BRB lunch
[19:35:14] <DeHackEd> UML allocates memory to the application processes as 1-page mmap() calls on the host... and the host only allows 65536 (by default) entries in /proc/$PID/maps, so the process hits the allocation limit and dies as its resident address space overflows.
[19:40:07] <kjetilho> DeHackEd: why not use the infohash instead of 1,2,3?
[19:40:25] <kjetilho> with your scheme, people can browse content without owning the torrent file.
[19:40:50] <DeHackEd> I said there was some authorization steps included. being logged in would be one step.
[19:41:33] <DeHackEd> it's an option though, and it will depend on how I do the on-disk storage. if I do use info_hash, I would use the hex version so it would be a 40 byte string
[19:42:52] <The_8472> would even allow you to organize the files into a tree
[19:44:08] <DeHackEd> one of my main ideas is that if a torrent is a batch torrent, the entry would be a directory and all the files would be arranged flat and just numbered sequentally. the ordering can be determined by .torrent file content examination.
[19:44:37] * DeHackEd is going out for a bit.
[20:01:40] <firefly2442> When a client requests a scrape, does the tracker give all information about all torrents to the client?
[20:01:56] <firefly2442> I could see this as being a security risk for those that was obscured or private trackers
[20:02:01] <firefly2442> *want
[20:05:48] <kjetilho> scrape isn't part of the protocol.
[20:13:07] <The_8472> <firefly2442> When a client requests a scrape, does the tracker give all information about all torrents to the client? <- there are 3 ways of scraping
[20:13:31] <The_8472> single-scrape, multiscrape, scrape on all torrents
[20:13:40] <The_8472> not all trackers support all of these features
[20:14:55] <firefly2442> ahh ok thanks
[20:42:19] <DeHackEd> a tracker is allowed to return empty file lists for a "full" scrape if it doesn't want to. it also doesn't have to return anything beyond the info_hash, which in theory isn't enough to download reliably but you could download blind with it.
[20:42:59] <The_8472> well, these days it is possible to obtain the .torrent for an infohash if the torrent is not marked as private
[20:43:08] <The_8472> with µT's meta data exchange and Az's DHT for example
[20:43:12] <The_8472> or a simple google search ^^
[20:43:45] <DeHackEd> and I could download a file from info_hash and tracker URL alone with 99% reliability.
[20:44:14] <DeHackEd> you just risk downloading garbage pieces and not knowing it (though you might be able to infer what you did wrong as you find yourself banned... hehe, that's nice)
[20:44:25] <DeHackEd> plus you don't know how large the file really is, so grabbing the last piece is going to be tricky
[20:45:48] <kjetilho> multi-file torrents are hard, too
[20:45:57] <DeHackEd> true that
[20:46:22] <DeHackEd> if you're lucky, you can get the filenames from the scrape and infer which are multifile and which are not
[20:47:25] <The_8472> you also don't know the piece size
[20:47:46] <The_8472> you'd have to guess by making requests with increasing offsets and see which get you disconnected
[20:47:50] <DeHackEd> but you can guess it by sacrificing a connection
[20:48:05] <The_8472> hehe, that makes me think of chess ^^
[20:48:23] <The_8472> sacrifice a bitcomet to capture a piece tower ^^
[20:48:40] <kjetilho> hehe, that's evil
[20:49:02] <kjetilho> knowingly break the spec and blame it on BitComet, that is
[20:50:07] <DeHackEd> no no. just find a bitcomet client. you don't WANT to stay connected, so you just do piece size guessing. when you overshoot the target, you get disconnected from the client. you now know the piece size and you are not connected to bitcomet. win-win.
[20:51:58] <kjetilho> nice :)
[20:52:45] <DeHackEd> now you go downloading all pieces of the same size (except the last one)
[20:53:15] <The_8472> you can reuse the same method for the last one
[20:53:18] <DeHackEd> for the last piece, you very carefully download small chunks, shrinking chunk sizes should you find yourself disconnected...  you can sortra binary search for the correct size
[20:53:28] <The_8472> yep
[20:54:02] <DeHackEd> the difference is that you can be reasonably sure the piece size will be a power of 2 which will be strictly larger than the slice size (which is itself a power of 2, hence the pice size is a multiple of the slice size)
[20:54:13] <DeHackEd> so that one would actually be easier and less error prone. most likely.
[20:56:06] <The_8472> well... actually... it doesn't have to be a power of two
[20:56:28] <The_8472> azureus can create 2^N + 2^(N-1) sized pieces too
[20:56:46] <DeHackEd> which is why I said "most likely", because I have never seen any other option, and many clients only give options for 2^n
[20:56:49] <The_8472> or theoretically any size that's the multiple of the chunk size (although we don't offer that as an option)
[20:56:58] <DeHackEd> okay, but for most values of N, a slice size increment would still be fine.
[20:57:21] <The_8472> yep, that would do
[20:58:32] <The_8472> hrrm... i think it's N-2 actually
[20:58:47] <The_8472> well, who cares
[20:59:16] <DeHackEd> N-1 makes the most sense, you get often seen non-powers-of-2 numbers like 48, 96, 192, 384, etc.
[20:59:28] <The_8472> yes, that's what i meant
[21:00:08] <The_8472> in my mind those always count as "sortof numbers of two" ... as they appear everywhere in computer related stuff
[21:00:12] <The_8472> err, powers
[21:00:14] <The_8472> damnit
[21:18:31] *** lymeca_ is now known as lymeca
[21:22:49] *** Elperion has joined #bittorrent
[21:55:55] <firefly2442> well, I updated rivettracker, the PHP + MySQL powered BT tracker based on PHPBTTracker from DeHackEd
[21:56:02] <firefly2442> here's the changelog:
[21:56:03] <firefly2442> http://forums.rivetcode.com/viewtopic.php?f=5&t=5&p=12#p12
[22:54:01] *** jlouis_ has joined #bittorrent
[23:07:08] *** jlouis has quit IRC
[23:25:10] *** init0_ is now known as init0
[23:30:08] *** Elperion has quit IRC
[23:53:43] *** danomac has quit IRC

top