Today I finished the ArTorr 0.1 Release.
Download:
ArTorr 0.1
News:
- New fancy gui, with pieces and availibility view and more 😉
- Piece-get randomization
- Can be started from shell with torrent file as parameter
- If no parameter given a requester to select will appear
Howto:
- Download the archive
- Unpack at a location you like (in the archive is a Folder with Icon)
- Navigate to the ArTorr Folder
- Start the Program by double click onto the icon
- Select the torrent file
- See how the download goes
- Press quit or close at any time to break the download, the download will be continue when you restart with the same torrent file
- Downloaded files can be found in „Output“ directory in the ArTorr folder
Atm I’m very busy to bring the AROS-fpc to the main branch and also play a little bit with the Amiga Freepascal (I want it for comparison with AROS). So there is only limited time for ArTorr.
But today I bring the new GUI to live already working rather nice. Ok behind the curtain still much to do. I have to think how to prevent, that the gui has to access deep into the Torrent class properties to get all the informations to show. There should be a better solution.
My original plan was to introduce first seeding to ArTorr and then create a GUI, but hmm I have more fun with GUI creating currently. First time I really use the uidesigner for a little bit gui creation (not only testing)
The final ArTorr LAyout looks at the moment like this:
(I filled the pieces view with some random Values, as it will be later, red missing, yellow downloading, green arrived)
My new tracker connection via UDP and HTTP works very nicely. So now all network actions are non blocking. I’m rather satisfied with the whole thing.. it does work without additional Threads. So all the GUI, Network, statistics, sha1 calculation and File writing in a single Thread (the main thread) and it still can reach 3 MB/s and more. Not bad I would say.
Besides this I played little bit the FPGUI Styles and created a very simple (from the Demo style) to make the buttons and so on a little bit modern 😉 looks much better I would say.
So today I made a new Release of ArTorr Version 0.02 new features in short:
- UDP Tracker
- HTTP tracker error handling (working redirect, thanks magorium for this file/report)
- Tracker communication non blocking 🙂
- a little bit nicer interface, but still the same debugging GUI 😉
Download:
ArTorr 0.02
The two words in the headline seems to have no connection… hmm yes its right, I’m just to lazy to write two seperate messages 😛
So the first: Syscall… or better SYSCALL 😉
I backported the SYSCALLs for AROS from the freepascal trunk to my branch. Karoly was so kind to implement (see here) but the rtl there is far from usable. And with the backported we can already begin to transfer the systemlibraries to the new schema, then later not so much work 😛
The porting was more difficult as i guessed, the changes in the code are not so many as you see in the commit, BUT some of this codeparts even does not exists in my branch… introduced in between. Seems also in this part much things happend. But after try to understand the code and compare with PowerPC-MorphOS I found the way to do it.
I tested it with some simple functions from exec.library and it works nicely.
for example:
function RawDoFmtLocal(const formatString: PChar;
const dataStream: Pointer; PutChProc: Pointer;
PutChData: Pointer): Pointer; syscall AOS_ExecBase 87;
I backed a release out of it, just to have it released, my primary target stays the torrent client.
The second point… UDP
I released the ArTorr yesterday and already some bug reports arrived at my desk. On of them was rather simple to understand … in the torrent file was no HTTP tracker, only UDP trackers. I didn’t know such things exist, my files always had at least one HTTP tracker included. The problem is, I have no idea how UDP or the Bittorrent Tracker UDP work. Very helpful is the official text: UDP Tracker Protocol for BitTorrent. But stays the problem how to do a UDP connection. I searched a lot for example codes but the most working with synapse or lNet, both I have not available. But i found this page: Programming with UDP sockets which explains, in C, how UDP connections are working. Its rather easy I would say, even simpler than a TCP connection, and much better for non blocking calls.
So I tried to implement the protocol, its rather straight forward. so tomorrow I will see how it will fit into ArTorr, if this is working I will see how to write myself the HTTP Protocol for the tracker and then make the whole thing non-blocking. 😉 Seems thats my major topic… make things non blocking.
Today I included the multiple trackers for my torrent client, this was again rather easy… I set the time to 30 minutes to reannounce at the tracker and get updated peer list.
With this all on the list to the first possible release is done. So I started today to prepare the release, included some more debugging features (especially debugoutput to file) If someone outthere has problems I can ask them to start with the „-l“ switch and a logfile is written. Ok mostly I think its enough if I get the torrentfile which does not work, to reproduce the error, but the logfile will be helpful as well.
I decided to call it ArTorr instead of ATorrent, 1st Atorrent is so long and boring 😉 2nd its already taken for an Android Torrent client. ArTorr hmm sounds like a „Talk like a pirate“ hommage.. ARRRRTor 😉
Its version 0.01 so very first alpha version.
How to use:
Download the file: http://www.alb42.de/prgs/ArTorr001.lha
unpack at a position you like (its a Folder ArTorr with Icon inside)
Start ArTorr, press „Load File“, choose your Torrent file (do not get disturbed by the console open together with the Program, its just some debugout from the GUI, just close it)
wait and see how the download goes, if any 😉
the upper panel shows the pieces to recieve (‚+‘ is already there), middle panel shows the peers you are connected (choked means the other side do not want to send any data, mostly because it send already to other peers), the lower panel shows some debugoutput.
you can stop the program at any time, you SHOULD click stop before you close the program, (then it will send a quit to all tracker and so on)
if you got a problem, of course you also can directly close the program.
Files can be found in the „Output“ Directory
if you stop the program and the file is not finished, no problem, after a restart and loading of the torrent file the program will check the file and continue the download where it stopped.
Some things not implemented until now:
- UDP:// Tracker (I have no knowledge about UDP, first need some docu about the torrent UDP)
- Seeding (Will be the next task on my list)
- GUI, its just a very simple GUI atm, more for debugging, still LCL, will be the point after seeding is done
- Choose Output directory (because thats also GUI)
- all advanced Bittorent extensions (not that important I would say, but maybe funny to learn some new technics) includes also magnet links, http downloads and so on.
- more than one .torrent file at once (this I’m not sure I will include at all, in principle the program could also stay as a one torrent client, if you have two, start the program two time)
I really like this number 28463 😉
Accomplished some advances for the torrent client… oh people we come closer to a release.
First I implemented the checking of Hash, when a piece is finished. To make sure there was no error in the network or so.
As next point on my list was the state machine for the Bittorrent network, which needs some simplification… so i made a little drawing of the states and transitions, its not very complicated, rather linear and implemented this behaviour. Works fine.
The next point is a big point on my list, a little bit complicated, multiple files. The pieces in the torrent stream are not stopping at a file border or so… just like a stream. On the first sight its not a big deal, if you have the pieces in an order.. but if they arrive randomly you have to calculate the position in the file and in the piece you recieved. For the sha1 calculation I have to keep the piece together, or I would just adjust the blocks inside a piece to get the file border. But I think I got it… downloaded just now a torrent file with 5 files in 3 different folders.. and it worked fine. Still needs a little bit finetuning for the recognition at start, if the file is already there and the sha1 check. Also I have to think about how to check if a file is finished (but not the whole torrent download) that I can rename it already and set free for the user.
I have one more point on my list before i would make a first alpha release…. tracker communication, multiple trackers, tracker reannounce, refresh peerlist, maybe also write an own communication with the tracker (non blocking you remember ;), the TfpHTTPClient is blocking of course)
Good news on the speed front, I transfered the ATorrent program to a native installation (Icaros in qemu) and there its really fast up to >3 MB/s (I have 50MBit so something like 5 MB/s possible), So 3 MB/s (Linux Mint DVD torrent Size: 1.3GB) in a virtual machine not bad, I would say. (Compiled on Linux it reaches also up to 3MB/s). This means my network handling is not that bad.
Seems the tap network for Linux hosted is very slow. there I always only get maximal 200 kb/s even on a local torrent network. (btw. the downloade file is 232.8 MB)
Until now I do not need any Threading. I’m still thinking if it later I will include some Threading possibilities. But my experience with threading at AROS are not so good, often not reproduceable crashes and deadlocks, even for code work perfectly on Linux/Windows.
To achieve this speed level I had to tweek the requests management and peer balancing. It work now much better but I’m still not satisfied. Some examples: If a request timed out (after a minute) I disconnect the peer. I noticed thats not always needed.. sometimes one packages time out but then later the other packages arrive (maybe a problem of the other side). Second: now all peers get the request with same priority, thats not bad but would make more sense if a fast peer is prefered over a slower/limited peer. I think about a kind of credit system for example: all peers start with 100 points. A request to a peer costs 10 Points, if the piece data arrive it earns again 10 points if it arrive at a specific time if it is faster it earns more point if it is slower it earns lower point.
A Timed out package costs 50 points. If the program has a slot free for it chooses the peer with highest points.
hmm maybe such thing would work, but difficult to test and set the right credits for every action.
I’m still playing with this debian torrent to see how real world torrent networks react. And one thing what happen, quiet often I would say, that the other side is not available from one second to the other. The problem is that my calls wait for data… the only solution make EVERY read from the socket non blocking, and thats exactly what I did.
Especially the handshake I noticed can need quiet a long time.. so in the past my handshake often failed, I just stopped to early… but I dont want to wait there until the answer arrive. Same for the payload, or better worse for payload.. I played around with the badwidth limiting of transmission (the other side of my torrent test network) and it seems that it makes this limit by slow down the piece sending.. so if he want to sent one block (usually 16kbyte Size) and the limit is set to 5kb/s it just divides the package in 3 part and send every second one of the parts ;). Nice idea btw. I will remember this when I include a limiter myself one day.
So I build something like a state machine for the network part, that can download the payload or make the handshake piece by piece. sometimes it makes it a little bit slower for one peer, but MUCH faster for multiple peers if one is limited or just slow.
In the video one can see how the download runs and at the same time new peers make the handshake. (before it always freezed at thisd position so I limited the handshake time to 100 ms)
To make some small speed tests I also begin to count the bytes transfered and measure the time… the speed is shown as the window title atm.
I only counts the real pieces bytes and not the network overhead.
Local I saw something like 700kb/s for small files and for big files something like 200kb/s on the real network.. mostly 60-70 but also once for two peers 180kb/s. Thats not very fast, but ok. Its not optimized and I notice the program waste much time for the debugoutput. First I bring it to work then I care about optimizations.
