Atm I’m very busy to bring the AROS-fpc to the main branch and also play a little bit with the Amiga Freepascal (I want it for comparison with AROS). So there is only limited time for ArTorr.
But today I bring the new GUI to live already working rather nice. Ok behind the curtain still much to do. I have to think how to prevent, that the gui has to access deep into the Torrent class properties to get all the informations to show. There should be a better solution.
My original plan was to introduce first seeding to ArTorr and then create a GUI, but hmm I have more fun with GUI creating currently. First time I really use the uidesigner for a little bit gui creation (not only testing)
The final ArTorr LAyout looks at the moment like this:
(I filled the pieces view with some random Values, as it will be later, red missing, yellow downloading, green arrived)
My new tracker connection via UDP and HTTP works very nicely. So now all network actions are non blocking. I’m rather satisfied with the whole thing.. it does work without additional Threads. So all the GUI, Network, statistics, sha1 calculation and File writing in a single Thread (the main thread) and it still can reach 3 MB/s and more. Not bad I would say.
Besides this I played little bit the FPGUI Styles and created a very simple (from the Demo style) to make the buttons and so on a little bit modern 😉 looks much better I would say.
So today I made a new Release of ArTorr Version 0.02 new features in short:
Download:
ArTorr 0.02
The two words in the headline seems to have no connection… hmm yes its right, I’m just to lazy to write two seperate messages 😛
So the first: Syscall… or better SYSCALL 😉
I backported the SYSCALLs for AROS from the freepascal trunk to my branch. Karoly was so kind to implement (see here) but the rtl there is far from usable. And with the backported we can already begin to transfer the systemlibraries to the new schema, then later not so much work 😛
The porting was more difficult as i guessed, the changes in the code are not so many as you see in the commit, BUT some of this codeparts even does not exists in my branch… introduced in between. Seems also in this part much things happend. But after try to understand the code and compare with PowerPC-MorphOS I found the way to do it.
I tested it with some simple functions from exec.library and it works nicely.
for example:
function RawDoFmtLocal(const formatString: PChar;
const dataStream: Pointer; PutChProc: Pointer;
PutChData: Pointer): Pointer; syscall AOS_ExecBase 87;
I backed a release out of it, just to have it released, my primary target stays the torrent client.
The second point… UDP
I released the ArTorr yesterday and already some bug reports arrived at my desk. On of them was rather simple to understand … in the torrent file was no HTTP tracker, only UDP trackers. I didn’t know such things exist, my files always had at least one HTTP tracker included. The problem is, I have no idea how UDP or the Bittorrent Tracker UDP work. Very helpful is the official text: UDP Tracker Protocol for BitTorrent. But stays the problem how to do a UDP connection. I searched a lot for example codes but the most working with synapse or lNet, both I have not available. But i found this page: Programming with UDP sockets which explains, in C, how UDP connections are working. Its rather easy I would say, even simpler than a TCP connection, and much better for non blocking calls.
So I tried to implement the protocol, its rather straight forward. so tomorrow I will see how it will fit into ArTorr, if this is working I will see how to write myself the HTTP Protocol for the tracker and then make the whole thing non-blocking. 😉 Seems thats my major topic… make things non blocking.
Today I included the multiple trackers for my torrent client, this was again rather easy… I set the time to 30 minutes to reannounce at the tracker and get updated peer list.
With this all on the list to the first possible release is done. So I started today to prepare the release, included some more debugging features (especially debugoutput to file) If someone outthere has problems I can ask them to start with the „-l“ switch and a logfile is written. Ok mostly I think its enough if I get the torrentfile which does not work, to reproduce the error, but the logfile will be helpful as well.
I decided to call it ArTorr instead of ATorrent, 1st Atorrent is so long and boring 😉 2nd its already taken for an Android Torrent client. ArTorr hmm sounds like a „Talk like a pirate“ hommage.. ARRRRTor 😉
Its version 0.01 so very first alpha version.
How to use:
Some things not implemented until now:
I really like this number 28463 😉
Accomplished some advances for the torrent client… oh people we come closer to a release.
First I implemented the checking of Hash, when a piece is finished. To make sure there was no error in the network or so.
As next point on my list was the state machine for the Bittorrent network, which needs some simplification… so i made a little drawing of the states and transitions, its not very complicated, rather linear and implemented this behaviour. Works fine.
The next point is a big point on my list, a little bit complicated, multiple files. The pieces in the torrent stream are not stopping at a file border or so… just like a stream. On the first sight its not a big deal, if you have the pieces in an order.. but if they arrive randomly you have to calculate the position in the file and in the piece you recieved. For the sha1 calculation I have to keep the piece together, or I would just adjust the blocks inside a piece to get the file border. But I think I got it… downloaded just now a torrent file with 5 files in 3 different folders.. and it worked fine. Still needs a little bit finetuning for the recognition at start, if the file is already there and the sha1 check. Also I have to think about how to check if a file is finished (but not the whole torrent download) that I can rename it already and set free for the user.
I have one more point on my list before i would make a first alpha release…. tracker communication, multiple trackers, tracker reannounce, refresh peerlist, maybe also write an own communication with the tracker (non blocking you remember ;), the TfpHTTPClient is blocking of course)
Good news on the speed front, I transfered the ATorrent program to a native installation (Icaros in qemu) and there its really fast up to >3 MB/s (I have 50MBit so something like 5 MB/s possible), So 3 MB/s (Linux Mint DVD torrent Size: 1.3GB) in a virtual machine not bad, I would say. (Compiled on Linux it reaches also up to 3MB/s). This means my network handling is not that bad.
Seems the tap network for Linux hosted is very slow. there I always only get maximal 200 kb/s even on a local torrent network. (btw. the downloade file is 232.8 MB)
Until now I do not need any Threading. I’m still thinking if it later I will include some Threading possibilities. But my experience with threading at AROS are not so good, often not reproduceable crashes and deadlocks, even for code work perfectly on Linux/Windows.
To achieve this speed level I had to tweek the requests management and peer balancing. It work now much better but I’m still not satisfied. Some examples: If a request timed out (after a minute) I disconnect the peer. I noticed thats not always needed.. sometimes one packages time out but then later the other packages arrive (maybe a problem of the other side). Second: now all peers get the request with same priority, thats not bad but would make more sense if a fast peer is prefered over a slower/limited peer. I think about a kind of credit system for example: all peers start with 100 points. A request to a peer costs 10 Points, if the piece data arrive it earns again 10 points if it arrive at a specific time if it is faster it earns more point if it is slower it earns lower point.
A Timed out package costs 50 points. If the program has a slot free for it chooses the peer with highest points.
hmm maybe such thing would work, but difficult to test and set the right credits for every action.
I’m still playing with this debian torrent to see how real world torrent networks react. And one thing what happen, quiet often I would say, that the other side is not available from one second to the other. The problem is that my calls wait for data… the only solution make EVERY read from the socket non blocking, and thats exactly what I did.
Especially the handshake I noticed can need quiet a long time.. so in the past my handshake often failed, I just stopped to early… but I dont want to wait there until the answer arrive. Same for the payload, or better worse for payload.. I played around with the badwidth limiting of transmission (the other side of my torrent test network) and it seems that it makes this limit by slow down the piece sending.. so if he want to sent one block (usually 16kbyte Size) and the limit is set to 5kb/s it just divides the package in 3 part and send every second one of the parts ;). Nice idea btw. I will remember this when I include a limiter myself one day.
So I build something like a state machine for the network part, that can download the payload or make the handshake piece by piece. sometimes it makes it a little bit slower for one peer, but MUCH faster for multiple peers if one is limited or just slow.
In the video one can see how the download runs and at the same time new peers make the handshake. (before it always freezed at thisd position so I limited the handshake time to 100 ms)
To make some small speed tests I also begin to count the bytes transfered and measure the time… the speed is shown as the window title atm.
I only counts the real pieces bytes and not the network overhead.
Local I saw something like 700kb/s for small files and for big files something like 200kb/s on the real network.. mostly 60-70 but also once for two peers 180kb/s. Thats not very fast, but ok. Its not optimized and I notice the program waste much time for the debugoutput. First I bring it to work then I care about optimizations.
I work on a Torrent client for AROS in Freepascal, as long as I didn’t find suitable sources for the BitTorrent Protocol, I used mainly the specification. This two pages were very helpful:
Wiki Theory, BitTorrent RFC. But even more useful was to listen to the connections of other Torrent clients with Wireshark. Wireshark knows and decodes the Bittorrent packages and also shows if my program produces an illegal package (wrong size for example).
The first stage was more or less just a dirty first try… very simple, just coded for my two torrent files, and my seeder and tracker on the server (transmission). At 10.08.2014 this was ready to run.. and it downloaded very nicely the little picture… in fact I was really curious if there really arrive a picture or just junk. But (already at first try) the picture looks very good and also a binary diff showed no differences.
If this proof was done I begun completely from zero, made a little sketch for the classes I would need to read/andalyze the torrent file, tracker communication, torrent communication.
I use Lazarus at Linux to write the code. The GUI is LCL for now, but later I think I will change to FPGUI native it just work much better on AROS. I could design own components, for the pieces view for example. Its just much easier to write the code in Lazarus with Code completion and hotkeys for find declaration and online debugging of course. So I try to keep the source compatible between AROS and Linux. And beside one part this worked very nicely.
The parts where I got problems, was the networking… of course… especially the non blocking calls via MSG_DONTWAIT seems not to work at AROS. (Now I know they can not work.. this constant is not present in the whole AROS TCPStack source code, only in the SDK includes ;)). I decided to check for other ways to non blocking calls.. of course one can set the complete socket to nonblocking, but all ways I found in Internet seems not to work on the AROS platform. By looking through the source for „recvfrom“ I found the hint to look for FIONBIO to set with the Ioctl() call (At AROS IoctlSocket()). I don’t know why I didn’t saw it before… Now when I search for nonblocking I can find this solution. maybe I just used the wrong keywords 😉 Nevertheless IOctl(socket, FIONBIO, 1) does work.
At 13.08.2014 the second stage was ready to run, with the nonblocking socket calls.
It downloads nicely from two seeder at the same time. I had to tweek a little bit on both torrent client, set limits and so on to get this situation.
We have there:
192.168.0.238: Desktop where also AROS runs.
192.168.0.128: Windows Notebook with µTorrent
192.168.0.254: the server where also the tracker run and transmission.
As next I tried to go to a real torrent file with a real world tracker and seeder. I tried with some trackers and client at home at it works nicely. I’m rather sure that my client will not disturb other people in the bittorrent network with wrong packages or so.
But.. how to say… I did not work at all.. first, some hosts are just not available. And I made a non blocking call for the transfer but the connect was still blocking… So I reprogrammed it to a non blocking socket connect. (If you also want to do it: google for „socket connect timeout“). The principle is rather easy set the socket to nonblocking connect the socket and check with Select() (At AROS WaitSelect()) if the socket connects, there you can set a timeout. and also can make a loop to poll messageloop in between.
The second problem appears that some peers in the world, unchoke me, I send them some requests for packages and they never answer to them. I limited the maximum numbers for requests, hence after a while, my request queue was full of never answered requests (I waited and hour and nothing happend). I’m not sure whats the reason for this „not answering“ peers.
The solution is simple I just also brought a time out time for every request. I think later such things must be set in config or so.
In the beginning I wanted to use an AROS related torrent package, so I tried Icaros and Broadway torrents, but both do not work.. no seeders (I rechecked at my windows machine). I changed to debian iso torrent. (Many torrent files will not work, because I did not include multi file support, so atm the torrent MUST consist of a single file, besides this I only support HTTP trackers, so no UDP). But the debian ISO is fine, good tracker (the ubuntu tracker refuse my connect, I think because it does not know my PeerID Format), and many many seeds. Ok this was an other problem.. Of course more seeds is better but very inefficient as I noticed.. So I limited the number to 5 connections at a time. But then I had the problem that dead peers blocking my list and sometimes I have no peers to download… So i also implemented a refresh of peers, kick dead peers from the list and connect some new. At 14.08.2014, today.. was the day to run my client successfully in the wild on a real life torrent network:
There are some more things to notice:
I’m already very happy about this result, even I’m still far away from a releaseable program. But it makes really fun to work with it.