The two words in the headline seems to have no connection… hmm yes its right, I’m just to lazy to write two seperate messages 😛
So the first: Syscall… or better SYSCALL 😉
I backported the SYSCALLs for AROS from the freepascal trunk to my branch. Karoly was so kind to implement (see here) but the rtl there is far from usable. And with the backported we can already begin to transfer the systemlibraries to the new schema, then later not so much work 😛
The porting was more difficult as i guessed, the changes in the code are not so many as you see in the commit, BUT some of this codeparts even does not exists in my branch… introduced in between. Seems also in this part much things happend. But after try to understand the code and compare with PowerPC-MorphOS I found the way to do it.
I tested it with some simple functions from exec.library and it works nicely.
for example:
function RawDoFmtLocal(const formatString: PChar;
const dataStream: Pointer; PutChProc: Pointer;
PutChData: Pointer): Pointer; syscall AOS_ExecBase 87;
I backed a release out of it, just to have it released, my primary target stays the torrent client.
The second point… UDP
I released the ArTorr yesterday and already some bug reports arrived at my desk. On of them was rather simple to understand … in the torrent file was no HTTP tracker, only UDP trackers. I didn’t know such things exist, my files always had at least one HTTP tracker included. The problem is, I have no idea how UDP or the Bittorrent Tracker UDP work. Very helpful is the official text: UDP Tracker Protocol for BitTorrent. But stays the problem how to do a UDP connection. I searched a lot for example codes but the most working with synapse or lNet, both I have not available. But i found this page: Programming with UDP sockets which explains, in C, how UDP connections are working. Its rather easy I would say, even simpler than a TCP connection, and much better for non blocking calls.
So I tried to implement the protocol, its rather straight forward. so tomorrow I will see how it will fit into ArTorr, if this is working I will see how to write myself the HTTP Protocol for the tracker and then make the whole thing non-blocking. 😉 Seems thats my major topic… make things non blocking.
I really like this number 28463 😉
Accomplished some advances for the torrent client… oh people we come closer to a release.
First I implemented the checking of Hash, when a piece is finished. To make sure there was no error in the network or so.
As next point on my list was the state machine for the Bittorrent network, which needs some simplification… so i made a little drawing of the states and transitions, its not very complicated, rather linear and implemented this behaviour. Works fine.
The next point is a big point on my list, a little bit complicated, multiple files. The pieces in the torrent stream are not stopping at a file border or so… just like a stream. On the first sight its not a big deal, if you have the pieces in an order.. but if they arrive randomly you have to calculate the position in the file and in the piece you recieved. For the sha1 calculation I have to keep the piece together, or I would just adjust the blocks inside a piece to get the file border. But I think I got it… downloaded just now a torrent file with 5 files in 3 different folders.. and it worked fine. Still needs a little bit finetuning for the recognition at start, if the file is already there and the sha1 check. Also I have to think about how to check if a file is finished (but not the whole torrent download) that I can rename it already and set free for the user.
I have one more point on my list before i would make a first alpha release…. tracker communication, multiple trackers, tracker reannounce, refresh peerlist, maybe also write an own communication with the tracker (non blocking you remember ;), the TfpHTTPClient is blocking of course)
Good news on the speed front, I transfered the ATorrent program to a native installation (Icaros in qemu) and there its really fast up to >3 MB/s (I have 50MBit so something like 5 MB/s possible), So 3 MB/s (Linux Mint DVD torrent Size: 1.3GB) in a virtual machine not bad, I would say. (Compiled on Linux it reaches also up to 3MB/s). This means my network handling is not that bad.
Seems the tap network for Linux hosted is very slow. there I always only get maximal 200 kb/s even on a local torrent network. (btw. the downloade file is 232.8 MB)
Until now I do not need any Threading. I’m still thinking if it later I will include some Threading possibilities. But my experience with threading at AROS are not so good, often not reproduceable crashes and deadlocks, even for code work perfectly on Linux/Windows.
To achieve this speed level I had to tweek the requests management and peer balancing. It work now much better but I’m still not satisfied. Some examples: If a request timed out (after a minute) I disconnect the peer. I noticed thats not always needed.. sometimes one packages time out but then later the other packages arrive (maybe a problem of the other side). Second: now all peers get the request with same priority, thats not bad but would make more sense if a fast peer is prefered over a slower/limited peer. I think about a kind of credit system for example: all peers start with 100 points. A request to a peer costs 10 Points, if the piece data arrive it earns again 10 points if it arrive at a specific time if it is faster it earns more point if it is slower it earns lower point.
A Timed out package costs 50 points. If the program has a slot free for it chooses the peer with highest points.
hmm maybe such thing would work, but difficult to test and set the right credits for every action.
I’m still playing with this debian torrent to see how real world torrent networks react. And one thing what happen, quiet often I would say, that the other side is not available from one second to the other. The problem is that my calls wait for data… the only solution make EVERY read from the socket non blocking, and thats exactly what I did.
Especially the handshake I noticed can need quiet a long time.. so in the past my handshake often failed, I just stopped to early… but I dont want to wait there until the answer arrive. Same for the payload, or better worse for payload.. I played around with the badwidth limiting of transmission (the other side of my torrent test network) and it seems that it makes this limit by slow down the piece sending.. so if he want to sent one block (usually 16kbyte Size) and the limit is set to 5kb/s it just divides the package in 3 part and send every second one of the parts ;). Nice idea btw. I will remember this when I include a limiter myself one day.
So I build something like a state machine for the network part, that can download the payload or make the handshake piece by piece. sometimes it makes it a little bit slower for one peer, but MUCH faster for multiple peers if one is limited or just slow.
In the video one can see how the download runs and at the same time new peers make the handshake. (before it always freezed at thisd position so I limited the handshake time to 100 ms)
To make some small speed tests I also begin to count the bytes transfered and measure the time… the speed is shown as the window title atm.
I only counts the real pieces bytes and not the network overhead.
Local I saw something like 700kb/s for small files and for big files something like 200kb/s on the real network.. mostly 60-70 but also once for two peers 180kb/s. Thats not very fast, but ok. Its not optimized and I notice the program waste much time for the debugoutput. First I bring it to work then I care about optimizations.
I work on a Torrent client for AROS in Freepascal, as long as I didn’t find suitable sources for the BitTorrent Protocol, I used mainly the specification. This two pages were very helpful:
Wiki Theory, BitTorrent RFC. But even more useful was to listen to the connections of other Torrent clients with Wireshark. Wireshark knows and decodes the Bittorrent packages and also shows if my program produces an illegal package (wrong size for example).
The first stage was more or less just a dirty first try… very simple, just coded for my two torrent files, and my seeder and tracker on the server (transmission). At 10.08.2014 this was ready to run.. and it downloaded very nicely the little picture… in fact I was really curious if there really arrive a picture or just junk. But (already at first try) the picture looks very good and also a binary diff showed no differences.
If this proof was done I begun completely from zero, made a little sketch for the classes I would need to read/andalyze the torrent file, tracker communication, torrent communication.
I use Lazarus at Linux to write the code. The GUI is LCL for now, but later I think I will change to FPGUI native it just work much better on AROS. I could design own components, for the pieces view for example. Its just much easier to write the code in Lazarus with Code completion and hotkeys for find declaration and online debugging of course. So I try to keep the source compatible between AROS and Linux. And beside one part this worked very nicely.
The parts where I got problems, was the networking… of course… especially the non blocking calls via MSG_DONTWAIT seems not to work at AROS. (Now I know they can not work.. this constant is not present in the whole AROS TCPStack source code, only in the SDK includes ;)). I decided to check for other ways to non blocking calls.. of course one can set the complete socket to nonblocking, but all ways I found in Internet seems not to work on the AROS platform. By looking through the source for „recvfrom“ I found the hint to look for FIONBIO to set with the Ioctl() call (At AROS IoctlSocket()). I don’t know why I didn’t saw it before… Now when I search for nonblocking I can find this solution. maybe I just used the wrong keywords 😉 Nevertheless IOctl(socket, FIONBIO, 1) does work.
At 13.08.2014 the second stage was ready to run, with the nonblocking socket calls.
It downloads nicely from two seeder at the same time. I had to tweek a little bit on both torrent client, set limits and so on to get this situation.
We have there:
192.168.0.238: Desktop where also AROS runs.
192.168.0.128: Windows Notebook with µTorrent
192.168.0.254: the server where also the tracker run and transmission.
As next I tried to go to a real torrent file with a real world tracker and seeder. I tried with some trackers and client at home at it works nicely. I’m rather sure that my client will not disturb other people in the bittorrent network with wrong packages or so.
But.. how to say… I did not work at all.. first, some hosts are just not available. And I made a non blocking call for the transfer but the connect was still blocking… So I reprogrammed it to a non blocking socket connect. (If you also want to do it: google for „socket connect timeout“). The principle is rather easy set the socket to nonblocking connect the socket and check with Select() (At AROS WaitSelect()) if the socket connects, there you can set a timeout. and also can make a loop to poll messageloop in between.
The second problem appears that some peers in the world, unchoke me, I send them some requests for packages and they never answer to them. I limited the maximum numbers for requests, hence after a while, my request queue was full of never answered requests (I waited and hour and nothing happend). I’m not sure whats the reason for this „not answering“ peers.
The solution is simple I just also brought a time out time for every request. I think later such things must be set in config or so.
In the beginning I wanted to use an AROS related torrent package, so I tried Icaros and Broadway torrents, but both do not work.. no seeders (I rechecked at my windows machine). I changed to debian iso torrent. (Many torrent files will not work, because I did not include multi file support, so atm the torrent MUST consist of a single file, besides this I only support HTTP trackers, so no UDP). But the debian ISO is fine, good tracker (the ubuntu tracker refuse my connect, I think because it does not know my PeerID Format), and many many seeds. Ok this was an other problem.. Of course more seeds is better but very inefficient as I noticed.. So I limited the number to 5 connections at a time. But then I had the problem that dead peers blocking my list and sometimes I have no peers to download… So i also implemented a refresh of peers, kick dead peers from the list and connect some new. At 14.08.2014, today.. was the day to run my client successfully in the wild on a real life torrent network:
There are some more things to notice:
- At the beginning it needs long time, looks like frozen, because he check the SHA1 of the file (which is already there and rather big, around 250MB) Later I will make a gui for it with progressbar or so.
- The file is already partial downloaded, I decided to use the same way as many other software so the partly downloaded get the extension „.part“, the sha1 check at start is used to determine which part still have to be downloaded
- In the beginning some peers are dead, later they are replaced with other peers and connected, with more peers the download runs rather nicely forward.
I’m already very happy about this result, even I’m still far away from a releaseable program. But it makes really fun to work with it.
I worked a little bit more on this mouse movement problem and I think now I got it and voila also the displacement bug of Menu disappeared. so now menu and combobox looks ok to me.
The maximus editor didn’t start at my machine anymore.. a little threading bug was the reason.. I didn’t know they use threading… in a EDITOR! how useless.. seems its meant to check for filesystem changes. But… to be honest.. a Timer would do it in this case.
I wanted to try maximus because I remember there was a strange font bug inside: The position of cursor does not fit to the char position. I tried a little bit, but in the end I have to say.. the maximus ist just poorly written. It just takes a width of a char to calculate the cursor position. By default a fixed width font is set, of course with fixes font it works. But this font seems not available by default in AROS so it uses the default font which is NOT fixed width. If no ini file is there for maximus it uses „#Edit2“ as the default font which is „Courier New“.. I changed the default to „ttcourier“ which is available by default in AROS. The #Edit1 I changed to „fixed“ which looks nicer for me. but also other fonts like sans mono or so work nicely.
The uidesigner is working.. but the movement of items is hard.. because it always jumps around. I noticed a hack in the GDI (Windows) implementation of FPGUI which captures the mouse when the editor is there. I also implemented this behaviour and now it moves smoothly.. In fact I did implement before already.. but the initial call to activate this function was missing.
As last I tried every single example in fpgui (/example/gui) and the most of them work really nice now. There I found a strange bug for TimerTest or animation examples.. seems the timer fire very seldom.
Initially I thought the reason is that there comes so seldom messages from aros.. but then I measured the time between messages and it is really short.. 0 to 20 ms, so no or one task switch. so thats not the reason it fires so seldom. In the timertest there is a Timer should fire every 50 ms but it looks more like 500ms-1s. Finally I found the reason, the time when the Timer should fire is determined by a Now() call. My Millisecond count in the result was just wrong, alawys an order to small. Type mismatch in the converter routine.
So now the animtest also works rather fast.. hmm I always thought my image drawing routine is so slow that I should improve it, seems it is not THAT slow 😉 OK in the colorwheel example you can see that it is VERY slow ;).
I think this changing are very important, it really improves the fpgui functionality a lot. So I created a release today for public again.
Available at the FPC DownloadPage
I try to find the displacement bug. I do not believe that it is some kind of fpgui bug.. and my debug output seems to tell me exactly the same.. it seems that my widget really try to send the right coordinates for a scroll bar (scrollbar is the most common problem) but the AROS Window does not move to there, but move some pixels more/less (depends on movement directions).
At this point I asked myself, how AROS knows where to paint the window?, because the window is REALLY there.. it reacts to clicks and so on, but the window coordinates tell its at a completely different position. Than I found a strange thing for the displaced case… the WindowLayer (PWindow^.WLayer) is at the wrong position. I can directly investigate after the movement for example the window^.top moved by 15 but the Window^.WLayer^.Bounds.MinY moved by 22 pixel. The difference is 17.. hmm suspicious this is width of the border decoration.. but this window has no border… but of course the top window has one. Solution unknown, I have to think about.
The second thing I investigated was the wrong mouse position in Menus/Combobox. In principle the solution was very easy.. even, again, very strange.
First things first. I have a Zero Zero window as top window usually. ZeroZero because then I don’t need to care about paint over the edge when drawing and do now need to calculate. The mouse position in this window work perfect. Also for borderless window, where I switched off the Zero Zero because do not need it (and it troubles the access, seems AROS still try to remove the borders, so I get negative values in side the Window).
So far it work.. but my Widgets are also borderless windows inside this top window (Parent-Children) and the mouse position in this windows seems to depend in what kind of Window they reside, if the parentwindow is a zerozero window with borders this borders also have influence to the mouseposition in this children. After some forward-backward calculations I think I got it right now. there I also found some bugs for Leave/Enter Events.. and Events to not Active Widgets (Mouse Move).
The result is that the Combobox now work much more smooth, always get the right entry when point to it, you can even close it by clicking outside the Combobox window. Also the Menudemo works much better, get the right selection as expected. If you click outside the menu the menu does not close (ESC close it) but this seems by purpose, but it also does not anylonger click to the nearest entry as before. The only Thing what seems to be a little bit strange: the menu opens directly above the MenuName. I thought its some kind of move displacement or WindowToScreen coordiate miscalculation. BUT for the ComboBox it works perfectly. So If I change it to the right position for menu its completly wrong for the combobox. Also something I have to compare, maybe compare in the FPGUI source code to see why it is different.
The commander like interface works very nicely already. Already possible Enter Directory(Return) and go one Directory up (Backspace), Copy(F5), Delete(F8) (Files and Directories) and Create(F7) Directories. Move(F6) I left out for now… I have to think about because of short cuts if on the same device could use the System function instead of recursive moving files.
‚+‘ key let you select File with a pattern, ‚-‚ key deselect with a pattern.
And the whole time I’m developing there was NO CRASH of the Program or AROS. :O I’m really surprised. Seems this wrong focus bug after modal window really helped a lot to bring stability to the program.
But of course there is also shadow.. the Alignment does not Work at all. If I make the Alignment by hand it works perfectly. but if I use Align (alTop, alBottom and so on) it looks VERY wrong… difficult to understand whats wrong there.
Still have no idea how to debug this situation.
After the Shift it-game project I switched to a new project I had in mind already long time before.
A two window file explorer, like total commander, midnight commander or so, for aros in fpc or course.
Yeah I know nobody needs such thing today there are already so many of them out there.
Two reason to start this project:
- I’m not really satisfied with the commander like programs at aros, they all missing a good keyboard control. Directory opus is VERY nice, I even use it at Windows, where it has a nice keyboard control. but at aros, it is an ancient version from the old Amiga there the mouse is for sure in the main focus of the programmer. The keybaord control is very difficult, you even have to activate it everytime you have a new window.
- the main reason is not really to create a useable program, (if this happen also ok) but is more or less and study object to search for fpgui-AROS or freepascal bugs. There still many inside. I tried with the example programs, but there often it just works.. or the program is already so complex, and not written by me. Its hard to track such bugs in code written by others.
I already found and fixedtwo nasty bugs on handling modal windows, so I think it worth to continue.
