Phex segment rating and prioritisation
For Phex 2.1 and later
implemented by Nick Farrell (n ecks f at sf)
What are segments?
If you click on the segments tab when a downloading file is selected,
you will see that each file is divided into a number of segments. Some
segments are complete (ie: all the data has been downloaded), incomplete,
or empty. Each candidate Phex connects to (that is, a remote computer
containing some data we're interested in) can service only one segment at a
time. Segments can be merged and split to try to maximise efficiency.
What do you mean by rating?
Each segment has a rating. It is used when a candidate is ready
to be assigned to a segment, to help determine the order in which Phex tries to
assign segments to the candidate.
Why bother?
Prior to version 2.1(beta), Phex more-or-less randomly allocated segments to
candidates. The most important reasons for adding this feature are:
- Previewing "streaming" data
- Getting rarest parts first
How does the segment rating work?
This is a little complicated, and you might want to skip on to the next questions. :-)
First a candidateRating list is assembled and maintained. Every candidate who is unable to
provide all of the file registers the parts of the file it has here. Phex watches the speed at
which it is able to download data from this candidate and records this in the same list.
Candidate name | Candidate speed | Part of file |
X | 1000 | 0-50 |
X | 1000 | 80-100 |
Y | 2000 | 0-20 |
Y | 2000 | 90-100 |
Every 10 seconds or so another list is created, based on the above list. It more-or-less
adds up the candidates' total speeds for each part of the file.
Part of file | Total candidate speed |
0-20 | 3000 |
21-50 | 1000 |
80-89 | 1000 |
90-100 | 3000 |
Segments are then assigned a value taken from this list.
How does the rating affect the prioritisation of segments?
Normally, Phex will assign the lowest-rated segment to
the first available candidate. Only unallocated and incomplete can be assigned,
for obvious reasons. In addition, and equally obvious, a segment will not be assigned if
the candidate in unable to provide that particular portion of the file. If two segments
are equally rated, the one closest to the start of the file will be assigned.
In the above example, that would mean a segment starting at position "21" would be
the highest-priority segment. If the candidate is candidate "Y", however, it will not
have this part of the file and will instead deliver a segment starting at position "0".
The above method is named Prioritise by rating and is the default method, when a file is
not recognised as being streamable. There are two other methods:
Prioritise by position and Prioritise for preview.
What do you mean by streamable and unstreamable?
Streamable just means that the file is somehow useful to you before
it's fully downloaded. For example, AVI or MP3 streams can be used before
the entire file has been downloaded.
Unstreamable is simply the opposite: if any of the file is missing the
whole thing is pretty much useless, regardless of whether it's the start, the
end, or somewhere in the middle that's missing.
What are these other 2 prioritisation methods?
Prioritise by position: ignore the rating for a segment, and always try to get the segments
closest to the start of the file first.
Prioritise for preview: a hybrid of the other two methods. Get the first 10% of the file
first, then prioritise by rating.
How can I use them?
When a new download is started, Phex, looks at the suffix of the filename and guesses whether
the file is streamable or not. If it is, it prioritises for preview. Otherwise, it
prioritises by rating.
You can change the current priority for a downloading file by right-clicking on the file and
selecting 'ordering'. Any change will take effect at the next calculating point (ie: roughly
every 10 seconds)
How can I change the default ordering method?
In your phex.cfg file you will see a variable orderingMethod, which is a 3-digit
number. The first value is used if a streamable file is detected, the second if an
unstreamable file is detected, and the first if neither is detected.
ordering method | value |
Prioritise by position | 1 |
Prioritise by rating | 2 |
Prioritise for preview | 3 |
How can I change the definition of streamed and unstreamed files?
Also in phex.cfg, look for streamableSuffixes_#. Simply add, delete or modify the values as
you prefer.
For advanced users: the suffixes are actually regular expressions, as you may guess.
If you create an entry starting with ^ and ending with $ it will match the whole file, not just
the suffix.
I don't know when to stop. Please tell me more!
Well OK. Some of the following borders on trivia, but you asked for it.
-
You can view the current ordering for segments in the Segment view by clicking
on a column until no sort arrow is shown.
-
Candidates do not contribute to the ratings until a complete segment has been downloaded
from them. This serves 2 purposes: we have a more reliable estimate of their transfer
speed, and a candidate has to "put its money where its mouth is" and deliver us some
data before it affects priorities.
-
The weighting of speeds can be tweaked. Until recently the candidates' speeds were not
summed, but rather their square roots were. The idea was that 4 candidates purporting to
deliver a segment at speed X was as good as one candidate working at 2X. Particularly
coupled with the above item, this might be overkill. I welcome any feedback though.
-
Completely-downloaded segments are handled specially by the sorting function. The
intention is keep them sorted by start position. This is purely for cosmetic reasons.
-
It's actually possible to re-allocate a segment which is not free. This only happens
if no other segments are available and the current candidate is much worse (ie: slower)
than the idle candidate. This 'hijacking' of a segment means you need not fret about
some 10bytes/second block being left after all the other segments are complete.
I have an interesting/useful idea relating to this!
Great! Email me, or even better post it in the Phex forums on sourceforge.
Some things I would particularly appreciate:
- Nice little icons for the 3 ordering modes. Maybe a little movie reel for streamable, a ZIP-like g-clamp for unstreamable,
and an eye for the preview mode?
- Suggestions on other useful download modes
- More/better ways of determining a file's type
- Better algorithm for calculating segments' ratings