Re: Squid store replacement policies

Alex Rousskov wrote:

> The number of indexes stays the same. The only difference is that each
> FS stores/loads its part of the main store index and the policy stores
> the order, if needed.

So you are proposing adding yet another persistent index where the
policy stores the order, and to somehow be able to store/load this
separately from the store index. Please elaborate on how you intend that
this should work.

The way I have inteded it there will only be one persistent index, and
it keeps both things. The policy tells the FS the order info while
writing the index, and the FS loads the index and inserts it into both
the store index and the policy.

> Ideally, there should be no "main store index" but a collection of FS
> indexes. FS should be able to decide on how to lookup an object based on
> request headers. The latter is probably a better design (with small
> overhead compared to the centralized index), but the difference is not
> that big, I think.

Maybe, but I am not convinced, and this most certainly is not something
which is actually relevant for this change.

> Depends on the policy. The general idea is that cache_dir index has the
> same interface for all cache_dirs and policies use that interface to
> re-index entries in their own index if needed. A policy may store more
> metadata than just a pointer to StoreEntry, depending on the policy
> needs.

Ok. I see where this are leading, but I am still not sure on how to
connect the two together if the loading/storing of FS and policy index
are separated from each other.

> For LRU, maintaining a queue of ids/pointers to StoreEntries (run-time)
> and storing/loading StoreEntry ids/filenos should be sufficient.

With a greatly added risk of inconsistency. Policy references to
non-existing objects are easy to handle, but store objects without
policy references is a bit harder..

> You win then. Sorry for wasting your time.

It is not a waste of time. You input have been very valuable. However,
much of what you are discussing require far greater changes than what I
am prepared to make right now. I prefer to take it one step at a time.

> > Again no. The "hot object cache" currently indexes MemoryObjects which
> > happens to have a StoreEntry. A StoreEntry cannot be indexed by the "hot
> > object cache" if it doesn't have a MemoryObject.
>
> I was describing how the things may be organized, not how they work now.

That is fine, and I fully agree with you. However, I have to live with
how it works at least for the moment. It is not very fruitful to design
an API which cannot be implemented without first redesigning most else
in Squid. If your design reasoning is applied recursively then mostly
everything in Squid needs to be redesigned before anything can get
implemented, and you are most likely better off starting a new project
from scratch where you don't have that amount of old luggage to care
for.

> > The "hot object cache" is not at all implemented as a FS type store. It
> > is completely different almost all aspects. It simply piggy-backs on
> > Squids object forwarding.
>
> Exactly. And, IMO, that is not the best design possible.

Certainly not. "Hot objects" are not equal to "transit objects".

> > It is called by the policy's users to create the policy before use.
>
> Thats exactly what I am against! That function should not be called to
> create policies (except by the parser), individual policy constructors
> should be called. "createPolicy(type) should be a "protected" method of
> the parser not an interface for others to use.

It must be called in the cache_dir creation phase, or there is nothing
to connect the policy to. Each cache_dir is responsible for the parsing
of that line. The policy user is the cache_dir, and is is the cache_dir
(or "store FS") who is the parser.

> It may be better to allow users to configure FS with a pre-created
> policy. The design where the policy can be created internally only is
> excessively rigid, IMO.

We have selected to have policies on a per cache_dir basis only. It you
have objections to this then say so.

Even then, changing this back to shared policy indexes later is no big
deal, at least not without the policy -> cache_dir association you
wanted to have...

/Henrik
Received on Fri Apr 28 2000 - 15:10:13 MDT