Re: storeGet() -> storeGetPublic() ?

----- Original Message -----
From: "Henrik Nordstrom" <hno@hem.passagen.se>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: "Adrian Chadd" <adrian@creative.net.au>; <squid-dev@squid-cache.org>
Sent: Monday, January 08, 2001 2:20 AM
Subject: Re: storeGet() -> storeGetPublic() ?

> Robert Collins wrote:
>
> > Hang on, I must have missed something.
> > Correct me if I'm wrong but to create a digest you set a bit x on in a large array if 1 or more cached URI's put through
filter(URI)
> > match that bit number.
>
> Sort of, except that there are more then one bit involved, and for any
> given bit there are many matching URLs.

"1 or more" :-]

...
> > As for rebuilding the digest, what about a schedulable background
> > process to create a new digest, and then swap the active digest
> > across when it completes?
>
> That is how it is typically done when it is time to clean up the digest.
> However, if you cannot retreive the index of stored objects then this
> gets a bit problematic.

But we can get the information(within the fs layer)... we just can't retrieve it as a full index, or terribly quickly.

>
> > Ie for a period of time you have two digests, one active, one being
> > built, and they are both registered to recieve URI deletion
> > messages until the new one becomes active.
>
> Or you could simply update the digest you have and hope that it is
> reasonably correct. But unless you can read the store index the digest
> will detoriate by time, typically forgetting older objects, and you
> might also end up with overpopulated digest which is not of any use to
> anyone, or a underpopulated one which uses to much space/bandwidth.

Which is why you said we need to rebuild the digest - that's what I was trying to address for reiser and similar systems.

>
> The good news is that digest sizes tends to stabilize once populated, so
> with some cleverness we might actually get away with a digest that is
> continously updated with additions/deletetions without having to rebuild
> it if your goal is to keep an approximation of what objects we have in
> the cache.

I wasn't aiming for published or internal digest generation specifically...

>
> For the published digest there is one more criteria which is quite
> important and which AFAICT requires one to have access to the store
> index: We do not wish to publish information about expired objects to
> our neighbours even if in the cache. This because they are not allowed
> to initiate refreshes/revalidations in our cache.

Good point. Given the above that implies we need an even on expiry of URI's from the store. hmm...
Rob
Received on Sun Jan 07 2001 - 08:32:15 MST