Re: storemanager

----- Original Message -----
From: "Adrian Chadd" <adrian@creative.net.au>
To: "Robert Collins" <robert.collins@itdomain.com.au>
Cc: <squid-dev@squid-cache.org>
Sent: Saturday, January 13, 2001 9:35 PM
Subject: Re: storemanager

> On Sat, Jan 13, 2001, Robert Collins wrote:
> > Adrian,
> > does the modio code you're doing include putting the http headers into metadata, rather than just part of the data stream
sent
> > to storeAppend? We must decode them something like three times during a typical MISS. (Once in http.c, once in client_side.c and
> > (i'm guessing once in store.c)
>
> The storage manager doesn't touch HTTP headers.

Ahh. Well how are we going to work with Varys/Content-negotiation/Content-Coding in the future? I think it probably needs to be
somewhat aware.

>
> > I'm finding myself shuffling/copying code between client_side.c and http.c do deal with the headers in a origin reply being sent
> > thru to storeAppend as part of the data stream (I have to remove the Transfer-encoding header).
> >
> > So if I could just
> > 1 decode the headers
> > 2 delheaderbyid (header, HDR_TRANSFER_ENCODING);
> > 3 remove Transfer-encoding from the conneciton header
> > 4 storeSetHeaders(header)
> > 5 storeAppend(buf+headerlength)
> >
> > it'd be much cleaner..
> >
> > I'm happy to tackle the knock on changes this will incur in client_side if you are willing to change the store manager ?
>
> Ok. All the storage manager does is cache and supply entire untouched
> responses from the network. From what I can gather, this works fine for
> HTTP/1.0 but not for some of the HTTP/1.1 features.

The Connection header and Hop-by-hop headers do put a crimp in that style. HTTP/1.1 is pretty clean on what needs to be processed at
each node. We have to update various headers when we successfully revalidate responses, and as we have (and I think we should keep
having) many store clients to a single store entry this needs to be done when the data is recieved not when it is sent.

The key reason for having many clients to one upstream is that applications with 'updating' ie anti virus updates built in will
often request the same file at the same time from multiple clients without the first copy having downloaded. Now I'm sure that
reading from disk you want several clients able to retrieve data at the same time, retrieving form network should be no harder: the
problem I percieve is that at the moment the data flows through unstructured - so the store _cannot_ accomodate the needs of
separate clients (ie client A wants bytes 0-999, 2000-2999, client b wants 1000 to 1999 would confuse the current code completely.

> Tell ya what, now that I've got the storage manager laid to waste in modio,
> how about we all figure out what the storage manager *should* supply so
> I can turn it into something useful for HTTP/1.1 ? There's no real point
> in trying to continue making more HTTP/1.1 features work with the storage
> manager we have.

Cool. I'll keep on muddling through and tweaking the layout.

Rob
Received on Sat Jan 13 2001 - 04:55:38 MST