Re: Bug in snapshot?

On Sun, Apr 23, 2000, Henrik Nordstrom wrote:
> Adrian Chadd wrote:
>
> > Yup, ditto from here. I'm bughunting a UFS rebuild-from-swaplog bug
> > where an object is set to be released but it doesnt come out of the
> > replacement policy (grr), so when that is fixed I'll make sure the
> > heap replacement code works with UFS.
>
> That might be an oldie. Found a bug like that in 2.2.STABLE5 where
> cancelled objects never got released. See my "Squid-2.3.DEVEL2: Release
> unused store entries during rebuild" patch.

Hrm. I'll give it a shot. The annoying bit is that when the storeentry
is to be pulled out, you can't issue an unlink() since the object
might be in use by something else. So you have to ''fake' it, by
removing the object from the repl policy manually, then set the
swap_filen / swap_dirn to -1, then call storeRelease{Request}.
I tried it quickly but it failed miserably, so I was going to worry about
it much later.

> > diskd will eventually be the async disk IO mechanism squid uses.
>
> On that I am not sure we agree. The design of diskd is basically plauged
> by the same issues as async-io (to small granularity of the operations,
> not very portable, not performing the best on all platforms where it
> builds).

Once I have defined an async io model, diskd is only one of the possible
asyncio mechanisms. Under FreeBSD it'd probably work better to use
the posix AIO or the kqueue stuff, under Linux it'd probably work better
to useclone()/threads, under Solaris libaio might bethe best choice ..
Implementing disk as the underlying mechanism forces me to clean up
the storage manager and its use of memory first, which isn't a bad
thing.

> > --enable-diskd and --disable-diskd shouldn't be applicable in
> > modio, I'm not sure what is the story in henrik's devel tree.
>
> Neither am I ;-) Well, this area of the code looks as it does in modio
> currently.
>
> > the maxobjsize is used as the initial hint a to where to place the
> > object. COSS requires it, and I wouldn't wind it above 100k right now.
>
> Is there a explanation of COSS somewhere apart from the actual code?

Heh. I was going to wait until I write up some storage manager documentation,
but COSS is a chunked LRU disk store. You define 'chunks' (default 1mb)
which is the size used to write to disk. Newly created objects enter the
chunk. When the chunk is full, it is written to disk. Any extra
space in the chunk is ignored, so you don't waste disk space. When an
object is read, if it is in the chunk being written to disk it comes
from the chunk. If it is not in the chunk being written to disk,
it is rewritten into the currentachunk. Whenever a chunk is allocate,
the objects on the disk where it currently is are freed. This makes
COSS an on-disk LRU store. There i one problem where on a busy cache you
can request an object from the disk which requires allocation of
a new chunk, and the new chunk would require the overwriting of the object,
which I currently fail and call a 'collision' but that will change
later.

COSS is a neat idea (thaanks Eric!) and it doesn't have the problem of
a straight cyclic store. It is full of memcpys, limited to 2gb stores
currently, and is not asyncio ready, but as it was a nice tease for
the storage io /replacement reorganisation, it will be a nice tease
for the storage manager memory reorganisation (one of the thing I
want to do with COSS and UFS is support clustered writes so COSS
doesn't require a membuf and lots of copying and UFS objects are written
in chunks larger than DISK_PAGE_SIZE or SM_PAGE_SIZE).

Adrian
Received on Sun Apr 23 2000 - 04:57:35 MDT