Re: unresponsive cache and stats with a loaded cache

I wrote:

>Andres Kroonmaa:
>
>> Once I chased the way squid uses its L1/L2 directory structures, and
>> it appeared to me that excessive amount of directories slows things
>> down quite a bit. I don't recall exactly, but I have an impression that
>> squid goes through every directory, before returning to any one for a
>> second time. On the way squid alters both L1 and L2 dirs in way to
>> distribute load evenly between disks, L1 and L2 directories on each disk.
>
>Hmmm, you're right. The patch below solves this problem by making sure
>that a configurable number of files (default 512, maybe a little too high)
>are written to a directory before going to the next one. Alternating
>among all cache_dirs is still done, too.
>
>Besides being more efficient for directory caching for writes it can also
>have a good effect for reads because with this scheme it is more likely
>that a page and it's pictures are stored in one directory instead of
>several different ones.

I decided to apply the patch on a real production cache this weekend and
have gathered one day of business-hours (08:00h-18:00h) usage data (data
has been gathered with my Squid timer patch). The results are quite good
if you compare the results of last Friday and today (Monday):

Friday November 14th (without patch):

Total time: 36001888 ms (10.00 hours)

       acl: 494550 ms 1.37% 1522151 calls 0.325 ms/call
   connect: 538307 ms 1.50% 929798 calls 0.579 ms/call
     diskr: 5931227 ms 16.47% 489457 calls 12.118 ms/call
     diskw: 1355709 ms 3.77% 1666468 calls 0.814 ms/call
      idle: 4064428 ms 11.29% 24301147 calls 0.167 ms/call
      netr: 211319 ms 0.59% 3510751 calls 0.060 ms/call
      netw: 533726 ms 1.48% 4090178 calls 0.130 ms/call
     openr: 8480700 ms 23.56% 354137 calls 23.948 ms/call
     openw: 8845615 ms 24.57% 254643 calls 34.737 ms/call

Monday November 17th (with patch):

Total time: 35980657 ms (9.99 hours)

       acl: 606635 ms 1.69% 1789964 calls 0.339 ms/call
   connect: 807657 ms 2.24% 1132707 calls 0.713 ms/call
     diskr: 4053126 ms 11.26% 480644 calls 8.433 ms/call
     diskw: 759850 ms 2.11% 1975015 calls 0.385 ms/call
      idle: 16682065 ms 46.36% 45496243 calls 0.367 ms/call
      netr: 272786 ms 0.76% 6368543 calls 0.043 ms/call
      netw: 680685 ms 1.89% 6725937 calls 0.101 ms/call
     openr: 2944895 ms 8.18% 358970 calls 8.204 ms/call
     openw: 1576401 ms 4.38% 315551 calls 4.996 ms/call

open(2)'s for write have dropped from 35ms to 5ms, open(2) for read had
dropped from 24ms to 8ms, read(2)'s have dropped from 12ms to 8ms. This
all leads to 46.3% idle time, i.e. Squid is waiting in select(2) for
something to happen. Waiting for disk I/O (diskr/w+openr/w) has dropped
from 68.37% to 25.93%.

I'll clean up the patch tonight and announce it on squid-users.

Arjan

PS: measurements were done on the BSD/OS 3.0 operating system, 2*4GB cache,
    PPro200 CPU, 256MB RAM, L1=32, L2=32, files_per_dir=384.

-- 
Arjan de Vet, Eindhoven, The Netherlands            <Arjan.deVet@adv.IAEhv.nl>
URL: http://www.IAEhv.nl/users/devet/       for PGP key: finger devet@IAEhv.nl