Re: unresponsive cache and stats with a loaded cache

--MimeMultipartBoundary
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit

Andres Kroonmaa:

> Once I chased the way squid uses its L1/L2 directory structures, and
> it appeared to me that excessive amount of directories slows things
> down quite a bit. I don't recall exactly, but I have an impression that
> squid goes through every directory, before returning to any one for a
> second time. On the way squid alters both L1 and L2 dirs in way to
> distribute load evenly between disks, L1 and L2 directories on each disk.

Hmmm, you're right. The patch below solves this problem by making sure
that a configurable number of files (default 512, maybe a little too high)
are written to a directory before going to the next one. Alternating
among all cache_dirs is still done, too.

Besides being more efficient for directory caching for writes it can also
have a good effect for reads because with this scheme it is more likely
that a page and it's pictures are stored in one directory instead of
several different ones.

> Whats bad news with it is that Unix FS tries to cache directory entries
> to speedup file searches (open) within directory and this kind of pass-through
> effectively busts to nothing unix's directory caching. By the time squid
> returns to once-used directory, quite alot of time passes and directory
> data is not in cache. By default squid creates 4096 dirs per disk, and
> with full cache each dir is about 4KB size. To keep all the dir cache
> in ram you'd need to have 16MB of spare ram per each disk which is very
> rarely the case.

> Thus, I believe that with every single file open-for-write squid has a
> directory cache miss and needs to do physical disk io for directory.

In the original case you would have 4096 dirs * (4K read + 4K write) =
32MB data to write with 8192 seeks, with my patch (assuming perfect cache
behavior) 4096/(256 files/dir) = 16 dirs * (4K read + 4K write) = 128KB
with 32 seeks. Although my computations may be a too optimistic, the
difference is clear.

> open-for-read is random in nature and this cannot be very much optimized.

Keeping related objects in one directory should help a little.

> All this results in the reasoning for the suggestion in squid.1.1.relnotes,
> you'd want to have the minimum amount of directories needed to hold max
> possible number of objects on your disks. Also, you'd want to increase
> DNLC cache and have lots of unused ram to make it possible for unix to
> cache directory io. Also, disabling inode-accesstime-updates and
> sync mode for dir-io helps alot, but you know what you are risking here...

That's almost default for things like proxy-caches and news servers
nowadays I think.

> Personally, I'd suggest to change squid's store logic a bit in a way that
> it would always use smallest numbered file available, instead of distributing

This sounds not so easy, although I haven't studied the exact details.

> load between L1/L2 dirs. Clearly, distributing load between disks is another
> issue and doesn't add very much to dir-cache load. Having fixed number
> of files per any directory there is no speed-loss, but for caches that have
> 100 times more directories created than there ever could be objects in
> them it is IMHO more reasonable to have few directorys full of 256 objects
> rather than have zillions of dirs with few files in each.

Yes, minimizing the number of directories is the way to go. One should
compute the expected number of objects in the cache (average object size
is around 13KB in my case) and solve the following equation:

        #objects = L1 * L2 * swap_files_per_dir * #cache_dirs

where L1*L2 should be kept as low as possible and swap_files_per_dir
should be such that the directory fits within one filesystem block (4K/8K).

Arjan

Here's the patch, relative to 1.1.10. NOTE: you will need to start with
an emtpy cache of course.

diff -uw /usr/local/src/squid/squid-1.1.10-cis/src/cache_cf.c src/cache_cf.c
--- /usr/local/src/squid/squid-1.1.10-cis/src/cache_cf.c Tue Jul 8 14:04:30 1997
+++ src/cache_cf.c Mon Nov 10 21:32:35 1997
@@ -203,6 +203,7 @@
 
 #define DefaultLevelOneDirs 16
 #define DefaultLevelTwoDirs 256
+#define DefaultFilesPerDir 512
 #define DefaultOptionsLogUdp 1 /* on */
 #define DefaultOptionsEnablePurge 0 /* default off */
 #define DefaultOptionsClientDb 1 /* default on */
@@ -1426,6 +1427,8 @@
             parseIntegerValue(&Config.levelOneDirs);
         else if (!strcmp(token, "swap_level2_dirs"))
             parseIntegerValue(&Config.levelTwoDirs);
+ else if (!strcmp(token, "swap_files_per_dir"))
+ parseIntegerValue(&Config.files_per_dir);
 
         else if (!strcmp(token, "netdb_high"))
             parseIntegerValue(&Config.Netdb.high);
@@ -1648,6 +1651,7 @@
     Config.Store.objectsPerBucket = DefaultObjectsPerBucket;
     Config.levelOneDirs = DefaultLevelOneDirs;
     Config.levelTwoDirs = DefaultLevelTwoDirs;
+ Config.files_per_dir = DefaultFilesPerDir;
     Config.Options.log_udp = DefaultOptionsLogUdp;
     Config.Options.res_defnames = DefaultOptionsResDefnames;
     Config.Options.anonymizer = DefaultOptionsAnonymizer;
diff -uw /usr/local/src/squid/squid-1.1.10-cis/src/cache_cf.h src/cache_cf.h
--- /usr/local/src/squid/squid-1.1.10-cis/src/cache_cf.h Fri Aug 1 15:10:20 1997
+++ src/cache_cf.h Mon Nov 10 21:25:49 1997
@@ -270,6 +270,7 @@
     } vizHack;
     int levelOneDirs;
     int levelTwoDirs;
+ int files_per_dir;
     struct {
         int high;
         int low;
diff -uw /usr/local/src/squid/squid-1.1.10-cis/src/store.c src/store.c
--- /usr/local/src/squid/squid-1.1.10-cis/src/store.c Mon Nov 3 22:41:50 1997
+++ src/store.c Mon Nov 10 21:28:23 1997
@@ -1079,8 +1079,9 @@
     fullpath[0] = '\0';
     sprintf(fullpath, "%s/%02X/%02X/%08X",
         swappath(fn),
- (fn / ncache_dirs) % Config.levelOneDirs,
- (fn / ncache_dirs) / Config.levelOneDirs % Config.levelTwoDirs,
+ (fn / ncache_dirs / Config.files_per_dir / Config.levelTwoDirs )
+ % Config.levelOneDirs,
+ (fn / ncache_dirs / Config.files_per_dir ) % Config.levelTwoDirs,
         fn);
     return fullpath;
 }

--MimeMultipartBoundary--
Received on Tue Jul 29 2003 - 13:15:44 MDT