Greetings!
I've been trying to locate a problem I've been
having with Squid 1.2beta22, 1.2beta25 and lately 2.0release.
Squid cache works fine until somewhere above 80% full, this starts
1998/10/10 10:18:28| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:18:28| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:18:28| WARNING: Shrinking cache_dir #1 to 1524642 KB
1998/10/10 10:30:08| diskHandleWrite: FD 16: disk write error: (28) No
space left on device
1998/10/10 10:30:08| storeSwapOutHandle: SwapOut failure (err code = -6).
That keeps happening, (shrinking cache) until I either notice, and restart
squid, or all but 40% ish has been purged.
I want to upgrade hard drive capacity and memory, but am not
keen on doing so until I can get this situation stable.
Any suggestions or further questions to better define the problem
would be most welcome.
I'm tending towards thinking faulty hard drive, or maybe not enough memory.
I've included details of all the things I've been watching.
Summary info
------------
Current squid version is 2.0release
OS is Linux 2.0.34, Redhat 5.1
All running with the help of a P233 CPU.
Boot, squid, squid logs drive is a small recycled IDE drive.
Logs are analysed, and rotated each night. Only 1 copy is kept.
Cache drives are on a single PCI scsi card, using ncr53c8xx module.
Main cache drives are 2 recycled 2gig scsi's, configured:
cache_dir /usr/local/squid/cache1 1800 16 256
cache_dir /usr/local/squid/cache2 1800 16 256
Midnight, last night :
df :
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/hda2 169819 122450 38599 76% /
/dev/sda1 2028862 1543746 380230 80%
/usr/local/squid/cache1
/dev/sdb1 2028862 1542416 381560 80%
/usr/local/squid/cache2
At the moment:
df
Filesystem 1024-blocks Used Available Capacity Mounted on
/dev/hda2 169819 108722 52327 68% /
/dev/sda1 2028862 1154317 769659 60%
/usr/local/squid/cache1
/dev/sdb1 2028862 1157121 766855 60%
/usr/local/squid/cache2
At midnight, last night,
Store Directory Statistics:
Store Entries : 264477
Maximum Swap Size : 3686400 KB
Current Store Swap Size: 3007967 KB
Current Capacity : 82% used, 18% free
Store Directory #0: /usr/local/squid/cache1
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1503960 KB
Percent Used: 81.60%
Filemap bits in use: 132595 of 335127 (40%)
Flags: SELECTED
Store Directory #1: /usr/local/squid/cache2
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1504007 KB
Percent Used: 81.60%
Filemap bits in use: 130227 of 335127 (39%)
Flags:
After last 'shrinking event' :
Store Directory Statistics:
Store Entries : 193008
Maximum Swap Size : 3686400 KB
Current Store Swap Size: 2167526 KB
Current Capacity : 59% used, 41% free
Store Directory #0: /usr/local/squid/cache1
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1083767 KB
Percent Used: 58.80%
Filemap bits in use: 96912 of 335127 (29%)
Flags: SELECTED
Store Directory #1: /usr/local/squid/cache2
First level subdirectories: 16
Second level subdirectories: 256
Maximum Size: 1843200 KB
Current Size: 1083759 KB
Percent Used: 58.80%
Filemap bits in use: 96091 of 335127 (29%)
Flags:
---
Complied with 256 file descriptors.
(I don't see this as currently being a problem. )
Midnight, last night:
File descriptor usage for squid:
Maximum number of file descriptors: 256
Largest file desc currently in use: 47
Number of file desc currently in use: 30
Available number of file descriptors: 226
Reserved number of file descriptors: 64
Disk files open: 4
Currently:
File descriptor usage for squid:
Maximum number of file descriptors: 256
Largest file desc currently in use: 45
Number of file desc currently in use: 26
Available number of file descriptors: 230
Reserved number of file descriptors: 64
Disk files open: 4
---
Memory is 96 meg, which also seems to be fine.
Midnight stats were :
free :
total used free shared buffers cached
Mem: 95480 94104 1376 13212 5584 14596
-/+ buffers/cache: 73924 21556
Swap: 32808 0 32808
vmstat :
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 1356 5584 14604 0 0 4 4 178 55 2 1 98
Stats at the moment are :
free
total used free shared buffers cached
Mem: 95480 82468 13012 12564 35416 13036
-/+ buffers/cache: 34016 61464
Swap: 32808 0 32808
vmstat
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
0 0 0 0 12904 35480 13060 0 0 4 5 173 56 2 1 98
---
Midnight last night :
mgr:info :
Squid Object Cache: Version 2.0.RELEASE
Start Time: Wed, 07 Oct 1998 08:47:48 GMT
Current Time: Fri, 09 Oct 1998 18:00:01 GMT
Connection information for squid:
Number of clients accessing cache: 70
Number of HTTP requests received: 189450
Number of ICP messages received: 2683280
Number of ICP messages sent: 2692803
Number of queued ICP replies: 1
Request failure ratio: 0.00%
HTTP requests per minute: 55.2
ICP messages per minute: 1566.4
Select loop called: 5189231 times, 39.685 ms avg
Cache information for squid:
Storage Swap size: 3007967 KB
Storage Mem size: 31120 KB
Storage LRU Expiration Age: 90.00 days
Mean Object Size: 11.44 KB
Requests given to unlinkd: 10215
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.39928 0.32154
Cache Misses: 1.54242 1.31166
Cache Hits: 0.04776 0.07014
Near Hits: 0.33943 0.68577
Not-Modified Replies: 0.00405 0.05331
DNS Lookups: 0.06083 0.11926
ICP Queries: 0.09506 0.10091
Resource usage for squid:
UP Time: 205933.908 seconds
CPU Time: 3707.220 seconds
CPU Usage: 1.80%
CPU Usage, 5 minute avg: 4.63%
CPU Usage, 60 minute avg: 1.33%
Maximum Resident Size: 0 KB
Page faults with physical i/o: 149526
Memory usage for squid via mallinfo():
Total space in arena: 63543 KB
Ordinary blocks: 63031 KB 2919 blks
Small blocks: 0 KB 0 blks
Holding blocks: 2264 KB 6 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 512 KB
Total in use: 65295 KB 103%
Total free: 512 KB 1%
Memory accounted for:
Total accounted: 54894 KB
File descriptor usage for squid:
Maximum number of file descriptors: 256
Largest file desc currently in use: 47
Number of file desc currently in use: 30
Available number of file descriptors: 226
Reserved number of file descriptors: 64
Disk files open: 4
Internal Data Structures:
264477 StoreEntries
6294 StoreEntries with MemObjects
6288 Hot Object Cache Items
262822 Filemap bits set
262822 on-disk objects
And just now :
mgr:info
Squid Object Cache: Version 2.0.RELEASE
Start Time: Sat, 10 Oct 1998 02:49:52 GMT
Current Time: Sat, 10 Oct 1998 04:12:14 GMT
Connection information for squid:
Number of clients accessing cache: 20
Number of HTTP requests received: 3615
Number of ICP messages received: 62503
Number of ICP messages sent: 62612
Number of queued ICP replies: 0
Request failure ratio: 0.00%
HTTP requests per minute: 43.9
ICP messages per minute: 1518.8
Select loop called: 124687 times, 39.641 ms avg
Cache information for squid:
Storage Swap size: 2167907 KB
Storage Mem size: 2532 KB
Storage LRU Expiration Age: 90.00 days
Mean Object Size: 11.23 KB
Requests given to unlinkd: 117
Median Service Times (seconds) 5 min 60 min:
HTTP Requests (All): 0.05046 0.16775
Cache Misses: 1.46131 1.38447
Cache Hits: 0.05046 0.05331
Near Hits: 0.00000 1.31166
Not-Modified Replies: 0.03427 0.04277
DNS Lookups: 0.06364 0.08717
ICP Queries: 0.12070 0.15326
Resource usage for squid:
UP Time: 4942.757 seconds
CPU Time: 82.550 seconds
CPU Usage: 1.67%
CPU Usage, 5 minute avg: 1.02%
CPU Usage, 60 minute avg: 1.46%
Maximum Resident Size: 0 KB
Page faults with physical i/o: 5384
Memory usage for squid via mallinfo():
Total space in arena: 22171 KB
Ordinary blocks: 22093 KB 3087 blks
Small blocks: 0 KB 0 blks
Holding blocks: 420 KB 1 blks
Free Small blocks: 0 KB
Free Ordinary blocks: 78 KB
Total in use: 22513 KB 102%
Total free: 78 KB 0%
Memory accounted for:
Total accounted: 16996 KB
File descriptor usage for squid:
Maximum number of file descriptors: 256
Largest file desc currently in use: 45
Number of file desc currently in use: 30
Available number of file descriptors: 226
Reserved number of file descriptors: 64
Disk files open: 6
Internal Data Structures:
193033 StoreEntries
482 StoreEntries with MemObjects
479 Hot Object Cache Items
193028 Filemap bits set
193026 on-disk objects
----
cache.log from when purging started :
1998/10/10 10:18:28| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:18:28| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:18:28| WARNING: Shrinking cache_dir #1 to 1524642 KB
1998/10/10 10:30:08| diskHandleWrite: FD 16: disk write error: (28) No
space left on device
1998/10/10 10:30:08| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:30:08| WARNING: Shrinking cache_dir #1 to 1436514 KB
1998/10/10 10:34:45| diskHandleWrite: FD 21: disk write error: (28) No
space left on device
1998/10/10 10:34:45| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:34:45| WARNING: Shrinking cache_dir #0 to 1382024 KB
1998/10/10 10:40:20| diskHandleWrite: FD 29: disk write error: (28) No
space left on device
1998/10/10 10:40:20| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:40:20| WARNING: Shrinking cache_dir #1 to 1341462 KB
1998/10/10 10:42:32| diskHandleWrite: FD 35: disk write error: (28) No
space left on device
1998/10/10 10:42:32| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:42:32| WARNING: Shrinking cache_dir #0 to 1279035 KB
1998/10/10 10:43:56| diskHandleWrite: FD 10: disk write error: (28) No
space left on device
1998/10/10 10:43:56| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:43:56| WARNING: Shrinking cache_dir #1 to 1248036 KB
1998/10/10 10:44:22| diskHandleWrite: FD 5: disk write error: (28) No space
left on device
1998/10/10 10:44:22| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:44:22| WARNING: Shrinking cache_dir #0 to 1204931 KB
1998/10/10 10:44:36| diskHandleWrite: FD 5: disk write error: (28) No space
left on device
1998/10/10 10:44:36| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:44:36| WARNING: Shrinking cache_dir #1 to 1173167 KB
1998/10/10 10:47:57| diskHandleWrite: FD 6: disk write error: (28) No space
left on device
1998/10/10 10:47:57| storeSwapOutHandle: SwapOut failure (err code = -6).
1998/10/10 10:47:57| WARNING: Shrinking cache_dir #0 to 1117214 KB
1998/10/10 10:48:38| Preparing for shutdown after 203171 requests
1998/10/10 10:48:38| Waiting 30 seconds for active connections to finish
1998/10/10 10:48:38| FD 11 Closing HTTP connection
1998/10/10 10:49:10| Shutting down...
1998/10/10 10:49:10| FD 46 Closing ICP connection
1998/10/10 10:49:10| FD 47 Closing SNMP socket
1998/10/10 10:49:10| Closing unlinkd pipe on FD 18
1998/10/10 10:49:10| storeDirWriteCleanLogs: Starting...
1998/10/10 10:49:10| 65536 entries written so far.
1998/10/10 10:49:11| 131072 entries written so far.
1998/10/10 10:49:11| Finished. Wrote 192482 entries.
1998/10/10 10:49:11| Took 1 seconds (192482.0 entries/sec).
[end]
.----------------------------------------------.------------------------.
| Mark Reynolds <mark@rts.com.au> | Phone 08 9474 1211 |
| Network Manager, Reynolds Technology Pty Ltd | Fax 08 9474 4772 |
| 8 Preston Street Como 6152 Western Australia | Pager 08 9480 5884 |
| PO Box 120 Como 6952 Western Australia | http://www.rts.com.au/ |
`----------------------------------------------^------------------------'
Received on Fri Oct 09 1998 - 21:30:46 MDT
This archive was generated by hypermail pre-2.1.9 : Tue Dec 09 2003 - 16:42:24 MST