Re: [squid-users] segfault at 0

On 18/05/2012 6:45 a.m., Pablo Arturi wrote:
>>> I'm having squid crash/restarting automatically from a week ago more
>>> or less, without any hardware or software change.
>>>
>>> I had the squid 3.1.16 release which I updated yesterday to 3.1.19
>>> from the gentoo package manager, and it keep logging the same errors:
>>>
>>> This is the evidence:
>>>
>>> firewall ~ # cat /var/log/kern.log | grep squid
>>>
>>> May 16 16:49:24 localhost kernel: squid[13702]: segfault at 0 ip
>>> 000000000053eae3 sp 000003df3c6c39b8 error 4 in squid[400000+23e000]
>>> May 16 16:49:24 localhost kernel: grsec: From 10.200.145.55:
>>> Segmentation fault occurred at (null) in /usr/sbin/squid[squid:13702]
>>> uid/euid:0/31 gid/egid:31/31, parent /usr/sbin squid[squid:12267]
>>> uid/euid:0/0 gid/egid:31/31
>>> May 16 16:49:28 localhost kernel: squid[14162]: segfault at 0 ip
>>> 000000000053eae3 sp 000003a1d7901b98 error 4 in squid[400000+23e000]
>>> May 16 16:49:28 localhost kernel: grsec: From 10.200.145.55:
>>> Segmentation fault occurred at (null) in /usr/sbin/squid[squid:14162]
>>> uid/euid:0/31 gid/egid:31/31, parent /usr/sbin squid[squid:12267]
>>> uid/euid:0/0 gid/egid:31/31
>>>
>>> firewall ~ # cat /var/log/messages | grep squid
>>>
>>> May 16 16:49:24 localhost squid[12267]: Squid Parent: child process
>>> 13702 exited due to signal 11 with status 0
>>> May 16 16:49:27 localhost squid[12267]: Squid Parent: child process
>>> 14162 started
>>> May 16 16:49:28 localhost squid[12267]: Squid Parent: child process
>>> 14162 exited due to signal 11 with status 0
>>> May 16 16:49:31 localhost squid[12267]: Squid Parent: child process
>>> 14201 started
>>>
>> <snip>
>>
>>> Any help will be appreciated.
>>
>>
>> Can you find or make a core dump to see where this crash is occuring?
>>
>> There may also be some extra information in cache.log from Squid when
>> it exited.
>>
>> http://wiki.squid-cache.org/SquidFaq/BugReporting
>>
>>
>> Amos
>>
>
> Amos, thanks for your reply.
>
> This is the debug information, I took it from the suggested method in
> SquidFaq/BugReporting which was creating a script with the following
> code:
>
> trap "rm -f $$.gdb" 0
> cat <<EOF >$$.gdb
> handle SIGPIPE pass nostop noprint
> handle SIGTERM pass nostop noprint
> handle SIGUSR1 pass nostop noprint
> handle SIGSEGV stop
> handle SIGABRT stop
> run -DNYCd3
> backtrace
> quit
> EOF
> while sleep 2; do
> gdb -x $$.gdb /path/to/squid 2>&1 | tee -a squid.out
> doneThis is the squid.out content after some crashes (it crashed a few
> times, but I think it's the same information, so I'll just put once;
> if you need it I'll post the entire file to an URL so you can check
> it).firewall ~ # cat /root/squid.out | less
>
> GNU gdb (Gentoo 7.3.1 p2) 7.3.1
> Copyright (C) 2011 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later
> <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law. Type "show
> copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-pc-linux-gnu".
> For bug reporting instructions, please see:
> <http://bugs.gentoo.org/>...
> Reading symbols from /usr/sbin/squid...(no debugging symbols
> found)...done.
> 2012/05/17 13:48:52| WARNING: -D command-line option is obsolete.
> 2012/05/17 13:48:52| Starting Squid Cache version 3.1.19 for
> x86_64-pc-linux-gnu...
> 2012/05/17 13:48:52| Process ID 16781
> 2012/05/17 13:48:52| With 1024 file descriptors available
> 2012/05/17 13:48:52| Initializing IP Cache...
> 2012/05/17 13:48:52| DNS Socket created at [::], FD 10
> 2012/05/17 13:48:52| DNS Socket created at 0.0.0.0, FD 11
> 2012/05/17 13:48:52| Adding nameserver 127.0.0.1 from squid.conf
> 2012/05/17 13:48:52| Adding nameserver 208.67.222.222 from squid.conf
> 2012/05/17 13:48:52| Adding nameserver 208.67.220.220 from squid.conf
> 2012/05/17 13:48:52| helperOpenServers: Starting 20/20 'squidGuard'
> processes
> 2012/05/17 13:48:52| Referer logging is disabled.
> 2012/05/17 13:48:52| Unlinkd pipe opened on FD 57
> 2012/05/17 13:48:52| Local cache digest enabled; rebuild/rewrite every
> 3600/3600 sec
> 2012/05/17 13:48:52| Store logging disabled
> 2012/05/17 13:48:52| Swap maxSize 16777216 + 524288 KB, estimated
> 1330884 objects
> 2012/05/17 13:48:52| Target number of buckets: 66544
> 2012/05/17 13:48:52| Using 131072 Store buckets
> 2012/05/17 13:48:52| Max Mem size: 524288 KB
> 2012/05/17 13:48:52| Max Swap size: 16777216 KB
> 2012/05/17 13:48:52| Version 1 of swap file with LFS support detected...
> 2012/05/17 13:48:52| Rebuilding storage in /var/cache/squid (CLEAN)
> 2012/05/17 13:48:52| Using Least Load store dir selection
> 2012/05/17 13:48:52| Current Directory is /root
> 2012/05/17 13:48:52| Loaded Icons.
> 2012/05/17 13:48:52| Accepting intercepted HTTP connections at
> 127.0.0.1:3128, FD 60.
> 2012/05/17 13:48:52| Accepting intercepted HTTP connections at
> 10.200.145.100:3128, FD 61.
> 2012/05/17 13:48:52| Accepting intercepted HTTP connections at
> 50.50.50.213:3128, FD 62.
> 2012/05/17 13:48:52| HTCP Disabled.
> 2012/05/17 13:48:52| Squid plugin modules loaded: 0
> 2012/05/17 13:48:52| Ready to serve requests.
> 2012/05/17 13:48:52| Store rebuilding is 8.92% complete
> 2012/05/17 13:48:52| Done reading /var/cache/squid swaplog (45906
> entries)
> 2012/05/17 13:48:52| Finished rebuilding storage from disk.
> 2012/05/17 13:48:52| 45906 Entries scanned
> 2012/05/17 13:48:52| 0 Invalid entries.
> 2012/05/17 13:48:52| 0 With invalid flags.
> 2012/05/17 13:48:52| 45906 Objects loaded.
> 2012/05/17 13:48:52| 0 Objects expired.
> 2012/05/17 13:48:52| 0 Objects cancelled.
> 2012/05/17 13:48:52| 0 Duplicate URLs purged.
> 2012/05/17 13:48:52| 0 Swapfile clashes avoided.
> 2012/05/17 13:48:52| Took 0.30 seconds (151289.75 objects/sec).
> 2012/05/17 13:48:52| Beginning Validation Procedure
> 2012/05/17 13:48:52| Completed Validation Procedure
> 2012/05/17 13:48:52| Validated 91837 Entries
> 2012/05/17 13:48:52| store_swap_size = 1045568
> 2012/05/17 13:48:53| storeLateRelease: released 0 objects
> 2012/05/17 14:12:08| squidaio_queue_request: WARNING - Queue congestion
> 2012/05/17 14:28:40| WARNING: swapfile header inconsistent with
> available data
>
> Program received signal SIGSEGV, Segmentation fault.
> 0x000000000053eae3 in storeRead ()
> #0 0x000000000053eae3 in storeRead ()
> #1 0x000000000053fbe9 in store_client::fileRead() ()
> #2 0x00000000005a1418 in UFSStoreState::readCompleted(char const*,
> int, int, RefCount<ReadRequest>) ()

This is being tracked in http://bugs.squid-cache.org/show_bug.cgi?id=3549

Thanks for the info this is being seen in 3.1.19.

Amos
Received on Fri May 18 2012 - 00:38:40 MDT