Re: [squid-users] Performance Issues Using NTML

Scott Anctil wrote:
> I have deployed a Squid server for a local school board to help with
> there ever increasing bandwidth issues. It is running Squid 2.6 Stable
> 16 under Ubuntu 7.10 server on a HP DL380. This server has 2GB of RAM, 2
> dual core 3.06 GHz processors and 288 GB of SAS 15k storage (RAID).

Two points, squid is still single-process, the core will only use one of
the CPU.

RAID on the squid cache drive is not a good idea. It is a waste of disk
IO maintaining the RAID. Cache can be re-generated as needed from the web.

> I am
> using NTLM authentication. We have only two schools running on it
> (300-500 concurrent connections) and the box is already running at
> 30-50% CPU consistently. The one time we tried all of the schools
> (1000-4000 concurrent connections) the box went to 100% CPU solid and
> users were dropping pages. I have been scouring the internet for answers
> and have made a number of changes, none of which have helped. Should I
> not be expecting more out of this box? Here are the options I used to
> compile.
>
> ./configure --with-maxfd=4096 --prefix=/usr/local/squid
> --enable-basic-auth-helpers="SMB" --enable-ntlm-auth-helpers="SMB"
> --enable-external-acl-helpers="wbinfo_group" --enable-auth="basic,ntlm"
> --with-winbind-auth-challenge
>
> Here is my squid.conf
>
> #######################
> # Basic Configuration #
> #######################
>
> visible_hostname ***************
> http_port 3128
> cache_dir ufs /usr/local/squid/cache 50000 15 256

Try aufs, its slightly faster on most systems.

> acl apache rep_header Server ^Apache
> broken_vary_encoding allow apache
> tcp_outgoing_address 10.1.10.211

outgoing should not be needed, your call though.

> append_domain *****************
> httpd_suppress_version_string on

why? be proud of your squid. :-)

> cache_effective_user squid
> authenticate_ttl 24 hours

Really people stay authenticated for 24 hrs?

> authenticate_ip_ttl 15 minutes
>
> #############
> # Log Files #
> #############
>
> cache_access_log /usr/local/squid/var/logs/access.log

that should be just "access_log"

> cache_log /usr/local/squid/var/logs/cache.log
> cache_store_log /usr/local/squid/var/logs/store.log

I'm not certain why this is even still around. Its only useful for
debugging the cache store. you an safely set it to "none" and reduce
load a lot.

>
> ###################
> # Control Caching #
> ###################
>
> hierarchy_stoplist cgi-bin ?
> acl QUERY urlpath_regex cgi-bin \?
> cache deny QUERY
>
> refresh_pattern cgi-bin 1 20% 2
> refresh_pattern \.asp$ 1 20% 2
> refresh_pattern \.acgi$ 1 20% 2
> refresh_pattern \.cgi$ 1 20% 2
> refresh_pattern \.pl$ 1 20% 2
> refresh_pattern \.shtml$ 1 20% 2
> refresh_pattern \.php3$ 1 20% 2
> refresh_pattern \? 1 20% 2
> refresh_pattern \.gif$ 10080 90% 43200
> refresh_pattern \.jpg$ 10080 90% 43200
> refresh_pattern \.bom\.gov\.au 30 20% 120
> refresh_pattern \.html$ 480 50% 22160
> refresh_pattern \.htm$ 480 50% 22160
> refresh_pattern \.class$ 10080 90% 43200
> refresh_pattern \.zip$ 10080 90% 43200
> refresh_pattern \.jpeg$ 10080 90% 43200
> refresh_pattern \.mid$ 10080 90% 43200
> refresh_pattern \.shtml$ 480 50% 22160
> refresh_pattern \.exe$ 10080 90% 43200
> refresh_pattern \.thm$ 10080 90% 43200
> refresh_pattern \.wav$ 10080 90% 43200
> refresh_pattern \.txt$ 10080 90% 43200
> refresh_pattern \.cab$ 10080 90% 43200
> refresh_pattern \.au$ 10080 90% 43200
> refresh_pattern \.mov$ 10080 90% 43200
> refresh_pattern \.xbm$ 10080 90% 43200
> refresh_pattern \.ram$ 10080 90% 43200
> refresh_pattern \.avi$ 10080 90% 43200
> refresh_pattern \.chtml$ 480 50% 22160
> refresh_pattern \.thb$ 10080 90% 43200
> refresh_pattern \.dcr$ 10080 90% 43200
> refresh_pattern \.bmp$ 10080 90% 43200
> refresh_pattern \.phtml$ 480 50% 22160
> refresh_pattern \.mpg$ 10080 90% 43200
> refresh_pattern \.pdf$ 10080 90% 43200
> refresh_pattern \.art$ 10080 90% 43200
> refresh_pattern \.swf$ 10080 90% 43200
> refresh_pattern \.mp3$ 10080 90% 43200
> refresh_pattern \.ra$ 10080 90% 43200
> refresh_pattern \.spl$ 10080 90% 43200
> refresh_pattern \.viv$ 10080 90% 43200
> refresh_pattern \.doc$ 10080 90% 43200
> refresh_pattern \.gz$ 10080 90% 43200
> refresh_pattern \.Z$ 10080 90% 43200
> refresh_pattern \.tgz$ 10080 90% 43200
> refresh_pattern \.tar$ 10080 90% 43200
> refresh_pattern \.vrm$ 10080 90% 43200
> refresh_pattern \.vrml$ 10080 90% 43200
> refresh_pattern \.aif$ 10080 90% 43200
> refresh_pattern \.aifc$ 10080 90% 43200
> refresh_pattern \.aiff$ 10080 90% 43200
> refresh_pattern \.arj$ 10080 90% 43200
> refresh_pattern \.c$ 10080 90% 43200
> refresh_pattern \.cpt$ 10080 90% 43200
> refresh_pattern \.dir$ 10080 90% 43200
> refresh_pattern \.dxr$ 10080 90% 43200
> refresh_pattern \.hqx$ 10080 90% 43200
> refresh_pattern \.jpe$ 10080 90% 43200
> refresh_pattern \.lha$ 10080 90% 43200
> refresh_pattern \.lzh$ 10080 90% 43200
> refresh_pattern \.midi$ 10080 90% 43200
> refresh_pattern \.movie$ 10080 90% 43200
> refresh_pattern \.mp2$ 10080 90% 43200
> refresh_pattern \.mpe$ 10080 90% 43200
> refresh_pattern \.mpeg$ 10080 90% 43200
> refresh_pattern \.mpga$ 10080 90% 43200
> refresh_pattern \.pl$ 10080 90% 43200
> refresh_pattern \.ppt$ 10080 90% 43200
> refresh_pattern \.ps$ 10080 90% 43200
> refresh_pattern \.qt$ 10080 90% 43200
> refresh_pattern \.qtm$ 10080 90% 43200
> refresh_pattern \.ras$ 10080 90% 43200
> refresh_pattern \.sea$ 10080 90% 43200
> refresh_pattern \.sit$ 10080 90% 43200
> refresh_pattern \.tif$ 10080 90% 43200
> refresh_pattern \.tiff$ 10080 90% 43200
> refresh_pattern \.snd$ 10080 90% 43200
> refresh_pattern \.wrl$ 10080 90% 43200
> refresh_pattern ^ftp:// 480 60% 22160
> refresh_pattern ^gopher:// 30 20% 120
> refresh_pattern . 480 50% 22160
>
> acl post_requests method POST
> cache deny post_requests
>
> acl No_Cache_Sites url_regex "/usr/local/squid/etc/squid-no_cache.acl"

There is a HUGE cause of CPU wastage.

Spit that file into two lists:
  a) with pure domains or partial domains ie ".gov" for everything in *.gov

  b) with URI that absolutely MUST be matched with regex pattern

Put (a) into a dstdomain ACL. It's amazingly fast and less CPU than
regex. Bonus points for removing (b) and regex entirely.

> no_cache deny No_Cache_Sites

That should be just "cache deny No_Cache_Sites"

>
> ##########################################
> # Enable the NTLM Authentication Program #
> ##########################################
>
> auth_param ntlm program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-ntlmssp
> auth_param basic program /usr/bin/ntlm_auth
> --helper-protocol=squid-2.5-basic
> auth_param basic children 25
> auth_param ntlm children 25
> auth_param ntlm keep_alive on

25 helpers. Are they really needed? can much of the load on these be
done concurrently?

>
> ########
> # ACLs #
> ########
>
> external_acl_type nt_group ttl=60 children=25 protocol=2.5 %LOGIN
> /usr/local/squid/libexec/wbinfo_group.pl

Ditto for these 25 helpers. Can it also be done concurrently?

> acl all src 0.0.0.0/0.0.0.0
> acl localhost src 127.0.0.1/255.255.255.255

Its nicer to read
   acl all src all
   acl localhost src 127.0.0.1

and 'all' will be pre-defined from 2.6stable17 on

> acl Safe_Ports port 80
> acl Safe_Ports port 21
> acl Safe_Ports port 9080
> acl Safe_Ports port 812
> acl Safe_Ports port 9090
> acl Safe_Ports port 8090
> acl Safe_Ports port 9000
> acl Safe_Ports port 22
> acl Safe_Ports port 88
> acl Safe_Ports port 8000
> acl Safe_Ports port 8008
>
> acl SSL_Ports port 443
>
> acl purge method PURGE
> acl CONNECT method CONNECT
>
> acl NTLMUsers proxy_auth REQUIRED
> acl Students external nt_group students
>
> acl Blocked_Sites url_regex "/usr/local/squid/etc/squid-block.acl"
> acl Bypass_Sites url_regex "/usr/local/squid/etc/squid-bypass.acl"

Again with the regex. See my comments earlier about using dstdomain as
much as possible instead.

>
> ##################
> # Control Access #
> ##################
>
> http_access allow Bypass_Sites
> http_access deny Blocked_Sites
> http_access deny SSL_Ports NTLMUsers Students
> http_access allow Safe_Ports NTLMUsers Students
> http_access deny NTLMUsers Students
> http_access allow NTLMUsers
>
> http_access allow purge localhost
> http_access deny purge
> http_access deny !Safe_ports
> http_access deny CONNECT !SSL_ports
> http_access allow localhost
> http_access deny all
> icp_access allow all

Amos
Received on Fri Nov 09 2007 - 18:34:48 MST