Re: trying cache post request for some sites failed

On Mon, 22 Jun 2009 12:03:02 +0700, johan firdianto <johanfirdi_at_gmail.com>
wrote:
> I already add refresh pattern
> refresh_pattern -i xxxxx 999888 100% 999999 ignore-no-cache
> ignore-reload override-expire override-lastmod reload-into-ims
> this could help to force cache the object ?
> I analyze the url of post request is unique. And no important data
> sent in post form just information of captcha image.
> which part in source code I could override the header to force to
> cache this object ?
> I also comment this in http.c line 246.
>
> if (EBIT_TEST(cc_mask, CC_NO_STORE))
> return 0;
>
> but the store.log still release the object.
> Any ideas ?

Not that I can think of offhand. But I would suggest at minimum working
from the current 2.HEAD snapshot.
It's relatively stable and has a lot more work related to this kind of
caching.

Amos

> Thanks
>
> Johan
>
>
> On Mon, Jun 22, 2009 at 11:34 AM, Amos Jeffries<squid3_at_treenet.co.nz>
> wrote:
>> On Mon, 22 Jun 2009 10:21:16 +0700, johan firdianto
>> <johanfirdi_at_gmail.com>
>> wrote:
>>> dear guys,
>>>
>>> I'm using squid 2.7 stable 5. Tring to cache some sites that use POST
>>> method as download method.
>>> I modif the source code of client_side.c at line 1550 in function
>>> clientCachable
>>>
>>>
>>> if ((method == METHOD_POST) && (!
>>> strncmp(uri,"http://xxxxxxxxxxxxx/",40)))
>>> return 1;
>>>
>>> in store.c line 1225 in function storeCheckCachable
>>> if ((e->mem_obj->method == METHOD_POST) && (!
>>> strncmp(e->mem_obj->url,"http://xxxxxxxxx/",40)) ) {
>>> store_check_cachable_hist.yes.Default++;
>>> return 1;
>>> }
>>>
>>> Here the debug result, flag REQ_CACHABLE is SET
>>> The request POST http://xxxxx/3705584/SD-Mat07.pdf is ALLOWED, because
>>> it matched 'test'
>>> 2009/06/22 09:49:03| clientRedirectStart:
>>> 'http://xxxxxxxxx/3705584/SD-Mat07.pdf'
>>> 2009/06/22 09:49:03| clientRedirectDone:
>>> 'http://xxxxxxxx/3705584/SD-Mat07.pdf' result=NULL
>>> 2009/06/22 09:49:03| clientStoreURLRewriteStart:
>>> 'http://xxxxxxxxxxxxx/3705584/SD-Mat07.pdf'
>>> 2009/06/22 09:49:03| clientStoreURLRewriteDone:
>>> 'http://xxxxxxxxx/3705584/SD-Mat07.pdf' result=NULL
>>> 2009/06/22 09:49:03| 0xb9f8d8 lookup for 37
>>> 2009/06/22 09:49:03| 0xb9f8d8 lookup for 7
>>> 2009/06/22 09:49:03| 0xb9f8d8 lookup for 7
>>> 2009/06/22 09:49:03| 0xb9f8d8 lookup for 40
>>> 2009/06/22 09:49:03| 0xb9f8d8 lookup for 52
>>> 2009/06/22 09:49:03| check clientcacheble atau tidak:
>>> http://xxxxxxxxxxx/3705584/SD-Mat07.pdf
>>> 2009/06/22 09:49:03| clientInterpretRequestHeaders: REQ_NOCACHE = NOT
>>> SET
>>> 2009/06/22 09:49:03| clientInterpretRequestHeaders: REQ_CACHABLE = SET
>>> 2009/06/22 09:49:03| clientInterpretRequestHeaders: REQ_HIERARCHICAL =
>> NOT
>>> SET
>>>
>>> Looks like create privatekey of the object,
>>>
>>> new_MemObject: returning 0xb9bd10
>>> 2009/06/22 09:49:03| new_StoreEntry: returning 0xb7bac0
>>> 2009/06/22 09:49:03| storeKeyPrivate: POST
>>> http://xxxxxxxxxxxxxxxxxxxxxx/3705584/SD-Mat07.pdf
>>> 2009/06/22 09:49:03| storeHashInsert: Inserting Entry 0xb7bac0 key
>>> '137B66F60D79074DC0AD75776DA66457'
>>> 2009/06/22 09:49:03| storeLockObject: (store_client.c:122): key
>>> '137B66F60D79074DC0AD75776DA66457' count=2
>>> 2009/06/22 09:49:03| storeClientCopy:
>>> 137B66F60D79074DC0AD75776DA66457, seen 0, want 0, size 4096, cb
>>> 0x42a0e0, cbdata 0xb88258
>>> 2009/06/22 09:49:03| cbdataLock: 0xb88258
>>> 2009/06/22 09:49:03| cbdataLock: 0xb9a4b8
>>> 2009/06/22 09:49:03| storeClientCopy2: 137B66F60D79074DC0AD75776DA66457
>>> 2009/06/22 09:49:03| storeClientCopy3: Waiting for more
>>> 2009/06/22 09:49:03| cbdataUnlock: 0xb9a4b8
>>> 2009/06/22 09:49:03| aclCheckFast: list: (nil)
>>> 2009/06/22 09:49:03| aclCheckFast: no matches, returning: 1
>>> 2009/06/22 09:49:03| fwdStart:
>>> 'http://xxxxxxxxxxxxxxx/3705584/SD-Mat07.pdf'
>>> 2009/06/22 09:49:03| storeLockObject: (forward.c:964): key
>>> '137B66F60D79074DC0AD75776DA66457' count=3
>>> 2009/06/22 09:49:03| peerSelect:
>>> http://xxxxxxxxxxxxxxxxx/3705584/SD-Mat07.pdf
>>> 2009/06/22 09:49:03| storeLockObject: (peer_select.c:155): key
>>> '137B66F60D79074DC0AD75776DA66457' count=4
>>> 2009/06/22 09:49:03| cbdataLock: 0xb9a548
>>>
>>> Here response of POST request
>>> Date: Mon, 22 Jun 2009 02:52:42 GMT
>>> Server: Apache
>>> Expires: Thu, 19 Nov 1981 08:52:00 GMT
>>> Cache-Control: no-store, no-cache, must-revalidate, post-check=0,
>>> pre-check=0
>>> Pragma: no-cache
>>> Content-Disposition: attachment; filename="SD-Mat07.pdf";
>>> Last-Modified: Tue, 03 Mar 2009 01:21:22 GMT
>>> Content-Length: 85654
>>> ETag: "800023d7dcac-14e96-4642cc0c02880"
>>> Connection: close
>>> Content-Type: application/pdf
>>>
>>> at last minute of downloading, status store_status=OK
>>>
>>> 2009/06/22 09:49:19| storeSwapOut:
>>> http://xxxxxxxxxx/downloadfiles/3705584/SD-Mat07.pdf
>>> 2009/06/22 09:49:19| storeSwapOut: store_status = STORE_PENDING
>>> 2009/06/22 09:49:19| storeSwapOut: mem->inmem_lo = 81920
>>> 2009/06/22 09:49:19| storeSwapOut: mem->inmem_hi = 86083
>>> 2009/06/22 09:49:19| storeSwapOut: swapout.queue_offset = 0
>>> 2009/06/22 09:49:19| storeSwapOutMaintainMemObject: lowest_offset =
>>> 84945
>>> 2009/06/22 09:49:19| fwdComplete:
>>> http://xxxxxxxxxxxxx/downloadfiles/3705584/SD-Mat07.pdf
>>> status 200
>>> 2009/06/22 09:49:19| fwdReforward:
>>> http://xxxxxxxxxxxx/3705584/SD-Mat07.pdf?
>>> 2009/06/22 09:49:19| fwdReforward: No, ENTRY_FWD_HDR_WAIT isn't set
>>> 2009/06/22 09:49:19| fwdComplete: not re-forwarding status 200
>>> 2009/06/22 09:49:19| storeComplete: '137B66F60D79074DC0AD75776DA66457'
>>> 2009/06/22 09:49:19| storeEntryValidLength: Checking
>>> '137B66F60D79074DC0AD75776DA66457'
>>> 2009/06/22 09:49:19| storeEntryValidLength: object_len = 86083
>>> 2009/06/22 09:49:19| storeEntryValidLength: hdr_sz = 429
>>> 2009/06/22 09:49:19| storeEntryValidLength: content_length = 85654
>>> 2009/06/22 09:49:19| storeSwapOut:
>>> http://xxxxxxxxxxxxxx/downloadfiles/3705584/SD-Mat07.pdf
>>> 2009/06/22 09:49:19| storeSwapOut: store_status = STORE_OK
>>> 2009/06/22 09:49:19| storeSwapOut: mem->inmem_lo = 81920
>>> 2009/06/22 09:49:19| storeSwapOut: mem->inmem_hi = 86083
>>> 2009/06/22 09:49:19| storeSwapOut: swapout.queue_offset = 0
>>> 2009/06/22 09:49:19| storeSwapOutMaintainMemObject: lowest_offset =
>>> 84945
>>> 2009/06/22 09:49:19| InvokeHandlers: 137B66F60D79074DC0AD75776DA66457
>>> 2009/06/22 09:49:19| InvokeHandlers: checking client #0
>>> 2009/06/22 09:49:19| comm_close: FD 35
>>> 2009/06/22 09:49:19| commCallCloseHandlers: FD 35
>>> 2009/06/22 09:49:19| commCallCloseHandlers: ch->handler=0x4489a0
>>> 2009/06/22 09:49:19| cbdataValid: 0xb8f768
>>> 2009/06/22 09:49:19| storeUnlockObject: (http.c:75): key
>>> '137B66F60D79074DC0AD75776DA66457' count=3
>>> 2009/06/22 09:49:19| cbdataFree: 0xb8f768
>>> 2009/06/22 09:49:19| cbdataFree: 0xb8f768 has 1 locks, not freeing
>>> 2009/06/22 09:49:19| cbdataUnlock: 0xb8f768
>>> 2009/06/22 09:49:19| cbdataUnlock: Freeing 0xb8f768
>>> 2009/06/22 09:49:19| commCallCloseHandlers: ch->handler=0x438180
>>> 2009/06/22 09:49:19| cbdataValid: 0xb9a548
>>>
>>> But i check in var/log/store.log, status of object is RELEASE not
>>> SWAPOUT
>>> 1245638959.814 RELEASE -1 FFFFFFFF 137B66F60D79074DC0AD75776DA66457
>>> 200 sh1245639162 1236043282 375007920 application/pdf 85654/85654 POST
>>> http://xxxxxxxxxxxxxxxxxxx/3705584/SD-Mat07.pdf
>>>
>>> What should i do in order this object could be saved on disk ?
>>> Which part in source code should I modify ?
>>
>> The web server is sending headers indicating the response MUST NOT be
>> cached. It is going to some length to ensure the object is not cached.
>>
>> Date: Mon, 22 Jun 2009 02:52:42 GMT
>> Expires: Thu, 19 Nov 1981 08:52:00 GMT
>> Cache-Control: no-store, no-cache, must-revalidate
>> Pragma: no-cache
>>
>> Any one of which will cause RELEASE.
>>
>> Amos
>>
>>
Received on Mon Jun 22 2009 - 05:11:37 MDT