Invalidate catalog cache contents by events

Usually when you setup a new cache strategy, you initially choose to apply a time based invalidation policy for your entries.

But starting from FEP5 if you are using search navigation and SOLR features, there is an interesting alternative.

Defining some specific dependency-ids in your cachespec.xml, now it’s possible to automatically invalidate catalog cache entries if the changes are generated from a dataload execution or from the Management Center.

Let me explain the full process.

You start generating a basic cachespec.xml to cache for example home, category and product pages but containing the dependencyIds you need to invalidate catalog contents.

When catalog contents will be changed either from Management Center or from dataload, the tables TI_DELTA_CATENTRY and TI_DELTA_CATGROUP (assuming you’re using a search navigation) will be updated with those changes to allow a delta reindex.

For example changing a catalog entry description (or a price value) in the Management Center (a boy pants in this case)

AuroraCatalogBoyPantsLongDescriptionChanged

an update record (“U” ) in TI_DELTA_CATENTRY table will be created.

AuroraCatalogBoyPantsLongDescriptionChanged_TI_DELTA_CATENTRYContents

And the same will happens when a dataload process will be executed using a specific FEP5 business object mediator

<_config:BusinessObjectMediator className=”com.ibm.commerce.catalog.dataload.mediator.CatalogEntrySearchIndexMediator” componentId=”com.ibm.commerce.catalog” >

Francesco described this very well in his blog

If you modified wc-component.xml

<_config:property name=”CacheInvalidationForCatalogEntry”
value=”ProductDisplay:productId:langId:$catEntryId$:$langId$” />
<_config:property name=”CacheInvalidationForCatalogGroup”
value=”CategoryDisplay:categoryId:langId:$catGroupId$:$langId$” />

for example for products and categories as the WC infocenter suggests and those patterns will match the dependencyIds you placed in cachespec.xml

<dependency-id>ProductDisplay:productId:langId
<component id=”” ignore-value=”true” type=”pathinfo”>
<required>true</required>
<value>/ProductDisplay</value>
</component>
<component id=”productId” type=”parameter”>
<required>true</required>
</component>
<component id=”DC_lang” type=”attribute”>
<required>true</required>
</component>
</dependency-id>

<dependency-id>CategoryDisplay:categoryId:langId
<component id=”” ignore-value=”true” type=”pathinfo”>
<required>true</required>
<value>/CategoryDisplay</value>
</component>
<component id=”categoryId” type=”parameter”>
<required>true</required>
</component>
<component id=”DC_lang” type=”attribute”>
<required>true</required>
</component>
</dependency-id>

the next UpdateSearchindex job execution will do the “magic”.

After changing the wc-component.xml, the UpdateSearchIndex job execution will create the correspondent invalidation entries in CACHEIVL table

AuroraCatalogBoyPantsLongDescriptionChanged_CACHEIVL_records

These records in the CACHEIVL will invalidate all the cache entries associated to the corresponding dependencyId at the next DynacacheInvalidation job execution and will do the trick.

Then this process is working good in a WCD environment: but what about a runtime and production environment.

In a real project catalog changes and delta index are managed in the staging environment. And these also means that CACHEIVL entries will be generated in staging not in production.

How to apply invalidations in the production environment then?

FEP6 solved this with new parameters in the indexprop utility: dbtype, destdb, destdb_user, destdb_passwd.

If you setup a search infrastructure following the best practices and then using a repeater and indexprop to propagate index changes from staging to production, the indexprop execution will write the CACHEIVL records to invalidate cache entries in the production environment for you.

If you didn’t apply FEP6 yet, there is a workaround based on stagingprop usage.

First of all you create a trigger for the CACHEIVL table to create STAGLOG entries when the invalidation records are created.

The records in STAGLOG will be created using a filter (1) for stagingprop. This will grant to execute a “filtered” stagingprop only for cache invalidation to production.

A stagingprop execution like (after a couple of INSERTs in the STGMERTAB both on production and staging environment)

stagingprop.sh -scope cacheInvalidation -configfile <pathToFile>/cacheinvalidation.xml -filter 1 -sourcedb ${STG_DB} -destdb ${PRD_DB} -sourcedb_user ${STG_DBUSER} -destdb_user ${PRD_DBUSER} -sourcedb_passwd ${STG_DBPWD} -destdb_passwd ${PRD_DBPWD}

will then propagate the changes to the correspondent CACHEIVL production table.

And again a DynacacheInvalidation job execution in the production environment will invalidate the corresponding cache entries.

Many thanks to Stefano for brainstorming with me the whole process and to Aleksander to help me figure out the UpdateSearchIndex job role in the process

Following a comment I received from Aleksander I have to specify an important thing about DynacacheInvalidation job execution and timestamps. As he pointed, there is a risk in this process: propagate CACHEIVL contents in the production system just AFTER last DynacacheInvalidation job execution with an older timestamp in the INSERTTIME table.

In that case invalidation entries will not be processed in the next execution.

But Toronto labs took a precaution to try avoid that situation: when UpdateSearchIndex job is executed and CACHEIVL entries are generated, the timestamp in INSERTTIME column is 10 minutes (the default interval between DynacacheInvalidation executions) later than UpdateSearchIndex. You can change this delay changing the property

<_config:property name=”CacheInvalidationDelay” value=”600000″ />

in wc-component.xml

This should allow to give you the time to propagate the changes in the production environment (using a very filtered stagingprop) in time for the next DynacacheInvalidation job scheduled execution.

Advertisements

Websphere Commerce and Portal Architect ✔ Motivated IT professional with more than ten years of experience, combining Java and JEE developer skills with systems and IBM products installation knowledge. ✔ Strong experience and skills on planning, architecting and implementing complex commerce and portal solutions based on IBM middleware products. ✔ Reliable with a strong network attitude, experience on leading developer's teams and manage international relationships. Specialties ✔ Pleasant manner, reliable. ✔ Ability to consider issues from different point of views. ✔ End-oriented work capacity and problem-solving attitude. ✔ Ability to work with deadlines and under pressure. ✔ Ability to prioritise tasks and manage people. ✔ Ability to increase the whole team skills. ✔ Ability to generate commercial leads

Tagged with: , , , ,
Posted in cache, search, websphere commerce
6 comments on “Invalidate catalog cache contents by events
  1. Aleksander says:

    The following redbook explains quite a lot about DynaCache:

    Mastering DynaCache in WebSphere Commerce

    http://www.redbooks.ibm.com/redbooks/pdfs/sg247393.pdf

  2. FSchettini.com says:

    Marco, first of all I would thank you cause your post represents the most complete guide about the process made- under the cover- by SOLR and DynaCache Invalidation.

    In addition, I found an interesting case when some further action could be needed to finalize properly the invalidation process. In fact, if you delete a product (CATENTRY and CATGPENREL are updated and a ‘D’ action is added to TI_DELTA_CATENTRY) how could the invalidation process knows the category of the product deleted?

    The product-category relationship (CATGPENREL) is gone, so I do not see any way for WCS to get this info. Do you?

    So,I was thinking that a possible workaround to overcome this inconvenient could be update the Data Load configuration in order to populate TI_DELTA_CATGROUP with ‘U’ action related to the product’s category the DL is going to delete. In this way, the invalidation will remove the cache entry related the removed product and the associated category.

    Do you see any other possible way- maybe more “elegant’- to achieve this missing invalidation task?

  3. FSchettini.com says:

    maybe with Dynacache dependencyIds?

  4. dstrasse says:

    Marco, thanks for the very helpful post. I tried this solution and it works well for product (i.e. the CACHEIVL table contains entries for products) but I am missing entries for catalog groups, i.e. when I change the name/description of a catalog group. Is there another customization I have to do?

  5. Giovanni says:

    Hi Marco, it is a very useful blog for WCS knowledge, thanks.
    Maybe out of scope: inserting a new allowed value on Attribute Dictionary via Management Center Catalog Upload ((with AttributeDictionaryAttributeAllowedValues.csv) the TI_DELTA_CATENTRY is automatically populated with all CATENTRY related to the ATTR.
    I was expecting that no records were inserted on TI_DELTA_CATENTRY because it is a new value associated with no CATENTRY.
    Is it possible to avoid this behavior?
    Thanks in advance.
    Giovanni

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: