Day 8 : Hot Folder Configuration

When we have to automatically process large volumes of data from another system, hot folder is the way. It scans the input directory periodically and converts the data in csv files to impex and loads the impex into Hybris.

It makes the entire data loading process much easier.

PC SAP Help

1. Major components are defined through spring configuration. We would be having two xml files — one for defining the base directory and adapter and the other for defining the headers. These xmls need to be included in our core extension’s spring file.

<import resource="classpath:/examplecore/integration/common/hot-folder-example-config-spring.xml"/>
<import resource="classpath:/examplecore/integration/common/hot-folder-example-spring.xml"/>

2. Define all the file configurations in hot-folder-example-config-spring.xml. This include the below steps.

a. Base Directory — Configuring the input directory which needs to be scanned frequently for the incoming feed.

<!-- Config a base directory -->
<bean id="exampleBaseDirectory" class="java.lang.String">
    <constructor-arg value="#{configurationService.configuration.getProperty('hotfolder.upload.example.csv')}"/>
</bean>

b. Next is configuring inbound channel adaptor to check for file name pattern

<!-- Scan for files inside the base directory with names matches the pattern ^(.*)-(\d+)\.csv -->
<file:inbound-channel-adapter id="exampleBatchFiles" directory="#{exampleBaseDirectory}"
                              filename-regex="^(.*)-(\d+)\.csv"
                              comparator="exampleFileOrderComparator">
    <!--  Periodic trigger in milliseconds -->
    <int:poller fixed-rate="#{configurationService.configuration.getProperty('example.hotfolder.poller.fixed-rate')}"/>
</file:inbound-channel-adapter>

c. If needed, custom file order comparator can be used to set the priority of processing. Higher the priority sooner it will be processed. And the key used should be mapped to the impex converter as well.

<!-- example from the accelerator -->
<bean id="exampleFileOrderComparator" class="de.hybris.platform.acceleratorservices.dataimport.batch.FileOrderComparator">
    <property name="prefixPriority">
        <map>
            <!-- default priority is 0 -->
            <entry key="base_product" value="2" />
            <entry key="variant" value="1" />
        </map>
    </property>
</bean>

d. Outbound configuration to pick up file from base directory and to move to the processing folder

<!-- Move the file to processing and start the flow -->
<file:outbound-gateway request-channel="exampleBatchFiles" reply-channel="exampleBatchFilesProc"
                       directory="#{exampleBaseDirectory}/processing"
                       delete-source-files="true"/>

e. Create a service-activator to feed the flow with relevant information.

<!-- Initialize the batch header with relevant information -->
<int:service-activator input-channel="exampleBatchFilesProc" output-channel="batchExampleFilesHeaderInit"
                       ref="exampleHeaderSetupTask"
                       method="execute"/>
<bean id="exampleHeaderSetupTask" class="de.hybris.platform.acceleratorservices.dataimport.batch.task.HeaderSetupTask">
    <property name="catalog" value="exampleProductCatalog"/>
    <property name="net" value="false"/>
    <property name="storeBaseDirectory" ref="exampleBaseDirectory"/>
</bean>
<!-- 2) move the file to processing and setup header, partly site dependent -->
<int:service-activator input-channel="batchExampleFilesHeaderInit" output-channel="batchFilesExampleHeader"
                       ref="headerInitTask"
                       method="execute"/>


<!-- 3) transform csv to impex file -->
<int:service-activator input-channel="batchFilesExampleHeader" output-channel="batchFilesExampleTran"
                       ref="batchExampleTransformerTask"
                       method="execute"/>

f. Add impex transformer

<bean id="batchExampleTransformerTask"
      class="com.sap.example.core.hotfolder.ExampleImpexTransformerTask"
      init-method="initConvertersMap">
    <property name="fieldSeparator" value=";"/>
    <property name="encoding" value="UTF-8"/>
    <property name="linesToSkip" value="1"/>
    <property name="cleanupHelper" ref="cleanupHelper"/>
    <property name="fileSeparatorDetails">
        <map>
            <entry key="A" value="|"/>
            <entry key="B" value="|"/>            
        </map>
    </property>
</bean>

3. Define converter mapping

<bean id="batchExampleConverterMapping"
      class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"
      p:mapping="A"
      p:converter-ref="batchExampleConverter"/>
<bean id="batchExampleConverter"
      class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.impl.DefaultImpexConverter">
    <property name="header">
        <value># ImpEx for INSERT_UPDATE Customer class
            INSERT_UPDATE Example;code(code)[unique=true];description
        </value>
    </property>
    <property name="impexRow">
        <value>;{+0};{1}</value>
    </property>
    <property name="type" value="Example"/>
</bean>

I would like to write about translators and decorators too. May be that’s for tomorrow :)