Hot Folder in Hybris

Hot Folder is one of the best way to import and migrate data into Hybris. you can use ImpEx files to import data in the system. Hybris provide hot folders concept which support data can be automatically load into hybris by placing the data file inside of the folder. It serves as staging area which is continuously poll a folder and when files are copied or dropped into it, they are automatically pick and processed file data and load in hybris. It is based on Spring integration framework. The Spring integration provides a pluggable, highly configurable service-based design that can be extended as required.

Benefit of Hot folder

This feature is intended for automated updates, such as Master data, stock status, price, and categories, product catalog, customer feeds that you may receive from external systems, or from other areas of your business

Data import or Data Migration Automation.
Fast and multithread based data load.
Support to File based System.

Data Importing

Hybris acceleratorservices extension template comes with a batch package that enables automated importing of data from hot folders. If you are using the Accelerator in a clustered environment, same hot folder on multiple nodes will not supported. Basic concept of hot folder, we transform .csv files using ImpexConverters to impexes and import them using ImportService.

General Flow

In Hot folder concept , hybris periodically poll the configured input directory for data files. If data files are found, they are moved to the processing subdirectory and then sent to the Batch Import pipeline. Once file data is import in Hybris. then file will move to achieve folder. if find any error new file will write in error folder with error. Below directory need to create for implementation.

processing
error
archive

Batch Import pipeline, which consists of the following tasks:

HeaderSetupTask: It is creates a new BatchHeader which used as a reference for file throughout the flow.
HeaderInitTask: It is retrieves a sequence ID and (optionally) a language from the file name.
ImpexTransformerTask: it is creates Impex files from the CSV input file and writes error lines in file that is in error subdirectory.
ImpexRunnerTask: It is processes all Impex files sequentially with multiple threads.
CleanupTask: It is deletes all transformed files and moves the imported file with an optionally appended timestamp to the archive subdirectory.
ErrorHandler: it is deletes all transformed files and moves the imported file with an optionally appended timestamp to the error subdirectory.

Step to standard Hot folder implementation

1) We create hot-folder-training-spring.xml in \hybris\bin\custom\example\trainingcore\resources\trainingcore\integration and import it in trainingcore-spring.xml

and add below import configuration in spring file.

2) We add Config base directory bean in hot-folder-training-spring.xml to specify where our files will be put and scanned.

<constructor-arg value="#{baseDirectory}/${tenantId}/training" />

</bean>

3) Below configuration define inbound channel adapter to scan input files based on file name regex pattern and poller rate in milliseconds. FileOrderComparator is used for compares files based on their file names. On this behalf a priorities are configurable for file name prefixes. Files with a lower priority value are considered more important and imported first.

<file:inbound-channel-adapter id="batchFilesTraining" directory="#{baseDirectoryTraining}"

filename-regex="^(.*)-(\d+)\.csv" comparator="fileOrderComparator">

<int:poller fixed-rate="1000" />

</file:inbound-channel-adapter>

4) Below configuration define outbound-gateway for move the file to processing folder

<file:outbound-gateway request-channel="batchFilesTraining" reply-channel="batchFilesTrainingProc"

directory="#{baseDirectoryTraining}/processing" delete-source-files="true" />

5) Below configuration define Service activator a referenced bean when receiving a message on a configured channel. The bean response is again wrapped in a message and sent to the configured output channel. The header is used throughout the pipeline as a reference and for cleanup.

<int:service-activator input-channel="batchFilesTrainingProc" output-channel="batchFilesHeaderInit" ref="trainingHeaderSetupTask" method="execute" />

</bean>

6) Below configuration define transformer converters mappings. To plug in the new converter, we must define it in Spring and define its mapping.

<bean id="batchTrainingProductConverterMapping" class="de.hybris.platform.acceleratorservices.dataimport.batch.converter.mapping.impl.DefaultConverterMapping"

p:mapping="base_product"

p:converter-ref="batchTrainingProductConverter"/>

7) Below configuration add add specific Converter. DefaultImpexConverter will converting a CSV file into impex.

<value>#{defaultImpexProductHeader}

# Insert Training Products

INSERT_UPDATE Product;code[unique=true];name[lang=$lang];description[lang=$lang];manufacturerName;unit(code)[default=pieces];$approved;sequenceId[translator=de.hybris.platform.acceleratorservices.dataimport.batch.converter.SequenceIdTranslator];$catalogVersion

</value>

</property>

</property>

</bean>

</property>

</bean>

ImpexConverter has two properties:

header : the header of the generated impex;
impexRow : the mapping between the column of the csv file and the impex file
1. (+) : means that column 0 will be mapped to code[unique=true] and etc.
2. (S) : can be used for writing the current sequence ID at the template position.

SequenceIdTranslator :The sequence ID is then imported by means of a specialized ImpEx translator, SequenceIdTranslator. The provided value is compared with the current value, and the row is rejected if the value is outdated

rowFilter : rowfilter is optional. it is filter to filter out rows not matching a specified filter expression. It is used DefaultImpexRowFilter.it is uses Groovy to evaluate the given row supplied as parameter "row". The expression is treated as a boolean.