I happened to notice that one of the datasources which executes in delta mode from ECC to BI was transferring a very high number of packages. What was strange was that, although the number of packages were high, the number of records that each package contained was significantly less. This aroused my curiosity!

I did know that BW internally allocates the package size, but I did not know the criteria on which this was being done.

On reading a few posts at SDN, I figured out the logic behind this golden number called Package size.

[Credits to original posters at SDN]

Package size = MAXSIZE * 1000 / size of the transfer structure,but not more than MAXLINES.

I tried simulating and it worked out like a gem!

Read on to see the steps involved in this simulation. . .

For my testing I used the datasource 0FI_GL_14, which is the new GL Line Item datasource.

On executing an infopackage against this DataSource, I constantly received a package size of 7924!



In my source system – ECC 6.0 , SBIW T-code, I have configured the MAXSIZE in kb to be at 20,000. This is done under SBIW -> General Settings -> Maintain Control Parameters for Data Transfer

To identify the Transfer structure for a particular DS, open up the DataSource in your source system. You could use RSA6 to do this. As my test case was for the DS - 0FI_GL_14, the transfer structure is FAGLPOSBW.

To get the length of the transfer structure, use T-code SE11

Once you are in the “Display Structure” view, select Utilities->Runtime Object->Display to identify the length.

For my test case, this is 2524

When the pieces are assembled. . .

MAXSIZE=20000

Size of Transfer Structure = 2524

Package Size = 20000*1000/2524 = 7924

Hence the theory holds good!

Changing the MAXSIZE to accommodate a better package size is not advisable for the shear reason that the MAXSIZE would in turn multiple systems and datasources. Hence this needs to be done cautiously.

If you have just one BI system, you could change the value for MAXSIZE in SBIW, at the same time limit the MAXSIZE within the InfoPackages were you do need a large package size. This can be done in an IP under Scheduler->Data Source Default Data Transfer.

When you have a configuration as below

1. Datastage job to post into BW (3.x datasource)
2. Infopackage triggered is of type – Update PSA and data target in parallel, typically a write optimized DSO
3. You use an external scheduling tool to trigger the datastage job
4. Have another BW chain to extract data from a DSO and post into a cube which is also triggered by an external scheduler

There is a possibility that your DTP load from the DSO to the cube could post empty data.

The reason why this happens is that, BW takes a lot of time to actually update the data in the WO DSO. BW responds back to datastage as soon as it recieves all the data and does not really wait for the data to get saved in the DSO.

If your BI system is slow for any reason, the external scheduler would trigger your DTP for the cube much before the data is actually saved in the DSO. This eventually results in zero records being transferred to the cube.

A solution to this problem is to only post to the PSA by datastage and have a process chain to push data to the DSO and DSO to the Cube.

Explanation

Multiple calls are made by datastage to the function module Z_RFC_DS_SERVICE with different parameters. Accordingly different activities such as create variant, delete variant, load program etc. is done.

Z_RFC_DS_SERVICE is the function module invoked remotely by datastage through the RFC.

Z_RFC_DS_SERVICE invokes the function module RS_VARIANT_DELETE

This function in turn invokes the form check_v_used_in_job.

perform check_v_used_in_job using rsvar-report rsvar-variant.

This function invokes the function module BP_VARIANT_USED_IN_JOB

Depending on the response from this function module the “unable to rename/delete variant” error is thrown. If the number of jobs in TBTCP is not equal to the number of entries in TBTCO which has finished or aborted status, the error will be thrown.

Consider the below screen-shot. These programs have been scheduled but not executed yet.

Let us consider BI_PROCESS_TRIGGER as an example. This program executes with the report name RSPROCESS and Variant name &0000000000076.

Below is the entry in TBTCP table

Below is the entry in TBTCO table

Below is the part of the program BP_VARIANT_USED_IN_JOB that calculates the value for l_no_jobs and l_finish_cancel.

The status field for these scheduled programs are blank in TBTCO. Hence the l_no_jobs=1 and l_finish_cancel=0 resulting in the error being thrown.

Simulation of variant deletion through SE38

Simulation of variant deletion through the program by which datastage is deleting.

Simulation of values thrown by the function module BP_VARIANT_USED_IN_JOB.

Deletion of job schedule in SM37 and validation of table entries

Entry has been deleted from TBTCP

Also from TBTCO

In a typical scenario following are the sequence of events when Datastage tries to create a variant in ECC and hence execute and ABAP program.

1. Check RFC connection
2. Create variant through Z_RFC_DS_SERVICE
3. Trigger the job through the BAPI_XBP_JOB_START_IMMEDIATELY
4. Check the status of the job in constant intervals
5. Delete all job related information through BAPI_XBP_JOB_DELETE
6. Finally delete the variant

Solution

In Datastage’s abort scenario or starting point, add a clean-up job. This job can do the following

• Execute the FM BAPI_XBP_JOB_DELETE with the respective ABAP program name and variant name
• Execute the FM Z_RFC_DS_SERVICE with parameter 10 to delete the variant

Scoring is a well known concept under data mining. For a newbie, Data Mining is the usage of data from a data warehouse for marketing. Data Mining involves complicate logics applied to get the maximum out of the data available with you on sales, guests, inventory etc.

Coming back to our topic, scoring is one of the methodologies under data-mining. Scoring can be applied to several dimensions such as guests, items etc.

Example :

Consider that you are planning to send out an offer on discount of Bakery items. Consider you being a retail giant, will have around 100 million customers. Sending offers to all of them would prove an expensive investment. Hence, you would rather prefer to send it to customers who have a history of purchasing bakery items. That is where scoring comes in. The probability that a customer will buy a bakery item again will be granted as a score to that guest. Say a score of 0 will be the least probability that he will purchase a bakery item and 1 the maximum. This directly depends on the number of times he/she has already purchased bakery items.

What is the Scoring Process?

The above picture summarizes the scoring process. I shall explain in detail each of the sections.

Segment

Segment is the category of data to which the scoring process is to be applied. An example would explain it well.

Example:

Say I decide to run the offer of discount in Bakery items to Women of the age category 30-40. This becomes the segment that forms as a candidate for the scoring process.

Model

Model is the logic or rule being applied to the data to decide what score it gets. Models are usually written in an XML type of language called PMML(Predictive Model Markup Language). Designing a model itself is a critical and complex process. Usually a dedicated team is available for this.

Example:

The model decides how to interpret the data. An example of a model would be

(Total amount spent on bakery Items by the guest)/(Total amount spent by the guest on purchase)

Basically it is just the calculation that needs to be applied on the data to generate the score.

Scoring Engine

The scoring engine is responsible for applying the model on the input segment of data. It could be a Procedure, Java based application etc.

Score

The output of the whole scoring process is the score. The score need not be 0-1 always. It could be any kind of desirable range value.

Campaign

Just having the scores does not complete the work. Now comes the work of campaigning. Once the scores have been assigned, a selection is made on the scored data to identify which need to be chosen or eliminated.

Example:

Once scoring is done for all women in the segment 30-40, we have a range of scores(0-1). The guests can be ordered by scores and the first 30% can be chosen for sending out the offers.

Summarizing everything

A segment of data is taken from a large Data Warehouse to perform the scoring. This selection is decided usually by the Business teams of the retail firm. Usually a view or table is created for each segment as there would be several segments and several models. A model is to be applied to these segments. The model is again done usually by a dedicated modeling team. Model is the statistical analysis that needs to be done for the segment to assign the score. Once the score is generated by the scoring engine the scores are saved in a ordered format. Again as there are multiple models and segments, usually dedicated tables are created. The campaigning team runs a selection criteria on the data based on the scores and decides the audience for the discounted bakery offer.

I found it quite difficult to set it up though, but after a few hick-ups I was able to make it. Even though I ran the command line setup, the engine never started when I gave the db2start statement. After a lot of googling I found a solution ironically on a Russian website.

De-package

sudo dpkg -i db2exc_9.5.0-1gutsy1_i386.deb

Set the annoying environment variable

sudo echo “kernel.shmmax=1610612736″ >>  /etc/sysctl.conf

Run the Set-up

/home/Prabhu/exp $ . ./db2_install

Start the Engine

db2start

Create the SAMPLE database

db2sampl

Refresh

db2stop

db2start

Connect and Enjoy

db2 connect to sample

• Edit the file /etc/inetd.conf file.
• Go to the line where ftp configuration is done. (Run a search in vi)
• add the following “-t <time-out limit>”
• for further details please go to the man pages of the ftp daemon. “man ftpd”.

UPDATE EMP

SET EMP_LOC_C = ’03′

WHERE EMP_LOC = ‘NY’

Now if this table has more than a billion records, the query might not succeed.

The ideal solution is to run an update partition by partition. You can get the partitions for your table by querying dba_tab_partitions.

SELECT TABLE_NAME,PARTITION_NAME

FROM DBA_TAB_PARTITIONS

WHERE TABLE_NAME LIKE ‘EMP’

P_001

P_002

. . . . .

P_030

Now what you could do is to let the query run against each partitions separately.

UPDATE EMP PARTITION(P_001)

SET EMP_LOC_C = ’03′

WHERE EMP_LOC = ‘NY’

Moreover if there is an index built on any of the columns you can use that as well and split the query even further.

UPDATE EMP PARTITION(P_001)

SET EMP_LOC_C = ’03′

WHERE EMP_LOC = ‘NY’

AND EMP_I > 8612751936

So this way you can split and update the table w.r.t. partitions.

This is an annoying situation you would come across in a production environment. Occasionally the volume of data would surge and the lookup stage would not be able to process it.

Sparse lookup stage basically fetches all the records from the driver link and the reference link and then loads it onto the RAM prior to performing the join. Definitely if the volume is above a certain extent the RAM would become a bottle neck. There are two ways by which this can be averted.

1) Modify the lookup type to “Sparse” from “Normal”. This can be done provided the input link count is considerable less. It should be below 1000 ideally.

2) Another option would be replace the lookup with a join. The join stage does not load the records into the RAM and performs quite well. The obvious question would be why not always use a join stage. Lookup stage is expected to work much faster than a join stage.

That’s it for now. . Happy Warehousing!