Data Migration Log Files

During a data migration, dataload.py updates the following log files in real time:

dataload.log

dataload_info.log

Dataload_info.log provides an abridged listing of events as the script runs. This log is also configurable in logging_config.json:

"info_file_handler": {
           "class": "logging.handlers.RotatingFileHandler",
           "level": "INFO",
           "formatter": "simple",
           "filename": "dataload)info.log",
           "maxBytes": 524288000

success.csv

Keeps a running tally of the records that were successfully migrated to the user profile store. For example, this excerpt shows that information was migrated, and a new user profile created, for the user karim.nafir@mail.com. That user has also been assigned the UUID bacfa66d-16e2-492e-a019-da6220df2ae9:

batch,line,uuid,email
1,2,bacfa66d-16e2-492e-a019-da6220df2ae9,karim.nafir@mail.com

Unlike dataload.log, success.csv is not cumulative: Each time dataload.py is executed a new success CSV file is created and a timestamp is appended to the name. For example:

success_May_22_2019_10_55.csv

fail.csv

Keeps a running tally of the records that were not successfully migrated to the user profile store. For example, here we see that the first record in the datafile (line 2, line 1 being the data header) failed because of a duplicate value; in this case, that means that there’s already a user profile using the email address referenced in the datafile:

batch,line,email,error
1,2,Attempted to update a duplicate value

Like success.cvs, fail.csv is not cumulative: Each time dataload.py is executed a new fail CSV file is created and a timestamp is appended to the name:

fail_May_22_2019_10_55.csv

In addition to logging all its activity, dataload constantly reports its progress on the command-line, letting you know how many records have been processed and some other useful real time measurements. For example:


Total Success (S)	Total Fails (F)	Success Rate (SR)	Average Records per Minute (AVG)
S:990	             F:30	          SR:97.06%	        AVG:12090rec/m : 50%