Announcement Announcement Module
Collapse
No announcement yet.
Pattern for converting from denormalized flat file to normalized structure Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • Pattern for converting from denormalized flat file to normalized structure

    Hi,

    I am looking for best practices in processing a flat file that holds a denormalized one-to-many structure. For example I'd like to get from the following flat file lines to the JSON document structure below.

    FROM:

    ID, Name, Color
    A, John, Blue
    A, John, Red
    A, John, Green
    A, John, Orange

    TO:

    {
    ID : "A",
    Name : "John",
    Color : ["Blue", "Red", "Green", "Orange"]
    }

    What I'd like to avoid is having to do intermediate lookups to fetch the object and then update with the additional data, as shown in the following sequence:
    1) Check if A exists. If it does not exist, create a new document A with the common fields (ID and Name) and the first color (blue).
    2) Check if A exists. If it does exist, add a new color to the array.

    Step 2 is the overhead I'd like to optimize so that we're not performing a lookup and then update for every single line in the file.

    Any advice is greatly appreciated.

    Thomas
Working...
X