Announcement Announcement Module
Collapse
No announcement yet.
The difficult part of SDN Neo4j Page Title Module
Move Remove Collapse
X
Conversation Detail Module
Collapse
  • Filter
  • Time
  • Show
Clear All
new posts

  • The difficult part of SDN Neo4j

    OK, I have spent a lot of time on Neo4J and SDN. My app isn't a difficult app and should have been finished weeks ago. I needed Pagination, but that can't be implemented with the Repos with Pageable (from my previous post). I needed to be able to write queries that were easy to understand and get exactly the data that I need, but the docs don't beyond the very simple types of queries. If there was lazy loading, then I might get around the cypher language to figure out how to write the query.

    But when you want certain data for a use case you typically retrieve exactly the amount of data you need for the entire use case so that when you get to a UI all the data is there ready for it to display it.

    However, I can't seem to get just that. I could run many simple queries within my use case and loop through objects and calculate whatever else I need. But that is piece-mealing it. I should be able to go to the db once or twice in a use case and get the data.

    It is really difficult to figure out what was loaded and what wasn't in your query as your code traverses through nodes and relationships. I thought I had it licked in my one use case that I posted in another thread, but my Event had its associated lists populated, but those vertices/nodes didn't have their associations populated even thought they are mapped with @Fetch. There was an object, but all their properties were set to null, even though there is actual data. I got the Event object from the repos findOne() method.

    I of course can never give up, that isn't in my nature.
    Thanks for all your help.

    Mark
    Last edited by bytor99999; Mar 7th, 2012, 12:17 AM.

  • #2
    Mark,

    thanks a lot for your feedback and all the contributions so far.

    Let me answer some of your questions.

    * Paging is not so easy as you would have to either write two queries or have to execute the query a second time for just counting the objects (both options might be implemented for 2.1)
    * I think that cypher is the best way to specify what you want to have loaded for a use-case, @Fetch handling is still very basic, you're right, what kinds of concrete requirements would you have there (e.g. specify fetch-depth on load or in the annotation etc.?)
    * What kind of have you written in the end, can we do something to help you get it done as you've imagined it?

    Thanks a lot again

    Michael

    Comment


    • #3
      First, I want to apologize. In no means do I want to belittle the amazing job you have done Michael. Spring Data Neo4j and the work with Neo4j you have accomplished goes way beyond what most developers could achieve.

      I had to rethink and refactor. I got template.fetch to work and I had to do a couple/few repository calls to get all the data that I needed because the Cypher Query was going to be more complicated than my knowledge of Cypher would allow me to write.

      It wasn't so much a fetch depth as Traversing many different relationships from the root node. And then have to do some comparisons.

      Here is what I ended up doing. I had 4 of these I needed. Each different, but this is a good example of it. The EventListItem is a UI object for my page, instead of returning the actual mapped domain objects to the UI. It actually made my UI template code a lot easier. And once I did one of these the other 3 were the same pattern, so easier to write. In the end I get exactly what I needed

      Code:
           @Override
          public ListOfDomains<EventListItem> getListOfEventsForUser() {
              User currentUser = userRepository.getUserFromSession();
              User populatedUser = userRepository.findOne(currentUser.getNodeId());
              template.fetch(populatedUser.getEventsIAmHosting());
              template.fetch(populatedUser.getEventsIAmInvitedTo());
              template.fetch(populatedUser.getEventsIAmAttending());
              ListOfDomains<EventListItem> listOfDomains = new ListOfDomains<EventListItem>();
              calculateListOfEvents(populatedUser, listOfDomains);
              return listOfDomains;
          }
          
          private void calculateListOfEvents(User populatedUser, ListOfDomains<EventListItem> listOfDomains) {
              List<EventListItem> itemsHolder = new ArrayList<EventListItem>();
              for (Event event: populatedUser.getEventsIAmHosting()) {           
                  itemsHolder.add(setEventInformationInListItem(event, User.HOSTING));
              }
              for (Event event: populatedUser.getEventsIAmInvitedTo()) {
                  itemsHolder.add(setEventInformationInListItem(event, User.INVITED));
              }
              for (Event event: populatedUser.getEventsIAmAttending()) {
                  itemsHolder.add(setEventInformationInListItem(event, User.ATTENDING));
              }
              listOfDomains.setDomainObjects(itemsHolder);
              listOfDomains.setType("Events you are involved with.");
          }
          
          private EventListItem setEventInformationInListItem(Event event, String type) {
              EventListItem listItem = new EventListItem();
              listItem.setEventDate(event.getEventDate());
              listItem.setEventId(event.getNodeId());
              listItem.setEventName(event.getTitle());
              listItem.setEventUserType(type);
              return listItem;
          }
      For those fetch calls, I couldn't use @Fetch because it could create circular fetches, and I saw what happens when that happens.

      Thanks

      Mark

      Comment


      • #4
        Mark,

        could you explain this:
        It wasn't so much a fetch depth as Traversing many different relationships from the root node
        Do you mean the graph-root or the entity-root node?

        Otherwise the code doesn't look too involved, one could extend template.fetch to take a varargs parameter which would shorten this part. And I would probably extract one more method and make EventList item immutable and pass all parameters in the constructor.

        For cypher, I think the query would look like:

        Code:
        start user=node:users(login={login})
        match user-[r]->event
        where type(r) = "ATTENDING" or type(r) = "INVITED" or type(r) = "HOSTING"
        return ID(event), event.eventDate, event.title, type(r)
        Then you could use the conversion-facilities that allows you to convert each of the returned rows into a EventListItem (much like the spring RowMappers).

        HTH

        Michael

        Comment


        • #5
          Thanks, That is the part of Cypher that I haven't learned yet. Mostly because I haven't found any docs or tutorials explaining or showing those types of more complex queries. Not that that should be in the SDN docs, but in the Cypher docs at Neo4J or somewhere else.

          With that query though, what objects get returned? Can I hook up the ConversionService to the Result set that gets returned by the query like a RowMapper, (answered already by you above)?

          So the part about traversal, I posted the easiest of the four use cases. This one will show how I have to go through many traversals on different relationships all through the same query.

          Code:
              @Override
              public ListOfDomains<ItemListItem> getListOfItemsForEvent(Long eventId) {
                  Event event = eventRepository.findOne(eventId);
                  User user = userRepository.getUserFromSession();
                  ListOfDomains<ItemListItem> listOfDomains = new ListOfDomains<ItemListItem>();
                  calculateListOfItems(event, user, listOfDomains);
                  return listOfDomains;
              }
              
              private ListOfDomains<ItemListItem> calculateListOfItems(Event event,
                                                                       User user,
                                                                       ListOfDomains<ItemListItem> listOfDomains) {
                  boolean hostOfEvent = isHostOfEvent(event, user);
                  List<ItemListItem> itemsHolder = new ArrayList<ItemListItem>();
                  for (ItemNeededForEvent neededItem : event.getItemsNeededForEvent()) {
                      ItemListItem listItem = new ItemListItem();
                      listItem.setEventId(event.getNodeId());
                      listItem.setItemNeededForEventId(neededItem.getNodeId());
                      listItem.setQuantity(neededItem.getQuantity());
                      Item item = itemRepository.findOne(neededItem.getItem().getNodeId());
                      listItem.setItemName(item.getDescription());
                      listItem.setEditableByHost(hostOfEvent);
                      setBooleanValuesInItemListItem(event, user, item, listItem);
                      itemsHolder.add(listItem);
                  }
                  listOfDomains.setDomainObjects(itemsHolder);
                  listOfDomains.setType("Items Needed for Event");
                  return listOfDomains;
              }
              
              private void setBooleanValuesInItemListItem(Event event, 
                                                          User user,
                                                          Item item,
                                                          ItemListItem listItem) {
                  // These are all null associations, need to get them another way
                  template.fetch(event.getItemsSignedUpToEvent());
                  
                  String usersSignedUp = "";
                  for (ItemUserSignedUpToBringToEvent signedUpToBringToEvent : event.getItemsSignedUpToEvent()) {
                      if (signedUpToBringToEvent.getUser().equals(user)) {
                          listItem.setCurrentUserSignedUp(true);
                          listItem.setItemSignedUpId(signedUpToBringToEvent.getNodeId());
                      }
                      if (signedUpToBringToEvent.getItem().equals(item)) {
                          usersSignedUp += signedUpToBringToEvent.getUserName() + ", ";
                      }
                  }
                  if (usersSignedUp.length() > 2) {
                      usersSignedUp = usersSignedUp.substring(0, usersSignedUp.length()-2);
                  }
                  listItem.setUsersSignedUpForItem(usersSignedUp);
              }
          
              private boolean isHostOfEvent(Event event, User user) {
                  return event.getEventHosts().contains(user);
              }
          The ListItem objects, about making a constructor and immutable. It really doesn't matter if it is immutable or not in my case, and I tend to avoid making constructors, I usually keep the no args constructor. Because I would end up with a constructor with many parameters that I never remember the order of them. I did try to make them Groovy objects, but I couldn't get GMaven to work in my build, but that is nothing of any concern to me anymore.

          Thanks Michael

          Mark
          Last edited by bytor99999; Mar 6th, 2012, 11:14 AM. Reason: editing some more and more

          Comment


          • #6
            Converters would only work if for each use case there is one and only one cypher query that can be performed. I think in the two examples I posted it might be possible for one query.

            Where as with this one with the boolean values, makes it difficult to write a single cypher query for it.
            This one is more of what i mean by complex traversal and comparing objects from one node to two different relationship traversals of different types of nodes and returning a Boolean based on that comparison.

            Code:
                @Override
                public ListOfDomains<ItemListItem> getListOfItemsForEvent(Long eventId) {
                    Event event = eventRepository.findOne(eventId);
                    User user = userRepository.getUserFromSession();
                    ListOfDomains<ItemListItem> listOfDomains = new ListOfDomains<ItemListItem>();
                    calculateListOfItems(event, user, listOfDomains);
                    return listOfDomains;
                }
                
                private ListOfDomains<ItemListItem> calculateListOfItems(Event event,
                                                                         User user,
                                                                         ListOfDomains<ItemListItem> listOfDomains) {
                    boolean hostOfEvent = isHostOfEvent(event, user);
                    List<ItemListItem> itemsHolder = new ArrayList<ItemListItem>();
                    for (ItemNeededForEvent neededItem : event.getItemsNeededForEvent()) {
                        ItemListItem listItem = new ItemListItem();
                        listItem.setEventId(event.getNodeId());
                        listItem.setItemNeededForEventId(neededItem.getNodeId());
                        listItem.setQuantity(neededItem.getQuantity());
                        template.fetch(neededItem.getItem());
                        listItem.setItemName(neededItem.getItemName());
                        listItem.setEditableByHost(hostOfEvent);
                        setBooleanValuesInItemListItem(event, user, neededItem, listItem);
                        itemsHolder.add(listItem);
                    }
                    listOfDomains.setDomainObjects(itemsHolder);
                    listOfDomains.setType("Items Needed for Event");
                    return listOfDomains;
                }
                
                private boolean isHostOfEvent(Event event, User user) {
                    return event.getEventHosts().contains(user);
                }
                
                private void setBooleanValuesInItemListItem(Event event, 
                                                            User user,
                                                            ItemNeededForEvent itemNeededForEvent,
                                                            ItemListItem listItem) {
                    // These are all null associations, need to get them another way
                    template.fetch(event.getItemsSignedUpToEvent());
                    
                    String usersSignedUp = "";
                    for (ItemUserSignedUpToBringToEvent signedUpToBringToEvent : event.getItemsSignedUpToEvent()) {
                        if (signedUpToBringToEvent.getUser().equals(user)) {
                            listItem.setCurrentUserSignedUp(true);
                            listItem.setItemSignedUpId(signedUpToBringToEvent.getNodeId());
                        }
                        if (signedUpToBringToEvent.getItem().equals(itemNeededForEvent.getItem())) {
                            usersSignedUp += signedUpToBringToEvent.getUserName() + ", ";
                        }
                    }
                    if (usersSignedUp.length() > 2) {
                        usersSignedUp = usersSignedUp.substring(0, usersSignedUp.length()-2);
                    }
                    listItem.setUsersSignedUpForItem(usersSignedUp);
                }
            Thanks

            Mark
            Last edited by bytor99999; Mar 6th, 2012, 11:14 AM. Reason: removing stupid stuff

            Comment


            • #7
              Mark,

              this is the typical code of converting a more involved and fine granular object network to simpler UI-view-object structures. It could be cleaned up a bit e.g. by moving calculateListOfItems to ListOfDomains, and by ItemListItem being able to hydrate itself from a ItemNeededForEvent.

              a cypher query for that could look like (so it is not much less involved than the code you have). It is probably just the complexity that lies in this conversion?).

              Code:
              start user=node({user}), event=node({eventId})
              match user-[host?:HOSTS]->event,
              event-[:ITEMS_NEEDED]->neededItem-[:ITEM]->item,
              event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user
              return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, item.name as itemName,  
              userSignedUp!=null as currentUserSignedUp, ID(signedUpToBringToEvent) as itemSignedUpId
              The convert I meant are the
              Code:
              List<ItemListItem> = template.query().to(ItemListItem.class, new ResultConverter<Map<String,Object>, ItemListItem >() {
               ItemListItem convert(Map<String,Object> row) {
                  create ItemListItem and set the fields from the row
               }
              }).as(List.class);
              HTH

              Michael

              Comment


              • #8
                OK, I was trying a different approach. When you said converter I thought you meant @MapResult and @ResultColumn. So I did the following method in my repository interface

                Code:
                    @Query("start user=node({0}) " +
                            "match user-[r]->event " +
                            "where type(r) = " + User.ATTENDING + " or type(r) = " + User.INVITED + " or type(r) = " + User.HOSTING + " " +
                            "return ID(event), event.eventDate, event.title, type(r) " +
                            "order by event.eventDate desc")
                    public Iterable<EventItemForAccountPage> getEventsForAccountPage(User user);
                and this is the domain object mapped to that query.

                Code:
                package com.perfectworldprogramming.eventgate.event;
                
                import com.perfectworldprogramming.eventgate.wrappers.JsonShortDateSerializer;
                import org.codehaus.jackson.map.annotate.JsonSerialize;
                import org.springframework.data.neo4j.annotation.MapResult;
                import org.springframework.data.neo4j.annotation.ResultColumn;
                
                import java.util.Date;
                
                @MapResult
                public interface EventItemForAccountPage {
                
                    @ResultColumn("event.title")
                    public String getEventName();
                
                    @ResultColumn("type(r)")
                    public String getEventUserType();
                
                    @ResultColumn("ID(event)")
                    public Long getEventId();
                
                    @ResultColumn("event.eventDate")
                    @JsonSerialize(using= JsonShortDateSerializer.class)
                    public Date getEventDate();
                }
                However, the query doesn't work, I got an error

                java.lang.IllegalArgumentException: object is not an instance of declaring class

                That second query looks cool. Definitely something you don't find in the docs. But I can see people having a use case for something relatively similar, maybe not as complex, but still with those possible options to have in a cypher query.

                I wish I could try the cypher queries directly on the database, but I am only using the embedded version. I don't have Neo4J server installed. Since I really don't see a need for the server version at this moment.

                Thanks

                Mark

                Comment


                • #9
                  Originally posted by MichaelHunger View Post

                  Code:
                  start user=node({user}), event=node({eventId})
                  match user-[host?:HOSTS]->event,
                  event-[:ITEMS_NEEDED]->neededItem-[:ITEM]->item,
                  event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user
                  return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, item.name as itemName,  
                  userSignedUp!=null as currentUserSignedUp, ID(signedUpToBringToEvent) as itemSignedUpId
                  Michael
                  I am confused about that query. In the match portion, say the first line. user-[host?:HOSTS]->event will this match only if the user that is currently logged in to my site, which is the user that I pass to the repo method, is the host of the event, or will this return all the events that the user is hosting?

                  Kind of the same question about "event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user"

                  "event" and "user" still represent the event for the repo's eventId param and user as the repo's user param. Or are they just aliases used in the query. Because I look at it and think it will give me the Users that have signed up to bring an item to an event, not a specific item. An event could have 10 items NEEDED FOR EVENT, and the current logged in user might only be signed up to bring 1 or 2 of those items and not the other items NEEDED FOR EVENT. and other users, not the one logged in might be signed up for those other items NEEDED FOR EVENT.

                  Thanks again, I feel it is close. After these uses cases, I just have forms and updates to write code for. All the other queries are all done.

                  Mark

                  Comment


                  • #10
                    You might have to quote the strings of your relationship-type constants.
                    Code:
                    @Query("start user=node({0}) " +
                                "match user-[r]->event " +
                                "where type(r) = \"" + User.ATTENDING + "\" or type(r) = \"" + User.INVITED + "\" or type(r) = "\" + User.HOSTING + "\" " +
                                "return ID(event), event.eventDate, event.title, type(r) " +
                                "order by event.eventDate desc")
                        public Iterable<EventItemForAccountPage> getEventsForAccountPage(User user);
                    Can you provide the full exception that you're running into?

                    You can run the neo4j-shell (which comes with server, or see willie wheelers post) which executes cypher queries, or use Neoclipse which is also able to work readonly on stores and can execute cypher as well.

                    Comment


                    • #11
                      user-[host?:HOSTS]->event

                      is an optional match which results in host being non-null for a successful match and null for a non-existing relationship (like an outer join in SQL). And as both user and event are already bound it will not spread the result to more than this single user + single event.

                      For your second questions, this will fetch all the relationships of the type ITEMS_SIGNED_UP_FOR and for the end nodes (signedUpToBringToEvent) it will also do the "optional relationship" check.

                      Comment


                      • #12
                        I got the first cypher query to work.

                        I had downloaded and unzipped the Neo4j 1.6 Community Server. So I copied my db from my Tomcat dir to the unzipped servers data directory and got it running with the web console. Took some time to figure the console out, when to the Power Tool console and learned some of that. Took the query I had on the repo method and tried to run it on the Power Tool console. Tweaked it to get it to return the correct results.

                        Had to change "match user-[r]->event" to "match user-[r]-event" since the Invited relationship is in the other direction. so I needed it to check both directions. The ResultSet had the correct data that I need.

                        I even got the @MapResult to populate the data, sort of. In debugging I noticed that I couldn't view any of the data that the returned object with @MapResult is a proxy, which makes sense since that object was just an interface. But now I am wondering when the data gets retrieved via the proxy. I guess just call the getters, and Jackson will call the getters and render them into json to send back to the client.

                        Thanks

                        Mark

                        Comment


                        • #13
                          Unfortunately, it doesn't look like I am able to get json from the proxies.

                          "Could not write JSON: No serializer found for class $Proxy69 and no properties discovered to create BeanSerializer (to avoid exception, disable SerializationConfig.Feature.FAIL_ON_EMPTY_BEANS) ) (through reference chain: com.perfectworldprogramming.eventgate.wrappers.Lis tOfDomains["domainObjects"]->java.util.ArrayList[0])"

                          the disable part is probably not a solution as there are no empty beans.

                          Do I have to implement those ResultConverters??? I like the annotations with @MapResult. Much less code to write.

                          Thanks

                          Mark

                          Comment


                          • #14
                            I don't think I can use those ResultConverters. I am not using the template to run queries, I have @Query on my repo interface methods.

                            Also earlier you had said something about moving the calculate method my ListOfDomain class. The ListOfDomains is just a wrapper class to store the List<T> domain objects, so all types for different requests not just this one, so a calculate method would be too specific for one use case to be put into the generic ListOfDomain class. But we already have that solved with the cool Cypher query.

                            The service class now would look like this for that first use case

                            Code:
                                @Override
                                public ListOfDomains<EventItemForAccountPage> getListOfEventsForAccount() {
                                    ListOfDomains<EventItemForAccountPage> listOfDomains = 
                                                new ListOfDomains<EventItemForAccountPage>();
                                    
                                    Collection<EventItemForAccountPage> events = 
                                                eventRepository.getEventsForAccountPage(userRepository.getUserFromSession().getNodeId());
                                    
                                    listOfDomains.setDomainObjects(new ArrayList<EventItemForAccountPage>(events));
                                    listOfDomains.setType("Events you are involved with.");
                                    return listOfDomains;
                                }
                            EventItemForAccountPage is just the EventListItem with a different name so I could have my old approach and the new approach in my code at the same time, till the new way works all the way to the UI.

                            Thanks

                            Mark

                            Comment


                            • #15
                              In the Web console Power Tool I tried

                              Code:
                              start user=node({user}), event=node({eventId})
                              match user-[host?:HOSTS]->event,
                              event-[:ITEMS_NEEDED]->neededItem-[:ITEM]->item,
                              event-[:ITEMS_SIGNED_UP_FOR]-> signedUpToBringToEvent-[userSignedUp?:USER]->user
                              return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, item.name as itemName,  
                              userSignedUp!=null as currentUserSignedUp, ID(signedUpToBringToEvent) as itemSignedUpId
                              and similar variations. It looks like Neo4j doesn't like things like "host!=null as isHost" I got messages that I hurt some puppies or like that.

                              Code:
                              neo4j-sh (0)$ start user=node(33), event=node(53) match user-[host?:HOSTING]-event, event-[?:ITEMS_NEEDED_FOR_EVENT]-neededItem, event-[?:ITEMS_SIGNED_UP_FOR]-signedUpToBringToEvent-[userSignedUp?:USER_SIGNED_UP]-user return host!=null as isHost, ID(neededItem) as neededItemId, neededItem.quantity as quantity, ID(signedUpToBringToEvent) as itemSignedUpId
                              ==> SyntaxException: string matching regex `\z' expected but `!' found
                              ==> Unfortunately, you have run into a syntax error that we don't have a nice message for.
                              ==> By sending the query that produced this error to [email protected], you'll save the
                              ==> puppies and get better error messages in our next release.
                              ==> 
                              ==> Thank you, the Neo4j Team.
                              When I removed those in the return statement the query would run. Not actually returning any results, but would not be the data I was looking for.

                              Code:
                              neo4j-sh (0)$ start user=node(33), event=node(53) match user-[host?:HOSTING]-event, event-[:IITEMS_NEEDED_FOR_EVENT]-neededItem, event-[:ITEMS_SIGNED_UP_FOR]-signedUpToBringToEvent-[userSignedUp?:USER_SIGNED_UP]-user return ID(neededItem) as neededItemId, neededItem.quantity as quantity, ID(signedUpToBringToEvent) as itemSignedUpId
                              ==> +------------------------------------------+
                              ==> | neededItemId | quantity | itemSignedUpId |
                              ==> +------------------------------------------+
                              ==> +------------------------------------------+
                              ==> 0 rows, 1 ms
                              ==> 
                              neo4j-sh (0)$ start user=node(33), event=node(53) match user-[host?:HOSTING]-event, event-[:ITEMS_NEEDED_FOR_EVENT]-neededItem, event-[:ITEMS_SIGNED_UP_FOR]-signedUpToBringToEvent-[userSignedUp?:USER_SIGNED_UP]-user return ID(neededItem) as neededItemId, neededItem.quantity as quantity, ID(signedUpToBringToEvent) as itemSignedUpId
                              ==> +------------------------------------------+
                              ==> | neededItemId | quantity | itemSignedUpId |
                              ==> +------------------------------------------+
                              ==> +------------------------------------------+
                              ==> 0 rows, 35 ms
                              ==> 
                              neo4j-sh (0)$ start user=node(33), event=node(53) match user-[host?:HOSTING]-event, event-[?:ITEMS_NEEDED_FOR_EVENT]-neededItem, event-[?:ITEMS_SIGNED_UP_FOR]-signedUpToBringToEvent-[userSignedUp?:USER_SIGNED_UP]-user return ID(neededItem) as neededItemId, neededItem.quantity as quantity, ID(signedUpToBringToEvent) as itemSignedUpId
                              ==> EntityNotFoundException: The property 'quantity' does not exist on Node[47]
                              The neededItem is a Relationship object. Well, I think I know where to fix that.

                              "event-[?:ITEMS_NEEDED_FOR_EVENT]-neededItem"

                              should be

                              "event-[neededItem?:ITEMS_NEEDED_FOR_EVENT]-item"

                              But I still get the puppies message with the !=null stuff for some reason.

                              Thanks

                              Mark

                              Comment

                              Working...
                              X