Making Child Documents Working with Spring-data-solr

The Problem
We use spring-data-solr in our project - as we like its conversion feature which can convert string to enum, entity to json data and etc, and vice versa, and recently we need use Solr's nested documents feature which spring-data-solr doesn't support.

Issues in Spring-data-solr
SolrInputDocument class contains a Map _fields AND List _childDocuments.

Spring-data-solr converts java entity class to SolrDocument. It provides two converters: MappingSolrConverter and SolrJConverter.

MappingSolrConverter converts the entity to a Map: MappingSolrConverter.write(Object, Map, SolrPersistentEntity)

SolrJConverter uses solr's DocumentObjectBinder to convert entity to SolrInputDocument,
it will convert field that is annotated with @Field(child = true) to child documents.
- This also means that spring-data-solt's convert features will not work with SolrJConverter

BUT SolrJConverter still just thinks SolrInputDocument is a map and add all into the destination: Map sink
- SolrJConverter.write(Object, Map)

After this, the child documents is discarded.

The Fix
We still want to use spring-data-solr's conversion functions - partly because we don't want to rewrite everything to use SolrJ directly.

So when save to solr: we uses spring-data-solr's MappingSolrConverter to convert parent entity as solrInputDocument, then convert child entities as solrInputDocuments and add them into parent's solrInputDocument.

When read from solr, we read the SolrDocument as parent entity, then read its child documents as child entities and add them into parent entity.
public class ParentEntity {
  @Field(child = true)
  private List<ChildEntity> children;
}
@Autowired
protected SolrClient solrClient;

// we add our own converters into MappingSolrConverter
// for more, please check 
// http://lifelongprogrammer.blogspot.com/2015/09/mix-spring-data-solr-and-solrj-in-solr.html
@Autowired
protected MyMappingSolrConverter solrConverter;

public void save(@Nonnull final ParentEntity parentEntity) {
    final SolrInputDocument solrInputDocument = solrConverter.createAndWrite(parentEntity);
    daddChildDocuemnts(parentEntity, solrInputDocument);
    try {
        solrClient.add(getCollection(), solrInputDocument);
        solrClient.commit(getCollection());
    } catch (SolrServerException | IOException e) {
        throw new BusinessException(e, "failed to save " + parentEntity);
    }
}

protected void daddChildDocuemnts(@Nonnull final ParentEntity parentEntity,
        @Nonnull final SolrInputDocument solrInputDocument) {
    solrInputDocument.addChildDocuments(parentEntity.getChildren().stream()
            .map(child -> solrConverter.createAndWrite(child)).collect(Collectors.toList()));
}

public List<T> querySolr(final SolrParams query) {
    try {
        final QueryResponse response = solrClient.query(getCollection(), query);
        return convertFromSolrDocs(response.getResults());
    } catch (final Exception e) {
        throw new BusinessException("data retrieve failed." + query);
    }
}
/*
 * Also return child documents in solr response as ChildEntity if it exists
 */
protected List<ParentEntity> convertFromSolrDocs(final SolrDocumentList docList) {
    List<ParentEntity> result = new ArrayList<>();
    if (docList != null) {
        result = docList.stream().map(solrDoc -> {
            final ParentEntity parentEntity = solrConverter.read(ParentEntity.class, solrDoc);
            final List<SolrDocument> childDocs = solrDoc.getChildDocuments();
            if (childDocs != null) {
                ParentEntity.setChildren(
                        childDocs.stream().map(childSolrDoc -> solrConverter.read(ChildEntity.class, solrDoc))
                                .collect(Collectors.toList()));
            }

            return parentEntity;
        }).collect(Collectors.toList());
    }

    return result;
}
Related
Mix Spring Data Solr and SolrJ in Solr Cloud 5
SolrJ: Support Converter and make it easier to extend DocumentObjectBinder

Eclipse: Add another Project as Dependency may Cause Unexpected Exception

The Problem
In local development, we run spring-boot application in eclipse tomcat - as we also deploy the project as a war.

But for some reason, one developer stills run it as a java application, and it fails with error - while it works well when run in (eclipse) tomcat.
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException:
Failed to instantiate [com.amazonaws.services.s3.AmazonS3]: Factory method 's3Client' threw exception;
nested exception is java.lang.NoSuchMethodError: com.amazonaws.handlers.HandlerChainFactory.getGlobalHandlers()Ljava/util/List;

The Root Cause
Maven uses nearest wins strategy to determine which version to use, So we explicitly specify what version of aws-java-sdk-s3 to use in admin's pom.xml, but one library also implicitly depends on aws-java-sdk-s3 in common-module project. 

run mvn dependency:tree -Dverbose -Dincludes=com.amazonaws:aws-java-sdk-core, which shows that maven chooses the right version.
[INFO] |  \- com.amazonaws:aws-java-sdk-cloudfront:jar:1.11.32:compile
[INFO] |     \- (com.amazonaws:aws-java-sdk-core:jar:1.11.32:compile - omitted for conflict with 1.11.98)

- We can also get which version maven uses and why in eclipse: 
open pom,xml then go to dependency hierarchy tab , select the library in right panel.

I tried to run it as a java application, it works fine. But why it failed in his environment?

I compared the difference between his eclipse setup and mine, and found out that he manually added common-module in admin's Java Build Path -> Projects tab.

Now it's kind of clear why it failed: when we add a project as dependency, Eclipse also includes all libraries it depends on to the project. So now the project includes both versions, and Eclipse chooses the wrong version to use.

I created one bug Bug 514094 - Adding another Project as Dependency Causes Unexpected Exception to track it.

Troubleshooting - JsonMappingException: Already had POJO for id

The Problem
We have two entities with one-to-many relationships which references each other, but it failed with the exception:
com.fasterxml.jackson.databind.JsonMappingException: Already had POJO for id

The Fix
To easily troubleshoot the issue, I created a sample class like below:
@Data
@Accessors(chain = true)
@EqualsAndHashCode(of = {"employeeId"}, callSuper = false)
@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "employeeId")
public static class Employee {
    private UUID employeeId;
    private String name;
    private Department department;
}
@Data
@Accessors(chain = true)
@EqualsAndHashCode(of = {"departmentId"}, callSuper = false)
@JsonIdentityInfo(generator = ObjectIdGenerators.PropertyGenerator.class, property = "departmentId")
@ToString(exclude = "employees")
private static class Department {
    private UUID departmentId;
    private String name;
    private Set<Employee> employees;

    public Set<Employee> getEmployees() {
        if (employees == null) {
            employees = new HashSet<>();
        }
        return employees;
    }
}


public static void main(String[] args) throws IOException {
    Employee e1 = new Employee().setEmployeeId(UUID.randomUUID()).setName("e1");
    Department d1 =
            new Department().setDepartmentId(UUID.randomUUID()).setName("oldD1").setEmployees(Sets.newHashSet(e1));
    e1.setDepartment(d1);
    ObjectMapper objectMapper = new ObjectMapper();

    String departmentStr = objectMapper.writeValueAsString(d1);
    objectMapper.writeValueAsString(e1);

    Department oldD1 = objectMapper.readValue(departmentStr, Department.class);

    Department newD1 = new Department().setDepartmentId(d1.getDepartmentId()).setName("newD1");
    newD1.getEmployees().addAll(oldD1.getEmployees());
    // without the following statements: it will throw
    // com.fasterxml.jackson.databind.JsonMappingException: Already had POJO for id
    // for (Employee e : oldD1.getEmployees()) {
    // e.setDepartment(newD1);
    // }

    departmentStr = objectMapper.writeValueAsString(newD1);
    System.out.println("new department: " + departmentStr);
    // now read it back will throw sonMappingException: Already had POJO for id
    Department newNewD1 = objectMapper.readValue(departmentStr, Department.class);
    System.out.println("---" + newNewD1);
}
This reproduces the issue, and from the output:
new department: {"departmentId":"e3e0e676-0c52-493d-8f49-bedde05cbb11","name":"newD1","employees":[{"employeeId":"6b7bbbec-8be6-4423-a4ef-af7924df177b","name":"e1","department":{"departmentId":"e3e0e676-0c52-493d-8f49-bedde05cbb11","name":"oldD1","employees":["6b7bbbec-8be6-4423-a4ef-af7924df177b"]}}]}
I found that after I changed the department to newD1, the employee still refers to old department object with department name: oldD1.

This leads to my fix like below: after I made change to the department object, make sure the employees refers to the new department object.

// without the following statements: it will throw
// com.fasterxml.jackson.databind.JsonMappingException: Already had POJO for id
for (Employee e : oldD1.getEmployees()) {
  e.setDepartment(newD1);
}

Miscs
We need exclude employees from Department's toString: @ToString(exclude = "employees")
- Otherwise it would throw java.lang.StackOverflowError
Likewise, we need exclude employees from @EqualsAndHashCode.

Labels

Java (159) Lucene-Solr (110) Interview (61) All (58) J2SE (53) Algorithm (45) Soft Skills (37) Eclipse (33) Code Example (31) Linux (24) JavaScript (23) Spring (22) Windows (22) Web Development (20) Nutch2 (18) Tools (18) Bugs (17) Debug (16) Defects (14) Text Mining (14) J2EE (13) Network (13) Troubleshooting (13) PowerShell (11) Chrome (9) Design (9) How to (9) Learning code (9) Performance (9) Problem Solving (9) UIMA (9) html (9) Http Client (8) Maven (8) Security (8) bat (8) blogger (8) Big Data (7) Continuous Integration (7) Google (7) Guava (7) JSON (7) ANT (6) Coding Skills (6) Database (6) Scala (6) Shell (6) css (6) Algorithm Series (5) Cache (5) Dynamic Languages (5) IDE (5) Lesson Learned (5) Programmer Skills (5) System Design (5) Tips (5) adsense (5) xml (5) AIX (4) Code Quality (4) GAE (4) Git (4) Good Programming Practices (4) Jackson (4) Memory Usage (4) Miscs (4) OpenNLP (4) Project Managment (4) Spark (4) Testing (4) ads (4) regular-expression (4) Android (3) Apache Spark (3) Become a Better You (3) Concurrency (3) Eclipse RCP (3) English (3) Happy Hacking (3) IBM (3) J2SE Knowledge Series (3) JAX-RS (3) Jetty (3) Restful Web Service (3) Script (3) regex (3) seo (3) .Net (2) Android Studio (2) Apache (2) Apache Procrun (2) Architecture (2) Batch (2) Bit Operation (2) Build (2) Building Scalable Web Sites (2) C# (2) C/C++ (2) CSV (2) Career (2) Cassandra (2) Distributed (2) Fiddler (2) Firefox (2) Google Drive (2) Gson (2) How to Interview (2) Html Parser (2) Http (2) Image Tools (2) JQuery (2) Jersey (2) LDAP (2) Life (2) Logging (2) Python (2) Software Issues (2) Storage (2) Text Search (2) xml parser (2) AOP (1) Application Design (1) AspectJ (1) Chrome DevTools (1) Cloud (1) Codility (1) Data Mining (1) Data Structure (1) ExceptionUtils (1) Exif (1) Feature Request (1) FindBugs (1) Greasemonkey (1) HTML5 (1) Httpd (1) I18N (1) IBM Java Thread Dump Analyzer (1) JDK Source Code (1) JDK8 (1) JMX (1) Lazy Developer (1) Mac (1) Machine Learning (1) Mobile (1) My Plan for 2010 (1) Netbeans (1) Notes (1) Operating System (1) Perl (1) Problems (1) Product Architecture (1) Programming Life (1) Quality (1) Redhat (1) Redis (1) Review (1) RxJava (1) Solutions logs (1) Team Management (1) Thread Dump Analyzer (1) Visualization (1) boilerpipe (1) htm (1) ongoing (1) procrun (1) rss (1)

Popular Posts