One of our projects required service to register portions of persons data processed by parallel enrichment services.
Data delivered from different sources in portions. Enrichment services retrieve all data required for processing the portion from database before processing. So to avoid double work and overwriting data processed in parallel we decided to create dispatcher service that keeps in memory data being processed. Real dispatcher takes care of data retention, hashing, etc. But for our demo it would be enough to take a look at simplified protocol and implementation of the service in different programming languages.
Also, in order to compare different languages we will build Docker images of the service and run some performance tests.
Now, about algorithm. Service should provide POST and DELETE endpoints that receive JSON array of objects with firstName and lastName attributes.
POST endpoint should check if passed first names and last names are present in service memory. If none present, the empty array should return. Any names found in memory returned back so calling service may avoid processing names being processed in parallel.
DELETE endpoint silently removes passed names.
We want to demonstrate the overhead related to runtimes for different programming languages as well as effectiveness of using hardware resources.
In series of related articles we’ll demonstrate solution of the task described above with related tests and measurements.