Skip to content

Microservices with Istio, gRPC, Redis, BigQuery, Spring Boot, Spring Cloud and Stackdriver

License

Notifications You must be signed in to change notification settings

kbhattac/coolretailer

Repository files navigation

Coverage Status Build Status License

CoolRetailer

Microservices application with Istio, gRPC, Redis, BigQuery, Spring Boot, Spring Cloud and Stackdriver

  • Smart product finder with autocomplete feature
  • Distributed Stackdriver Tracing with Log correlation across gRPC calls
  • Istio based Stackdriver Service Monitoring
  • Automatic synthetic load generation with Locust
  • Complete CI/CD to GKE with Google Cloud Build
  • Redis in Master/Slave replication mode (image from Bitnami)
  • Autocomplete logic based on ZRANGEBYLEX Redis command
  • Tunable cache size
  • Dataset from BestBuy (converted to ndjson using JsonProcessor)
  • Developed completely with VSCode
  • Inspired by Next '18 Microservices Demo
  • Covering slides on Slideshare

Architecture:

Architecture

Automatic Installation:

Open in Cloud Shell

Guided one click installation from Google Cloud Shell. Complete stack (incl. all dashboards shown below + Istio fault simulation). No client tooling required.

Manual Installation:

  • Get the Best Buy products dataset and save it to a GCS bucket:
  • BQ requires the JSON to be new line delimited. Use the provided utility JSON processor for this:
java -jar src/queryservice/target/queryservice-1.0.0.jar \
    --spring.profiles.active=JSON \
    --input.json="$PWD"/products.json --exit
  • Load it to BQ as dataset coolretailer.products from GCS
  • Create a GCP service account with the following roles:

BigQuery Data Viewer, BigQuery Job User, Stackdriver Debugger Agent, Cloud Trace Agent, Context Graph Asserter, Error Reporting Admin, Logs Writer, Monitoring Metric Writer

and install the private key as app-gac.json as kubernetes secret app-gac.

  • apply Istio manifests in: setup/istio-manifests
  • set PROJECT_ID, TAG_NAME, REDIS_HOST in setup/kubernetes-manifests/ and apply them

Notes:

  • Spring Cloud Sleuth Stackdriver Trace and Logging enabled.
  • Complete CI/CD pipeline with Google Cloud Build.
  • Test locally using:
$mvn spring-boot:run \
    -DGOOGLE_APPLICATION_CREDENTIALS=path-to-key-file \
    -DPROJECT_ID=coolretailer \
    -DGOOGLE_CLOUD_PROJECT=coolretailer \
    -Dspring-boot.run.arguments=--spring.redis.host=localhost
  • Using docker from Cloud Shell:

(If tunneling to the redis service is required)

gcloud container clusters get-credentials coolretailer2 --zone europe-west4-a --project coolretailer \
 && kubectl port-forward $(kubectl get pod --selector="app=redis-master,role=master,tier=backend" --output jsonpath='{.items[0].metadata.name}') 6379:6379

gcloud auth configure-docker -q &&\
docker run \
    --network=host \
    -v /home/kbhattacharya:/etc/app-gac \
    -e GOOGLE_CLOUD_PROJECT=coolretailer \
    -e PROJECT_ID=coolretailer \
    -e GOOGLE_APPLICATION_CREDENTIALS=/etc/app-gac/app-gac.json \
    -e --spring.sleuth.sampler.probability=1.0 \
    -e --spring.application.name=ux-local \
    -e --spring.cloud.gcp.trace.enabled=true \
    -e --spring.cloud.gcp.logging.enabled=true \
    -e --spring.cloud.gcp.project-id=coolretailer \
    -it gcr.io/coolretailer/queryservice
  • Use Cloud Memorystore instead of the Redis deployments (src/setup/kubernetes-manifests/archive/redis):
  • Istio TLS setup
  • See setup/istio-manifests/secure for simple TLS configuration
  • The static content is recommended to be served from a GCS bucket with Cloud CDN. Just drop off the content of src/ui/static/ui.* and src/ui/static/lib in a GCS bucket and provide allUsers READER access.

Screenshots:

Application

CoolRetailer

Cloud Trace with Log correlation

Trace

Istio fault injection

Mesh

Grafana

Grafana

Istio Service monitoring

Istio

Kubernetes CPU telemetry

CPU

Disclaimer:

This is not an official Google product