chore:schema change documentation (#51)

* chore:schema change documentation

* chore: change doc

* chore: update api headers

---------

Co-authored-by: gagan.dhand <[email protected]>

docs/docs/guides/1_quickstart.md

@@ -10,6 +10,7 @@ This quick start will explore how to use Stencil command line interface and clie
 - [Docker](../installation#using-docker-image) or a [local installation](../installation#binary-cross-platform) of the Stencil binary.
 - A development environment applicable to one of the languages in this quick start (currently Go, Java, and JavaScript).
 - Postgres database and [protoc](https://github.com/protocolbuffers/protobuf#protocol-compiler-installation) if your schema format is protobuf.
+- Kafka
 
 ## Step 1: Start server
 
@@ -43,7 +44,7 @@ Note: Below command assumes `stencil_dev` db present in your postgres instance.
 $ docker run -e PORT=8000 -e DB_CONNECTIONSTRING=postgres://[email protected]:5432/stencil_dev?sslmode=disable -p 8000:8000 gotocompany/stencil server migrate
 
 # Stencil server at port 8000
-$ docker run -e PORT=8000 -e DB_CONNECTIONSTRING=postgres://[email protected]:5432/stencil_dev?sslmode=disable -p 8000:8000 gotocompany/stencil server start
+$ docker run -e PORT=8000 -e KAFKAPRODUCER_BOORSTRAP_SERVER=localhost:9092 SCHEMACHANGE_ENABLE=true SCHEMACHANGE_KAFKATOPIC=schema_change DB_CONNECTIONSTRING=postgres://[email protected]:5432/stencil_dev?sslmode=disable -p 8000:8000 gotocompany/stencil server start
 
 # Check if server running
 $ curl -X GET http://localhost:8000/ping

docs/docs/guides/3_manage_schemas.md

@@ -43,7 +43,7 @@ echo "syntax=\"proto3\";\npackage stencil;\nmessage One {\n  int32 field_one = 2
 protoc --descriptor_set_out=./file.desc --include_imports ./**/*.proto;
 
 # now try to upload this descriptor file with same name as before. This call should fail, giving you reason it has failed.
-curl -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc";
+curl -H "X-SourceURL:www.github.com/some-repo" -H "X-CommitSHA:some-commit-sha" -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc";
 
 # now let's try fixing our proto add a new field without having any breaking changes.
 echo "syntax=\"proto3\";\npackage stencil;\nmessage One {\n  int32 field_one = 1;\nint32 field_two = 2;\n}" > one.proto;
@@ -52,7 +52,7 @@ echo "syntax=\"proto3\";\npackage stencil;\nmessage One {\n  int32 field_one = 1
 protoc --descriptor_set_out=./file.desc --include_imports ./**/*.proto
 
 # now try to upload this descriptor file with same name as before. This call should succeed
-curl -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc"
+curl -H "X-SourceURL:www.github.com/some-repo" -H "X-CommitSHA:some-commit-sha" -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc"
 ```
 
 ## List versions
@@ -63,5 +63,5 @@ curl -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binar
 curl -X GET http://localhost:8000/v1beta1/namespaces/quickstart/schemas/example/versions
 
 # upload schema can be called multiple times. Stencil server will retain old version if it's already uploaded. This call won't create new version again. You can verify by using versions API again.
-curl -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc"
+curl -H "X-SourceURL:www.github.com/some-repo" -H "X-CommitSHA:some-commit-sha" -X POST http://localhost:8000/v1/namespaces/quickstart/schemas --data-binary "@file.desc"
 ```

docs/docs/guides/5_schema_change.md

@@ -0,0 +1,103 @@
+# Detect Schema Change
+
+## Enabling Schema Change
+
+To enable schema change toggle on the flag `SCHEMACHANGE_ENABLE` and provide `KAFKAPRODUCER_BOOTSTRAPSERVER`
+and `SCHEMACHANGE_KAFKATOPIC`.
+
+## Create a schema
+
+```bash
+# create namespace named "quickstart"
+curl -X POST http://localhost:8000/v1beta1/namespaces -H 'Content-Type: application/json' -d '{"id": "quickstart", "format": "FORMAT_PROTOBUF", "compatibility": "COMPATIBILITY_BACKWARD", "description": "This field can be used to store namespace description"}'
+
+# create descriptor file v1
+protoc --descriptor_set_out=./protos/1.desc --include_imports ./guide/protos/1.proto
+
+It will be version 1
+# upload generated proto descriptor file to server with schema name as `example` under `quickstart` namespace.
+curl -H "X-SourceURL:www.github.com/some-repo" -H "X-CommitSHA:some-commit-sha" -X POST http://localhost:8000/v1beta1/namespaces/quickstart/schemas/example --data-binary "@1.desc"
+
+# create descriptor file v2
+protoc --descriptor_set_out=./protos/1.desc --include_imports ./guide/protos/2.proto
+
+It will be version 2
+
+# upload generated proto descriptor file to server with schema name as `example` under `quickstart` namespace.
+curl -X POST http://localhost:8000/v1beta1/namespaces/quickstart/schemas/example --data-binary "@2.desc"
+
+# Detect Schema Change
+Now, when the schema is uploaded in DB. Below things happened parallely in separate go routine
+1. It will get the previous schema data corresponding to that namespace and schema name.
+2. It will check all the messages in that schema whether they are impacted(directly/indireclty) or not. 
+3. For change, it will check if any fileds got added/deleted. In this case only `address` field is added in `friend` message. So, `friend` proto is only impacted.
+4. In this way it gets all impacted messages. And do below steps for all messages.
+5. Then it will get the all indirectly impacted messages due to the message `friend`. 
+6. So, it will get `User` as it is using `Friend` message as compostion.
+7. It will do the same thing for `User` also until all impacted messages got proccessed.
+8. Then it will make `SchemChangedEvent` object and push it to a kafka topic.
+9. After that it will update the `notification_events` database table with `success=true` after pushing the object to kafka. 
+```
+
+## Handling Failures
+
+In case of any failure while publishing to Kafka or other issues, the `notification_events` table can be utilized to
+manage these failures. If the `success` field is `false` for a specific `namespace_id`, `schema_id`, and `version_id`,
+it indicates that the notification was not sent. In such scenarios, the `reconciliation API` can be used to resend the
+notifications.
+
+## Depth
+
+In schema changes, we use the `depth` parameter to determine the level of impacted schemas to retrieve. If `depth` is
+set to an empty string `""` or `-1`, it will fetch all indirectly impacted schemas. However, if a specific depth is
+provided, it will only retrieve schemas up to that specified level. For example, if `depth` is set to `3`, it will fetch
+up to three levels of impacted schemas, even if there are more levels of actual impacted schemas.
+E.g. If depth is `2` then the `impacted_schemas` for `Address` in below sample would contain only `Address` and `Friend` message.
+
+## Sample SchemaChangedEvent Object
+
+```
+{
+  "namespace_name": "quickstart",
+  "schema_name": "example",
+  "updated_schemas": [
+    "test.stencil.main.Friend",
+    "test.stencil.main.Address",
+  ],
+  "updated_fields": {
+    "test.stencil.main.Friend": {
+      "field_names": [
+        "address"
+      ]
+    },
+    "test.stencil.main.Address": {
+      "field_names": [
+        "city"
+      ]
+    }
+  },
+  "impacted_schemas": {
+    "test.stencil.main.Address": {
+      "schema_names": [
+        "test.stencil.main.Address",
+        "test.stencil.main.Friend",
+        "test.stencil.main.User",
+      ]
+    },
+    "test.stencil.main.Friend": {
+      "schema_names": [
+        "test.stencil.main.Friend",
+        "test.stencil.main.User",
+      ]
+    }
+  },
+  "version": 1,
+  "metadata": {
+    "source_url": "https://github.com/some-repo",
+    "commit_sha": "some-commit-sha"
+  }
+}
+```
+
+## Showing MR information 
+While calculating the impacted schemas, the `SchemaChangedEvent` will also include information about the source URL and commit SHA. The `source_url` represents the repository URL, and the `commit_sha` corresponds to the commit SHA associated with that version.

docs/docs/guides/protos/1.proto

@@ -0,0 +1,17 @@
+syntax = "proto3";
+
+package gotocompany.test.stencil.main;
+
+option java_multiple_files = true;
+option java_package = "com.test.stencil.main";
+option java_outer_classname = "FriendsProto";
+
+message User {
+  string  name = 1;
+  repeated  Friend friends = 2;
+}
+
+message Friend {
+  string name = 1;
+}
+

docs/docs/guides/protos/2.proto

@@ -0,0 +1,22 @@
+syntax = "proto3";
+
+package gotocompany.test.stencil.main;
+
+option java_multiple_files = true;
+option java_package = "com.test.stencil.main";
+option java_outer_classname = "FriendsProto";
+
+message  User{
+  string  name = 1;
+  repeated  Friend friends = 2;
+}
+
+message Friend {
+  string name = 1;
+  Address address = 2;
+}
+
+message Address {
+  string city = 1;
+}
+