Add robots.txt file for API, Embed, Widget & Web Applications

[chavda-bhavik]

Describe the bug
Due to API routes getting indexed in the Google search console, it's harming the application's reputation. Also, API, Embed, Widget & Web applications are not meant to be crawled by Google or any other search engines.

Expected behavior
API, Embed, Widget & Web applications should have robots.txt files where it should block crawlers from crawling. Expect these paths in web applications,

• https://web.impler.io/auth/signin
• https://web.impler.io/auth/signup

Additional context
Add any other context about the problem here.