Hey Champs, this is Tanmay again. And today's topic is a little different, in the sense We are coders but trying to embrace low-code tools also. It is not that we are trying to transition to this kind of tool but to increase productivity as a developer.

Using this tool, we will remove the repeated same tasks for building a rest API of a low-complexity project. Suppose you are building a health tracking project and you need a simple backend service for performing various logic of available data, in that scenario building a backend server in the project is comparatively time-consuming and hectic.

And that position, we can use tools like Syncloop to build a Rest API with ease and use it in our project. In that way, we can remove the need for a dedicated backend server and the time of a developer to code a backend.

With that being said, let's see what are the possibilities with Syncloop.

Overview of Syncloop

Syncloop is an innovative no-code API service that simplifies the process of creating and managing APIs. With its user-friendly interface, developers and non-developers alike can effortlessly design, deploy, and monitor APIs without writing a single line of code.

Syncloop offers a wide range of features, including data integration, authentication, and endpoint customization, enabling users to seamlessly connect their applications and services. It's intuitive workflow and robust security measures make it an ideal choice for businesses and individuals seeking a quick and efficient way to build powerful APIs without the complexities of traditional coding.

Let's get started with Syncloop 🚀

Now I'll walk you through the whole process to build an API which will give us a list of quotes and the authors' names.

1. Create an account with Syncloop:

   Visit https://www.syncloop.com/ and create a free account there. Now log in to the dashboard with your account.

   

   As you can see, you will find useful resources to understand the tool in their documents and training videos. If you are serious about using it, you should watch and understand the tool more.

2. Head over to Workspace:

   Here in the workspace, we will work. Everything related to building APIs is here.

   

   You will find `packages` there, inside it there are demos and pre-built services. Now maybe you are thinking about what are services.

   💡

   The service in the Syncloop tool behaves like functions in the programming languages. A function is first defined and then called for its proper usage. The main advantage of using services is that it aids in better reusability of program code. Find more about the service here.

3. Create the folders and files:

   First, you need to create a dedicated package for our quote API by right-clicking and selecting new.

   

   Second, you'll see the new package quoteAPI with a dependency folder inside it. Now create a folder there. I'm naming it generateQuotes

   

   Third, create a last file which is a service of the API category. And in this file, we will drag and drop stuff to make the quote generation work.

   

   We can use a dedicated database to fetch quotes to our Rest API but I don't want to make this beginner-friendly tutorial more complex. So I will fetch quotes from another free available API and route it to our API, so that we can use our API to fetch quotes but deep inside our API will also fetch the quote from another API.

   As you can see there are Consumers and Developers fields, it is for better role-based permissions.

   

   With that, we have created our first API service which has an endpoint that we will use as an API link. Now we have to customize it to our needs. Let us see what to be done now.

   

4. Build API Service File:

   You understood the input and output parameters area. Now first we have to use a Map or Transformer to map flow data in further steps.

   💡

   Transformer is a Syncloop term, this is one of the most frequently used statements on the platform. A Transformer statement is created by right-clicking on the control flow section and selecting Transformer from the context menu strip. A Transformer is created in the control flow section of the Syncloop workspace.

   So, right-click on the centre and choose Transformer.

   

   With that, you will see a new layout. Each transformer has two fields also input and output. You can right-click to create new files or fields there.

   💡

   We plan to make a get request to an existing quote API, get the quotes, convert the quotes from JSON to object and output the response with it.

   First, we are making a transformer with output parameters url with https://type.fit/api/quotes and method with GET.

   

   You can just select any parameter and click on the pen icon to set default values to the parameters. So just set url value and method value.

   With that, you are sure that our transformer is outputting these two parameters.

   Second, right-click on the dashboard area and this time select a service and this popup will arise. So without interfering with the pop-up, search in the packages section with 'request', select the requestV2 and click on Done.

   

   Now we have a transformer and a service in our API in respective order. So the transformer will send its output data to the service now.

   Select the requestV2 service in the dashboard, you will see a new layout in the bottom part. You have to just make the route lines just by dragging.

   

   So, url from the previous transformer will go to the requestV2 service's URL param and method to method param. And also the service's respPayload to a newly created payload output parameter string.

   Now, create the same name param in the main output section and that param will hold the payload also.

   

5. Now let's test our API with Postman:

   Before that, extract our API's endpoint by clicking on the configuration icon.

   

   And, get the bearer token from the sidebar,

   

   Now, just put the endpoint of the token in Postman and perform a GET request.

   

Video Solution 📀

If you found the article a little confusing, I've got your back too. Watch this video:

Conclusion

This tool proves especially useful for projects with straightforward backend requirements, eliminating the need for dedicated backend servers and saving valuable time for developers.

As we wrap up, I hope this blog has sparked your interest in delving into the world of low-code tools. We covered the following topics:

• What is Syncloop?

• Who should use it?

• How to use it.

• Building a very simple quote-generating API.

I hope this blog has inspired you to get involved in open source also! Because as an active contributor, I stay updated with the community. And I'm writing this blog with the challenge from the WeMakeDevs community. If you have any questions, please feel free to comment or reach out to me on Twitter.

With that have an amazing day, sayonara 👋🏼

Follow me on @sarkartanmay393 🐦
Direct mail me at sarkartanmay393@gmail.com 📧

Hey Devs, What's up? Today I'm back with another blog and it's mainly for beginners in Open Source. You might be trying to get involved in Open Source but you don't have any heads-up. For that reason, you should read it to the end.

Open source is a great way to learn new skills, meet new people, give back to the community, and definitely, the journey will be fruitful for you. As you're new to open source, you might be wondering where to start. One of the best ways to get involved is to find and fix a good first issue.

A good first issue is a bug or feature request that is relatively easy to fix and doesn't require in-depth prior knowledge of the project. These issues are a great way to get your train rolling in open source and make a meaningful contribution.

I'll be mentioning some best places where you can find your first open-source issue in your comfortable tech stack.

Before you move forward

I want to talk about who should read this article and find an issue for themself. Obviously, who has practical knowledge of any tech stack (e.g. web development) and knows that he/she can contribute to other similar projects. After satisfying this requirement and only then you should search for projects.

As I'm currently into web development with MERN, TypeScript, Next.js, etc. So I can find a web project built with any of these technologies and I'm good to go with contributing to that project.

Some might be:

1. https://github.com/layer5io/layer5

2. https://github.com/meshery/meshery.io

3. https://github.com/InternetHealthReport/ihr-website

Let's find an issue for you 🙌

Here are some of the best ways:

• GitHub: GitHub is the most popular website for hosting open-source projects. You can find good first issues by searching for the good-first-issue label.

  In the GitHub search bar, enter the following query:

  is:issue is:open label:"good first issue" label:"your-tech-stack"

  Replace your-tech-stack with the name of your specific tech stack or programming language. For example, if you are working with JavaScript, you would use label:"javascript"

  

• Good First Issue: This is a website that curates good first issues from popular open-source projects. They have a search engine that lets you filter issues by language, framework, and other criteria. They are totally elegant looking.

  Here: https://goodfirstissue.dev/

• Up for Grabs: Up for Grabs is also a website but it lists open-source projects that are looking for contributors. They have a section for good first issues, as well as a forum where you can ask questions and get help from other contributors.
  Here: https://up-for-grabs.net/

• First Timers Only: Last but not least, First Timers Only is a website that helps new contributors find open-source projects that are welcoming to first-timers. They have a list of projects with good first issues, as well as tutorials and resources to help you get started. You should head over to their website sometime.

Once you've found a good first issue, you can start working on it. The first step is to make sure that you understand the issue. Read the issue description carefully and ask questions if you're not sure what it means.

Once you understand the issue, you can start working on a fix. If you're not sure how to fix the issue, you can ask for help from the project maintainers or other contributors.

Once you've fixed the issue, you can submit a pull request. A pull request is a way of submitting your changes to the project's codebase.

The project maintainers will review your pull request and either approve it or reject it. If your pull request is approved, your changes will be merged into the project's codebase.

If you're new to open source, I encourage you to find a good first issue and start contributing today!

Conclusion

In this blog, we discussed how to find and fix your first open-source issue. We covered the following topics:

• Who should read this article and find an issue for themselves.

• What is a tech stack?

• How to find good first issues on GitHub, Good First Issue, and Up for Grabs.

• How to submit a pull request and never wait for getting assigned in issues.

I hope this blog has inspired you to get involved in open source! If you have any questions, please feel free to comment or reach out to me on Twitter.

Here are some additional tips for you:

• Start small. Don't try to fix a big issue on your first try. Start with something small and easy to understand.

• Be patient. It may take some time for your pull request to be approved. Don't get discouraged if it doesn't happen right away.

• Be helpful. If you see someone else struggling with an issue, offer to help them.

• Have fun! Open source is a great way to learn new things and meet new people. So enjoy the process!

With that have a great day, sayonara 👋🏼

Follow me on @sarkartanmay393 🐦
Direct mail me at sarkartanmay393@gmail.com 📧

Hey Guys, It's Tanmay here, and today I'll be talking about How you can run a smaller version of LLM on your machine directly, totally free of cost.

As we all know, Using and building on top of LLMs is costlier. Some students and researchers can't pay much amount of money to further study these models. That's why GPT4ALL is here.

Today, we'll use this quantized model to build a very simple QnA program instead of OpenAI's GPT. So stay along and enjoy the journey.

Also, if you are not the reader type, watch this video Here. I covered all everything in that video but this blog is necessary because I can write in detail here.

How Quantized GPT4ALL works?

GPT4ALL is trained on top of Facebook's LLaMA. We can't run Facebook's model on our machine because it is very big and has more than 7 billion parameters. That's why engineers used tricks like fine-tuning and inference to make smaller versions.

The underlying algorithm that takes place is Quantization, where it maps every continuous infinite value to a smaller set of discrete values. It basically converts the pre-trained model weights to 4-bit precision using the GGML format. So, the model uses fewer bits to represent the numbers. Therefore, reducing memory usage and faster inference.

Running GPT4ALL Locally 👷🏻

Requirements :

• Read: https://hashnode.com/post/cljoj6t5v000908l3fzm89hn5

Steps :

1. Download the model: You have to download the quantized model weight from this link (https://the-eye.eu/public/AI/models/nomic-ai/gpt4all/). The model name is gpt4all-lora-quantized-ggml.bin.

2. Clone converter repo: Run the command to clone llama.cpp that is used to convert our model weight to a newer format. Because langchain only supports the newer formatted models.

    git clone https://github.com/ggerganov/llama.cpp.git

3. Checkout to a specific stage: You've to set the repo to a previous stage 2b26469. Because the latest version llama.cpp doesn't quite work well and throws a lot of errors.

    cd llama.cpp && git checkout 2b26469

4. Convert the model: To convert it, you have to run the convert.py argument value path-to-our-downloaded-model So you can just move the model to llama.cpp/models the directory.

    # Inside llama.cpp directory python3 convert.py models/gpt4all-lora-quantized-ggml.bin

   

   Our conversion to a newer format is now successfully done, you can check that out in the /models/ directory. There must be a new file named similar to ggml-model-q4_0.bin. Now, we can build a simple program using GPT4ALL.

5. Now, Coding the program: Our program will take any question and try to answer in a step-by-step manner in 100 words.

   Before starting, you have to install langchain the library by pip install -q langchain==0.0.152

    from langchain import PromptTemplate, LLMChain from langchain.llms import GPT4All from langchain.callbacks.base import CallbackManager from langchain.callbacks.streaming_stdout import StreamingStdOutCallbackHandler # Create a callback manager and add a callback handler for streaming stdout callback_manager = CallbackManager([StreamingStdOutCallbackHandler()]) # Create an LLM using the GPT4All model llm = GPT4All(model="llama.cpp/models/ggml-model-q4_0.bin", callback_manager=callback_manager, verbose=True) # Create a template for prompting the LLM template = """Question: {question} Answer: Let's think step by step in 100 words. """ prompt = PromptTemplate(     template=template,     input_variables=["question"] ) # Create an LLM chain chain = LLMChain(llm=llm, prompt=prompt) # Prompt the chain with a question q = "How rain falls?" chain.run(q)

6. Running our program: You can change the q variable to ask any question you want. Try out different questions, and see our this model performs on your machine. I have an MBA 8GB with 256GB SDD, and I still think a 16GB version would do a lot better in terms of faster output generation time.

   

    pip3 test.py # will run the program

   Running the code may take a lot of time, so you've to be patient at first.

7. Understanding the Output: Let us see the output first.

   

   This 100 words output took a lot of time on my machine, if there is any trick to make it faster, please do tell us in the comments.

   In the code, we used the StreamingStdOutCallbackHandler and verbose=True but why? It is because we want to see the output in real-time and with a lot of extra system details.

   

Conclusion

In this blog post, I showed you how to run a smaller version of LLM on your machine directly, totally free of cost. I used GPT4ALL, which is an open-source software ecosystem that allows anyone to train and deploy powerful and customized large language models (LLMs) on everyday hardware.

I also showed you how to build a simple QnA program using GPT4ALL. Our program takes any question and tries to answer in a step-by-step manner in 100 words.

Here are some additional thoughts:

• The GPT4ALL model is still under development, so the output may not always be perfect. However, it is a great way to get started with LLMs and see how they can be used to build real-world applications.

• The code in this blog post is just a starting point. You can experiment with different questions and prompts to see how the model performs. You can also try to improve the performance of the model by using a faster machine or by fine-tuning the model.

I hope you learned something new today. If you have any questions, feel free to ask me in the comments below. I will be happy to help you. It would be great if you share this article with your developer friends who are new to tech.

For reference, I wrote this article after learning these topics from a course called LangChain & Vector Databases in Production. I highly suggest you join it and learn in-depth about these topics. I've used some code snippets and an image from their coursework.

Follow me on @sarkartanmay393
Direct mail me at sarkartanmay393@gmail.com

Hi Fellows, today is your special day as it is for me too. I learned more about AI, LLMs and Langchain framework. You may be thinking what are these terms?

LLM stands for Large Language Model and Langchain is a framework to build applications using LLMs(GPT-3, GPT-4).

As I'm learning advanced development, building applications using AI tools is necessary in today's world. The tech community nowadays sees various awesome apps built on top of AI models and gives users a wonderful experience. So today, I'll be talking about this topic and helping you kickstart your journey in this field.

As you are already aware the constant bombard of new information on us every single day. It can be challenging to keep up with all the news and articles published daily. This is where a news article summarizer can be helpful, What do you think? 🤓
A news article summarizer can read a long article and generate a shorter summary of the key points. This can save you time and help you to stay up-to-date on the latest news.

And how will it happen? Definitely using OpenAI services. But there's a catch, it takes money to give you API services. Heyyy no worries, I've got a solution for you. You can signup at OpenAI and avail $5 for the first three months. I have another way also, where we leverage quantized chat models to run locally on our machine and get services directly from it, but it's another topic. I will walk you through that way also, but in the next blog, so stay updated with me.

LLMs and Langchain

What are basically LLMs? Large language models (LLMs) are a type of artificial intelligence (AI) that are trained on massive datasets of text and code. This allows them to generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.

Some of the most popular LLMs include:

• GPT-3: Developed by OpenAI, GPT-3 is one of the most powerful LLMs available. It has been trained on a dataset of text and code that is over 500 times larger than the dataset used to train its predecessor, GPT-2.

• LaMDA: Developed by Google AI, LaMDA is a factual language model that can answer your questions in an informative way. It has been trained on a dataset of text and code that is over 1.56 trillion words long.

LLMs are still under development, but they have the potential to revolutionize many industries. They can be used for a variety of tasks, including:

• Text summarization: LLMs can be used to generate summaries of long pieces of text. This can be helpful for people who want to quickly get the gist of a document without having to read the whole thing.

• Translation: LLMs can be used to translate text from one language to another. This can be helpful for people who want to communicate with people who speak other languages.

• Creative writing: LLMs can be used to generate creative text, such as poems, stories, and scripts. This can be helpful for writers who are looking for inspiration or who want to automate the writing process.

• Question answering: LLMs can be used to answer your questions in an informative way. This can be helpful for students who are doing research or for people who want to learn more about a particular topic.

LLMs are still a relatively new technology, but they have the potential to change the way we interact with computers. As they continue to develop, they will become more powerful and capable, and they will be used for a wider range of tasks.

And Langchain is just a framework (e.g. React) but it simplifies the development of applications using LLMs. It provides a number of functions that allow you to interact with the LLMs and generate cool stuff. That is what it's all about in a general sense. We will use Python as our coding language to build an article summarizer today with the help of OpenAI ChatModel by using the Langchain framework.

Building the Summarizer 👷🏻

Requirements :

• Basic CS Fundamentals

• Knowledge of Any Programming Language

Code :

1. First of all, you need to install these packages langchain, openai, python_dotenv, requests, newspaper3k

    pip install langchain openai python_dotenv requests newspaper3k

2. Inside your project directory named news-summarizer-python and create a Python file

   

3. Now start writing code. You need to follow along with my code sample below, I've written the documentation alongside.

   #GetNewsArticle.

    import requests  from newspaper import Article   # this function takes article URL and return it in Article object.  # I included a default value for the url, it never matters.  def GetNewsArticle(article_url="https://labs.hakaioffsec.com/nginx-alias-traversal/"):       # Fetches the news article from the given URL       # header object for the request we have make to article url      headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/89.0.4389.82 Safari/537.36'}       # Create a session object that we can use for calling get, post function with that url.      session = requests.Session()        # try catch block      try:              # making the request with url, header, and timeout parameter.              response = session.get(article_url, headers=headers, timeout=10)              if response.status_code == 200:                      # if response comes with no issues, we make the article object                      # it takes the url and does everything we did using request and session on its own.                       article = Article(article_url)                      article.download()                      article.parse()                       # after downloading and parsing the data, we're returning it.                      return article              else:                      # this is just for error checking                      print("Error fetching article url")       # this is also for error checking      except Exception as e:          print("Error occured while fetching the news article")

   #summarizer.py

    from GetNewsArticle import GetNewsArticle  # importing ChatOpenAI (Wrapper around OpenAI Chat large language models) from langchain.chat_models import ChatOpenAI # schema for formatting users input for the model from langchain.schema import HumanMessage # if a .env file, you've to pass OPENAI_API_KEY from dotenv import load_dotenv load_dotenv() # Taking user news article url article_url = input("Enter any article link: ") # Fetch the article we will be summarizing from our article = GetNewsArticle(article_url) # Create a chatbot instance that will summarize the article # verbose for more details to print along # temperature is the randomness the model can play with # 0.0 is only for real facts, 1.0 is for full creativity. llm = ChatOpenAI(model_name="gpt-3.5-turbo", verbose=True, temperature=0.0) # Create the template for the prompt # Crafting prompt is the main thing in LLM workings. # using this template, we're asking the chat model to give us the summary of any long article. template = """You are a smart assistant that summarizes online articles in 100 words. Here's the article you want to summarize. ================== Title: {article_title} {article_text} ================== Write a summary of the given article. """ # Format the prompt with the article details we've got prompt = template.format(article_title=article.title,                          article_text=article.text) # Create a list of messages to send to the chatbot # but we passing only one message in HumanMessage object form. messages = [HumanMessage(content=prompt)] # Send the messages to the chatbot and print the response output = llm(messages) # we can print our summary print("\n" + output.content + "\n")

   Note: Code is available here with a code explanation

   https://github.com/sarkartanmay393/ArticleSummarizer-Python

4. That's it. You can now build an AI-based summarizer. Now let us run our program,

   %[https://vimeo.com/842226438]

Note: You need to create a .env file and put OPENAI_API_KEY

And it's not just a summarizer, it can be more than that. Like, you can get the summarized text in bulleted points. You ask how?

Just tweak the prompt message a little,

template = """You are an advanced AI assistant that summarizes online articles into bulleted lists.Here's the article you need to summarize.==================Title: {article_title}{article_text}==================Now, provide a summarized version of the article in a bulleted list format."""

That is it, now the output will,

Conclusion

In this blog post, we embarked on a journey to explore the world of AI, LLMs (Large Language Models), and the Langchain framework. We discovered the potential of LLMs, particularly the powerful GPT-3, in generating text and even assisting with creative writing. These advancements can revolutionize various industries and change the way we interact with computers.

To simplify the development of applications using LLMs, we introduced the Langchain framework, which provides a range of functions for interacting with LLMs and generating fascinating outputs. Leveraging Python as our coding language, we set out to build an article summarizer using OpenAI's ChatModel in conjunction with Langchain.

By delving into the world of AI and leveraging the Langchain framework, we have taken the first steps toward the future of technology.

So, let's embrace the possibilities that AI, LLMs, and frameworks like Langchain offer, and embark on our journey of innovation and exploration in this fascinating field.

I hope you learned something new today. If you have any questions, feel free to ask me in the comments below. I will be happy to help you. It would be great if you share this article with your developer friends who are new to tech.

For reference, I wrote this article after learning these topics from a course called LangChain & Vector Databases in Production. I highly suggest you join it and learn in-depth about these topics. I've used some code snippets and an image from their coursework.

Follow me on @sarkartanmay393
Direct mail me at sarkartanmay393@gmail.com

Welcome to the fascinating world of binary trees! If you're new to this topic or looking to reinforce your understanding, you're in the right place. In this tutorial, I'll take you on a step-by-step journey through the fundamentals of binary trees. By the end of this article, you'll have a solid understanding of binary trees and feel confident in your ability to tackle more advanced data structures.

Understanding Binary Trees 🎄

Let's start by demystifying binary trees. Don't worry if you're unfamiliar with the concept I'll break it down for you. Binary trees consist of nodes connected by edges, forming a hierarchical structure. Imagine it as a tree with branches spreading out, where each node can have up to two children. Simple, right? We'll illustrate this with easy-to-understand diagrams to make it crystal clear.

A binary tree is a hierarchical structure composed of nodes. Each node can have at most two children: a left child and a right child. The topmost node is called the root, and it serves as the starting point of the tree. The nodes below the root are referred to as internal nodes, while those at the bottom without any children are called leaf nodes.

Now, let's break down the components of a binary tree:

1. Root: The topmost node of the tree, serving as the starting point.

2. Internal Node: Nodes other than the root and leaf nodes, having both a left and right child.

3. Leaf Node: Nodes at the bottom of the tree without any children.

4. Left Child: The node directly below and to the left of another node.

5. Right Child: The node directly below and to the right of another node.

6. Parent: The node above and connected to another node.

7. Subtree: A smaller binary tree within the main binary tree, having its root.

The time complexity of binary tree operations is typically determined by the height in balanced trees, while it is determined by the number of nodes in unbalanced trees. A well-balanced binary tree, such as an AVL tree or Red-Black tree, ensures that the height remains logarithmic. This balance leads to efficient operations, such as insertion, deletion, and searching, all of which have a time complexity of O(logN) in balanced trees.

Now a question arises, Why should one use binary trees over other data structures?

Binary trees excel in scenarios that require efficient searching, sorting, and hierarchical organization. They provide a balance between ordered structure and fast access, making them suitable for a wide range of applications. Compared to linear structures like Linked Lists or Arrays, Binary trees reduce the search space significantly, resulting in faster operations. Additionally, binary trees can handle dynamic data effectively, allowing for efficient insertion and removal of elements.

Use cases of Binary Trees

Prepare to be amazed by the countless applications of binary trees! From efficient searching with binary search trees to evaluating mathematical expressions with expression trees, and even data compression with Huffman coding. All these is being possible by Binary Trees.

Let's explore some common use cases where binary trees shine.

1. Searching and Sorting: One of the primary applications of binary trees is in searching and sorting algorithms. Binary Search Trees (BST), a specific type of binary tree, enable efficient searching for a particular value. They maintain an order where the left child of a node is smaller, and the right child is greater. This property allows for quick comparisons and reduces the search space, resulting in logarithmic time complexity O(logN) for searching operations.

2. Hierarchical Structures: Binary trees are ideal for representing hierarchical relationships. File systems, organization charts, and family trees can all be efficiently modelled using binary trees. The parent-child relationship between nodes simplifies the navigation and management of hierarchical data.

3. Expression Evaluation: Expression trees utilize binary tree structures to represent mathematical expressions. Operators reside in internal nodes, while operands reside in leaf nodes. Expression trees allow for efficient evaluation of complex expressions by performing operations in a specific order, guided by the tree structure.

4. Huffman Coding: Binary trees are central to Huffman coding, a compression technique widely used in data compression algorithms. Huffman trees are constructed based on the frequency of characters in a text, with frequent characters occupying shorter paths in the tree. This results in optimal compression, reducing file sizes while maintaining data integrity.

5. Decision Trees: Binary trees are instrumental in decision tree algorithms used for classification and regression tasks in machine learning. Decision trees divide data based on features at each node, ultimately leading to leaf nodes representing specific classes or regression values. Decision trees are interpretable and enable efficient data classification and prediction.

6. Game Trees: Binary trees serve as the backbone for game trees, used in game theory and artificial intelligence. Game trees represent the possible moves and outcomes in games, facilitating decision-making and strategy development for both players and intelligent agents.

Implementation of Binary Tree 🌱

Let's roll up our sleeves and build our very own binary tree. I'll guide you through the process, step by step. You'll be constructing binary trees like a pro in no time!

In the above Replit code, I coded everything you need to know to get started using Trees. I wrote explanations also with the code itself, you must take a look at it to understand Binary Tree implementation completely.

My DSA with Java Repository: DSA-Java BinaryTrees (Github). Give it a proper look and you may find useful codes and explanations.

Traversing Binary Trees 🎋

Time to explore the exciting world of traversing binary trees. Think of it as navigating through the tree and visiting each node. There are three popular techniques: pre-order, in-order, and post-order traversal.

1. Pre-order traversal is primarily used for obtaining prefix expressions, creating copies of trees, and generating prefix notations. It provides a systematic way of visiting nodes in a binary tree, allowing for various applications in expression evaluation, tree manipulation, and programming.

2. In-order traversal is a valuable technique used for retrieving data in sorted order, evaluating expressions, creating copies of binary trees, and validating the structure of binary search trees. Its properties and applications make it an essential tool in various algorithms and data-processing tasks involving binary trees.

3. Post-order traversal is primarily used for evaluating expressions in expression trees, deleting nodes while maintaining proper dependencies, and generating postfix representations of expressions. Understanding the different traversal orders, including post-order, provides valuable tools for efficiently manipulating and analyzing binary trees.

We'll take a greater look and code each traversal technique in our upcoming post where I'll explain Binary Search Tree aka BST.

Short Note of Binary Search Tree 📝

A Binary Search Tree (BST) is a binary tree with an ordering property where the left child contains smaller values and the right child contains greater values. BSTs enable efficient searching, insertion, and deletion operations with a logarithmic time complexity of O(log n). They are widely used in applications requiring fast searching and sorting, such as dictionaries, symbol tables, and databases. An in-order traversal of BSTs provides a sorted sequence of values. While the efficiency of BST operations depends on maintaining balance, they serve as the foundation for more advanced balanced tree structures. Overall, BSTs offer an ordered and efficient way to store and retrieve data.

I'll be coming up with a new blog very soon, explaining Binary Search Tree and more about Binary Trees.

Conclusion 🙌🏼

Congratulations! You've successfully started the journey through the world of binary trees, and you've done it with confidence and ease. Remember, mastering binary trees takes time and practice, but you're well on your way. Keep exploring advanced topics and challenging yourself with new projects. With each step, you'll enhance your problem-solving skills and become a proficient programmer.

So, go ahead and embrace the power of binary trees. You're now equipped with the knowledge and understanding to tackle complex data structures. Happy coding and enjoy your newfound binary tree prowess!

Follow me on 🐦 @sarkartanmay393
Direct mail me 📧 sarkartanmay393@gmail.com

1. Optimize Images: One of the most common causes of slow website speed is oversized images. To optimize images, compress them using an image compression tool. There are many tools available online that can help you compress images without sacrificing quality.
   Example: Before uploading an image to your website, use an image compression tool such as TinyPNG to compress it. This will significantly reduce the image file size, making it load faster and there are a lot of other ways present also.

2. Minimize HTTP Requests: HTTP requests are made when a browser loads a web page. The more requests a page has to make, the longer it takes to load. To minimize HTTP requests, combine CSS and JavaScript files, and minimize the use of external scripts.
   Example: Combine all your CSS files into one file and all your JavaScript files into another file. This will reduce the number of HTTP requests required to load your website.

3. Use a Content Delivery Network (CDN): A CDN is a network of servers around the world that can help deliver content faster to website visitors. Using a CDN can reduce the distance between your website visitors and your server, resulting in faster load times.
   Example: Cloudflare is a popular CDN that can help improve your website's load speed. By using Cloudflare, your website's content is cached on their servers, and it's delivered to visitors from the server closest to their location.

4. Enable Browser Caching: Browser caching allows your website visitors to store the files of your website on their computer, so they don't have to download them every time they visit your site. This can significantly reduce load times for repeat visitors.
   Example: Use a caching plugin such as WP Super Cache if you're using WordPress. This plugin allows you to enable browser caching, which can significantly reduce load times for repeat visitors. A web developer's guide to browser caching, read this to get an overview of caching. I'll write a tutorial for enabling browser caching in React-based websites.

5. Minimize Redirects: Redirects may introduce extra HTTP requests, potentially slowing down your website. Aim to reduce the number of redirects on your site, ensuring they are properly executed.
   Example: If you've recently updated your website's URL structure, make sure to set up proper redirects. Instead of using multiple redirects, use a single redirect to the new URL.

6. Optimize Your Code: Optimizing your code is crucial to reducing load times. Make sure your HTML, CSS, and JavaScript files are minified, and remove any unnecessary code. You can also use code optimization tools to help with this task.
   Example: Use a code optimization tool such as HTMLMinifier to minify your HTML code. This will remove any unnecessary white space and comments, reducing file size and improving load times. And you can find lots of plugins to use in your React-based websites.

7. Choose a Fast Web Host: Your web host plays a significant role in the speed of your website. Choose a fast and reliable web host that can handle your website's traffic and provide a fast server response time.
   Example: If you're looking for a fast and reliable web host, consider using a managed hosting service such as WP Engine. They offer high-performance hosting optimized for WordPress, with fast server response times. There are other free options like Firebase, Vercel and Netlify.

To wrap things up, boosting your website's loading speed is key to offering an awesome user experience and enhancing your site's performance. By putting the handy tips we've shared into action, you can notably cut down on load times and keep more visitors coming back for more. After all, we create websites for our lovely visitors, so making their experience delightful is what matters most to us!

Thank you for your time 🧡 and hope to see you in my coming articles.

What are the options?

1. MLH Swags + Bonus

   This is the best and easy-to-get swag source of all. For the past 2-3 months, MLH is organising a Global Hack Week event and in that event, you can avail of MLH swags, a Bonus Swag Kit and various other benefits from sponsors of the event. So be sure to check it out.

   Visit: https://ghw.mlh.io

2. DigitalOcean: Hacktoberfest

   You may already know about Hacktoberfest, it runs in the month of September every year. Here you need to perform 4 pull requests at least and if they get merged, you get the opportunity to get Swags or plant a tree with your name. This event is very popular in India and Globally also and it is also easy-to-get.

   Visit: https://hacktoberfest.com

3. Blockstack: Zero to DApp

   In addition to learning a new approach and platform, developers who complete the Zero-to-Dapp tutorial and submit it to app.co get a free limited edition t-shirt.

   Visit: https://docs.stacks.co/docs/build-apps

4. Kong/kong

   Kongs Contributor Program is designed to encourage knowledge sharing amongst Kongs community of users and to recognize and reward the important work our contributors are doing. We have a variety of ways through which you can contribute and earn points towards some amazing Kong swag!

   Visit: https://konghq.com/community/open-source-contribution

... at this time, these four are good to go. And in future, If more comes up, I'll add them somehow. But you have to help by commenting below some more swag sources.

And with that being said, I'm taking off from here, see you on my next blog.

Thank you for your time 🧡

We have been learning new topics with Go, and before getting into DevOps topics in our blogs, today lets build a simple fun project and gear up your algorithms skills a little.

As you know, Sudoku is a board game where there is 81 boxes, each with a specific number between 1-9 or it can be a empty box. We have to fill those empty boxes in a way that no number is repeated in its row, column and in 3*3 area of board. Dont worry, google a little about Sudoku if you dont know what this game is in general.

We know that there is definitely a algorithm we humans use while solving a sudoku pattern in paper, and now we have to just think about that algorithm and try to write it for our computers to understand. It is going to be interesting and brainstorming, so now we should proceed into it.

Prerequisites

You should have a basic knowledge of

• Go
• Recursion (optional)

Lets Begin

Very important part of building a sudoku solver using programming languages is Recursion. Briefly I can say, Recursion is a method where a function calls itself. Lets see an example of that,

// len returns the length of a string using Recursion.func len(str string) int {   // base condition where our function stops.   if str == "" {     return 0   }   // if str = "golang"   substring := str[1:]   // substring = "olang"   // for each len func call, we are increasing the result int by 1 and    //calling the same function by reducing the string length    //by one character (or rune).   return 1 + len(substring)}

Output: len("abcde") = 5

As you can see, len function is returning 1 + (len(substring). If str is not empty then it must have a length of 1 and passing the reduced str to len again, and running until str becomes empty. When it gets empty, it returns 0 where our programme ends.

For better understanding, here is reference link to you. Learn More

Now you understand Recursion a little, so we can just head into our sudoku engine. You can see our program in running state now, go/sudoku-engine.

1. Setup Sudoku Engine Project

1. Create a new folder for your project and name it sudoku-engine.
2. Create a new file in that folder and name it main.go.
3. Run go mod init sudoku-engine in terminal to initialize your project.

2. Writing Engine Code

1. First create a parse function to convert our sudoku pattern into a 2D slice. We will use this slice to solve our sudoku pattern.

   // parseInput converts a string to 2D slice/array. func parseInput(input string) [][]int {     board := [][]int{         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},         {0, 0, 0, 0, 0, 0, 0, 0, 0},     } // dummy board     // scanner is set up to read input string.     scanner := bufio.NewScanner(strings.NewReader(input))     scanner.Split(bufio.ScanRunes)     // go for an iteration over the whole string of length 81.     // if the rune is a number, then we convert it to int and add it to our board.     for row := 0; row < 9; row++ {         for col := 0; col < 9; col++ {             scanner.Scan()             tmp := scanner.Text()             val, err := strconv.Atoi(tmp)             if err != nil {                 // error here             }             board[row][col] = val         }     }     return board }

2. Create isSafe function to check if a number we are trying to put in a box is safe or not.

It means, if the number is already present in its row, column or 3*3 area of the board, then it is not safe to put that number in that box.

// isSafe returns true if it is possible to put n in that (row, col) position in the given board.func isSafe(board *[][]int, row int, col int, num int) bool {  // tasks to do  // 1. check if num is present in row  // 2. check if num is present in col  // 3. check if num is present in 3*3 box  // checking in the row  for _, slice := range *board {    if slice[col] == num {      return false    }  }  // checking in the col  for _, value := range (*board)[row] {    if value == num {      return false    }  }  // checking in the 3*3 box  rowStart := row - row%3 // (row / 3) * 3  colStart := col - col%3 // (col / 3) * 3  for i := rowStart; i < rowStart+3; i++ {    for j := colStart; j < colStart+3; j++ {      if (*board)[i][j] == num {        return false      }    }  }return true}

1. Create a solve function to solve our sudoku pattern. We will use recursion and backtracking in this function to solve our sudoku pattern.

// solve the sudoku board using backtracking.// It returns solved board if the board is solved.func solve(board *[][]int) bool {  // tasks to do  // 1. find the empty cell  // 2. try to put a number in that cell  // 3. check if it is safe to put that number in that cell  // 4. if it is safe then put that number in that cell and solve the rest of the board  // 5. if it is not safe then try another number  // 6. if all the numbers are tried and none of them is safe then return false  row := -1  col := -1  isEmpty := false  // finding the empty cell  for r, slice := range *board {    for c, value := range slice {      if value == 0 {        isEmpty = true        row = r        col = c        break      }     if isEmpty {      break     } // found empty cell.    }  }  if !isEmpty {    return true  } //base condition  // if there is no empty cell then the board is solved  for i := 1; i <= 9; i++ {    // trying to put a number in that cell    // checking if it is safe to put that number in that cell    if isSafe(board, row, col, i) {      // putting that number in that cell      (*board)[row][col] = I      // solving the rest of the board      if !solve(board) {        // if the board is not solved then we need to backtrack         (*board)[row][col] = 0 // backtrack      } else {      // board it solved!      return true      }    }  }// if all the numbers are tried and none of them is safe then return falsereturn false}

1. Create a print function to print our solved sudoku pattern.

// display prints sudoku board on console.func display(board [][]int) {  for _, slice := range board {    for _, val := range slice {      fmt.Printf("%d, ", val)    }    fmt.Println()  }}

Whole function with input taking part in main.go,

import ("bufio""fmt""log""os""strconv""strings")func main() {  fmt.Printf("Enter the name of your file containing the Sudoku Pattern :        ")  scanner := bufio.NewScanner(os.Stdin)  scanner.Scan()  var file string  file = scanner.Text()  if !strings.Contains(file, ".text") {    file = "pattern-1.text"  }  input, err := os.ReadFile(fmt.Sprintf("./inputs/%s", file))  if err != nil {    log.Println(err)  }  board := parseInput(string(input))  fmt.Println("Defined Sudoku Pattern")  display(board)  fmt.Println()  fmt.Println("Engine starts solving the pattern...")  if solve(&board) {  //display(board)  file, err := os.Create(fmt.Sprintf("./outputs/%s", file))  if err != nil {    log.Println(err)  }  for _, slice := range board {    s := ""    for _, v := range slice {      s= s + fmt.Sprintf("%v, ", v)    }    _, err = file.WriteString(fmt.Sprintf("%v\n", s))    if err != nil {      log.Println(err)    }   }  }  fmt.Println(fmt.Sprintf("Result is saved in \"./outouts/%v\"", file))}

3. Performing Tests

1. Git clone this repository: github/SudokuSolver-Golang
2. Follow the steps performed in this video
   https://vimeo.com/763376424
3. Output will be:

    $ go run main.go 3, 1, 6, 5, 7, 8, 4, 9, 2,  5, 2, 9, 1, 3, 4, 7, 6, 8,  4, 8, 7, 6, 2, 9, 5, 3, 1,  2, 6, 3, 4, 1, 5, 9, 8, 7,  9, 7, 4, 8, 6, 3, 1, 2, 5,  8, 5, 1, 7, 9, 2, 6, 4, 3,  1, 3, 8, 9, 4, 7, 2, 5, 6,  6, 9, 2, 3, 5, 1, 8, 7, 4,  7, 4, 5, 2, 8, 6, 3, 1, 9,

4. You can see the test here also, go/sudoku-engine

4. Issue to Solve for you

Solve a project issue now: sudoku-engine/issue#1

If you never tried open source, this is a good place to start and learn more of Git and Github.

Final Note

I hope you learned something new today. You can also contribute to this project. If you have any questions, feel free to ask me in the comments below. I will be happy to help you. It would be great if you share this article with your developer friends who are new to tech.

Follow me on: @sarkartanmay393
Direct mail me: hello@tanmaysarkar.tech

Today I am back with a new article on Go, but this is slightly in production side. Today we are going to deploy a simple containerized Go web app on Heroku. I am assuming, you are well aware about how web servers are created in Go and how they work. That means, now you have created your own web server which can serve us a website locally, but you want to make it available for public with a target url right. How do you do that ? The answer is been coveyed in this blog. So start reading and following the learn more links to get to the topics that are neccessary for our deployment process.

What we will be doing today ? 🧐

1. Write a Dockerfile
2. Write a heroku.yml
3. Deploy our app on Heroku.

Prerequisite

1. Go
2. Docker

Let us not waste time

If you don't know much about Docker then you should learn about it now and for today's tutorial you can just install docker and play it with a little. I have suggested a video for a brief understanding about Docker, check that out definitely. Now, I am supposing you are aware of Docker and have basic knowledge of it. Because we are writing a Dockerfile and if you don't even know what Docker is, then it's not possible.

I can give you a brief introduction of Docker. It is like a virtual machine manager, that takes hardware resource from your machine and creates one or more hidden boxes inside your computer with those resources and provide a operating system in each of those boxes, this boxes are running containers, and this containers are created in docker from docker images. The goal of Docker is to make multiple isolated virtual machines in your computer.

Learn basics of Docker before proceeding. Learn More

Here is my project structure

 main.go go.mod go.sum Dockerfile [yet to write] heroku.yml [yet to write] static     index.html

Visit my Github repo of this project to get the idea, How I build a web server and used HTML with it.

We can go forward now

We will start by writing Dockerfile and end with installing Heroku CLI to deploy our app effortlessly. This will be knowledge journey and I hope after this, you will understand the deployment process of Go web applications.

• Write Dockerfile

A Dockerfile is a declarative way of automatically creating a docker image, and we will create a alpine linux container where the app will run at a specific port.

Create a file with this name and follow my code below,

# Using this minimal base image for our work.# It will consist of golang tool and nothing much more.FROM golang:1.19-alpine# Setting our current directory to /app inside our base image.WORKDIR /app# Copying everything from our Web-Server directory to our image's /app directory.COPY . .# Now all are necessary files are in our image.# So we can start our commands to run the go app.RUN go mod downloadRUN go build main.go# We have now a executable, named main.# First let us expose our docker container.# EXPOSE 8080# We hosting our app on heroku, and heroku provides a particular port# for us via enironment variables. And so here docker, we don't need to # our container, heroku takes care of it.# Run the executable.CMD ./main

Now with a simple command docker build -t webapp . you will get your fully functional docker image.

• Write heroku.yml

This file is nothing but a declarative way of saying Heroku that here is how you build and how you run the app. This file is necessary for deploying a containerized app in Heroku automatically. This file is a manifest that defines our app and allows us to specify add-ons and config vars to use during app provisioning by Heroku.

build:  docker:   web:    Dockerfilerun:  web:   ./main

• Getting started with Heroku

Heroku is a platform as a service (PaaS) that enables developers to build, run, and operate applications entirely in the cloud. We will host our application, and it will generate a URL to our deployed app. There is three different ways of deploying on Heroku.

• Using GitHub
• Heroku CLI
• Container Registry

Now in this tutorial, we are gonna push our whole directory to heroku via git, and heroku.yml starts it's work to help heroku by telling how to build and run the application. Don't worry, I am gonna explain everything, but there is little task for you. Go and complete the mentioned prerequisite.

• Create account in Heroku
• Learn more about Heroku Deployments from Docs
• Install Heroku CLI

• Using Heroku to Deploy

Now we are all set to host our application front public, open project directory and follow my lead.

Our Git status is clean and all necessary files are committed safely on Git. Now, let us use heroku cli tool to do the rest of work. But before that, go to your heroku dashboard and create a app with your preferred name.

Now, switch to terminal and command,

heroku login

It will take you to a website and demands for login. After successfully login, we have set our heroku app stack to container. Here is how,

heroku stack:set container -a sample-web-09

With -a alias, we specify the app that we have created earlier. Change the sample-web-09 to your app name.

Now we need to add git remote of heroku by,

heroku git:remote -a sample-web-09

This is the final look after these three commands.

And, now the final command, but before this you may have to command git add . and git commit -s -m "last push 101" and then,

git push heroku main

To directly open up your web application link,

heroku open

Checkout the app that I deployed in this blog, here is sample-web-09.

Finishing note 📝

Hope you understood the process of deploying a Go based web server. I always try to explain things in detail, and still if you find any position where I should explain more then tell me in comments or ping me on twitter. I am always there for my article people.

Thanks you everybody for being with me. New blogs on Go and other techinal topics are yet to come out in upcoming days. So stay tuned, and follow me here on @Hashnode.

Follow me on 🐦 @sarkartanmay393

Mail me anytime on 📬 hello@tanmaysarkar.tech

Hi fellows, I have been learning about web scrapers these days and saw there is no many articles on Go itself on Hashnode and building a scraper with it is far away. So I thought about let's build a simple cli tool which will take birthday as input and crawl a tiny part of the website IMDB to look for celebrities who born on that date and stores various data into a file outputs/mm-dd.json inside output directory.

If you do not know about IMDB, it is a website which contains information about movies, TV shows, and celebrities. It is a very popular website and has a huge database of information. So, it is a good place to start learning about web scraping.

As you might not know, building a web scraper consists of four main steps :

1. Crawling : Crawling is the process of finding all the links on a website and adding them to a queue.
2. Scraping : Scraping is the process of extracting data from a website.
3. Parsing : Parsing is the process of converting the raw data into a structured format.
4. Storing : Storing is the process of saving the data into a file or database.

And we will look deep into all of them step by step, so sit tight and let's get started.

Prerequisites 👩🏻🏫

• Go
• JSON

And also you should have basic knowledge of crawlers and scrapers in general. Learn here

Let us begin then

1. Project Setup

At first, Create a directory and make a main.go file inside and open terminal in that directory to command :

go mod init github.com/sarkartanmay/IMBD-Scraper-Golang

I am using my GitHub repo link in my module name as a convention, so you should change that to your desired module name. E.g. github.com/username/project-name and you can learn more about Modules in Go here.

Then, install colly by commanding in terminal :

go get github.com/gocolly/colly

And this is how our project setup is done, let us now go straight into coding our web scraper.

2. Coding the Scraper

I hope as you write the code, your code editor will import the packages that are necessary.And with that, let us start coding our scraper.

First, we will make customized structs that will hold our scraped data in program memory. We will make a struct for each type of data we want to scrape. For example, we will make a struct for each celebrity and a struct for each movie.

//  Star holds a celebrety's name and other imdb information.type Star struct {    Name      string  `json:"name"`    Photo     string  `json:"photo"`    JobTitle  string  `json:"job_title"`    Birthdate string  `json:"birthdate"`    Bio       string  `json:"bio"`    TopMovies []Movie `json:"top_movies"`}// 🎦 Movie holds basic information about a movie.type Movie struct {    Title   string `json:"title"`    Release string `json:"release"`}

This is our two structures that will hold our scraped data. We will use the json package to convert our data into JSON format. We will use the json tag to specify the name of the key in the JSON object, and to tell Go that when we convert our struct into a JSON object, it must name them as we specified in the json tag.

For example, the Name field in the Star struct will be converted to the name key and TopMovies to top_movies in the JSON object.

Now, we will create function that will take two parameters (day, month) to crawl into IMDB with specified date and return scraped data in a slice of stars []stars .

// 🕷 Crawl crawls the imdb website for a given page and returns a list of stars.func Crawl(month int, day int) []Star {    // stars will hold all the scraped data.    var stars []Star    // Collectors are used to scrape data from a website.    // Main collector instance is initialized here.    // Our main collector c will only look for profile links of celebreties    // and go on next pages if there is next button on current page.    c := colly.NewCollector(        // Colly collector will only visit the given domains.        colly.AllowedDomains("www.imdb.com", "imdb.com"),        // Colly collector stores cache and uses from here.        colly.CacheDir("./.imdb_cache"),        // Cache is disabled by default, so we need to enable it.    )    // ic stands for Info Collector, is just cloned from main collector.    // This collector will be used to scrape data from the info page.    // It will go inside a profile link and look for data in that only.    // This collector will be on work everytime main collector find a new profile link.    ic := c.Clone()    // Where crawler sees 'mode-detail' class attribute, it will call back the function.    c.OnHTML(".mode-detail", func(e *colly.HTMLElement) {        // Getting the profile url from the href attribute, it is goquery selector string.        // 'div.lister-item-image > a' is path of profile url in HTML.        // At fist there is a Div tag with class 'lister-item-image' and        // inside that there is a tag 'a' where href attribute is profile url.        profileUrl := e.ChildAttr("div.lister-item-image > a", "href")        // But the profile url is relative, so we need to add the base url.        // profileUrl is "/name/nm0000123/" corrently.        profileUrl = e.Request.AbsoluteURL(profileUrl)        // Now profileUrl is "https://www.imdb.com/name/nm0000123/".        // Asking info collector to visit the profile url.        ic.Visit(profileUrl)    })    // This crawler function gets into next page, if there is one.    // On IMDB, there is a button with text 'Next' on each page with    // class 'lister-page-next next-page' and we are looking only for that.    c.OnHTML("a.lister-page-next", func(e *colly.HTMLElement) {        // Getting the next page url from the href attribute.        nextPageUrl := e.Attr("href")        nextPageUrl = e.Request.AbsoluteURL(nextPageUrl)        // Asking main collector to visit the next page url.        c.Visit(nextPageUrl)    })    // This info collector crawler function gets the information of the celebrety inside the 'profileURL' page.    ic.OnHTML("#content-2-wide", func(e *colly.HTMLElement) {        // Getting all details using info collector and storing them in a Star struct.        temporaryStar := Star{            Name:      e.ChildText("h1.header > span.itemprop"),            Photo:     e.ChildAttr("#name-poster", "src"),            JobTitle:  e.ChildText("#name-job-categories > a > span.itemprop"),            Birthdate: e.ChildAttr("#name-born-info > time", "datetime"),            Bio:       strings.TrimSpace(e.ChildText("#name-bio-text > div.name-trivia-bio-text > div.inline")),            TopMovies: []Movie{},        }        // Now iterating over all the top movies of the profile url page.        e.ForEach("div.knownfor-title", func(_ int, el *colly.HTMLElement) {            temporaryStar.TopMovies = append(temporaryStar.TopMovies, Movie{                Title:   el.ChildText("div.knownfor-title-role > a.knownfor-ellipsis"),                Release: el.ChildText("div.knownfor-title > div.knownfor-year > span.knownfor-ellipsis"),            })        })        // Now appending the temporaryStar to the stars slice.        stars = append(stars, temporaryStar)    })    // Printing text of every request made on info collector.    ic.OnRequest(func(r *colly.Request) {        fmt.Println("Visiting:    ", r.URL.String())    })    // Our link to crawl with user specified date and month.    startUrl := fmt.Sprintf("https://www.imdb.com/search/name/?birth_monthday=%d-%d", month, day)    fmt.Println("Starting crawling into    ", startUrl)    // Starting the main collector.    c.Visit(startUrl)    // Returning all the scraped data in a slice of stars.    return stars}

If you are overwhelmed by the code, don't worry. As I always write lots of comments in code, if you read them, you will understand what is going on.

Let's break down one important part of scraping.

• HTML Structure Recognition and Query String 🧵🔎

Q. How we can recognize the HTML structure of a website ?

A. There are two ways to do that. One is to use the browser's developer tools manually and the other is to use the colly package's InspectElement function.

I like the first one, manually going through web pages and finding the HTML elements. Searching in HTML is kind of a skill itself, you need to know how to find the right element in HTML.

Let us see how we do that ?

Here is video that I made for you to understand how to find right HTML component of a website using query strings.

Now, we can go further into the code. Let us now deal with the []Star slice that we are returning from the Crawl function.

// main takes two arguments, month and day. // Parse them into integers and call Crawl function.// Crawl function returns a slice of stars.// That slice is converted into JSON object.// And then writes the object into a file.func main() {    // Printing prompt for user to input values.    var input string    // User should type just like this. e.g.. 12-23    fmt.Println("Type a date and month in (MM-DD) format:    ")    _, err := fmt.Scanf("%s", &input)    if err != nil { // Error handling        // Fatalln will print the error and exit the program.        log.Fatalln("Unable to read input because    ", err)    }    // Splitting the input into month and day.    sliceOfInput := strings.Split(input, "-")    // Converting the month into integer.    month, err := strconv.Atoi(sliceOfInput[0])    if err != nil {         log.Fatalln("Unable to convert month into int because    ", err)    }    // Converting the month into integer.    day, err := strconv.Atoi(sliceOfInput[1])    if err != nil {         log.Fatalln("Unable to convert date into int because    ", err)    }    // Crawling the website with the given month and day and stores the data into a slice of Star.    // Calling Crawl func with month and day.    // Storing the returned slice of Star into stars variable.    stars := Crawl(month, day)    // To print all the scraped data in terminal.    // This code part is commented out, because it is not necessary.    // You can play with it to show the scraped data on terminal.    // encoder := json.NewEncoder(os.Stderr)    // encoder.SetIndent("", "   ")    // err := encoder.Encode(stars)    // Converting Star struct into JSON.    data, err := json.MarshalIndent(stars, "", "    ")    if err != nil {        log.Println("Error while marshalling JSON: ", err)    }    // Writing JSON into a file called 'mm-dd.json'.    // file name will vary according to the month and day.    // We are using os pkg to write the file.    fileName := fmt.Sprintf("%s.json", input)    err = os.WriteFile(fmt.Sprintf("./outputs/%s", fileName), data, 0644)    if err != nil {        if err.Error() == "open ./outputs/12-23.json: no such file or directory" {            log.Println("Making directory named outputs.")            os.Mkdir("./outputs", 0755) // making a directory.            err := os.WriteFile(fmt.Sprintf("./outputs/%s", fileName), data, 0644)            if err == nil {                log.Fatalln("Web is scraped and JSON file is created.")            }        }        log.Fatalln("Error while writing JSON: ", err)    }    log.Println("Web is scraped and JSON file is created.")}// This is where our code ends.

For the whole code, you can check out the IMDB-Scraper-Golang repository.

3. Testing our Scraper

Now, we run our program to test it. We will run it with the date and month of my birthday. So, I will run the main.go then type 12-23 and hit enter.

Conclusion :

This was a simple yet interesting scraper project. We scraped the IMDB website and got the data of all the celebrities who were born on the same day as me. We used colly package to scrape the website. We also used the os package to write the scraped data into a file.

I hope you liked this blog and learned something new. If you have any inquiry or suggestions, feel free to tell me in the comments section or on Twitter. I will be very happy to get connected with you.

A special thanks to Akhil Sharma, I learned from him.

Follow me on @sarkartanmay393

Go is a statically typed, compiled programming language and useful for carrying out programming for scalable servers and large software systems. Golang programming language was built to fill in the gaps of C++ and Java that Google came across while working with its servers and distributed systems.

On the other hand, PostgreSQL, also known as Postgres, is a free and open-source relational database management system emphasizing extensibility and SQL compliance and used as the primary data store or data warehouse for many web, mobile, geospatial, and analytics applications.

And today we will interact with a running database using this very language, and print out every step change in database in our terminal.

Prerequisites

You should have a basic knowledge of

• Go
• PostgreSQL

Lets Begin

1. Setup Postgres on Local Machine

Firstly, download PostgreSQL for you specific operating system. Mac users can just use this postgresapp, it is very light and easy to use.

After downloading, install the software and open up to start your server.

PostgreSQL server runs on port 5432 by default.

Now, download and install a visual database viewer and editor like Postico or even DBeaver.

Pestico is very easy to use but paid, still I like to use it in free version.

Our server is now running in background, now we can start building a Go program to interact with the running database. In my case the name of the database is test-database. You can create new database in Pestico itself and I hope you are familiar with databases.

2. Setup Go Project

Open a new folder in your favorite code editor and create a main.go file containing the code

package mainfunc main() {    // Connect to database    // Test Connection    // View Rows    // Insert Row    // View Rows    // Update Row    // View Rows    // Delete Row    // View Rows    // View Row by ID}

And hit the command on terminal to initiate a module with module name and here I put my GitHub repository link as a convention.

go mod init github.com/sarkartanmay393/Go-PostgreSQL-Test

As you can see our main function with a lot of areas with specified tasks, we are going to connect database with our program then test it, view its data, perform insert, delete, update and view its data again to see the changes on every step in the specific table of our database.

3. Writing Code

• Now, lets connect our database with Go program

  // Connect to databasedsn := "host=localhost port=5432 dbname=test-database user=postgres password=" // replace fields with your own database credentialsconn, err := sql.Open("pgx", dsn) // Opening connection to database with the driver 'pgx'if err != nil { // Error handling  log.Fatalf("Unable to open database: %v\n", err)}defer conn.Close() // This db connection will stop when our program finishes not before that.

  For using pgx driver, we have to import _ "github.com/jacke/pgx/v5/stdlib" in our function.

• Time to test our connection with program

  // Test Connectionerr = conn.Ping() // Ping function will test the connection and return error if any problem occurs.if err != nil { // Error handing  log.Fatalln("Unable to ping database: ", err)}log.Println("Pinged Database!")

• For viewing all rows of our table from database, lets create a function so that we can use it repeatedly.

// viewRows prints all available information from database.func viewRows(conn *sql.DB) error {    // Remember to use `` instead of '' or "" for multiline strings    // Query to select all rows from table, SQL language is used here.    query := `SELECT id, first_name, last_name, email FROM users ORDER BY id;`     rows, err := conn.Query(query) // Query function will execute the query and return rows and error if any.    if err != nil { // Error handling        log.Println("Unable to query result.")        return err    }    // We will use defer to close the rows after the function finishes.    defer rows.Close()    var id int8    var first_name, last_name, email string    for rows.Next() { // Looping through all rows        // scanning the rows and storing in variables        err := rows.Scan(&id, &first_name, &last_name, &email)        if err != nil {            log.Println("Something went wrong while scanning rows.")            return err        }        fmt.Printf("ID: %v, FIRST: %v, LAST: %v, EMAIL: %v\n", id, first_name, last_name, email)    }    fmt.Println("----------------------------------")    if err = rows.Err(); err != nil { // Rechecking for error in rows.        log.Println("Something went wrong while iterating over rows.")        return err    }    return nil}

• Now we can see our data with a just viewRows(conn) function call. But I am writing some extra codes with it to get better look in our terminal.

  // View Rowsfmt.Printf("Initialising with these two entries.\n")err = viewRows(conn) // Only call the function to view rows.if err != nil {  log.Println("Unable to view database: ", err)}

• Now we can see data from database in terminal, hit command go run main.go and as because I had put a row with my own name, it is shown in terminal.

1. How can we insert a row in our database ?

INSERT INTO users (first_name, last_name, email) VALUES ($1, $2, $3)

1. How can we update a row in our database ?

UPDATE users SET email = $1 WHERE id = $2

1. How can we delect a row in our database ?

DELETE FROM users WHERE id = $1

These are SQL codes to perform tasks in database, and $1 or $2 are placeholders, we have to populate these placeholders with appropriate values. Just like these,

• Lets see how can we insert a row in database in Go

  // Insert Rowquery := `INSERT INTO users (id, first_name, last_name, email) values ($1, $2, $3, $4)`// query is the SQL statement to perform in database using GO_, err = conn.Exec(query, 3, "Amit", "Nandi", "amit@ac.org") // Executing a query with 4 placeholders.if err != nil { // Error handling  log.Println("Unable to insert into database: ", err)}

As you can see, we can execute any query just calling conn and all other tasks are shown in below main.go, try to understand reading the comments.

package mainimport (    "database/sql"    "fmt"    "log"    _ "github.com/jackc/pgx/v5/stdlib" // to just use the driver name `pgx`, we have to import it.)func main() {    // Connect to database    dsn := "host=localhost port=5432 dbname=test-database user=postgres password=" // replace fields with your own database credentials    conn, err := sql.Open("pgx", dsn)                                              // Opening connection to database with the driver 'pgx'    if err != nil {                                                                // Error handling        log.Fatalf("Unable to open database: %v\n", err)    }    defer conn.Close() // This db connection will stop when our program finishes not before that.    // Test Connection    err = conn.Ping() // Ping verifies a connection to the database is still alive, establishing a connection if necessary.    if err != nil {   // Error handling        log.Fatalln("Unable to ping database: ", err)    }    log.Println("Pinged Database!")    // View Rows    fmt.Printf("Initialising with these two entries.\n")    err = viewRows(conn) // Only function call to view all data in terminal.    if err != nil {      // Error handling        log.Println("Unable to view database: ", err)    }    // Insert Row    query := `INSERT INTO users (id, first_name, last_name, email) values ($1, $2, $3, $4)`    // query is the SQL statement to perform in database using GO    _, err = conn.Exec(query, 3, "Amit", "Nandi", "amit@ac.org") // Executing a query with 4 placeholders.    if err != nil {                                              // Error handling        log.Println("Unable to insert into database: ", err)    }    // View Rows    err = viewRows(conn) // View all data again.    if err != nil {      // Error handling        log.Println("Unable to view database: ", err)    }    // Update Row    query = `UPDATE users SET email = $1 WHERE id = $2`    // query to update something in our database, where id matches with our given id.    _, err = conn.Exec(query, "deba@slg.org", 2) // Executing query with 2 placeholders, email and id.    if err != nil {                              // Error handling        log.Println("Unable to update database: ", err)    }    // View Rows    err = viewRows(conn) // Viewing all rows again.    if err != nil {      // Error handling        log.Println("Unable to view database: ", err)    }    // Delete Row    query = `DELETE FROM users WHERE id = $1`    // query to delete a specific row fromd database where id matches.    _, err = conn.Exec(query, 1) // Executing the specific query to delete thw row whose id is 1.    if err != nil {              // Error handling        log.Println("Unable to execute delete query.")    }    // View Rows    viewRows(conn)    // View Row by ID    var id int    var first_name, last_name, email string    // declaring some variables to hold data and will come from database    query = `SELECT id, first_name, last_name, email FROM users WHERE id = $1`    // SELECT will show us data like id, first, last and email FROM a table called users WHERE id is placeholder.    err = conn.QueryRow(query, 3).Scan(&id, &first_name, &last_name, &email)    // QueryRow executes a query that is expected to return at most one row.    // We are scaning our returned row to extract the values that came.    fmt.Println("QUERYROW \t", first_name, last_name, email) // Printing everything in terminal.    // Close Connection}func viewRows(conn *sql.DB) error {    // Remember to use `` instead of '' or "" for multiline strings    // Query to select all rows from table, SQL language is used here.    query := `SELECT id, first_name, last_name, email FROM users ORDER BY id;`    rows, err := conn.Query(query) // Query function will execute the query and return rows and error if any.    if err != nil {                // Error handling        log.Println("Unable to query result.")        return err    }    // We will use defer to close the rows after the function finishes.    defer rows.Close()    var id int8    var first_name, last_name, email string    // Rows is like a list of row and looping through it to extract values from all indivisual row.    for rows.Next() { // Looping through all rows        // scanning the rows and storing in variables        err := rows.Scan(&id, &first_name, &last_name, &email)        if err != nil {            log.Println("Something went wrong while scanning rows.")            return err        }        fmt.Printf("ID: %v, FIRST: %v, LAST: %v, EMAIL: %v\n", id, first_name, last_name, email)    }    fmt.Println("----------------------------------")    if err = rows.Err(); err != nil { // Rechecking for error in rows.        log.Println("Something went wrong while iterating over rows.")        return err    }    return nil}

I have created a repository on GitHub, Go-Postgres-Test where you can visit can check out whole project.

And now the final terminal output :

Final Note

Learning to interact with databases through Go is useful in these days when Go is getting popular every day and more people are using it for there server side works. I am making a Room Reservation project where it needs to talk with database to parse and manipulate data, and from there I have learnt using database query statements in Go.

Hope that you have learnt something new today. If you like my presentation through this article, it would be great if you share this article with your developer friends who are new to Go and databases.

Follow me on: @sarkartanmay393

What is Argo CD ?

Argo CD is a Kubernetes-native continuous deployment (CD) tool. Unlike external CD tools that only enable push-based deployments, Argo CD can pull updated code from Git repositories and deploy it directly to Kubernetes resources. It enables developers to manage both infrastructure configuration and application updates in one system.

Argo CD offers the following key features and capabilities:

• Manual or automatic deployment of applications to a Kubernetes cluster.
• Automatic synchronization of application state to the current version of declarative configuration.
• Web user interface and command-line interface (CLI).
• Ability to visualize deployment issues, detect and remediate configuration drift.
• Role-based access control (RBAC) enables multi-cluster management.
• Single sign-on (SSO) with providers such as GitLab, GitHub, Microsoft, OAuth2, OIDC, LinkedIn, LDAP, and SAML 2.0
• Support for webhooks triggering actions in GitLab, GitHub, and BitBucket.

With fewer words, we can say Argo CD is a GitOps agent that maintains the desired state of a cluster.

Let me explain, suppose there is an application you built, and you want it to run on a cluster with 3 Pods and 2 Services for external IPs. This is the desired state you want and whenever changes happen in your application's repository or if at any moment, your desired state is not present in that cluster, you need to check and do some changes to the cluster to get your desired state running, this is a manual process.And here comes Argo CD to automate the whole process with GitOps principles where we use Git as the data provider of that application and cluster state.

How does Argo CD work ?

As we know, Agro CD ensures that the cluster is running in the desired state at all times. Then a question rises how does it work then ?

It's simple, there will be two repositories, one holding the application you build and the other one holding the declaratively written desired state of the cluster. Argo CD as a Kubernetes controller can be installed in the same cluster where our application is or it can also perform its tasks from an external cluster. It checks whether the current state of the running cluster matches with the written desired state, if it doesn't then Argo CD will try to make the desired state and synchronize everything.

For a better understanding of its architecture, visit Argo CD Documentation.

When Argo CD checks that the live and desired states are different, it shows OutOfSync in its UI, yes it does have a UI as well as CLI, whatever you wish to use. And if it matches then Everything is fine, its status return Synced.

It supports plain Kubernetes manifests, Helm charts, Kustomize definitions, and other templating mechanisms.

By default, Argo CD looks for any changes in our repository every 3 minutes. If there are any changes in our repositories, Argo CD will update our cluster with the new changes we desired.But if we want Argo CD to look for changes every second, we can do that too and if you don't like this mechanism of checking for changes then you can use webhooks too where Argo CD receives notifications from your git provider and when it receives a notification from your application's repo, it will look for changes.

Check core concepts of Argo CD from this link core-concepts.

Prerequisites

1. Basic knowledge of Git, K8s, and Networking.
2. Tools - kubectl, minikube (optional), docker (optional).

Implementation of Argo CD

• Q. What will we be doing today ?
• A. Deploy a Docker application in a Kubernetes cluster which is declaratively described in plain Kubernetes manifest files in a Github Repo.

In this article, I have used this repo Webserver-CI-CD-Golang

For this demo, I am using Minikube for a local cluster. Or you can create your Argo CD setup in Codefresh Course.

• Installation of Argo CD

Argo CD can be used to deploy and manage other applications, do not forget that it is also an application by itself. There are many ways to install it on a Kubernetes cluster.

For the experiment, You can deploy Argo CD by directly using the manifest,

kubectl create namespace argocdkubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

For a production setup, we suggest you use Autopilot, a companion project that not only installs Argo CD but commits all configuration to git so Argo CD can manage itself using GitOps.

For more traditional installation options, the manifests directory offers different modes of installation with or without High Availability. The installs marked as namespace allow you to install Argo CD on a single namespace without cluster-wide privileges. This is best when you want to install Argo CD on one cluster but have it manage other external clusters.

See all resources created by Argo CD -

kubectl get all -n argocd

• Expose Argo CD UI

By default, Argo CD is only accessible from within the cluster. To expose the UI you can utilize any of the standard Kubernetes networking mechanisms such as -

1. Ingress (recommended for production)
2. Load balancer (affects cloud cost)
3. NodePort (simple but not very flexible)

Once the external URL is ready, you need to decide how users will access the Argo CD UI. There are mainly two approaches:

1. Use a small number of local users. Authentication is handled by Argo CD itself. Best for very small companies (e.g. 2-5 people).
2. Use an SSO provider. Authentication is handled by the provider. Ideal for companies and large organizations.

To expose using Ingress,

kubectl port-forward svc/argocd-server -n argocd 8085:443

Now, visit https://localhost:8085 for Argo CD UI.

For admin username and password,Open another tab in the terminal and command -

kubectl -n argocd get secret argocd-initial-admin-secret -o jsonpath="{.data.password}" | base64 -d > admin-pass.txt

Now, login to the admin account -

• Creating and Syncing Application in Argo CD

An application can be created in Argo CD from the UI, CLI, or by writing a Kubernetes manifest that can then be passed to kubectl to create resources.

Using UI

1. Click on NEW APP

2. Put only necessaries as mentioned below

• General

  Application Name: demo-app

  Project Name: default

• Source

  Repository URL: https://github.com/sarkartanmay393/Sample-GO-WebApp

  Revision: HEAD

  PATH: .

• Destination

  Cluster URL: https://kubernetes.default.svc

  namespace: default

YAML File -

apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:  name: demo-appspec:  destination:    name: ''    namespace: default    server: 'https://kubernetes.default.svc'  source:    path: .    repoURL: 'https://github.com/sarkartanmay393/Sample-GO-WebApp'    targetRevision: HEAD  project: default

Check your YAML by hitting EDIT AS YAML and try to understand.

Now, Press on Create and You will see this on screen.

Now, click on SYNC and Proceed without any change.

Using CLI

1. Install Argo CD CLI

Run the following command -

For Linux :

curl -sSL -o /usr/local/bin/argocd https://github.com/argoproj/argo-cd/releases/latest/download/argocd-linux-amd64chmod +x /usr/local/bin/argocd

For Mac :

brew install argocd

I am using this mac command as I am working with a local cluster.Use this reference to install Argo CD CLI.

Argo CD CLI Help Screen -

2. Login Argo CD CLI

Run the command -

argocd login localhost:8085

Then continue with y and,Username: adminPassword: check your admin-pass.txt

I am login into our localhost server because it is the argocd-server that was port forwarded earlier.For Further reference, tap here.

3. Create an app in CLI

Run the following command -

argocd app create demo-app --project default --repo https://github.com/sarkartanmay393/Sample-GO-WebApp --path . --dest-namespace default --dest-server https://kubernetes.default.svc

This will be output -

Some other Argo CD commands -

  argocd app list    #See all apps in Argo cdNAME      CLUSTER                         NAMESPACE  PROJECT  STATUS     HEALTH   SYNCPOLICY  CONDITIONS  REPO                                                 PATH  TARGETdemo-app  https://kubernetes.default.svc  default    default  OutOfSync  Missing  <none>      <none>      https://github.com/sarkartanmay393/Sample-GO-WebApp  .       argocd app get demo-app    #Get demo-app infoName:               demo-appProject:            defaultServer:             https://kubernetes.default.svcNamespace:          defaultURL:                https://localhost:8085/applications/demo-appRepo:               https://github.com/sarkartanmay393/Sample-GO-WebAppTarget:             Path:               .SyncWindow:         Sync AllowedSync Policy:        <none>Sync Status:        OutOfSync from  (2ee5cba)Health Status:      MissingGROUP  KIND        NAMESPACE  NAME               STATUS     HEALTH   HOOK  MESSAGE       Service     default    sample-service     OutOfSync  Missing        apps   Deployment  default    sample-deployment  OutOfSync  Missing

4. Sync app in CLI

As this app is only created, the default sync status will be OutOfSync. To perform synchronization run the command -

  argocd app sync demo-app    #Sync demo-appTIMESTAMP                  GROUP        KIND   NAMESPACE                  NAME    STATUS    HEALTH        HOOK  MESSAGE2022-08-18T14:13:47+05:30   apps  Deployment     default     sample-deployment  OutOfSync  Missing              2022-08-18T14:13:47+05:30            Service     default        sample-service  OutOfSync  Missing              2022-08-18T14:13:48+05:30            Service     default        sample-service  OutOfSync  Missing              service/sample-service created2022-08-18T14:13:48+05:30   apps  Deployment     default     sample-deployment  OutOfSync  Missing              deployment.apps/sample-deployment created2022-08-18T14:13:48+05:30   apps  Deployment     default     sample-deployment    Synced  Progressing              deployment.apps/sample-deployment created2022-08-18T14:13:48+05:30            Service     default        sample-service    Synced  Healthy                  service/sample-service createdName:               demo-appProject:            defaultServer:             https://kubernetes.default.svcNamespace:          defaultURL:                https://localhost:8085/applications/demo-appRepo:               https://github.com/sarkartanmay393/Sample-GO-WebAppTarget:             Path:               .SyncWindow:         Sync AllowedSync Policy:        <none>Sync Status:        Synced to  (2ee5cba)Health Status:      ProgressingOperation:          SyncSync Revision:      2ee5cba61a4446ec05241edce5f781a7f4aa26dePhase:              SucceededStart:              2022-08-18 14:13:47 +0530 ISTFinished:           2022-08-18 14:13:48 +0530 ISTDuration:           1sMessage:            successfully synced (all tasks run)GROUP  KIND        NAMESPACE  NAME               STATUS  HEALTH       HOOK  MESSAGE       Service     default    sample-service     Synced  Healthy            service/sample-service createdapps   Deployment  default    sample-deployment  Synced  Progressing        deployment.apps/sample-deployment created

Now the deployed application is synced.

This will be the final view of our deployed application in the Argo CD User Interface.

• Access the Deployed App

Now check for services running under default namespace in kubernetes.Run the command -

 kubectl get services    #Get cluster services under default namespace.NAME             TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGEkubernetes       ClusterIP   10.96.0.1     <none>        443/TCP          40dsample-service   NodePort    10.96.22.35   <none>        8080:31000/TCP   17m

You need to port forward the sample-service to access the app.Run the command -

kubectl port-forward svc/sample-service 8086:8080

Now, visit http://localhost:8086 to see.

This is the Argo CD Beginners Article. Hope that you have learned something new. See you in my next article.

PS. Learning About Agro CD from Codefresh.For this article, I have taken references from Codefresh.