Automate your tasks - crewAI tutorial for beginners

Hi folks!

The past couple of weeks have been wild. I've been testing some popular multi-agent frameworks and the potential use cases are limitless. I'm getting a billion-dollar idea, every second.

Today, I'm writing this to share everything you need to know to get started with a very popular multi-agent framework: crewAI. I will skip a few things here and there to make this a distilled tutorial going over the key concepts and essentials to get you started.

🍿 Feel like watching instead?

What is crewAI?

crewAI is a framework created by João Moura. It is designed to build a team of AI agents that work together to complete tasks. It is built on top of LangChain and provides an intuitive easy-to-use API.

Why do I need agents?

Let's say you're a blogger and spend lots of time between researching and writing content. Could you automate the process and save countless hours?

Or maybe, you're an SEO specialist and waste hours generating and interpreting reports. Is there a way to automate this so that you can focus on the strategic aspects of SEO?

The answer to both questions is: Yes, there is.

While at the infancy stage, agents have the potential to replace many real-world processes done by humans due to their efficiency and speed.

Core crewAI components

We're going to go over the most important components of crewAI. Here's the breakdown of what we're going to cover:

• Agents
• Tasks
• Tools
• Crews (and Processes)

Now if I were to put the points above together in a sentence to explain how they're linked together that's what this sentence would be:

Crews, made up of Agents perform Tasks using Tools by following Processes. *

* Couldn't fit the Memory component in that sentence - If you can, share in the comments! 🫤

Meet the crew

Suppose that you're a blogger (like me) looking to optimize your post-promotion process. To keep things simple, let's assume that this is a 3-step process that looks like this:

1. Get the latest post on the blog.
2. Write a tweet based on the post.
3. Compose a newsletter email based on the post.

Now I'm going to show you how crewAI can help me achieve this above in seconds instead of minutes or hours that it would normally take.

Agents in crewAI

Agents work together, each one uses one or more tools to contribute to the group to solve a common goal.

If we go back to our example, we'll need to create two agents. The first one will pull the latest post from my blog, then the second one will use the content to transform it into a Twitter (or X) post and a newsletter email.

Ok, cool - let's take a look at what the first agent looks like in Python:

from crewai import Agent

...

extractor = Agent(
    role='Content Retriever',
    goal='Given a URL you will retrieve the content.',
    backstory='''As an expert at retrieving complete and accurate
    information, you are responsible for presenting the content of webpages
    that will be used to create engaging content for twitter and a newsletter.
    ''',
    verbose=True
)

And the second one:

writer = Agent(
    role='Content Writer',
    goal='You are responsible to transforming long text into engaging content ready for promotion on different channels.',
    backstory="""
        You are an excellent communications specialist, known for your
        exceptional skill of transforming complex subject into easy to
        understand stories that attract people.
        """,
    verbose=True
)

• The role attribute specifies the function of the agent.
• The goal attribute specifies what the agent must achieve.
• The backstory attribute adds context to what the agent is and does.

That's all you need to know to create agents. Now let's assign tasks.

Tasks in crewAI

Tasks are the things agents do. At a minimum, a task is made up of a description, an expected output, and a reference to an agent that will perform the task.

In our case, we'll need the following tasks:

• Task 1: Fetch content from the blog
• Task 2: Compose the tweet based on the content
• Task 3: Write the newsletter based on the content

Task 1 and the writer agent will take care of, and since an agent can perform multiple tasks, we will assign Task 2 and 3 to our extractor.

We're going to start with Task 1: fetch

from crewai import Task

...

fetch = Task(
    description=f'''
        Given a URL, retrieve the content of the webpage.
        It is important that you do not miss any information.
        
        Make sure that:
         - The content does not include html, css, or javascript.
         - The content is complete and accurate.
         - You do not include headers, footers, or sidebars.
    ''',
    agent=extractor, 
    expected_output='''
        Title: [The title of the article]
        Author: [The author of the article]
        Date: [The date the article was published]
        Content: [The content of the article]
    '''
)

Here's what Task 2 looks like:

twitterize = Task(
    description='''
        Given a long text, transform it into engaging content ready for promotion on Twitter.
        Make sure that:
         - The content is engaging and informative.
         - The content is less than 280 characters.
         - The content includes relevant hashtags - Limit to one.
    ''',
    agent=writer,
    expected_output='''
        Title: [Engaging catchy title for the tweet]
        Content: [Engaging content for the tweet]
    '''
)

Ok, cool! Now you know what tasks are and how they work. Next: Tools.

crewAI Tools

As you've seen, our extractor agent is tasked with pulling the information from a given URL. But how?

Hint: By using a tool.

In crewAI there are three ways you could use tools:

• Custom Tools: The ones you write yourself, essentially a Python function.
• Built-in Tools: crewAI ships with lots of included tools.
• LangChain Tools: Since crewAI is built on top of LangChain, you'll get all the LangChain goodies as well.

For our example, there are plenty of existing tools that help us extract the information from the blog. One such tool is the ScrapeWebsiteTool built into crewAI. So in this case, we don't need to build our own.

To use it, we'll just pass it within the list of tools used by the extractor as such:

from crewai_tools import ScrapeWebsiteTool

site_url = 'https://www.gettingstarted.ai/crewai-beginners-tutorial
scrape_tool = ScrapeWebsiteTool(url=site_url)

extractor = Agent(
    ...
    tools=[scrape_tool] # <----
    ...
)

Make sure to install the optional tools package using pip:

pip install crewai[tools]

Cool - we're nearly done which means that you're almost a superstar! ⭐

crewAI Crews

Now that we've defined the tasks, tools, and agents. We'll have to group them all so they work together. That's where we define a crew, but before - let me explain how the agents work together.

Crew Processes

So now you know that a crew is composed of agents. But these agents must know how to talk to each other, like which one starts the conversation. In crewAI, there are two supported processes and a third one that's coming soon. Current processes are:

• Sequential: One task after the other, in an orderly fashion.
• Hierarchical: A manager will orchestrate the conversation flow.

Since our tasks could be done sequentially, we'll go for the Sequential process.

Let's do it:

from crewai import Crew

...

crew = Crew(
    agents=[extractor, writer],
    tasks=[fetch, twitterize, newsletterize],
    Process=Process.sequential
)

That's all, pretty simple, eh?

Finally, we call the kickoff() method to set everything in motion:

result = crew.kickoff()

print("#### USAGE ####")
print (crew.usage_metrics) # <-- Optional

print("#### RESULT ####")
print(result)

Check this out:

That was awesome. Now you're officially a superstar! Congrats 😀

Conclusion and thoughts

Honestly, the beauty of crewAI is undoubtedly its simple structure. You could potentially add another agent that could be responsible to post the message on Twitter and send an email to your subscribers.

You would do this by leveraging tools, either using an existing one or creating your own integration with a third-party API.

Will agents replace human teams? Yes, and no.

Yes, if you or your company could achieve results faster for less, wouldn't you go for it?

And No, because nothing still comes close to human interpretation of contexts, events, and emotions. 

Efficiency is king, but the human touch reigns supreme.

Found this useful? Support my work by:

• Following me on X / Twitter
• Subscribing to my YouTube channel
• Become a member of the blog

Want me to write about something specific? More crewAI posts? Let me know in the comments below.

See you soon!

Further readings

More from the Web

• crewAI website
• crewAI github repo
• crewAI official docs

Source code for the geek in you 🤓