Skip to content

Method: Search Endpoints

Jump to bottom
Matthew R. DeVerna edited this page Aug 1, 2021 · 5 revisions

You can use the search() method to access either the Full Archive search endpoint (Academic Track only) or the Recent search endpoint (standard access).

This endpoint takes a single string that tells Twitter how to find tweets.

Check out How to Build a Query for all that can be done with Twitter's search operators, as well as the best way to construct your query string.

Basic usage:

import osometweet

# Initialize the OSoMeTweet object
bearer_token = "YOUR_TWITTER_BEARER_TOKEN"
oauth2 = osometweet.OAuth2(bearer_token=bearer_token)
ot = osometweet.OsomeTweet(oauth2)

# Get tweets from Twitter CEO Jack Dorsey's account @jack
response = ot.search(
            query = "from:jack",
            since_id = "1360109997242216450",
            until_id = "1360720695337000962",
            )
print(response)

As you can see above, I pass since_id and until_id into the method as additional parameters - these are not built-in method parameters). The method automatically reads in additional parameter values as additional parameters for our search query.

For example, the print line above returns the typical dictionary with "data" and "meta" objects.

{ 'data': [ { 'id': '1360114881978982403',
              'text': 'JAY-Z/@S_C_ and I are giving 500 BTC to a new endowment '
                      'named ₿trust to fund #Bitcoin development, initially '
                      'focused on teams in Africa & India. It‘ll be set up '
                      'as a blind irrevocable trust, taking zero direction '
                      'from us. We need 3 board members to start: '
                      'https://t.co/L4mRBryMJe'}],
  'meta': { 'newest_id': '1360114881978982403',
            'oldest_id': '1360114881978982403',
            'result_count': 1}}

In this example, I explicitly selected two tweet IDs from @jack's page that were directly before and after a single tweet that I wanted to select. These tweet IDs were then passed into the since_id and until_id parameters, creating a time-based boundary around where Twitter will search - defined by the tweet IDs. This illustrates a bit of the control that the search endpoint allows, however, there are many other operators/parameters that can be used. See the How to Build a Query page for all options).

As another example, you can include max_results to control how many tweets are returned in each response (default = 10, system limit is currently 500). For example, the below will return up to 50 tweets per query call (if there are only 30 tweets matching your request, you will get all 30).

response = ot.search(
            query = "from:jack",
            max_results = 50
            )

Recent search vs. Full archive search

By default, this method utilizes the recent search endpoint. To use the full archive search endpoint (only available if Twitter has given you access to the Academic Product track) simply add full_archive_search = True to your query, like below

import osometweet

# Initialize the OSoMeTweet object
bearer_token = "YOUR_TWITTER_BEARER_TOKEN"
oauth2 = osometweet.OAuth2(bearer_token=bearer_token)
ot = osometweet.OsomeTweet(oauth2)

# Get tweets from Twitter CEO Jack Dorsey's account @jack
response = ot.search(
            query = "from:jack",
            full_archive_search = True,       # <~~~~~~~~~~~~ Add this to search with the full archive endpoint (Academic Product track only)
            since_id = "1360109997242216450",
            until_id = "1360720695337000962",
            )
print(response)

Getting the full data object

There is a lot of data missing from the above response object. However, this method follows the same structure as all of the other osometweet methods, allowing a user to pass the boolean True to the everything parameter and automatically download all available tweet/user fields and expansions. Or, you can pass explicit data object fields and expansion objects. See Method: Specifying-fields-and-expansions for a more detailed walk-through of how this works, however, here is a quick example of the everything option.

import osometweet

# Initialize the OSoMeTweet object
bearer_token = "YOUR_TWITTER_BEARER_TOKEN"
oauth2 = osometweet.OAuth2(bearer_token=bearer_token)
ot = osometweet.OsomeTweet(oauth2)

# Get tweets from Twitter CEO Jack Dorsey's account @jack
response = ot.search(
            query = "from:jack",
            everything = True,      #<~~~~~~~~~~ Include to pull all available data objects and expand anything that can be expanded.
            since_id = "1360109997242216450",
            until_id = "1360720695337000962",
            )
print(response)

Pagination

If you have a query that will return more tweets than Twitter can return in one query, you can utilize the next_token returned in the meta-data of the response. For example, after one query, retrieve the 'next_token' parameter and pass it to a subsequent query.

First, make the original query. Query all tweets including "trump" since 2018.

# Make sure to authorize as shown above.

# This is how you can create a date object that works with Twitter's query system
start_date = datetime.datetime(2018,1,1).astimezone()
start_date = start_date.isoformat()
end_date = datetime.datetime.now().astimezone()
end_date = end_date.isoformat()

# Make first query
response = ot.search(
        query = "trump",
        start_time = start_date,
        end_time = end_date,
        full_archive_search = True
)

Since we searched for all tweets containing "trump" since the beginning of 2018, there are many more tweets than those that will be returned in one request. We can keep track of where we left off in that request by grabbing the next_token from the response object and passing it to your next query.

next_token = response["meta"]["next_token"]  #<~~~~ Grab the `next_token` from the response above

response_2 = ot.search(
        query = "trump",
        start_time = start_date,
        end_time = end_date,
        next_token = next_token,   #<~~~~ This will tell twitter to get ten more tweets from this point.
        full_archive_search = True
)

Fields & expansions

V2 API, by default, only returns limited information. To fetch more, you will need to specify the fields and expansions parameters in the requests. OSoMeTweet contains several classes to handle them. See Method: Specifying fields and expansions for examples of how to work with them.

Clone this wiki locally